ONDCP Media Campaign: Contractor's National Evaluation Did Not Find that the Youth Anti-Drug Media Campaign Was Effective in Reducing Youth Drug Use
Published: 2006-08-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Between 1998 and 2004, Congress appropriated over $1.2 billion to the Office of National Drug Control Policy (ONDCP) for the National Youth Anti-Drug Media Campaign. The campaign aimed to prevent the initiation of or curtail the use of drugs among the nation's youth. In 2005, Westat, Inc., completed a multiyear national evaluation of the campaign. GAO has been mandated to review various aspects of the campaign, including Westat's evaluation which is the subject of this report. Applying generally accepted social science research standards, GAO assessed (1) how Westat provided credible support for its findings and Westat's findings about (2) attitudes, beliefs, and behaviors of youth and parents toward drug use and (3) youth self-reported drug use.
section_title: What GAO Found
paragraphs: GAO's review of Westat's evaluation reports and associated documentation leads to the conclusion that the evaluation provides credible evidence that the campaign was not effective in reducing youth drug use, either during the entire period of the campaign or during the period from 2002 to 2004 when the campaign was redirected and focused on marijuana use. By collecting longitudinal data--i.e., multiple observations on the same persons over time--using generally accepted and appropriate sampling and analytic techniques, and establishing reliable methods for measuring campaign exposure, Westat was able to produce credible evidence to support its findings about the relationship between exposure to campaign advertisements and both drug use and intermediate outcomes. In particular, Westat was able to demonstrate that its sample was not biased despite sample coverage losses, maintained high follow-up response rates of sampled individuals to provide for robust longitudinal analysis, established measures of exposure that could detect changes in outcomes on the order of magnitude that ONDCP expected for the campaign and that could reliably measure outcomes, and used sophisticated statistical methods to isolate causal effects of the campaign. Westat's findings on the effects of exposure on intermediate outcomes--theorized precursors of drug use--were mixed. Specifically, although sampled youth and parents' recall of campaign advertisements increased over time, they had good impressions of the advertisements, and they could identify the specific campaign messages, exposure to the advertisements generally did not lead youth to disapprove of using drugs and may have promoted perceptions among exposed youth that others' drug use was normal. Parents' exposure to the campaign led to changes in beliefs about talking about drug use with their children and the extent to which they had these conversations with their children. However, exposure did not appear to lead to increased monitoring of youth. Moreover, the evaluation was unable to demonstrate that changes in parental attitudes led to changes in youth attitudes or behaviors toward drug use. Westat's evaluation indicates that exposure to the campaign did not prevent initiation of marijuana use and had no effect on curtailing current users' marijuana use, despite youth recall of and favorable assessments of advertisements. Although general trend data derived from the Monitoring the Future survey and the Westat study show declines in the percentage of youth reportedly using marijuana from 2002 to 2004, the trend data do not explicitly take into account exposure to the campaign, and therefore, by themselves, cannot be used as evidence of effectiveness. In Westat's evaluation of relationships between exposure and marijuana initiation the only significant finding was of small unfavorable effects of the campaign exposure on marijuana initiation during some periods of data collection and in some subgroups.

section_title: Letter
paragraphs: Congressionally mandated under the Treasury and General Government  Appropriations Act of 1998, the National Youth Anti-Drug Media  Campaign had the primary goals of preventing the initiation of drug use— particularly the use of entry-level drugs marijuana and inhalants—among  the nation’s youth and stopping youth that have begun using drugs from  continuing their use. Administered through the Office of National Drug  Control Policy (ONDCP) and implemented in three phases, the campaign  featured as its centerpiece a paid advertising effort in which campaign  funds were used to purchase media time and space for advertisements that  delivered anti-drug messages to the campaign’s target audience—youth  aged 9 to 18 and their parents—through strategic placement of anti-drug  advertisements on television and radio and in print media. In addition to  the advertising, the campaign included community outreach, work with  the entertainment industry to encourage the accurate depiction of the  consequences of drug use, outreach to faith-based organizations, and work  with youth organizations. The campaign’s first two phases, which ran from  January 1998 through the summer of 1999, were pilot phases that focused  primarily on informing the planning and development for phase III and  included a 12-city pilot (phase I) and nationwide advertising (phase II).  Phases I and II aimed to increase public awareness of anti-drug messages.  Phase III of the campaign, which began in mid-1999, continued the  nationwide advertising campaign begun during phase II and integrated the  advertising with outreach efforts. From fiscal year 1998 through fiscal year  2006, Congress appropriated over $1.4 billion to support the campaign. For  fiscal year 2007, the President’s budget requested $120 million for the  campaign, an increase over the fiscal year 2006 appropriation, to purchase  additional media time and space to increase the reach and frequency of  the campaign’s messages, which would restore appropriations to their  fiscal year 2005 level.
Congress first authorized funding for the campaign in fiscal year 1998 with  the expectation that demonstrable changes in youth drug behaviors would  be apparent within 3 years, and Congress required ONDCP to assess  whether the campaign’s efforts have been effective in changing the drug  use behaviors of America’s youth. ONDCP also indicated that it  anticipated that it would take 2 to 3 years for the campaign to affect drug  use behavior, although ONDCP also indicated that it was with the  implementation of phase III of the campaign, beginning in mid-1999, that  ONDCP expected to see improvements in anti-drug attitudes that would  lead to decreases in youth drug use within 3 years. We previously reported  that ONDCP’s evaluations of the first two phases of the campaign  produced inconclusive results because of various evaluation  implementation problems and limitations of the analyses used to support  findings about effects during these pilot phases. In particular, we noted  that the impact evaluations of phases I and II did not adequately gauge the  overall level of anti-drug awareness generated by the campaign—the  principal outcome measure for these two phases—and we identified site  selection problems, unknown parent response rates, low school response  rates, and data analysis issues contributing to the inconclusive results.
To implement the phase III evaluation, ONDCP entered into an  interagency agreement with the National Institute on Drug Abuse (NIDA),  which in turn awarded contracts to Westat, Inc., through June 2005 for  $42.7 million to conduct the evaluation. Westat subcontracted with the  Annenberg School for Communication at the University of Pennsylvania,  and staff from Westat and Annenberg were coprincipal investigators for  the study. Westat’s phase III evaluation covered the period from  September 1999 through June 2004 and studied the impact of the  effectiveness of the nationwide campaign in reaching its target audience;  affecting youth beliefs, attitudes, intentions, and behaviors with regard to  drug use; and affecting their parents’ beliefs and attitudes toward drug use  and affecting parents’ behaviors associated with interacting with their  children and monitoring their activities.
Westat evaluated the campaign using a longitudinal panel survey—the  National Survey of Parents and Youth (NSPY)—that aimed to measure the  campaign’s effectiveness by assessing changes in various outcomes within  individuals over time in relation to their exposure to campaign messages.  Westat’s evaluation assessed the effect of exposure to the campaign on  youth drug use and on several key intermediate outcomes—such as youth  and parent attitudes toward and beliefs about drug use and parental  involvement with their children—that were believed to influence youth  drug use. In conducting the evaluation, Westat submitted several interim  reports that were used in part to inform decisions about the direction of  the campaign. Westat submitted a draft of the final evaluation report to  NIDA in February 2005.
Both Congress and ONDCP recognized the need for a separate evaluation  of the campaign, because of limitations associated with existing national  surveys of drug use. Congressional conferees acknowledged their  intention to rely on the evaluation to gauge the impact of the campaign,  and also indicated that if the campaign failed to show its effectiveness,  they would be compelled to reevaluate the use of taxpayer money to  support it. ONDCP acknowledged that existing national surveys of drug  use would not be able to answer the critical question of whether changes  in drug use behavior and attitudes were the result of the campaign. These  surveys do not ask respondents about their exposure and reactions to the  messages of the campaign that can then be linked to their drug-related  attitudes and behavior. For example, in a 2001 report on youth drug use  and the campaign, ONDCP officials noted that while national surveys of  youth drug use showed flattening or declining youth marijuana use in 1999  and 2000 and these trends suggested that the campaign may be having the  desired impact, it was necessary to await the results of the campaign’s  independent evaluation before drawing any definitive conclusions  regarding the campaign’s contribution to changes in youth drug use.
In a committee report for the fiscal year 2004 appropriations cycle, the  Senate Appropriations Committee directed us to review how consultants  were used in support of the media campaign. This is the second of two  reports responding to this mandate. The first report provided information  concerning ONDCP’s use of consultants in the campaign. This second  report addresses three questions related to Westat’s evaluation of phase III  of the media campaign: (1) How did Westat ensure that it could report  credible results in its evaluation of the campaign? (2) What did the  evaluation find about the effect of exposure to the campaign on key  intermediate outcomes that were intended to lower youth drug use?   (3) What did the evaluation find about the effect of exposure to the  campaign on youth drug use?
In addressing our objectives, a team of GAO social scientists reviewed and  assessed materials related to Westat’s phase III evaluation, applying  generally accepted social science research standards, including such  elements as when and how the sample data were collected, adjustments  made to the sample to address nonresponse, how program effects were  isolated (i.e., the use of statistical controls), and the appropriateness of  outcome measures. The materials reviewed included interim and final  evaluation reports, documentation and analyses provided by Westat to us  in response to several sets of questions that we submitted about the details  of its methodology, documentation pertaining to meetings of scientific  panels that provided guidance on the evaluation, and documentation  prepared by ONDCP about the design and implementation of the  campaign. We conducted our work in accordance with generally accepted  government auditing standards from October 2005 through June 2006.
subsections: section_title: Background
subsections: section_title: The National Youth Anti- Drug Media Campaign
paragraphs: As part of the Treasury and General Government Appropriation Act of  1998, the Drug Free Media Campaign Act of 1998 required, among other  things, the Office of National Drug Control Policy to conduct a national  media campaign for the purpose of reducing and preventing drug abuse  among young people in the United States. The National Youth Anti-Drug  Media Campaign may be the most visible federal effort devoted to  preventing drug use among the nation’s youth. It aims to educate and  enable America’s youth to reject illegal drugs; to prevent youth from  initiating use of drugs, especially marijuana and inhalants; and to convince  occasional users of these and other drugs to stop using drugs.  Administered by ONDCP, and implemented in three phases, the campaign  has as its centerpiece a paid advertising effort in which campaign funds  were used to purchase media time and space for advertisements that  delivered anti-drug messages to the campaign’s target audiences—youth  aged 9 to 18 and their parents and adult caregivers—through strategic  placement of anti-drug advertisements on television and radio and in   print media.
The campaign’s first two phases were pilot phases that had as their  objectives developing advertising concepts, creating limited  advertisements, testing public awareness of the advertisements in   12 metropolitan areas, and eventually extending the pilot program  nationwide. Phase III of the campaign, which began in mid-1999, continued  the nationwide advertising campaign begun during phase II and integrated  the advertising with outreach efforts. In addition to the advertising, the  fully integrated phase III campaign included community outreach, work  with the entertainment and media industries to encourage the accurate  depiction of the consequences of drug use, outreach to faith-based  organizations, and work with youth organizations.
During phase III, ONDCP had overall responsibility for developing and  implementing the campaign, and to do so, it enlisted the support of  nonprofit organizations, trade associations, private businesses, and federal  agencies. Appropriated media campaign funds were to be used to cover  the costs of actually making the advertisements as well as the costs for  planning of purchase of media time and space. The campaign also  included public outreach and specialized communications efforts. The  purpose of public outreach and communications was to extend the reach  and influence of the campaign through nonadvertising forms of marketing  communications. Examples of these nonadvertising forms of  communication included submitting articles related to key campaign  messages such as effective parenting or the effects of marijuana on teen  health to newspapers and magazines; building partnerships and alliances,  for example, coordinating positive activities for teens with local schools  and community groups; creating Web sites and exploring alternative media  approaches; and entertainment industry outreach.
According to the campaign’s communications strategy, youth aged 9 to   18 were segmented into three school and age risk-level categories: late  elementary school adolescents, aged 9 to 11; middle school children, aged  11 to 13; high school youth, ages 14 to 18. The campaign originally targeted  youth aged 9 to 18 with a focus on middle school age adolescents (roughly  11-to 13-year-olds); its secondary focus was on high school-aged youth  (approximately 14 to 18 years of age). In 2001, the campaign shifted its  creative focus to 11- to 14-year-olds in order to more effectively reach  youth at the time they are most at risk for trying drugs. In 2002, the  campaign altered its target age group to focus primarily on 14- to   16-year-olds. For all age groups, the communications strategy identified  the primary focus of the campaign as at-risk nonusers and occasional  users of drugs. For all groups, it was designed to give consideration to  differences arising from gender, race, ethnicity, and regional and  population density factors.
From fiscal year 1998 through fiscal year 2004, Congress appropriated  $1.225 billion to support the campaign (table 1).
For fiscal year 2007, the President’s budget requested $120 million for  campaign activities. The 2007 request represents an increase of $21 million  above the fiscal year 2006 budget authority. The additional resources were  requested to help to purchase additional media time and space to increase  the reach and frequency of the campaign’s messages.
section_title: Planning and the Underlying Logic of the Campaign
paragraphs: According to ONDCP, its planning for the campaign’s communications  strategy included reviews of published studies on the etiology and  prevention of adolescent drug use, drug prevention campaigns, other  public health campaigns, and general consumer marketing campaigns  targeting youth and their parents. ONDCP also supplemented its research  evidence with an extensive expert consultation process that included  input from over 200 experts in academia, civic and community  organizations, government agencies, and the private sector. A campaign  design expert panel that included experts in the fields of drug use and  prevention, public health communication, advertising, market research,  consumer marketing, and public policy met over a 4-day period during the  fall of 1997 and played a key role in integrating diverse sources of  information and guiding the development of the communications strategy  for the campaign.
The planning process resulted in a statement of ONDCP’s communications  strategy for the campaign, which described the premises of the campaign.  Among these were the following: First, that the media can influence  people in a variety of ways, such as informing and alerting them to  important developments and shaping subsequent actions; satisfying leisure  time needs, thereby influencing individuals’ views and beliefs about the  world; and stimulating interest in commercial goods and services, thereby  influencing where and how people shop. Second, that media messages  have more potential to reinforce rather than to alter existing attitudes and  beliefs. Third, to the extent that youth attitudes, beliefs, and intentions  toward drug use vary with their age, the potential of a media campaign to  influence drug use may be directly related to the age of the youth. Fourth,  the campaign had to be sustained over time and to have a significant  media presence, and its central messages have to be repeated often and in  a variety of ways. Citing research showing that attitudinal and behavioral  change took time to occur, ONDCP reported that it expected to observe  “improvements in anti-drug attitudes that would lead to decreases in youth  drug use within three years” of the implementation of phase III of the  campaign. Fifth, as parents and adult caregivers play a vital role in youth  drug use behaviors, and by also targeting parents, the campaign would aim  to affect the nature of their interaction with their children, thereby  strengthening their children’s capacity to resist using illicit drugs.
The campaign focused on primary prevention—that is, preventing those  who did not use drugs from starting to use drugs. According to ONDCP, a  media campaign that focused on primary prevention targets the underlying  causes of drug use and therefore has the greatest potential to reduce the  scope of the problem over the long term. Further, a primary prevention  campaign also has greater potential to affirm and reinforce anti-drug  attitudes of nonusers than to persuade experienced users to change their  behaviors, and a primary prevention campaign would also, over time,  lessen the need for drug treatment services. With a focus on young, non- drug-using adolescents, an expectation underlying the campaign’s  potential success was that as these young, non-drug-using adolescents  aged, the campaign’s messages would intervene, retard the development of  more pro-drug attitudes, and enable adolescents to continue to maintain  their preexisting anti-drug attitudes. By maintaining these attitudes, or  preventing the development of pro-drug sentiments, the campaign would  affect drug use rates by lowering the rate at which youth initiated drug  use, particularly the use of marijuana or inhalants.
The campaign was designed to have a significant and sustained media  presence. During planning, ONDCP acknowledged that the campaign  would have to be sustained for a period of time sufficient to bring about a  measurable change in the beliefs and behaviors of youth in the target  audience. On the basis of the experiences of successful social marketing  campaigns, ONDCP reported that it expected that changes in awareness or  recall of the campaign would be detectable within a few months of the  start of the campaign, that changes in perceptions and attitudes would be  detectable within 1 to 2 years of the start of the campaign, and that  changes in behavior would be detectable within 2 to 3 years.
section_title: Campaign Activities during Phase III
paragraphs: From mid-1999, the start of phase III, through June 2004, the end of the  phase III evaluation, campaign activities included extensive media  dissemination of campaign messages to a national audience of youth and  parents; an interactive media component, which involved using content- based Web sites and Internet advertising; use of experienced individuals  and organizations with expertise in marketing to teens, advertising and  communications, behavior change, and drug prevention to inform the  campaign strategy and implementation; use of multicultural initiatives that  focused on sufficiently exposing campaign messages to African  Americans, Asian Americans, Pacific Islanders, Hispanic Americans,  American Indians, and Alaskan Natives; and the implementation of the  integrated social marketing and public health communications campaign  through the creation of partnerships with civic, professional, and  community groups and outreach to media, entertainment, and sports  industries. Through the partner organizations, the campaign attempted to  strengthen local anti-drug efforts, and through outreach, it encouraged the  news media to run articles that conveyed campaign messages. Youth and  parent exposure to campaign messages could come from the direct, paid  and donated advertising or from content delivered by news media and  entertainment industries through the outreach efforts. Additional  opportunities for exposure to anti-drug messages could be enhanced  through personal involvement with organizations that became partners as  a result of campaign outreach or by interaction with the campaign’s Web  site. Further, youth exposure to anti-drug messages could also occur  through interactions with friends, peers, parents, or other adults that  occurred directly from either campaign ads or outreach efforts.
section_title: Campaign Themes and Messages
paragraphs: Campaign messages for both youth and their parents and caregivers were  to focus on common transitions—such as the transition from elementary  to secondary school—and common situations—such as the amount of  time spent in settings without adult supervision—that were believed to  heighten adolescents’ vulnerability to drug use initiation. In addition,  messages were to focus on altering mediating variables—such as beliefs  and intentions—that were known to have a significant impact on  adolescent drug use. Finally, campaign messages were designed to create  a “brand identity” in the minds of target audience members and through  brand identity position campaign messages as credible and important.  Throughout phase III, themes such as parents as “The Anti-Drug” and the  “My Anti-Drug” theme for youth were designed to promote identification  and positive associations with the campaign’s messages.
While they evolved throughout the campaign, the central strategic  messages or themes for youth focused on resistance skills and self-efficacy  to refuse drugs, normative education and positive messages, negative  consequences of drug use, and early intervention. Resistance skills and  self-efficacy advertisements were designed to enhance the personal and  social skills of youth that promote lifestyle choices and to help build  youth’s confidence that they could resist drugs. Normative education  themes attempted to instill the beliefs that most young people do not use  drugs or convey messages that “cool people don’t use drugs,” while  positive message themes reinforced the idea of positive uses of time as  alternatives to illicit drugs. Negative consequences themes aimed to  enhance youth perceptions that drug use is likely to lead to a variety of  negatively valued consequences, such as loss of parental approval,  reduced performance in school, and negative social, aspirational, and  health effects. Negative consequences themes were the primary focus of  the Marijuana Initiative, which was introduced during 2002. An early  intervention theme sought to motivate youth to intervene with friends who  they perceived as having problems with drugs or alcohol and tried to  convince youth of their ability to take action and to give them the tools  and skills they needed to intervene.
For parents, the campaign’s themes included messages that every child,  including their own, was at risk of doing drugs; that they can learn  parenting skills to help them help their children avoid drugs; that they  need to be aware of the harmful effects of drugs including marijuana and  inhalants; and, as part of the Early Intervention Initiative, that it was  important that they intervene at the earliest possible opportunity in their  child’s life if their child was using drugs or alcohol.
section_title: Design of the Evaluation, Interim Evaluation Reports, and Redirection of the Campaign
paragraphs: ONDCP recognized the need for a separate evaluation of the campaign and  for ongoing reporting of evaluation results. The need for a separate  evaluation stemmed in part from the limitations of existing national  surveys that monitor drug use, such as Monitoring the Future, which  provides data on drug use by high school students, the National Household  Survey on Drug Abuse, and the Youth Risk Behavior Survey, which  addresses health risk behaviors including drug use. These recurring  surveys provide very little information with which to evaluate the impact  of the campaign, because they were not designed to evaluate it. As ONDCP  has written, these surveys contain no questions about target audience  exposure and response to the campaign, and as a result, any changes in  attitudes, beliefs, and behaviors toward drug use could not be associated  directly with the campaign. By comparison, ONDCP acknowledged that it  was using the Westat evaluation to assess the extent to which changes in  anti-drug attitudes and beliefs or drug-using behavior can be attributed to  the campaign, as opposed to other socioeconomic factors. In addition,  ONDCP indicated that for the campaign, data from Westat’s evaluation  would enable ONDCP to assess whether the campaign is working.
The primary tool of the Westat evaluation was the National Survey of  Parents and Youth. The NSPY is a longitudinal panel study of children and  their parents’ exposure and response to the campaign. The NSPY was  designed to collect initial and follow-up data from nationally  representative samples of youth aged 9 to 18 and from the parents of these  youth. The sample was designed to represent youth living in homes in the  United States and their parents. Data collection began in November 1999  and was conducted over four rounds—each of which was about 1 year  apart from the next round—in nine waves of interviews. An interview  wave refers to the fielding of a survey round to a specific subsample in the  NSPY. An interview round refers to the completion of interviews with the  entire sample. Data for each of the nine waves were collected using a  laptop computer and a combination of computer-assisted interview  technologies. To collect sensitive data, audio computer-assisted self- interview technology was used, allowing respondents to self-administer  the questionnaire in total privacy. The final wave of data collection was  completed in June 2004 (fig. 1). Eligible youth and parents were to be  interviewed four times.
The evaluation aimed to assess whether exposure to the campaign  affected the self-reported knowledge, attitudes, beliefs, and drug use of  youth. Because the campaign reached out to all youth nationwide, the  evaluators could not assess its effects using experimental methods, in  which some subjects are randomly assigned to the intervention and others  are randomly assigned to control groups that were not exposed to the  intervention. Westat’s evaluation was designed to take into account the  variation in self-reported exposure to the campaign messages and to  assess how this variation in exposure was correlated with outcomes that  the campaign intended to affect. To attribute changes in drug use attitudes  and behaviors to the campaign, the evaluation was designed to assess  exposure to the campaign and to compare differences in outcomes for  groups of persons that were exposed to varying levels of the campaign’s  messages, and to use statistical controls to account for individual-level  differences among survey respondents.
Westat’s evaluation assessed youth self-reported drug use and  intermediate outcomes—such as youth and parent attitudes and beliefs  toward drug use and parental involvement with their children—that were  believed to influence youth drug use. The evaluation of phase III  addressed issues related to (1) whether the campaign was reaching its  target populations, (2) whether the desired outcomes moved in favorable  or unfavorable directions, (3) whether the campaign was influencing  changes in the desired outcomes, and (4) what could be learned from the  overall evaluation to support ongoing decision making for the campaign.  These issues led to the five major objectives for the evaluation:  to measure changes in drug-related knowledge, attitudes, beliefs, and  behavior in youth and their parents;  to assess the relationship between changes in drug-related knowledge,  attitudes, beliefs, and behavior and self-reported measures of media  exposure, including the salience of the measures;  to assess the association between parents’ drug-related knowledge,  attitudes, beliefs, and behavior and those of their children;  to assess changes in the association between parents’ drug-related  knowledge, attitudes, beliefs, and behavior and those of their children  that may be related to the campaign; and  to compare groups of people with high exposure to other groups with  low exposure.
Westat submitted semiannual and special topic reports to NIDA, as the  findings from these interim evaluation reports were to be used to support  ongoing decision making for the campaign. Westat submitted the first  semiannual report in November 2000. By December 2003, Westat had  submitted six additional reports, four of which were labeled as semiannual  reports, and the other two included a special report on historical trends in  drug use and a 2003 report of findings. Westat submitted its first draft of  its final report to NIDA in February 2005.
In addition to Westat’s evaluation of the relationship between exposure  and outcomes, Westat also prepared a report on the environmental  context of the campaign. In May 2002, Westat reported findings from this  qualitative study of views of representatives from major national  organizations and state prevention coordinators about the messages  conveyed by the campaign and the role of the campaign as an organizing  partner in helping to bolster local substance abuse prevention efforts.  According to Westat, representatives felt that the campaign’s messages  reinforced their own messages that encouraged youth to find healthy  alternatives to drug use and to raise public awareness of the issue of illicit  drugs among youth. Westat also reported that representatives were less  enthusiastic about the role of the campaign as an organizational partner in  helping with local substance abuse prevention efforts. “There is little evidence of direct favorable Campaign effects on youth. There is no  statistically significant decline in marijuana use to date, and some evidence for an  increase in use from 2000 to 2001. Nor are there improvements in beliefs and  attitudes about marijuana use between 2000 and the first half of 2002. Contrarily,  there are some unfavorable trends in youth anti-marijuana beliefs. Also there is no  tendency for those reporting more exposure to Campaign messages to hold more  desirable beliefs.”
Westat further reported that there were unfavorable delayed effects of  campaign exposure on subsequent intentions to use marijuana and on  other beliefs. By delayed effects, Westat referred to the relationship  between exposure to the campaign measured in one survey round having  an effect on intentions or beliefs outcomes at a subsequent survey round.  For parents, Westat reported that the evidence was consistent with  favorable campaign effects, as it found that there were favorable changes  for three of five parents’ belief and behavior outcome measures. However,  Westat also reported that it found no evidence for favorable indirect  effects on youth behavior as the result of their parents’ exposure to the  campaign.
Congressional appropriators expressed concerns about the findings of  Westat’s fifth semiannual report. In the conference report for fiscal year  2003 omnibus appropriations, the conferees reported that they were  “deeply disturbed by the lack of evidence that the National Youth Anti- Drug Media Campaign has had any appreciable impact on youth drug  use.” The conferees further acknowledged that while the evaluation  conducted under NIDA’s auspices showed “slight and sporadic impact on  the attitudes of parents, it has had no significant impact on youth  behavior.” The conferees further acknowledged that while other surveys  of youth drug use—such as Monitoring the Future, a survey of high school  youth—showed recent declines in drug use, “the NIDA study was  undertaken to measure the specific impact of the Media Campaign, not  simply to gauge general trends,” and the conferees stated that they “intend  to rely on the scientifically rigorous NIDA study to gauge the ultimate  impact of the campaign” and to reevaluate the use of taxpayer money to  support the campaign if the campaign continued to fail to demonstrate its  effectiveness.
In 2002, the strategy for the campaign was redirected. In the spring, the  target age group of the campaign became 14- to 16-year-olds—youth who  have higher rates of marijuana initiation than younger youth—from its  original targeting of 11- to 13-year-olds. The shift to teens in the 14- to   16-year-old range aimed to allow the campaign to more effectively reach  youth during the time at which they are most at risk for trying drugs.  ONDCP also required more rigorous copy test procedures of all television  advertisements before they were aired, and ONDCP increased its oversight  in guiding the development and production of advertisements. In October  2002, ONDCP launched a new initiative called the Marijuana Initiative.  This initiative contained more focused advertising to address youth  marijuana use. In a hearing before the House Committee on Government  Reform, ONDCP announced that it would reverse the ratio of campaign  advertising expenditures directed to adults and youth, respectively.  Previously, about 60 percent of expenditures were directed to adults and  40 percent toward youth. Finally, during February 2004, it expanded the  campaign’s communications goals to include the Early Intervention  Initiative. This intervention was targeted toward both parents and teen  friends, and ONDCP intended to use parental and peer pressure to stop  drug and alcohol use among teens.
section_title: Assessment of the Campaign by the Office of Management and Budget and ONDCP’s Current Approach
paragraphs: To strengthen the linkages between resources and performance  envisioned in the Government Performance and Results Act of 1993  (GPRA), the Office of Management and Budget (OMB) developed the  Program Assessment Rating Tool (PART) to bring performance  information into the executive budget formulation process. PART is  designed to determine the strengths and weaknesses of federal programs  by drawing upon available program performance and evaluation data so  that the federal government can achieve better results. The PART  therefore looks at factors that affect and reflect program performance,  including program purpose and design; performance measurement,  evaluations, and strategic planning; program management; and program  results. Because the PART includes a consistent series of analytical  questions, it allows programs to show improvements over time and allows  comparisons between similar programs.
OMB’s PART rating of the campaign addressed issues related to its  purpose and design, strategic planning, program management, and  program results and accountability. OMB indicated that the purpose was  clear—giving ONDCP a 100 percent score on this factor—and it rated the  campaign’s planning and management with scores of 67 percent and   70 percent, respectively. In its assessment of ONDCP’s strategic planning,  OMB noted that in response to its 2002 PART review, ONDCP revised the  campaign’s logic model and significantly changed its long-term and annual  performance measures.
However, OMB’s assessment rating for the campaign was “results not  demonstrated.” OMB indicated that its assessment of the campaign’s  progress toward both the long-term goals and annual performance goals  will be reviewed against the results of the NIDA-managed evaluation. OMB  noted that while there is no federal program closely comparable to the  campaign, evaluations of other health behavior change efforts found short- term effects after exposure to media. While acknowledging that a final  assessment of the effects of the campaign awaited the final report from the  NIDA-managed evaluation, OMB also indicated that “outcome data from  the evaluation suggest little or no direct positive effect on youth behavior  and attitudes attributable to the campaign to date. Perhaps some positive  effect on parental attitudes/behavior but that has not yet translated into an  effect on youth.”
ONDCP has credited the campaign, along with a variety of collective  prevention efforts, with contributing to “significant success in reducing  teen drug use, as evidenced by the 19 percent decline from 2001 to 2005.”  It has introduced a new youth brand approach to connect youth with  aspiration themes. ONDCP also has indicated that while it awaits our  formal assessment of the evaluation, that it will use existing national  surveys to evaluate the campaign and suspend its request for proposals for  a new evaluation contract. Specifically, ONDCP indicated that it would use  the MTF survey to track improvements in perception of the risk of drug  use—a predictor of lower drug use by youth—and it would use a special  analysis of the PATS survey—the Partnership for a Drug-Free America’s  Attitude Tracking Survey—data on anti-drug messages. According to the  2005 data from MTF, there were no significant 1-year declines in marijuana  use for youth in any grade levels, and while gradual declines in the upper  grades continued, declines halted for youth in the 8th grade. Additionally,  for 8th graders, perceived risk of marijuana use held steady, while for  youth in 10th and 12th grade, there was an increase in perceived risk of  marijuana use.
section_title: Recent Research on the Effects of the Campaign in Local Settings
paragraphs: Two recently released studies have reported that exposure to the  campaign was associated with changes in past-month marijuana use under  certain conditions for certain groups of students exposed to the campaign.  In one of the studies, 45 South Dakota high schools and their middle- school feeder(s) were randomly assigned to three groups: (1) a basic  prevention curriculum, (2) a group given this curriculum with booster  lessons, and (3) a control group. All schools were exposed to the  campaign during the fall of 1999 and spring of 2000. This permitted the  researchers to test for a synergistic effect between exposure to the  campaign’s anti-drug messages and participation in the school-based drug  prevention curriculum. The sample of about 4,100 youth were asked how  often they had seen anti-drug advertisements in recent months in five  media outlets that were used by the campaign, and the researchers  measured exposure to the campaign that indicated whether or not the  adolescents reported seeing ads at least one to three times per week in any  of the five media outlets. Consistent with Westat’s fifth interim report, the  evaluation of the South Dakota drug prevention curriculum found no  direct effects of exposure to the campaign on its sample of adolescents’  use of illegal drugs. However, the evaluation also found that marijuana use  in the past month was significantly less likely among adolescents who  received both the curriculum with booster lessons and weekly exposure to  the campaign’s messages. In other words, neither the enhanced curriculum  nor the campaign alone had a substantial effect on marijuana use in the  absence of the other. In addition, this evaluation’s measure of exposure  was based on weekly exposure, suggesting that the synergistic effect of  the campaign observed in these South Dakota schools was based on the  delivery of repeated messages.
The second study used monthly random samples of 100 youth from the  enrollment lists of 4th to 8th graders in the public schools in the spring of  1999 in two moderate-sized communities—Fayette County (Lexington),  Kentucky, and Knox County (Knoxville), Tennessee—over 48 months from  April 1, 1999, through March 31, 2003. The study period included  advertisements aired under the campaign’s Marijuana Initiative. Students  in the samples aged over time and were 13 to 17 years of age at the  beginning of the Marijuana Initiative. Youth in the samples were measured  on marijuana use during the past 30 days, as well as on their attitudes  toward marijuana. Exposure to television and radio advertisements was  measured by self-reported past-month exposure. The study found that  among high-sensation-seeking youth—that is, youth who desire novel,  complex, and intense sensations and experiences and who are willing to  take social risks to obtain them—exposure to the first 6 months of the  campaign’s Marijuana Initiative led to reductions in marijuana use. The  study’s authors reasoned that the effects that they found for the Marijuana  Initiative were consistent with an approach termed SENTAR (for  sensation-seeking targeting), in which high-sensation-seeking youth are  targeted with high sensation value messages to prevent risky behaviors.
section_title: Westat’s Evaluation Design, Use of Generally Accepted and Appropriate Sampling and Analytic Techniques, and Reliable Methods for Measuring Campaign Exposure Produced Credible Evidence to Support Its Findings
paragraphs: Westat was able to produce credible evidence to support its findings about  the relationship between exposure to campaign advertisements and both  drug use and intermediate outcomes by employing a longitudinal panel  design—i.e., collecting multiple observations on the same persons over  time—using generally accepted and appropriate sampling and analytic  techniques and establishing reliable and sufficiently powerful measures of  campaign exposure. Westat encountered various challenges and threats to  validity that are commonly associated with large-scale longitudinal  studies, including lack of an opportunity to use experimental methods,  lack of baseline data, and changes in campaign focus that were not timed  with data collection; issues with ensuring adequate sample coverage and  controlling for sample attrition over time; establishing measures that were  sufficient to detect and reliably measure campaign effects; and  disentangling causal effects without being able to employ an experimental  design where subjects would have been randomly assigned to different  levels of exposure. Our review of Westat’s evaluation report and  associated documentation leads us to conclude that the design and  methodology used in its evaluation responded appropriately to these  challenges, resulting in credible findings.
subsections: section_title: Although Elements of the Campaign Limited Choices of Evaluation Designs and Affected Data Collection, Westat’s Design Was Rigorous and Provided a Means to Test for Campaign Effects
paragraphs: The nationwide scope of the campaign precluded Westat from using  experimental methods or obtaining baseline data, and the timing of the  introduction of some new campaign initiatives limited some of the data  available to evaluate them. However, Westat’s longitudinal panel survey  design provided a framework for developing strong evidence of within- respondent changes in outcomes over time as a result of exposure to the  campaign. The consensus of a scientific panel convened by NIDA in  August 2002 to review the evaluation was that Westat’s use of a national  probability sample to study change arising from the campaign was  preferable as the “gold standard” to a study based on other alternatives,  such as in-depth community-based studies of the mechanisms of change  and campaign effects. Additionally, the theoretical underpinnings of  behavioral change through advertising, along with statistically significant  outcomes in some but not all groups, suggest that the absence of baseline  data and introduction of new campaign initiatives did not invalidate the  evaluation’s findings. Finally, despite the introduction of new campaign  initiatives that were not timed with data collection cycles, Westat was able  to assess change in the NSPY data and generate statistically significant  findings using these data.
Westat’s longitudinal panel design was based on the premise that effects of  exposure to the campaign on outcomes could be measured and detected  within individuals over time, after controlling for various other factors that  could have influenced outcomes. The design called for measuring the  same respondents up to four times to assess how the natural variation in  exposure to the campaign correlated with campaign outcomes. Westat’s  approach—an exposure (or dose)-response model—is based upon a  premise that respondents’ recall of advertisements (exposure or dose) is  related to outcomes (response). In two recent studies of the effects of the  campaign on specific groups of youth in local areas, an exposure-response  approach has been shown to be an effective method for detecting effects  of the campaign in reducing youth drug use in local settings. One of the  studies reported a synergistic effect of exposure to the campaign and a  classroom-based drug prevention curriculum among 9th grade students in  45 South Dakota high schools. The other study reported reductions in drug  use during the period of the redirected campaign among high-sensation- seeking youth in schools in Knoxville, Tennessee, and Lexington,  Kentucky. To assess the possibility of preexisting differences between  groups of exposed youth and parents that might explain both the variation  in exposure to the campaign and variation in outcomes, Westat included in  the NSPY structured interview many questions on personal and family  history, and it used the responses to control statistically for differences in  attributes of respondents in order to attempt to isolate the relationship  between exposure to the campaign and outcomes.
The absence of baseline data—that is, precampaign data on outcomes— was beyond Westat’s control, as phase III of the campaign began before  the first wave of data collection for the phase III evaluation began. The lag  between the start of phase III of the campaign in mid-1999 and the  completion of the evaluation’s first round of data collection—around mid- 2001—leaves open the possibility that there were effects of the campaign  that occurred very early on in the campaign, prior to when Westat began  data collection. Several factors suggest that the absence of pre-phase III  baseline data was not fatal to the evaluation’s findings. First, if there were  effects of the campaign that could not be detected because of the absence  pre-phase III baseline data, those effects must have occurred very rapidly  and then endured throughout the remainder of the campaign, from 1999  through 2004. However, rapid changes in youth drug use were not  observed in MTF data; rather, the overall trend in MTF past year drug use  was flat between 1998 and 1999. Second, rapidly occurring effects were  not expected by ONDCP in designing the campaign. As we reported in  2000, and as ONDCP wrote in 2001, ONDCP believed that it would take   2 to 3 years for changes in drug use to be evident as a result of the  campaign.
Another campaign design factor that affected Westat’s evaluation was the  implementation of new campaign initiatives, such as the Marijuana  Initiative, which were implemented at times that officials at ONDCP  considered to be important, and therefore they may not have coincided  with planned data collection for the evaluation, nor should they  necessarily have done so. For example, the Marijuana Initiative was  implemented in October 2002, and the NSPY data available to evaluate  outcomes during it were limited to three complete survey waves. For its  longitudinal analysis of change during the Marijuana Initiative, Westat was  limited to data from two survey waves. Despite these limitations, the  evaluation produced data that enabled Westat to detect effects during the  period of the Marijuana Initiative.
section_title: Sample Coverage Issues Did Not Invalidate Westat’s Assessment of the Effectiveness of Exposure to the Campaign on Intermediate and Drug Use Outcomes
paragraphs: During the enrollment phase of the NSPY, Westat experienced sample  coverage problems, in that it enrolled—or rostered—a smaller percentage  of households with youth in the targeted age range than would be  expected based on comparable Current Population Survey (CPS)  estimates—the data that Westat used to develop its expectations about the  percentage of households having youth in the targeted age ranges.  Coverage refers to the extent to which a sample is representative of the  intended population on specified characteristics, and it is important  because the omission of segments of the intended population from a  sample—or undercoverage—can lead to biased results, in that omitted  segments may differ in some important respect from those segments  included. Westat estimated the extent of undercoverage in the NSPY to be  about 30 percent as compared to the CPS estimates, and according to  Westat and NIDA, the undercoverage arose during the stage of sampling in  which Westat was developing rosters of households that were believed to  contain youth in the target age range. At this stage, the survey rostering  process required entry into the household, which may have led  respondents in potentially eligible households to refuse to participate.
Our review of Westat’s documentation leads us to conclude that there was  no evidence of biased results due to undercoverage and that the sample  was sufficiently reliable both for the purposes of estimating changes over  time in individual outcomes and for assessing the effectiveness of  exposure to the campaign on outcomes. Westat’s comparisons of the  estimated population characteristics of the NSPY—such as race and  ethnicity of head of household and race and ethnicity of youth in  households—with the estimated population characteristics from the CPS  show that they are generally similar. That is, the distributions of  characteristics of eligible households with youth included in the NSPY  were broadly consistent with a variety of corresponding distributions from  the 1999 CPS. These comparisons suggest that the NSPY estimated  population by race and ethnicity was similar to that of CPS. Westat also  used multivariate modeling techniques to develop weighting adjustments,  and it developed weights to adjust its sample for nonresponse that were  reasonably effective in reducing nonresponse bias.
An additional test for bias in a sample is to compare estimates derived  from it with estimates on the same variable derived from another sample.  If the NSPY results were biased, then one would expect that estimates  derived from it would differ from estimates derived from unbiased  samples. For example, if eligible households refused to participate in the  NSPY because they contained teens with drug issues and as a result  avoided participation at a higher rate than did households containing teens  without drug issues, then these higher refusal rates by households  containing teens with drug issues would lead to NSPY estimates of the  percentage of youth reporting that they used drugs that were lower than  those obtained from other, comparable national surveys. According to  data provided by NIDA officials and our review of Westat’s final report,  estimated self-reported drug use rates from the NSPY are comparable to  estimates derived from other major surveys of drug use, such as the  National Survey on Drug Use and Health. For example, in the NSPY, rates  of past-month marijuana use among 12- to 18-year-olds were 7.2 percent in  2000, 8 percent in 2001, 8.9 percent in 2002, and 7.9 percent in 2003. These  rates were similar to those reported for 12- to 17-year-olds in the National  Survey on Drug Use and Health (NSDUH) of 7.2 percent in 2000, 8 percent  in 2001, 8.2 percent in 2002, and 7.9 percent in 2003. If youth with known  drug use problems consistently opted out of both the NSPY and the  NSDUH—a hypothesis that is not testable with the available data—then  the estimates from both the NSPY and the NSDUH of the true prevalence  of youth drug use would be biased underestimates.
section_title: Sample Attrition across NSPY Interview Rounds Was Sufficiently Low to Allow for Reliable Assessments of the Effect of Campaign Exposure on Outcomes
paragraphs: As the NSPY was a longitudinal survey—in which eligible sample  respondents were re-interviewed up to three times after their enrollment  interviews—attrition was a concern with which Westat had to contend. If  comparatively large numbers of sample respondents were not retained  across successive rounds of the survey, the capacity of the NSPY to  provide data to assess changes in outcomes in response to exposure over  time would be greatly diminished. Further, if attrition was specific to  certain groups, then the NSPY estimates would also be biased.
For the purpose of estimating within-respondent changes in outcomes in  response to changes in exposure across sample periods—the main use of  the NSPY data—Westat achieved follow-up longitudinal response rates of  between 82 percent and 94 percent for waves 4 through 9, the follow-up  waves to the first three enrollment waves. The longitudinal response rate  consists of two elements: (1) the percentage of prior survey respondents  that are tracked and for whom eligibility is determined and (2) the  percentage of those eligible that actually complete an interview. Across  the three follow-up survey rounds, Westat tracked and determined the  eligibility to participate in a follow-up survey of between 92 percent and   96 percent of the youth and parents who completed a survey in the prior  round. Of these, Westat obtained consent and completed extended  interviews with between 94 percent and 96 percent of youth and parents  for whom eligibility for a follow-up survey had been determined.
In our view, Westat’s follow-up response rates resulted in a sample that  was sufficient to provide reliable findings about the effects of exposure on  outcomes. In addition, Westat’s nonresponse adjustment methodology  compensated for effects of differential response rates related to the  percentage of persons in certain age groups, of certain races and  ethnicities, of those that owned homes versus rented, those that were   U.S. citizens versus noncitizens, and those with incomes below the   poverty level.
section_title: The NSPY Data Could Be Used to Detect Reasonably Small Effects, and Westat’s Measurement of Exposure and Outcomes Were Valid and Could Detect Effects, if They Occurred
paragraphs: The NSPY sample could be used to detect changes in outcomes that were  on the order of magnitude of changes expected by ONDCP for the  campaign, and its measures of exposure were valid and reliably predicted  outcomes. In early meetings on the design of the evaluation of the media  campaign, ONDCP officials reported that it had a specific Performance  Measures of Effectiveness system and that the campaign was embodied  within the first goal of the National Drug Strategy, which was to “educate  and enable America’s youth to reject illegal drugs as well as the use of  alcohol and tobacco.” Under this goal, ONDCP’s proposed targets for  reducing the prevalence of past-month use of illicit drugs and alcohol  among youth from a 1996 base year—by 2002, reduce this prevalence by   20 percent, and by 2007, reduce it by 50 percent. ONDCP officials  identified other specific targets, again from the base year 1996—by 2002,  increase to 80 percent the proportion of youth who perceive that regular  use of illicit drugs, alcohol, and tobacco is harmful; and by 2002, increase  to 95 percent the proportion of youth who disapprove of illicit drug,  alcohol, and tobacco use. To achieve a goal of 80 percent of 12th grade  youth who perceive that regular use of marijuana is harmful would require  increasing the 1996 baseline percentage of youth perceiving marijuana as  harmful from 60 percent, as measured by MTF, or by about 3.3 percentage  points per year from 1996 to 2002. Westat’s sample could be used to detect  this amount of annual change in youth attitudes.
In order to detect changes in outcomes due to exposure to the campaign,  it also was necessary that Westat accurately measure and characterize  exposure to the campaign. Westat provided evidence for the validity of its  measures of self-reported exposure, and the evidence suggests that the  measure of exposure was both valid and reliable. To measure exposure to  the campaign for both youth and parents, NSPY interviewers asked  respondents about their recall of anti-drug advertisements (general  exposure) and their recognition of specific current or very recent  television and radio advertisements (specific exposure). To facilitate  measures of recall, respondents viewed television and radio  advertisements on laptop computers. Youth and parents were only shown  or listened only to advertisements targeted to their respective groups. In  addition, both youth and parents were asked some general questions about  their recall of advertisements seen or heard in various media, including  television, radio, newspapers, magazines, movie theaters, billboards, and  the Internet.
Westat’s assessments of the validity of its measure of exposure to  campaign advertisements confirm that the NSPY data were able to  measure exposure. First, Westat examined respondents’ recall of  campaign advertisements using “ringer” television advertisements— advertisements that never had appeared. According to Westat’s analysis of  ringer advertisements, youth were more likely to recognize an  advertisement as a campaign advertisement when presented with an  actual campaign advertisement than a bogus one. For example, a far lower  percentage of respondents (11 percent) claimed to have seen a ringer, or  bogus, advertisement than the percentage who claimed to have seen the  broadcast advertisements (45 percent), particularly the advertisements  that were delivered with high frequency. The result held for youth and for  parents.
Second, comparing data on advertisement time purchases with self- reported exposure to these advertisements in the NSPY, Westat found a  high correlation between advertising and exposure. Specifically, on the  basis of analysis of individual advertisements’ gross rating points (GRP)—  a measure of the underlying reach and frequency of each advertisement— and self-reported exposures by respondents, Westat found a high  correlation between GRPs purchased by the campaign and self-reported  exposure to advertisements among youth. The correlation for parents was  somewhat smaller, but was also significant. Third, Westat also compared  self-reported exposure with recall of the correct brand phrase and found a  strong association between self-reported exposure and correct recognition  of the brand phrase. This is further evidence for the validity of its  measures of self-reported exposure.
Westat measured a variety of outcomes for youth and parents and took  steps to ensure that the measures were consistent with existing research.  The youth questionnaires included numerous questions that were designed  to measure exposure to the campaign advertisements and other anti-drug  messages. The youth question areas included exposure propensity to  media; current and past use of tobacco, alcohol, marijuana, inhalants, and  Ecstasy; past discussions with and communication of anti-drug messages  from parents and friends; expectations of others about respondent’s drug  use; knowledge and beliefs about the positive and negative consequences  of drug use; exposure to campaign messages; family and peer factors;  personal factors; and demographic information. Westat used separate  questionnaires for youth of different ages; one questionnaire was used for  children (aged 9 to 11) and another one was used for teens (aged 12 to 18).
section_title: Westat’s Analytic Methods Aimed to Isolate Causal Effects of the Campaign and Did So Using Sophisticated Techniques That Enhanced the Strength of Its Findings
paragraphs: In it analysis, Westat used three types of evidence to draw inferences  about the effects of the campaign: (1) trend data—data that describe  increases or decreases in drug use and other outcomes over time;   (2) cross-section analysis—measures of association between exposure to  campaign messages and individual drug use beliefs, intentions, and  behaviors, at the time data were collected; and (3) longitudinal analysis— measures of association, for youth and parents who were observed at two  points in time, between exposure to campaign messages at the earlier time  on outcomes at the later time. Westat indicated that trends over time, by  themselves, could not be used to provide definitive support for campaign  effects. Rather, the trends needed to be supported by measures of  association. Westat also indicated that measures of association, whether  cross-sectional or longitudinal, needed to control for variables that could  influence outcomes independently of the campaign or otherwise confound  the association between exposure and outcomes. Cross-section  association between exposure and outcomes measured at the same time  would provide stronger evidence of campaign effects than would trend  data alone, provided that controls for other variables were introduced into  the associational analyses. However, even if cross-section associations  between exposure and outcomes hold after controlling for the effects of  other variables, as Westat pointed out, there may remain an alternative  explanation for cross-section associations: For example, an outcome—like  perceptions of others’ use of drugs—may be the cause of exposure rather  than an effect of it. Westat’s longitudinal analysis attempts to address the  ambiguities that exist with cross-sectional associations. With longitudinal  data, if, after controlling for other confounding variables, exposure  measured at an earlier time is associated with an outcome at a later time,  the inference made is that the causal direction is from exposure to  outcome, since an effect cannot precede a cause in time.
As the campaign was implemented nationally and it was therefore not  possible to assign youth and their parents randomly to treatment and  control groups, a major threat to the validity of the conclusions from the  evaluation is that the observed correlations between exposure to the  campaign and self-reported attitudes and behaviors could reflect  preexisting differences among individuals in their underlying susceptibility  to campaign messages. The evaluation’s associations between exposure to  the campaign and self-reported initiation of marijuana use took into  account statistically the individual differences in attributes among youth  who were exposed to various levels of campaign messages, and they  adjusted for the influence of other variables that could determine  marijuana initiation—called confounder variables. As such, Westat’s  evaluation of the associations between campaign exposure and marijuana  initiation have accounted for individual differences among youth and can  be viewed as comparisons of outcomes for statistically similar individuals.  Further, the statistical test Westat used in assessing the relationship  between exposure and initiation did not rely upon assumptions of linearity  between levels of exposure and initiation. Instead, it tests for an ordered  relationship between exposure and an outcome such as marijuana use  initiation.
Westat used statistical methods to address the possibility that preexisting  differences between individuals could have caused both reported levels of  exposure and respondent outcomes, and its use of these methods  contributed to the validity of its findings about the effects of the campaign  on outcomes. If, independently of the campaign, individuals differed in  their underlying tendencies to accept and recall campaign messages, and if  the individuals who were more likely to recall advertisements also were  those who were more likely to respond to advertisements, then, absent  efforts to address this confounding factor, the findings about the  evaluation would be questionable. This type of bias is often called a  selection effect. If selection effects occurred in the campaign, then both  exposure and reported changes in attitudes and behaviors could reflect  underlying beliefs that were not affected by the campaign, despite the  presence of statistical correlations between self-reported exposure and  changes in attitudes and behaviors.
To control for selection effects and the many factors that could have  influenced both exposure and outcomes independently of, or in  conjunction with, the campaign, Westat used propensity scoring methods.  These methods limit the influence of preexisting differences among  exposed groups by controlling for a wide range of possible confounding  variables. Propensity score methods are used to create comparison groups  that are similar on measured and potentially confounding variables but  that differ on their levels of treatment. In the evaluation of the campaign,  the comparison groups were similar on confounding variables but differed  on their level of exposure to campaign messages. Propensity score  methods replace a set of confounding variables with a single function of  these variables, which is called the propensity score. In Westat’s analysis,  an individual’s propensity score is considered to represent an individual’s  probability of being assigned to a particular level of exposure to the  campaign, conditional upon the individual’s values of the confounding  variables. By including relevant, potentially confounding variables and  matching individuals on their propensity scores, Westat was able to  minimize bias due to selection effects. The comparison groups that Westat  created by using propensity score methods can be considered as statistical  analogues to randomly assigning individuals to different levels of  exposure. After creating these groups, Westat then analyzed outcomes  among the groups having different propensities to be exposed to campaign  messages.
Our assessment of Westat’s methods leads us to conclude that Westat took  reasonable steps to develop valid propensity models, and as a result of its  models, its analysis identified the effects of the campaign, net of other  factors included in its propensity score models. First, rather than simply  compare individuals who were exposed to campaign messages with those  who were not exposed, Westat estimated and compared groups of  individuals with different levels of exposure, where the number of  exposure groups was measured alternatively as a three- or four-level  variable—e.g., low, medium, or high exposure. Second, for the results of  propensity methods to be valid, it is important that the propensity scoring  models include all relevant variables that could otherwise explain  differences in both exposure and outcomes, as evaluators can adjust only  for confounding variables that are observed and measured. If an important  variable is omitted from the propensity model, the results of analyses may  be affected. Westat’s models included many relevant and potentially  confounding variables. For example, in the youth models, the propensity  score models included measures of demographic attributes, educational  attainment and educational aspiration, family and parent background,  parental consumption of television and other media, income and  employment, reading habits, Internet usage, location of residence in an  urban area, among other variables. Third, for propensity models to remove  the effects of confounding variables from the association between  exposure and response, it is necessary that the population means of the  confounder variables not vary across exposure levels. If a confounder is  successfully balanced, then it will have the same theoretical effect across  all exposure levels. After estimating models, Westat also assessed and  demonstrated the balance of variables in its propensity models.
section_title: The Phase III Evaluation Provided Mixed Evidence of the Campaign’s Effectiveness on Intermediate Outcomes, but It Found No Effect of the Campaign on Parental Monitoring of Youth
paragraphs: Westat reported mixed evidence about the effectiveness of the campaign  on intermediate outcome measures—such as recall and identification of  campaign messages, youth anti-drug attitudes, and parents’ beliefs and  behaviors—that were thought to be causal factors influencing youth drug  use, the ultimate target of the campaign. Most parents and youth recalled  exposure to campaign anti-drug messages, and for both groups, recall  increased during the September 1999 to June 2004 period covered by the  phase III evaluation. For current, non-drug-using youth—whose resistance  to initiating marijuana use the campaign intended to affect—although  NSPY data showed some favorable trends in anti-drug attitudes and beliefs  and in the proportion of youth who said that they would definitely not try  marijuana, there was no evidence that exposure to the campaign  influenced these trends. Conversely, among current, non-drug-using youth,  evidence suggested that exposure to the campaign had unfavorable effects  on their anti-drug norms and perceptions of other youths’ use of  marijuana—that is, greater exposure to the campaign was associated with  weaker anti-drug norms and increases in the perceptions that others use  marijuana. On three of five parent belief and behavior outcome  measures—including talking with children about drugs, doing fun  activities with children, and beliefs about talking with children—the  evidence pointed to a favorable campaign effect on parents. However,  while there was mixed evidence on the effect of the campaign on parents’  beliefs and attitudes about monitoring children’s behaviors, there was no  evidence to support a claim that the campaign actually affected parents’  monitoring behaviors—an area of the campaign’s focus for parents—and  there was little evidence for favorable indirect effects on youth behavior  or beliefs as the result of parental exposure to the campaign.
subsections: section_title: Youth and Parents’ Recall of Campaign Advertisements Increased over Time, Their Impressions of the Advertisements Were Favorable, and They Could Identify the Campaign Brand
paragraphs: According to Westat, the campaign purchased enough advertising time  over the 58-month period from September 1999 to June 2004 to achieve an  average exposure of 2.5 youth-targeted ads per week for youth and an  average of 2.2 parent-targeted advertisements per week for parents.  Westat’s estimates include campaign advertisements intended for either all  youth or all parents, but they do not include exposure of youth to parent  advertisements or parents to youth advertisements, nor do they account  for separate advertising targeted to specific race- or ethnicity-defined  audiences.
Using exposure indexes, Westat measured trends in general and specific  exposure to campaign advertisements. The general exposure index was  based on questions that asked about exposure to anti-drug messages in  recent months through a variety of channels, including movies, television,  radio, and billboards, and was not limited to campaign advertisements.  The specific exposure index was based on recall of specific  advertisements broadcast during the 60 days prior to the respondent’s  interview, and was limited to advertisements that targeted the respondent.  For example, for youth, only youth advertisements were sampled to  measure specific exposure. Youth aged 12½ to 18 and their parents  reported increasing levels of recall of specific but not general exposure to  campaign advertisements over time. For both parents and youth, there was  a sharp increase over time in the recall of specific exposure of television  ads across the campaign. Westat speculated that the increase in specific  recall may have arisen from better-placed, more memorable, or longer- aired advertisements rather than only to an overall increase in television  advertisements. However, recall of all general anti-drug advertising was  fairly stable over time, as there was no overall detectable change in  reported general exposure over the course of the campaign.
Beginning in 2001, when the evaluation started to measure brand phrase  recall, and continuing through 2004, the evidence indicates that youth, in  particular, exhibited increases in brand phrase recall. Advertising  campaigns may use a brand phrase to provide a recognizable element, and  to the extent that the brand is recognized and positively regarded, its  familiarity may lead to a positive response to a new advertisement or  increase the perception that each advertisement is part of a larger  campaign. The campaign included both a parent and a youth brand. Brand  messages may have involved a series of phrases or the portrayal of an  activity or lifestyle as positive (e.g., participating in team sports) to set up  the brand phrase of “The Anti-Drug.” Westat reported that the evidence  from the NSPY shows that the greater the exposure to media campaign  advertising, the more likely respondents were to recall the brand phrase.  In addition, the more that respondents recalled specific ads, the more  likely they were to recognize the brand phrase, although over time even  those with less exposure had learned the brand phrase.
Overall, youth reported favorable impressions of the subset of campaign  television advertisements that they were asked to evaluate, and their  favorable impressions increased over time. Responses to the  advertisements—whether they were attention getting, convincing, or said  something important to the respondent—were positive among both youth  and their parents. Parents’ evaluations of the advertisements were  generally more positive than those of youth, and parents’ positive views  also increased over time.
In addition to distributing messages directly in media advertisements, the  campaign aimed to reach its target audiences indirectly through other  institutions and routes, such as community groups, in-school and out-of- school anti-drug education, and discussions among youth and parents, and  youth and friends, concerning drug use and the drug advertisements. The  NSPY data indicated that the campaign’s messages were not accompanied  by similar increases in exposure to messages from other sources. Both  youth and parents reported receiving anti-drug messages from other  sources, but they did not consistently report increases in exposure to  messages from these sources. For example, from the 2000 to 2004 samples,  the percentages of youth reporting receiving in-school drug education  messages and attending out-of-school drug education both declined.
section_title: Westat Found That the Campaign Generally Had No Effect on the Attitudes of Youth Not Using Marijuana toward Its Use but That Exposure to the Campaign Was Associated with Unfavorable Effects on Youth Perceptions of Others’ Use of Marijuana
paragraphs: Westat generally found no significant effects of campaign exposure on the  cognitive outcomes of adolescent nonusers of marijuana—  i.e., development of anti-drug attitudes and beliefs. For current nonusers,  the evaluation reported on four cognitive measures and a fifth measure of  their perceptions of others’ use of marijuana. Three of the four measures— attitudes and beliefs about the consequences of marijuana use; perceived  social norms or pressures from parents, friends, and peers about  infrequent or regular marijuana use; and perceived self-efficacy to avoid  using marijuana, or their confidence to turn down use of marijuana under  various circumstances—were premised to affect the fourth—youth  intentions to use marijuana at all during the next year. The fifth outcome,  perceptions of other youths’ use of marijuana, was included to examine  whether exposure to the campaign was leading to increased perception  among youth that others use marijuana, and whether this perception, in  turn, affected their own behaviors.
Westat reported that the evidence from the analysis of trend data from  2000 to 2004 for two of the youth cognitive measures—attitudes and  beliefs about the consequences of marijuana use and intentions to use  marijuana—showed significant increases in youth believing that marijuana  use had negative consequences and significant increases in the percentage  of youth that reported that they had no intention to use marijuana.  However, evidence from both cross-section and longitudinal associations  between exposure and these two cognitive outcomes did not substantiate  that the favorable trends arose from exposure to the campaign.  Specifically, the cross-sectional associations between both general and  specific exposure to the campaign and intentions not to use marijuana  show no significant favorable effects of exposure on this outcome. None  of the cross-section associations between either general or specific  exposure and intention to use marijuana are significant, and none of the  longitudinal associations between specific exposure and intentions are  significant. Two of the longitudinal associations between general exposure  and intentions are significant, but the direction of the effect is unfavorable,  in that greater exposure led to declines in intentions not to use marijuana.  The evidence from the associational analyses between exposure and  attitudes and beliefs about the consequences of marijuana use generally  did not show an effect of the campaign. While there was one significant  cross-section association between general exposure and attitudes and  beliefs about consequences during the final two waves of survey data,  there were no significant cross-section associations between specific  exposure and attitudes and beliefs about consequences, nor were there  any significant longitudinal associations with either general or specific  exposure.
The associational analysis also produced some evidence of unfavorable  effects of exposure on social norms—i.e., social pressures from parents,  peers, and other important persons about marijuana use. Westat’s cross- section associations showed no significant effects of exposure on social  norms, but its longitudinal associations showed that across all survey  rounds, there was a significant relationship between specific exposure and  weaker social norms. Westat’s analysis of associations between exposure  and perceptions of others’ use of marijuana also produced significant  results. Cross-section associations between specific exposure and  perceptions of others’ use were significant, as were longitudinal  associations of this relationship. In other words, among youth who  reportedly did not use marijuana at the time of their interview, there was a  significant effect of specific exposure on the perception that others used  marijuana, and the direction of the effect was unfavorable—that is, those  reporting higher exposure to anti-drug ads were more likely to believe that  their peers used marijuana regularly. A significant and unfavorable  relationship between specific exposure and perceptions of others’ use of  marijuana was obtained for the data covering the entire period of the  evaluation as well as for the period of the redirected campaign, from   2002 to 2004.
section_title: The Evaluation Reported Favorable Effects of the Campaign on Three Parent Outcomes but Not on Parental Monitoring
paragraphs: A theme of the campaign was to encourage parents to engage with their  children to protect them against the risk of drug use, and parent skills  were a focus of parent advertising almost since the start of the campaign.  The campaign encouraged parents to monitor their children’s behavior by  knowing where they were and with whom, and to make sure that they had  adult supervision. It also encouraged parents to talk with their children  about drugs and to a lesser degree to engage in fun activities with their  children. The evaluation observed five outcomes for parents, and for four  of the five found significant and favorable effects of exposure to the  campaign. For three outcomes—parent-child conversations about drugs  (talking behavior), parents’ beliefs and attitudes about talking with their  children about drugs (talking beliefs), and parents’ engagement with their  children in in-home and out-of-home activities (fun activities)—both cross- section and longitudinal associations between exposure and outcomes  were generally significant and favorable to the campaign. For parents’  beliefs and attitudes toward monitoring their children’s behaviors, Westat  reported favorable trend and cross-sectional associations but no  significant overall longitudinal effects of either general or specific  exposure on this outcome. For the fifth outcome, parent monitoring  behaviors—that is, parents’ knowing or having a pretty good idea about  what their child was doing or planned to do—the evidence did not support  a finding of an effect of the campaign. There were no significant favorable  trends in parents’ reports of monitoring behaviors, and there were no  significant cross-section or longitudinal associations of either general or  specific exposure on monitoring behaviors.
section_title: No Evidence of Favorable Effects of the Campaign on Youth Outcomes through Campaign Effects on Parental Outcomes
paragraphs: Despite evidence of some favorable parental outcomes for the campaign,  Westat found no significant evidence for the overall evaluation that these  favorable parent outcomes affected youth attitudes and behaviors toward  drug use. Specifically, for the entire period covered by the evaluation,  Westat found no evidence of overall, indirect campaign effects on parents  leading to changes in marijuana use, intentions to use marijuana, social  norms, self-efficacy, or cognitions among youth who were not marijuana  users. Westat found that there were some significant indirect effects of  parental specific exposure on some youth outcomes for some subgroups.  For example, parental specific exposure was favorably associated with  intentions to use marijuana for 14- to 18-year-olds and for boys, and it was  also associated favorably with attitudes and beliefs about the  consequences of marijuana use for Hispanics. Westat also found  significant but unfavorable indirect effects of parents’ general exposure on  subgroups of youth in other youth outcomes. For example, parental  general exposure was unfavorably associated with youth social norms for  14- to 16-year-olds and for girls.
section_title: The Phase III Evaluation Found No Significant Effects of Exposure to the Campaign on Youth Drug Use Outcomes Other than Limited Unfavorable Effects on Marijuana Initiation
paragraphs: Westat reported that the NSPY data showed some declines in self-reported  lifetime and past-month use of marijuana by youth over the period from  2002 to 2004, and these trends in NSPY were consistent with trends in  other national surveys of drug use over these years. Westat also reported  that the NSPY data showed declining trends in youth reports of offers to  use marijuana. However, Westat cautioned that because trends do not  account for the relationship between campaign exposure and changes in  self-reported drug use, drug use trends alone should not be taken as  definitive evidence that the campaign was responsible for the declines. On  the basis of the analysis of the relationship between exposure to campaign  advertisements and youth self-reported drug use in the NSPY data— assessments that used statistical methods to adjust for individual  differences and control for other factors that could explain changes in self- reported drug use—for the entire period covered by its evaluation, Westat  found no significant effects of exposure to the campaign on initiation of  marijuana by prior nonusing youth. The only significant effect indicated in  Westat’s analysis of the relationship between campaign exposure and self- reported drug use was an unfavorable effect of exposure on marijuana  initiation—that is a relationship between campaign exposure and higher  rates of initiation—for one round of NSPY data and similar unfavorable  effects of campaign exposure on marijuana initiation among certain  subgroups of the sample (e.g., 12½- to 13-year-olds and girls). Westat found  no effects of campaign exposure on rates of quitting or use by prior users  of marijuana.
subsections: section_title: Westat Tracked Trends in Marijuana Use from Several Sources and Reported That the Trend Data by Themselves Were Insufficient to Demonstrate Effects of Exposure to the Campaign
paragraphs: Westat tracked trends in self-reported use of marijuana by youth and  trends in youth reports of offers to use marijuana for the period from   2000 to the first half of 2004 to determine if there were significant declines.  Westat also assessed these trend data for changes occurring since 2002,   or during the period of the redirected campaign. Westat’s trend analysis  was designed to provide supportive but not definitive evidence for  campaign effects.
In its trend analysis, Westat compared trends in self-reported drug use— lifetime, past year, and past month—in the NSPY with trend data on self- reported drug use from three other nationally representative surveys of  drug use—Monitoring the Future, the Youth Risk Behavior Surveillance  System (YRBSS), and the National Survey on Drug Use and Health. Both  MTF and YRBSS are school-based surveys, and NSDUH is a household  survey that provides estimates of drug use by the civilian,  noninstitutionalized population of the United States aged 12 years and  older. Methodological differences between the school-based surveys— MTF and YRBSS—and the household surveys—NSPY and NSDUH—have  been shown to account for the some of the differences in estimates of  marijuana use.
According to Westat’s analysis, the surveys of self-reported marijuana use  show some similarities and differences in trends depending upon the  measure, age group, or subperiod covered within the longer 2000 to 2004  period. For example, the MTF data generally show declines in lifetime,  past-year, and past-month self-reported drug use for 8th, 10th, and   12th graders over the years from 2000 to 2004, although only some of the  year-to-year differences in the MTF self-reported drug use data were  statistically significant. Nonetheless, for the subperiod from 2002 to 2004,  MTF data show statistically significant declines in past-year and past- month use for 8th graders and past-year use for 10th graders, and the  NSPY data also show statistically significant declines in past-month use  from 2002 to 2004 for youth aged 12½ to 18 years old and for 14- to 18-year- olds. On the other hand, the MTF data suggest a decline in past-year and  past-month use by 10th graders from 2000 to 2002, but the NSPY data  suggest an increase in past-month marijuana use during this period.  Further, the data from NSDUH for 2000 and 2001 also show statistically  significant increases in lifetime, past-year, and past-month marijuana use  among youth aged 12 to 17, statistically significant increases in lifetime  and past-year marijuana use for youth aged 16 to 17, and a statistically  significant increase in past year use for youth aged 14 to 15. The pattern of  increase in NSDUH data from 2000 to 2001 is consistent with the 2000 to  2002 increases in past-month use in NSPY, but they differ from the MTF  trends over this period.
All four surveys generally show declines in marijuana use beginning in  2002, but not all of the declines are statistically significant. Both MTF and  NSPY show some statistically significant declines since 2002, and while  NSDUH and YRBSS show declines, the declines were not statistically  significant. These declines starting in 2002 coincide with the redirected  campaign and the introduction of the Marijuana Initiative. “They provide policy makers with broad indicators of the success of  policy…However, they will not be able to answer the critical question of whether  these changes were the result of the Media Campaign. These surveys do not ask  respondents about their exposure and reactions to the messages of the Media  Campaign that can then be linked to their drug-related attitudes and behavior.”
section_title: Westat Reported That Trends in Marijuana Offers Declined over Time, but Factors Other than the Campaign Contributed to Changes in Offers
paragraphs: Westat assessed trends in youth reports of receiving offers of marijuana— whether anyone had ever offered youth marijuana and the frequency of  offers within the past 30 days. Marijuana offers are closely related to  marijuana use, and the campaign aired messages that encouraged  resistance to offers of marijuana. Over the 2000 to 2004 period, Westat  found significant increases in the percentage of youth reporting that they  had never received offers, and it also found significant decreases in the  percentage of youth reporting that they had received offers in the prior  month. Westat also found significant changes in offers over 2002 to 2004,  during the period of the redirected campaign, and these changes were  generally consistent with the trends for the overall 2000 to 2004 period.  Further, on the basis of longitudinal analysis of the relationship between  offers in one period and marijuana use in the subsequent period among  youth who were nonusers in an initial survey round—an analysis that  assesses whether offers precede use or are simply a correlate of it— Westat found that youth who reported having received a marijuana offer at  one period were much more likely—between three and seven times more  likely, depending upon age group—to have initiated marijuana use at a  following period than nonusing youth who reported never having received  such an offer. However, as Westat reported, while the findings on offers  are favorable, they cannot be ascribed to the campaign because they may  be caused by other factors, as the analysis of the relationship between  offers and use did not take into account other factors that could affect use.
section_title: On the Basis of Its Analysis of the Association between Exposure and Drug Use Outcomes, Westat Found No Evidence That Exposure to the Campaign Affected Initiation or Cessation of Marijuana Use
paragraphs: From its longitudinal analysis of associations between exposure and  initiation of marijuana use, Westat found no evidence that increased  exposure to the campaign reduced youth’s initiation of marijuana use.  Westat’s longitudinal analysis assessed the effects of exposure at one  survey wave on marijuana initiation at a subsequent survey wave,  controlling for potential confounding variables that could affect the  exposure initiation relationship. Westat assessed the effects of two types  of exposure on initiation of marijuana use—general exposure and specific  exposure. General exposure represents the sum of recalled exposure to  anti-marijuana advertising in four types of sources of advertisements— television and radio, movies and videos, print media including newspapers  and magazines, and outdoor media. Specific exposure represents the sum  of recalled exposure to youth-targeted individual campaign television  advertisements that had been aired in the 60 days prior to an interview.
Westat found no significant effects of the level of general exposure on  marijuana use initiation, either over the entire period of the campaign or  between subperiods as defined by survey rounds. Westat also found no  overall effects of levels of specific exposure on marijuana initiation during  the entire period of the campaign, but it found one significant association  between specific exposure and marijuana use initiation that occurred in  the data from wave 7 and its wave 9 follow-up, or during the period of the  Marijuana Initiative. Wave 7 was the first complete survey wave covering  exposure to the Marijuana Initiative. The significant association from this  analysis was that higher levels of specific exposure were associated with  higher levels of initiation of marijuana use among previously nonusing  youth.
Westat also examined the longitudinal relationships between exposure  and initiation for nine subgroups of youth (two sexes, three race/ethnicity  groups, two risk groups, and two nonoverlapping age groups). For several  subgroups, it found significant associations between specific exposure and  marijuana initiation. These associations were in a direction that was  unfavorable to the campaign, in that greater specific exposure was  associated with higher levels of initiation. The subgroups for which these  unfavorable associations were most pronounced included 12½- to 13-year- olds, girls, African Americans, and lower risk youth.
On cessation and reduction of marijuana use, Westat assessed two  outcomes among current marijuana users: the rate at which they quit using  marijuana and their frequency of use. The frequency of use measure  allowed for campaign effects to be observed if users did not quit but  reduced their use of marijuana. Westat estimated that the quit rate—the  percentage of prior-year users reporting that they no longer used  marijuana—among prior-year users of marijuana was 24.8 percent.  However, it found no statistically significant association between general  exposure and quitting or between specific exposure and quitting. It also  found that among adolescent marijuana users, the frequency of use— increase, decrease, or no change—was not affected by exposure to the  campaign.
section_title: Conclusions
paragraphs: A well-designed and executed multiyear study of the impact of the ONDCP  anti-drug media campaign on teen initiation of drug use, or cessation of  drug use, shows disappointing results for the campaign. The study  provides no evidence that the campaign had a positive effect in relation to  teen drug use, and shows some indications of a negative impact. Some  intermediate outcomes, such as parents talking with children about drugs,  and doing fun activities with their children, showed positive results in that  the media campaign encouraged parents to adopt these behaviors.  However, other intermediate outcomes, such as parents’ monitoring of  their children’s behavior, were not shown to be affected by the campaign.  Moreover, the evaluation did not provide evidence that intermediate  outcomes that showed positive results translated into greater resistance to  drugs among the teenage target population.
Unfavorable preliminary findings from the evaluation were reported by  Westat in 2002. Beginning in 2002, ONDCP took a number of steps that  were intended to strengthen the power of the campaign to achieve positive  results. These steps included more rigorous ad copy testing and a  concentration on anti-marijuana messages. However, the post-2002 results  yielded no evidence of positive impacts and some evidence of negative and  unintended consequences in relation to marijuana use. Specifically,  exposure to advertisements during the redirected campaign was  associated with higher rates of marijuana use initiation among youth who  were prior nonusers of marijuana.
Most parents and youth recalled exposure to the campaign messages and,  further, they recognized the campaign brand. Thus, the failure of the  campaign to show positive results cannot be attributed to a lack of  recognition of the messages themselves. This raises concerns about the  ability of messages such as these to be able to influence teen drug  attitudes and behaviors. It raises questions concerning the understanding  of the factors that are most salient to teens’ decision making about drugs  and how they can be used to foster anti-drug decisions.
Westat’s evaluation is centered on this particular configuration of a media  campaign as it was presented from 1999 to 2004, and its results pertained  to the campaign nationwide. It cannot be construed to mean that a media  campaign that is configured differently from this one cannot work. Nor  should its results be construed to mean that in some locations, for some  groups of youth, the campaign did not have an effect on drug use.  However, substantial effort and expertise were brought to the task of  designing the advertisements from the outset, and the 2002 redirection of  the campaign placed even greater emphasis on copy testing and enhanced  ONDCP oversight. This casts some doubt on the notion that a better media  campaign can lead to positive results.
It is also important to note that two recent smaller studies in three  locations have provided evidence of a limited effect of the campaign for  some youth, and it is quite possible that additional analyses of the NSPY  data using different methods or measures may find other effects of the  campaign, at least for some adolescents, than have been produced by  Westat’s evaluation team. The data from the evaluation have only recently  been made available to academic and other researchers, and while the  analyses undertaken by Westat are, as we have noted elsewhere,  appropriate and thorough, they are not exhaustive.
It is heartening that surveys intended to measure teen drug use, such as  Monitoring the Future, are showing declines in marijuana use in recent  years. Indeed, NPSY also shows some evidence of a decline in drug use  among teens. However, Monitoring the Future and other surveys of teens  concerning drug use are not linked to exposure to the media campaign,  and NPSY shows no relationship between anti-drug media campaign  exposure and favorable drug outcomes for teens. This seems to indicate  that other unidentified factors, other than the anti-drug media campaign,  are affecting drug use decisions among teens.
Although ONDCP has pointed to declines in teen drug use and credited the  campaign along with other prevention efforts as contributing to significant  success in reducing teen drug use, trend data derived from the Monitoring  the Future survey that show declines in teen marijuana use from 2001 to  2005 do not explicitly take into account exposure to the campaign, and  therefore, by themselves, cannot be used as evidence of effectiveness.
ONDCP has indicated in the past, and we concur, that because these  surveys cannot link their results with the media campaign, they do not  measure campaign effectiveness. The evaluation of the media campaign  reinforces the lack of linkage between the media campaign and teen drug  use behavior.
It is important to note that virtually all social science research is  imperfect. Attempting to systematically observe and document human  behavior in real-world settings is a daunting task given the extremely wide  variation in both humans and settings. We believe that the evaluation of  the ONDCP media campaign is credible in that it was well designed given  the circumstance of the campaign, and appropriately executed.
section_title: Matter for Congressional Consideration
paragraphs: In light of the fact that the phase III evaluation of the media campaign  yielded no evidence of a positive outcome in relation to teen drug use and  congressional conferees’ indications of their intentions to rely on the  Westat study, Congress should consider limiting appropriations for the  National Youth Anti-Drug Media Campaign beginning in the fiscal 2007  budget year until ONDCP is able to provide credible evidence of the  effectiveness of exposure to the campaign on youth drug use outcomes or  provide other credible options for a media campaign approach. In this  regard we believe that an independent evaluation of the new campaign  should be considered as a means to help inform both ONDCP and  Congressional decision making.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of the Office of National  Drug Control Policy for comment on July 31, 2006. ONDCP provided us  with written technical comments on the report, which we incorporated  where appropriate. In addition, ONDCP provided written comments about  our report in which it raised a question about our matter for congressional  consideration and outlined a number of concerns that it had with our  report on Westat’s findings. These written comments are reproduced in  appendix II. In our evaluation of ONDCP’s written comments, we address  each of the other concerns in the order ONDCP presented them.
subsections: section_title: Westat Evaluation’s Role in Judging the Impact of the Advertising Campaign
paragraphs: “ONDCP, on the other hand, is measuring the impact of the Media Campaign with  a thorough, rigorous, and independent evaluation. The nationally representative  evaluation is being conducted for ONDCP by the National Institute on Drug Abuse  (NIDA).…The evaluation is a 4-year longitudinal study of parents’ and their  children’s exposure and response to the Media Campaign.…ONDCP will be able  to assess the extent to which changes in anti-drug attitudes and beliefs or drug  using behavior can be attributed to the Media Campaign.”
ONDCP officials had opportunities during the evaluation to raise concerns  about Westat’s design and its efforts to establish a link between exposure  to the campaign and outcomes, but we are not aware of their having done  so. However, we are aware of ONDCP’s participation in a NIDA-sponsored  expert panel review of Westat’s evaluation that was held in August 2002.  Our review of the minutes of that meeting reveals that while an ONDCP  official raised concerns about issues such as assessing the nonadvertising  components of the campaign and the number of interim reports, ONDCP  officials did not at that time raise concerns that the evaluation was  fundamentally flawed. The consensus of the expert panel was that  Westat’s evaluation was “pretty impressive” given the challenges presented  by the absence of baseline data and of an experimental design. Panel  members also asserted that Westat’s use of propensity score models to  isolate the effects of the campaign was termed both “sensible” and “state- of-the-art.”
ONDCP further states that major advertisers evaluate the success of their  campaigns by rigorously testing advertisements prior to airing and by  developing correlations between messages and consumer attitudes and  behavior. While we do not dispute whether this is a commonly used  approach among major advertisers, we believe that in assessing the  expenditure of public funds researchers should attempt, where feasible, to  establish causal relationships or use research designs that attempt to  isolate the effects of federally funded interventions. While we  acknowledge that establishing causal relationships is difficult, we maintain  that Westat used sophisticated and appropriate statistical methods that  aimed to isolate the effects of recalled exposure to the campaign on youth  drug use. Further, adopting a methodology that relies upon correlations  between advertising messages and an outcome, such as reductions in  youth drug use, without attempting to take into account many of the other  factors that could affect drug use allows for too many post hoc  explanations of findings. Westat’s analysis included socioeconomic  factors, parent characteristics, television viewing habits, risk of using  drugs, and sensation-seeking tendencies to be able to determine whether  exposure was related to drug use net of the influences of these factors. We  conclude, on the basis of our assessment of Westat’s methods, that  exposure to campaign messages generally did not influence youth drug  use net of these other influences.
ONDCP notes that correlational findings have been used to assess anti- tobacco advertising campaign results. We have not reviewed the anti- tobacco campaign and cannot comment on its relationship to youth  smoking prevalence. We notice, however, that in ONDCP’s comments on  “Consequences of Further Budget Cuts,” it appears to contradict its  statements about establishing causal relationships to determine the effect  of advertising campaigns. ONDCP writes, “Previous studies have  established a relationship between exposure to anti-tobacco messages and  smoking rates among teens.” ONDCP goes on to draw an analogy between  anti-smoking messages and anti-drug messages to write, “We should  expect similar results for illicit drug use if anti-drug messages decline.”  These statements emphasize very directly the same kind of causal  relationships that ONDCP cites as not appropriate in its opening  comments.
We also note that ONDCP indicates in its comments that it has made  multiple refinements to the media campaign on the basis of earlier findings  from the Westat study. This seems to be inconsistent with a position of  major concerns with the fundamental soundness of the study.
Finally, the three research papers that ONDCP cites on page 2 of its  comments on “Conflicting Evidence from Other Research” all use  exposure-response methodologies that are analogous to Westat’s and all  attempt to isolate the causal effects of exposure either to ONDCP’s  campaign or to other media campaigns. Thus, it would seem that ONDCP’s  comment that efforts to isolate causal effects of media campaigns are  fundamentally flawed would also apply to these three studies.
section_title: ONDCP Made Campaign Changes as a Result of Westat Interim Findings
paragraphs: “These surveys [MTF, the National Household Survey on Drug Use, and the Youth  Risk Behavior Survey] will permit the determination of whether drug use behavior  and related attitudes and beliefs changed after the launching of Phase III of the  Media Campaign in mid-1999. However, they will not be able to answer the critical  question of whether these changes were the result of the Media Campaign. These  surveys do not ask respondents about their exposure and reactions to the  messages of the Media Campaign that can then be linked to their drug-related  attitudes and behavior.”
More recently in late 2005, ONDCP launched a newly designed campaign.   The impact of this campaign is not known and should be independently  evaluated.
section_title: Other Youth Drug Use Findings
paragraphs: ONDCP believes we did not provide adequate discussion of studies that  report findings contrary to those of Westat. Our report mentions two of  the three studies that ONDCP identifies—the Longshore and Palmgreen  studies. Our report does not mention the third study, Slater, because it  focused on a different anti-drug media campaign approach and not on the  ONDCP media campaign. Overall, these studies’ findings are not  necessarily “contrary” to Westat’s findings. Rather, they assess small slices  of the youth population or particular circumstances (such as other  programs that could reinforce an anti-drug message) and find some  positive results. The Westat national findings do not preclude the findings  of positive results for some subpopulations of youth. The Palmgreen study,  for example found a positive effect for the media campaign on high- sensation-seeking youth, but did not find an effect on non-high-sensation- seeking youth in the two moderate size communities in which the study  was conducted. The distribution of these youth in the nationwide  population could be consistent with both studies being correct. Our  objective was to assess the Westat study as a national evaluation of the  impact of the national campaign. In the Slater study, after being trained in  the use of campaign media materials, leaders in each of eight communities  that received a media campaign were allowed to develop their own media  strategies and were able to use whatever materials they chose or  developed on their own. This approach emphasized the flexibility to adopt  different media strategies deemed appropriate by individual communities  and not the use of a single national strategy.
ONDCP expressed concern that we had not discussed Westat’s hypothesis  concerning why the campaign might have contributed to youth  experimentation with marijuana. We are unable to draw a conclusion  about this hypothesis based on Westat’s report, nor do we have additional  information upon which to base an assessment. ONDCP also faults our  report for not discussing other potential competing explanations for the  substantial downturn in teen drug use and increase in anti-drug attitudes.  Although this is beyond the objectives of this report, we note that multiple  other indicators of youth responsibility also seem to be trending in a  positive direction at the same time that MTF reports declines in youth drug  use. For example, from 1991 through 1999, the teen pregnancy rate  declined by 27 percent and from 1991 through 2002, the teen birth rate fell  30 percent. Similarly, the number of juvenile homicides declined by   44 percent from 1993 to 2002, and the juvenile violent crime arrest rate fell  by more than 40 percent from 1994 to 2003. All of these trends—including  declines in drug use—could be related to broader environmental, familial,  or other influences. The coincidence of these trends with drug use trends  indicates that factors other than the campaign could be responsible for the  decline in drug use and points to the necessity of trying to isolate the  effects of the campaign, rather than relying upon simple correlations.
section_title: Steps Taken to Remedy Potential Problems
paragraphs: ONDCP states that it has taken extensive “due diligence” steps that are  briefly acknowledged in our report, but that our report “fails to  acknowledge the thoroughness of our actions to identify, assess, and  attenuate any possible negative consequences of the campaign once  Westat reported the possibility of such an effect.” Apart from those actions  described in Westat’s evaluation reports, a full discussion of the steps that  ONDCP took in response to Westat’s interim evaluation reports that  highlighted the possibility of unintended negative consequences of  exposure to the campaign on youth initiation of marijuana was not salient  to our assessing whether Westat took appropriate steps to address the  evaluation implementation challenges that it faced. However, Westat’s  findings for the period from 2002 to 2004 showed that the campaign also  was not effective after ONDCP took these steps.
section_title: ONDCP Cites Major Changes in Campaign
paragraphs: ONDCP states that the campaign is substantially different from what it  was when the last data were collected by Westat more than 2 years ago.  We are not in a position to comment on ONDCP’s new campaign (“Above  the Influence”), launched in November 2005, as these current efforts are  beyond the scope of our report and outside the time frame of the Westat  data collection. At this time, neither we nor ONDCP have empirical  information with which to assess this revised campaign. However,  Westat’s evaluation showed that neither the campaign as initially  implemented nor the redirected campaign implemented after 2002 was  effective. Hence, although a new and improved campaign may be effective,  Westat’s findings raise concerns about whether any campaign can affect  youth drug use, especially since the lack of effect does not seem to be  related to recognition of campaign ads, but rather to subsequent impact on  attitudes and behaviors. Finally, ONDCP cites the receipt of awards from  both the advertising and communications industry for its newest  campaign. While laudable, these awards are not evidence that the new  campaign will change youth drug attitudes and behavior. Only an  independent evaluation can assess the current campaign’s effectiveness.
section_title: ONDCP Offers an Alternative Explanation for Counterintuitive Results
paragraphs: ONDCP stated that there is growing research evidence showing that  asking people a question about their future behavior influences the  subsequent performance of the behavior in question. ONDCP then  indicates that the use of a panel design for the Westat study with repeated  interviews of youth concerning drug attitudes and behaviors might, itself,  have resulted in increased perceptions that drug use is widely pervasive  among youth. If, during the course of the Westat study, ONDCP and NIDA,  who acted as monitor for the study, felt that the study itself—that repeated  interviews of youth by Westat concerning the campaign and drug attitudes  and behavior—was resulting in a negative effect, it would have been  appropriate for them to discontinue the study to avoid potential harm to  subjects. Although ONDCP raised this issue in its comments to us, neither  ONDCP nor NIDA mentioned this issue in any of our previous meetings  specific to this engagement.
section_title: ONDCP Takes Issue with the Timing of Our Review
paragraphs: ONDCP said that the “long delay” in receiving our assessment of the  Westat report has prevented it from making progress on the next round of  evaluation. We note that Westat’s draft final report was not made available  to us until spring 2005 (not 2 years ago as seems to be indicated in  ONDCP’s comments). The volume of reports from the 4½-year study, and  the complexity of the review required a great deal of time from our most  skilled social scientists and statisticians. Time was required to ensure that  our review of the Westat study was both comprehensive and correct.
section_title: Points Concerning Our Matter for Congressional Consideration
paragraphs: ONDCP said that our matter for congressional consideration—that  Congress consider limiting appropriations until ONDCP is able to provide  credible evidence of the effectiveness of exposure to the campaign on  youth drug use outcomes—offers insufficient detail concerning how to  demonstrate satisfactory evidence of progress and that it was puzzled by  our lack of recommendations to ONDCP for improving the campaign. Our  mandate was to assess Westat’s evaluation and to draw conclusions about  the reliability of its findings so that Congress could make decisions about  funding for the campaign, and developing suggestions for improvements to  the media campaign itself was beyond our scope. In so doing, we focused  on Westat’s methods and efforts to address challenges in implementing the  evaluation. Our matter for congressional consideration was intended to  allow ONDCP to explore a number of approaches to providing credible  evidence of campaign effectiveness to Congress. Our report clearly  indicates that one approach is the one applied in the Westat evaluation,  which is the focus of this report, but we do not want to rule out other  approaches. At the same time, we acknowledge that providing such  evidence is not easy.
section_title: ONDCP Posits Consequences of Further Budget Cuts
paragraphs: ONDCP states that further budget cuts to the campaign could have far- reaching and unfavorable consequences in youth drug use. Given that the  Westat findings show that the campaign was not having a positive impact,  we found no evidence that a reduction in campaign advertisements would  have a negative impact. ONDCP cites the 2005 MTF as an indicator of  media campaign effectiveness by indicating that the reduction in anti-drug  messages has resulted in a flattening of 8th graders’ perception of risk.  Again, as ONDCP has indicated, the relationship cannot be assessed with  MTF because it does not ask respondents about their exposure and  reactions to the messages of the media campaign that can then be linked  to their drug-related attitudes and behaviors.
Failure to continue the media campaign’s efforts, according to ONDCP, is  “raising a white flag to those who favor drug legalization, with the  expectation that youth drug use soon would begin to rise, reversing years  of hard-earned positive news.” In our view, on the other hand,  continuation of programs that have been demonstrated not to work diverts  scarce resources from programs that may be more effective.
We are sending copies of this report to other interested congressional  committees and the Director of the Office of National Drug Control Policy.  We will make copies of the report available to others upon request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  either Nancy Kingsbury at 202-512-2700 or by e-mail at  KingsburyN@gao.gov or Laurie Ekstrand at 202-512-8777 or by e-mail at  EkstrandL@gao.gov. Contact points from our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors are listed in appendix III.
section_title: Appendix I: Westat’s Methods for Addressing Evaluation Implementation Issues
paragraphs: This appendix provides additional details about how Westat’s addressed  evaluation implementation issues related to the coverage of the National  Survey of Parents and Youth (NSPY), sample attrition, and its analytic  methods.
subsections: section_title: Coverage in the NSPY
paragraphs: The NSPY was a nationwide household survey of youth aged 9 to 18 and  their parents. Westat used a dual-frame sampling frame—or list of the  members of the population from which the sample was ultimately  selected. One frame—the area frame—consisted of housing units that had  been built by late 1991; the second frame—the building permit frame— consisted of building permits issued between January 1990 and December  1998 for new housing. Combined, these frames constituted an estimated  98 percent of dwelling units nationwide that existed by the end of 1998.
A household had to meet two criteria in order to be eligible to be included  in the NSPY sample: It had to (1) contain children within a specified age  group and (2) be a housing unit that was built before April 1, 1990, was a  mobile home, or was selected from a roster of building permits for new  housing units issued between January 1990 and December 1998. To  identify households that met these conditions, Westat drew a sample of  dwelling units and from this sample it screened households to determine  their eligibility for inclusion in the NSPY, that is, whether a household  contained children in a specified age group, where the specified age  groups were children aged 9 through 13, 12 and 13, or 9 through 18.
According to estimates provided by Westat, after completing enrollment in  the NSPY—which occurred during waves 1 through 3—the NSPY sample  covered more than an estimated 95 percent of occupied dwelling units  (households) nationwide. From its sample of occupied dwelling units,  Westat developed rosters of households that were believed to contain  youth in the target age range. At this second stage of sample enumeration,  Westat experienced a drop-off in the coverage of households that were  believed to be eligible for inclusion in the sample. The number of eligible  households enumerated in the NSPY was 30 percent smaller than the  number expected from the 1999 Current Population Survey (CPS) data.
According to Westat, coverage losses in the NSPY could have occurred for  several reasons: (1) because an interviewer may have decided to classify a  household as an ineligible household rather than as a nonresponding  household, (2) because the household respondent took cues from the  screening questions to avoid selection into the sample by giving an  incorrect answer, or (3) because the doorstep enumeration process was  considered to be intrusive. Westat reported that it could not conclusively  rule out the first explanation for coverage losses. However, it undertook  sample validation procedures that examined whether ineligible  households in the recruitment waves were misclassified, and it found  none. Neither Westat nor the National Institute on Drug Abuse reported  that undercoverage was primarily due to respondents avoiding selection  into the sample by taking cues from the screening questions and giving  incorrect answers as a way to avoid selection into the sample. Overall,  Westat reported that the main reason for undercoverage was the rostering  component of the survey, which required actual entry into the home, and  led to “a great many respondents” asking the interviewer to come back at a  later date, only to repeat the request when the interviewer reappeared.  Westat inferred that this represented passive refusal to participate.  Therefore, according to Westat, most of the coverage losses occurred  during the doorstep screening process in which simple, focused screening  questions about the composition of the household were used to identify  households from which to sample eligible youth.
section_title: NSPY and CPS Comparisons of Distributions on Analyzed Variables
paragraphs: In response to questions from us, Westat provided data that indicated that  the coverage losses in the NSPY did not result in differences in the  estimated distributions of population characteristics from the NSPY as  compared with those estimated from the CPS data. In other words, the  distributions of characteristics of eligible households with youth included  in the NSPY were broadly consistent with a variety of corresponding  distributions from the 1999 Current Population Survey.
The comparisons of NSPY-estimated populations to CPS-estimated  populations were based on weighted NPSY estimates, where the weights  adjusted for nonresponse at the doorstep and household enumeration  (roster) stages, and the weights also reflected the differential probabilities  of retaining a household for the NSPY depending on the screener group to  which it was applied. These weights were calculated prior to Westat’s  poststratification calibration techniques, which brought the estimated  NSPY population totals into line with the estimated CPS population totals.  Hence, if upon using the weights based only on the probability of selection  and nonresponse adjustments, the population characteristics in the NSPY  differed widely from those derived from the CPS, this would constitute  evidence of potential bias in the NSPY sample due to undercoverage.
Westat compared NSPY and CPS distributions for each of the three  enrollment waves of the NSPY (waves 1 through 3) on several variables,  including the race/ethnicity of the householder and the presence of males  28 years of age or older, the distribution of eligible households by the age  of the youth in the household, the age and gender distributions of youth,  and the age distributions of youth by race and ethnicity. Each of these  comparisons involved discrete subgroups within the focused  subpopulation of the NSPY. The largest differences between the NSPY and  CPS estimates arose in the comparison of the distributions by  race/ethnicity of household and the presence of a male 28 years of age or  older in the household. Some of these differences could also arise from  sampling variance, as both the NSPY and CPS estimates are based on  samples that are subject to sampling errors. Although Westat did not  provide sampling errors with the estimates that it provided to us, some of  the differences in distributions could be apparent, as opposed to real,  differences, in statistical terms.
section_title: Undercoverage in the NSPY and Other Widely Known and Used Longitudinal Surveys
paragraphs: Coverage issues are not an uncommon problem with surveys that focus on  relatively small subpopulations within a larger population, such as  occurred with the NSPY’s focus on youth aged 9 to 18. The NSPY’s target  population of households with youth aged 9 to 18 focused on a  subpopulation that, according to 1999 CPS data, constituted about   25 percent of the roughly 104 million households in the United States.
The estimated extent of undercoverage of eligible youth in the NSPY was  comparable to the extent of undercoverage in other well-known and  widely used longitudinal surveys. Both the National Longitudinal Survey of  Youth (NLSY)—sponsored by the Bureau of Labor Statistics—and the  National Immunization Survey of Children (NIS)—sponsored by the  National Immunization Program (NIP) and conducted jointly by the NIP  and the National Center for Health Statistics of the Centers for Disease  Control and Prevention—focus on specific subpopulations, and both  experienced undercoverage that was comparable to that of the NSPY. The  1979 NLSY is a nationally representative sample of men and women born  in the years 1957 to 1964 who were ages 14 to 22 when first interviewed in  1979. It had a coverage rate of 68 percent. The 1979 NLSY has been widely  used and cited to examine a wide variety of policy issues. As documented  in the National Longitudinal Surveys’ annotated bibliography, about   3,100 journal articles, working papers, monographs, and other research  documents have been catalogued as having used the 1979 NLSY data. The  target population for the NIS is children between the ages of 19 and   35 months living in the United States at the time of the interview, and it  has been conducted annually since 1994. The survey involves the selection  of a quarterly probability sample of telephone numbers, and the coverage  has been about 20 percent lower than estimated by two other benchmark  surveys. Survey data are used primarily to monitor immunization coverage  in the preschool population in the nation and to provide national, state,  and selected urban area estimates of vaccination coverage rates for these  children.
section_title: Sample Attrition across NSPY Interview Rounds
paragraphs: In the NSPY, respondents initially recruited into the sample were to be  tracked for three additional survey rounds that covered about a 3-year  period following the recruitment round. By the final survey round of the  NSPY, the cumulative response rate—the percentage of youth or parents  in eligible households that completed all four interviews—reached  between 50 percent and 55 percent. These cumulative response rates after  four survey rounds were determined largely by the response rates during  the enrollment waves, as postenrollment, Westat was able to track,  contact, determine eligibility for reinterview, and complete interviews for  between 82 percent and 94 percent of previously interviewed respondents  between two successive interview waves. The response rates achieved for  the first three survey waves—the enrollment waves—were generally  similar. Specifically, about 74 percent to 75 percent of the dwelling units  determined to be eligible for the survey in waves 1 through 3 completed  the household enumeration (or rostering of youth). After obtaining  consent to conduct interviews from parents and youth, interviewers  completed extended interviews—that is, completed the full NSPY  questionnaire—with about 91 percent of the sampled youth in each of  waves 1 through 3. Among sampled parents, about 88 percent gave  consent and completed extended interviews in the enrollment waves.   (See table 2.)
Across the three follow-up rounds of the NSPY, Westat achieved between  an 82 percent and a 94 percent longitudinal response rate. Follow-up  required that respondents be tracked over time and across places, as  persons enrolled in the sample could move, and their eligibility for a  follow-up interview had to be determined. For example, youth who turned  19 years of age between survey rounds would no longer be eligible for  reinterview, as they were beyond the target age of the campaign. Efforts to  track individuals prior to the second survey round included verifying  address change information with the U.S. Postal Service and obtaining  location information from a national database company. Westat obtained  updated location information from these sources, and telephone  interviewers placed calls to these households to verify the identity of  respondents. According to Westat, a high proportion of the households  that moved were contacted and respondents verified their new addresses.  During the third and fourth survey rounds, Westat used procedures to  track and verify addresses that were similar to those used to track  respondents from the first to second survey rounds, although Westat  modified these procedures as necessary. The key eligibility requirement  for youth for a follow-up interview was the youth had to be 18 years of age  or younger at the time of the interview.
For the first follow-up round—waves 4 and 5—Westat located individuals  and determined eligibility for 92 percent of the youth and 92 percent of the  parents who completed an initial interview during the first round of the  survey—that is, in waves 1, 2, and 3, and of these youth who were still  eligible, 94 percent completed an interview. Among parents from the first  round who were tracked and determined to be eligible in the second  round, 92 percent completed a second round interview. In the third and  fourth survey rounds of the NSPY, between 96 percent and 97 percent of  the youth and parents who had completed prior round surveys were  tracked and determined to be eligible, and of these, the youth response  rates were 96 percent and the parent rates were 95 percent.
section_title: Comparisons of Respondents and Nonrespondents across NSPY Survey Waves
paragraphs: Even with the relatively high follow-up response rates that Westat  achieved, it is possible that respondents could differ from nonrespondents  in follow-up rounds, and if so, the NSPY estimates of the effects of  exposure on outcomes would be biased. Westat provided data that  compared nonrespondents to the respondents across the three enrollment  waves, indicating that with some differences, nonrespondents were  generally similar to respondents with respect to characteristics that might  affect survey outcomes. Nonrespondents were compared to respondents  on gender, age at interview, whether both parents were in the household,  the number of youth in the household, the type of household dwelling, and  the type of area in which the household was located. For example, apart  from the three differences below, nonrespondents and respondents were  similar in characteristics across survey waves: In the three enrollment  waves, nonrespondents were proportionately older youth than  respondents; in waves 2 and 3, there were proportionately more youth  living in cities among nonrespondents than respondents; and in wave 1,  there were proportionately more youth in the building permit sample  among nonrespondents than respondents.
section_title: Differences in Sampling Methodologies between NSPY and MTF
paragraphs: Westat compared estimates of drug-use prevalence from the NSPY data  with those obtained from other national surveys such as Monitoring the  Future (MTF). While the NSPY estimates of marijuana use prevalence  differ over some periods covered by the NSPY from those derived from the  MTF survey of youth in school, differences between the two surveys’  sampling frames and methodologies mean that direct comparisons  between the two surveys must be made with caution and must take the  methodological differences into account. Specifically, MTF showed a  decline in marijuana use for some teenage groups during the 2000 to 2002  period, while the NSPY showed the increases reported above. However,  the difference in drug use rates reported from the two surveys could  plausibly arise from differences in the sampling frames. The MTF sampling  frame covers only youth who are in school and not those who drop out of  school, who are truant on the survey day, or who are 17- and 18-year-olds  who have graduated from high school. To the extent that high school  dropouts and truants have more involvement with drugs than those who  stay in school, the MTF estimates of drug use may underrepresent drug  use among all youth of high school age. By comparison, the NSPY  household survey includes youth who are not enrolled in school in its  sampling frame. To the extent that dropping out of high school is  correlated with drug use, and given that dropouts are excluded from the  MTF sampling frame, differences in drug use between MTF and NSPY  could reflect the fact that youth enrolled in high school reported drug use  at different rates from all youth in the general population covered by the  NSPY, which would include dropouts who may be at higher risk of using  drugs.
section_title: The Capacity of the NSPY to Detect Reasonably Small Effects
paragraphs: One challenge in designing surveys to evaluate changes in outcomes as the  result of an intervention lies in selecting a sample with sufficient power to  detect differences between groups—including the same individuals at two  points in time—or significant associations among variables, such as  between levels of exposure to the campaign and outcomes. Sample size is  a major factor determining a study’s power to detect differences, and  while larger sample sizes will generally allow researchers to detect smaller  differences over time, as the size and power of a sample to detect changes  increases, so too generally does its cost.
In consultation with the National Institute on Drug Abuse (NIDA), Westat  chose to compute power for analyses of annual change in a prevalence  statistic—that is, change in the percentage of a population that reported an  outcome. For purposes of its power analysis, Westat chose to assume  different baseline prevalences for parents and for youth of all ages and to  assume that the study should be able to detect reliably declines of  specified sizes. For example, for youth of all ages, Westat assumed a  baseline prevalence of 10 percent and determined the power of its sample  for detecting a minimum downswing in an outcome—such as past-month  drug use—of 2.3 percentage points over a year. The power of the sample  to detect this difference was well within conventional power criteria.
As reported above, the sizes of differences that Westat’s sample could  detect were consistent with the Office of National Drug Control Policy’s  (ONDCP) goals for the campaign. In early meetings on the design of the  evaluation of the media campaign, ONDCP officials reported that ONDCP  had a specific Performance Measures of Effectiveness (PME) system and  that the campaign was embodied within the first goal of the National Drug  Strategy, which was to “educate and enable America’s youth to reject  illegal drugs as well as the use of alcohol and tobacco.” Under this goal,  ONDCP’s PME proposed targets for reducing the prevalence of past-month  use of illicit drugs and alcohol among youth from a 1996 base year: by  2002, reduce this prevalence by 20 percent, and by 2007, reduce it by   50 percent. ONDCP officials further identified specific targets for the  media campaign, again with respect to a base year of 1996: by 2002,  increase to 80 the percentage of youth who perceive that regular use of  illicit drugs, alcohol, and tobacco is harmful; and by 2002, increase to   95 the percentage of youth who disapprove of illicit drug, alcohol, and  tobacco use. To achieve a goal of 80 percent of youth who perceive that  regular use of marijuana is harmful would require increasing the 1996  baseline percentage of youth perceiving marijuana as harmful from   60 percent, as measured by MTF, or by about 3.3 percentage points per  year from 1996 to 2002. Westat’s sample had sufficient power to detect this  amount of annual change in youth attitudes.
The power of the NSPY to detect changes in outcomes due to exposure to  the campaign also presumes that it was possible to accurately measure  and characterize exposure to the campaign by the reported number of  advertisements recalled by respondents. While the general question of  how exposure to advertisements affected respondents was beyond the  scope of the evaluation, if by exposure is meant a recognition-based  task—or encoded exposure—then the NSPY measures of exposure can be  viewed as valid. According to communications researchers, often what is  of interest to campaign planners and evaluators is whether the  presentation of campaign content generates at least a memory trace in  individuals. At this point, a potential audience member can be said to have  engaged the campaign’s presentation in a meaningful sense, and this is  what is meant by encoded exposure. To measure exposure to the  campaign for both youth and parents, NSPY interviewers asked  respondents about their recall of specific current or very recent television  and radio advertisements.
There was variation in recall of advertisements by both youth and parent  respondents, and this type of variation is needed in order to examine  associations between levels of exposure and outcomes. For example, for  the entire campaign, youth reported a median of 12 exposures per month,  and 76.7 percent reported 4 or more exposures per month. Comparatively  few youth—about 6 percent—reported less than 1 exposure per month.  Youth recall of specific exposure also varied, as 41.2 percent of youth  reported 12 or more television exposures per month throughout the  campaign while reporting a median of 4.4 exposures to television  advertisements. Additionally, Westat’s measures of exposure and  outcomes have demonstrated sensitivity to detect favorable campaign  effects among parents.
Westat’s test for associations between exposure and outcomes—the  gamma coefficient—was an ordinal test statistic for whether two variables  (e.g., exposure and marijuana use initiation) have a montonic, but not  necessarily a linear, relationship. Therefore, were there nonlinear  relationships, its test would have allowed for them. Finally, nonrandom  measurement error in the measure of exposure is unlikely to have biased  estimates of campaign effects, as if the nonrandom measurement error  were constant, it would not affect measures of association, and if it was  not constant, it would be addressed by Westat’s statistical methods.
section_title: Westat Methods to Measure Outcomes
paragraphs: Westat measured a variety of outcomes for youth and parents and took  steps to ensure that the measures were consistent with existing research.  The youth questionnaires included numerous questions that were designed  to measure exposure to the campaign advertisements and other anti-drug  messages. The youth question domains included exposure propensity to  media; current and past use of tobacco, alcohol, marijuana, inhalants, and  Ecstasy; past discussions with and communication of anti-drug messages  from parents and friends; expectations of others about respondent’s drug  use; knowledge and beliefs about the positive and negative consequences  of drug use; exposure to campaign messages; family and peer factors;  personal factors; and demographic information. Westat used two separate  questionnaires for youth of different ages; one questionnaire was used for  children (aged 9 to 11) and another one was used for teens (aged 12 to 18).
The NSPY parent questionnaire also included numerous questions that  were intended to measure parents’ exposure to the campaign’s messages  and other anti-drug messages. The question domains for parents included  media consumption; past discussions with child about drug attitudes and  avoidance strategies; past child monitoring behaviors; self-efficacy of  discussing drugs with child and monitoring of child’s actions; belief that  the child is at risk of drug use; belief that drug use has bad consequences;  exposure to the campaign’s advertising, including brand recognition;  parent’s own current and past use of tobacco, alcohol, and drugs; and  demographic information.
Westat followed generally accepted procedures in developing the survey  instruments for the NSPY by using information from a prototype prepared  by NIDA and using information from other surveys that addressed youth  drug use and prevention. Prior to the phase III evaluation, and in  preparation for the NSPY, NIDA convened an expert panel to assist in the  development of the youth and parent questionnaires. The panel, which  consisted of experts in adolescent drug use prevention and parenting  behaviors, drafted NSPY survey questionnaires for children, teens, and  parents, and NIDA shared these prototypes with Westat at the beginning of  Westat’s evaluation contract. In developing the final questionnaire for the  NSPY, Westat created a questionnaire development team consisting of  evaluation experts. In developing the final NSPY questionnaires, the  Westat team reviewed NIDA’s prototype and other surveys.
Westat measured youth drug use by self-reported data on use. We have  previously cautioned about limitations associated with self-reported data  on youth drug use. Additionally, the National Research Council (NRC) of  the National Academy of Sciences also has pointed out limitations  associated with self-reported drug use in national surveys such as the  National Survey of Drug Use and Health (NSDUH) and MTF. As NRC has  pointed out, while self-reported data on drug use may have limitations for  estimating the actual levels of use at a particular point in time, they may  not suffer from these same limitations when they are used to assess  changes in use over time, unless there is reason to believe that attitudes  about drug use change in ways that affect respondents’ willingness to  honestly report drug use, or stigma.
Specifically, if there is a stigma associated with self-reporting drug use,  that stigma may affect the levels of use reported, as some have argued that  the propensity of respondents to give valid responses may be affected by  social pressures. In particular, the incentive to give false negative reports  may increase over time if drug use becomes increasingly perceived as  harmful or socially unacceptable. Using data from NSDUH and MTF, NRC  showed an inverse relationship between the percentages of respondents  who either disapproved of illegal drug consumption or perceived it to be  harmful. Thus, as stigma increased, self-reported drug use decreased. As  NRC cautioned, one could interpret this relationship as indicating that  changes in stigma are associated with changes in invalid reporting, or as  stigma increases, false negative reports increase, rather than necessarily  indicating that as stigma increases, drug use decreases.
The NRC analysis leads to two inferences: First, if social stigma remains  constant over time, changes in the propensity to give valid responses  would be unaffected and estimates of change in self-reported drug use  would not be biased by social stigma. For the evaluation results, this  would imply that its measures of changes in self-reported drug use would  provide valid measures of changes in use, so long as factors other than  stigma did not affect the propensity to self-report use. Second, if the social  stigma associated with reporting drug use is inversely related to  disapproval of illicit drug use or increased perceptions that it is harmful,  then the estimates of self-reported drug use are likely to decrease as a  result of the stigma. According to results from the evaluation, trends in  youth attitudes and beliefs about illicit drugs changed significantly over  the entire campaign in a direction that was favorable to the campaign.  Specifically, the trends in youth attitudes and beliefs about illicit drug use  meant that youth were more likely to believe, as the campaign went on,  that use of illicit drugs was likely to have negative consequences.  Alternatively, the social stigma associated with drug use increased over  time. If the relationship between stigma and reporting that NRC found  held and applied to the data in the evaluation of the campaign, this would  imply that the increased stigma associated with drug use would lead to  decreases in self-reports of drug use over time.
section_title: Westat’s Analytic Methods
paragraphs: To control for the many factors that could have influenced both exposure  and outcomes independently of, or in conjunction with, the campaign,  Westat used propensity scoring methods to match individuals based on  numerous measured attributes and to create groups of individuals who  differed on their underlying propensity to be exposed to different levels of  campaign advertisements. A propensity score is a weighted sum of the  individual effects of variables in a model that predicts the likelihood of  exposure to campaign messages. Westat’s propensity scoring methods  resulted in the creation of groups of individuals who were statistically  similar on exposure propensities. These groups can be considered as  statistical analogues to randomly assigning individuals to different levels  of exposure. After creating these groups, Westat then analyzed outcomes  between the groups having different propensities to be exposed to  campaign messages.
Westat used ordinal logit models to estimate the chances of being  exposed, where exposure was measured alternatively as a three- or four- level variable—e.g., low, medium, or high exposure. Westat used a myriad  of variables to predict exposure levels in both the youth and parent  models. For example, in the youth models, the propensity score models  included measures of demographic attributes, educational attainment and  educational aspiration, family and parent background, parent consumption  of television and other media, income and employment, reading habits,  Internet usage, location of residence in urban areas, among other  variables. After estimating models, Westat also assessed the balance of  variables in its propensity models. For propensity models to remove the  effects of confounding variables from the association between exposure  and response, it is necessary that the population means of the confounder  variables not vary across exposure levels. If a confounder is successfully  balanced, then it will have the same theoretical effect across all exposure  levels.
The net result of the propensity scoring models is to provide each  individual with a score that reflects the individual’s propensity to recall  advertisements based upon a weighted sum of all of the variables in the  model. Therefore, while two individuals may differ on the likelihood that a  particular variable affects their chances of being exposed to messages or  on their levels of a certain variable—such as age or education—they could  be similar in their overall propensity to be exposed to campaign messages  if the differential effects of any individual variables sum to the same total  propensity.
In order for the results of propensity methods to be valid, it is important  that the propensity scoring models include all relevant variables that could  otherwise explain differences in both exposure and outcomes. Propensity  score models can adjust only for confounding variables that are observed  and measured. In other words, they are built upon the assumption that all  relevant variables are measured and controlled for. If an important  variable is omitted from the propensity model, the results of analyses may  be affected. Westat made reasonable attempts to identify and control for a  variety of confounding variables, include them in its models, and reduce  bias.
section_title: Appendix II: Comments from the Office of National Drug Control Policy
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the contacts named above, contributors to this report  included David P. Alexander, Billy Commons, James Fields,   Kathryn Godfrey, Mary Catherine Hult, Jean McSween, Karen V. O’Conor,  Mark Ramage, William J. Sabol, Barry J. Seltser, and Douglas Sloane.