Performance Metrics for the Integrated Marketing and Communications
Campaign

**Jennifer Marks**

Census Advisory Committee of Professional Associations Meeting

American Marketing Association Committee

October 26, 2006

# Abstract

Measuring the success of communications, especially communications to
effect behavioral changes, presents a unique challenge. Evaluations of
the Census 2000 integrated partnership and marketing campaign suggested
a successful effort -- the decline since 1970 in the mail response rate
was reversed; and the total and differential undercounts dropped
substantially. This paper examines evaluation strategies for other
government campaigns -- military recruiting and the anti-drug campaign
\-- to see if their recent experiences might be useful in informing
evaluation plans for the 2010 communications campaign.

# Questions for Committee

1)  Should the evaluations done for 2000 (surveys of attitudes and
    intention and modeling) be repeated for 2010?

2)  Is the measurement of factors that lead to behavior (awareness,
    attitude, intention) adequate as an evaluation strategy?

3)  If not, what other options are there?

4)  Are there additional measures related to specific campaign tactics
    that should be considered?

**Disclaimer.** This report is released to inform interested parties of
ongoing work and to encourage discussion of issues related to the
project. The views expressed on methodological issues are those of the
author and not necessarily those of the U.S. Census Bureau.

**Background**

The paid advertising campaign in support of Census 2000 was a first-time
effort and was generally seen as highly successful. Along with an
ambitious partnership effort, the Partnership and Marketing Program was
considered to have played a major role in raising the mail response rate
from an expected 61percent to 67percent. Quantifying that relationship
proved to be difficult. An independent evaluation[^1] as well as a
sophisticated model created to examine the effects of partnership and
marketing on response[^2] failed to demonstrate the expected
relationship. In the world of social marketing, this outcome is not
unusual.

As we prepare for the 2010 Census Communications Campaign, we must
decide 1) how to evaluate the effectiveness of the overall campaign, and
2) how to assess the performance of the Contractor.

# Goals

The Census Bureau in 2010 will make every effort to improve the accuracy
of census coverage by enumerating each household and person. The overall
communications strategy must support the following goals:

-   Increase mail response

-   Improve cooperation with enumerators

-   Improve accuracy/ reduce the differential undercount

Each of these goals is measurable but making the connection to
communications will be difficult.

# The Experience of Other Government Advertisers

The 2010 Census Communications Campaign will encompass advertising,
public relations, grassroots efforts, etc. For the purposes of this
discussion, we looked at other advertising campaigns, recognizing that
measurement issues may differ somewhat for advertising-only campaigns.

The question of how best to evaluate advertising has been examined by
Rand, the National Research Council of the National Academy of Sciences,
and the Government Accountability Office (GAO) for military recruitment
advertising and by GAO for the anti-drug campaigns.

## Military recruitment

In 2003, GAO studied the adequacy of measures that the Department of
Defense (DOD) used to evaluate advertising effectiveness. GAO
recommended "DOD set clear, measurable advertising objectives; develop
outcome measures to evaluate advertising programs' performance; and use
these measures to monitor advertising's performance and make choices on
recruiting investments."[^3] In response, DOD turned to the National
Academy of Sciences to develop an advertising strategic framework.[^4]
The National Academy of Sciences' recommended framework recognizes that
different research designs should be used to answer different types of
research questions, or to address different measures of success. A
version of their framework, adapted for the census, is shown below:

Evaluation Framework

  ----------------- ----------------- ------------------- --------------------
                    Outcome Not       Specific Attitudes  Actual Behavior
                    Specified A       or Behavioral       
                    Priori            Intentions          

  Question          "What does a      "What is the effect "What is the effect
                    target audience   of the campaign on  of the campaign on
                    see as attractive specified attitudes response to the
                    or unattractive   or behavioral       census?"
                    features of a     intentions?"        
                    campaign?"                            

  Method            Focus groups;     Surveys;            Surveys;
                    unstructured or   quasi-experiments   quasi-experiments;
                    open-ended                            modeling
                    surveys and                           
                    interviews                            
  ----------------- ----------------- ------------------- --------------------

If the Census Bureau were to use this framework, we would measure
effectiveness based on:

1\) the effectiveness of the messages or creative concepts (through
focus groups or interviews),

2\) increases in awareness (measured by surveys),

3\) changes in attitudes (measured by surveys), and

4\) increases in the intention to respond and actual response (measured
by quasi-experiments/models).

### The Anti-Drug Campaign

The National Youth Anti-Drug Media Campaign, administered by the Office
of National Drug Control Policy (ONDCP), has the goal of preventing drug
use among the nation's youth. Westat, Inc., under the auspices of the
National Institute on Drug Abuse (NIDA), evaluated the campaign and
found, according to the GAO, "no significant favorable effects of
campaign exposure on marijuana initiation among non-drug-using youth or
cessation and declining use among prior marijuana users."[^5] Although
GAO found Westat's analysis to be sound, ONDCP pointed out a number of
difficulties associated with proving a causal relationship between
advertising and behavior.

ONDCP has modified its program goals to include:

1\) Increase awareness

2\) Improve attitudes

3\) Change intentions

4\) Improve management (implement improved communications, training,
best practices, reviews, and quality controls)

**Implications for Census**

In the draft Request for Proposals for the 2010 Census Communications
contract, we suggested evaluating the extent to which the communications
campaign moves people toward the goal of responding to the census. This
might consist of measuring the effectiveness of specific messages for
target audiences (copy research), measuring increases in awareness of
the census and changes in attitudes toward the census (survey research),
and measuring changes in intention to return the census form and actual
return of the form (survey research; modeling). These are general
measures of effectiveness, and when used together, provide a good
indication of how well a campaign does in support of the overall
objectives.

The Contractor's first assignment will be to develop a research-based
national communications campaign plan. As part of that task, the
Contractor will recommend specific campaign strategies and tactics. The
Census Bureau will develop specific additional performance measures
based on the type of campaign developed. For example, a campaign that
uses advertising heavily might be evaluated based on such measures as
reach and frequency, or recall of messages. A campaign that is built
around grassroots Word-of-Mouth or the Internet would require different
specific measures. The combination of the broad measures (awareness,
attitudes, and behavior) with the specific, task-related measures are
intended to be strong tools for evaluation.

Individual task orders will describe additional performance measures and
how each will be measured, as well as the specific numeric goals.

[^1]: Census 2000 Topic Report No. 6, *Evaluations of the Census 2000
    Partnership and Marketing Program*, February 2004.

[^2]: Bentley, Michael, *Evaluation of Partnership and Marketing on
    Improving Census 2000 Mail Return Rates*, April, 2003.

[^3]: United States Government Accountability Office, Report to the
    Senate and House Committees on Armed Services. *Military Recruiting
    DOD Needs to Establish Objectives and Measures to Better Evaluate
    Advertising's Effectiveness.* GAO-03-1005, September 2003.

[^4]: National Research Council. *Evaluating Military Advertising and
    Recruiting: Theory and Methodology.* Committee on the Youth
    Population and Military Recruitment---Phase II. Paul R. Sackett and
    Anne S. Mayer, editors. Board on Behavioral, Cognitive, and Sensory
    Sciences, Division of Behavioral and Social Sciences and Education.
    Washington, DC: The National Academies Press.

[^5]: United States Government Accountability Office, Report to the
    Subcommittee on Transportation, Treasury, the Judiciary, Housing and
    Urban Development, and Related Agencies, Committee on
    Appropriations, U. S. Senate. *ONDCP Media Campaign Contractor's
    National Evaluation Did Not Find That the Youth Anti-Drug Media
    Campaign Was Effective in Reducing Youth Drug Use*. GAO-06-818,
    August 2006.