Putting Disclosures to the Test September 15, 2016

Impact on Decision Making and Behavior

Moderator: Janis Pappalardo

Asst. Director for Consumer Protection

Bureau of Economics, FTC

Lillian Ablon

Information Scientist

RAND Corporation

Idris Adjerid

Mendoza College of Business

University of Notre Dame

Ginger Zhe Jin

Director

Bureau of Economics, FTC

Adair Morse

Finance Group, Haas School of Business

UC Berkeley
Putting Disclosures to the Test September 15, 2016

Lillian Ablon

Information Scientist

RAND Corporation
INTO THE BREACH:

Sasha Romanosky

Into the Breach: Consumer Attitudes Toward 

Breach Notifications and Data Loss

Lillian Ablon

Paul Heaton, Diana Lavery, Sasha Romanosky

Into the Breach: Consumer Attitudes Toward 

Breach Notifications and Data Loss

Lillian Ablon

Paul Heaton, Diana Lavery, Sasha Romanosky

The research presented was funded by the RAND Corporation’s Institute for Civil Justice
Ablon - 5

5

76 million accounts 110 million

80 million 22 million
Ablon - 6
Ablon - 7

Are data breach notifications helping?

• All but 3 states now require companies to notify people 

about the loss of personal data

• Purpose is two-fold

• Allow people to take quick action to reduce risk

• Create incentives for companies to improve data 

security

• Very little research on consumer response
Ablon - 8

This study focused on the consumer experience

• Frequency of breach notifications and type of data lost

• Consumer response

• Perceived cost of the breach to consumers
Ablon - 9

This study focused on the consumer experience

• Frequency of breach notifications and type of data lost

• Consumer response

• Perceived cost of the breach to consumers

We used the American Life Panel survey

Instrument for this study
Ablon - 10

• Nationally representative panel of over 6,000 

individuals

• Internet-based survey, allowing for a “real-time pulse” of 

the American public

• Yields a relatively high response rate

American Life Panel (ALP) survey was 

a useful instrument for our study
Ablon - 11

• Our survey method is useful for policymakers to get a 

pulse of the American public

– Repeatable, nationally representative, high response 

rate

• Responses are based on consumer recall

– Consumer recall is likely not perfect

– Consumers may say one thing but act a different way

– Consumer response and behavior may change over 

time

Benefits and costs of our methodology
Ablon - 12

Survey details

• Last 2 weeks 

of May 2015

• (OPM breach 

disclosed June 4; 

notifications sent in 

July & August)

• 2,618 adults • 2,036 

respondents

• 78% response 

rate
Ablon - 13

Findings
Ablon - 14

How often does this happen?

What kind of data is lost?
Ablon - 15

Of those surveyed, 26% recalled receiving 

a breach notification in the last 12 months

An estimated

64 million Americans
Ablon - 16

No. of 

notification

s

1

3+

2

3

Of those alerted, over 50% recalled receiving 

more than one notification

26%
Ablon - 17

56%

• Media reports

• Bank or other third party

• Identified suspicious 

activity on their own

Through other means 44%

From the company

Many respondents learned of the breach 

before they received the notice
Ablon - 18

Most common types of data compromised

• Credit card information 49%

• Health information 21%

• Social Security number 17%

• Other personal data 13%
Ablon - 19

How did consumers respond?
Ablon - 20

77% were satisfied with the company’s response

Dissatisfied

Satisfied
Ablon - 21

Most remained loyal to the company

11%

23%

65%

1%

stopped doing business

 with the company

decreased business 

no change

increased business
Ablon - 22

AllClear ID protect your identity for two (2) years at no cost to you.
Ablon - 23

23

When free credit monitoring was offered, 

62% accepted

When free credit monitoring was offered, 

62% accepted
Ablon - 24

Many took steps to improve their data security

• Changed PINs or passwords

• Became more diligent

• Closed or switched accounts

• Notified others

• Started using a password manager

51%

24%

24%

17%

4%
Ablon - 25

Consumer recommendations for breached firms 

“to greatly improve satisfaction”

1. Apologize

2. Notify consumers immediately

3. Take measures to prevent future breach

4. Donate money to a cyber security organization

5. Compensate for financial loss

6. Offer free credit monitoring, similar services

24%

63%

68%

11%

54%

64%
Ablon - 26

How do consumers estimate the cost 

of the data breach?

26
Ablon - 27

Overall, they estimated a modest cost

• 32% reported no cost at all

• $500 was the median cost

• 6% put inconvenience cost at $10,000 or more
Ablon - 28

Conclusions and implications
Ablon - 29

Are data breach notifications 

serving their purpose?

Do they allow people 

to take quick action 

to reduce risk? 

Do they create incentives 

for companies to improve 

data security?
Ablon - 30

Are data breach notifications 

serving their purpose?

Do they allow people 

to take quick action 

to reduce risk? 

Do they create incentives 

for companies to improve 

data security?

No: 44% already knew of the breach 

Yes: 78% took additional action
Ablon - 31

Are data breach notifications 

serving their purpose?

Do they allow people 

to take quick action 

to reduce risk? 

No: 44% already knew of the breach 

Yes: 78% took additional action

No: Most were satisfied and loyal

Yes (no? maybe?): Breaches 

appear to be on the rise

Do they create incentives 

for companies to improve 

data security?
Ablon - 32

Thank you

Lillian Ablon

lablon@rand.org

@LilyAblon

Sponsored by Juniper Networks

Available at: http://www.rand.org/pubs/research_reports/RR1187.html
Putting Disclosures to the Test September 15, 2016

Idris Adjerid

Mendoza College of Business

University of Notre Dame
FRAMING, DISCLOSURES, AND THE 

RATIONALITY OF PRIVACY 

CHOICES

IDRIS ADJERID – UNIVERSITY OF NOTRE DAME

EYAL PEER – BAR ILAN UNIVERSITY

ALESSANDRO ACQUISTI – CARNEGIE MELLON UNIVERSITY

THE RESEARCH PRESENTED WAS FUNDED IN PART BY THE NATIONAL SCIENCE 

FOUNDATION, THE ALFRED P. SLOAN FOUNDATION, AND THE CARNEGIE 

CORPORATION OF NEW YORK.
Motivation

35

 When disclosures are effective: Objective differences in the content 

of privacy disclosures are the main determinants of consumers’ 

choices

 Why: consumers use disclosure to weigh the expected benefits of privacy 

choices against their potential costs, and make privacy choices 

accordingly

 When they aren’t: Factors independent of the objective features of 

privacy disclosures can consistently and powerfully impact consumers’ 

choices

 Why: predictable and replicable deviations from rational models of 

privacy choice arise due to their susceptibility to behavioral heuristics and 

decision biases
Relative Privacy Protection

36

Source: Acquisti, A., Brandimarte, L., & Loewenstein, G. (2015). Privacy and human 

behavior in the age of information. Science, 347(6221), 509-514.
Overview

37

Hypothesis: Consumer response to privacy disclosures can be predictably 

manipulated by framing that alters the relative perception (but not 

objective content) of privacy disclosures 

Results: Holding the objective privacy disclosures constant, individuals are 

significantly more likely to share personal information if they perceived a 

relative increase in privacy protection and vice versa

Implication: Subtle changes to the framing and presentation of privacy 

disclosures can have powerful (and sometimes perverse) impacts, limiting 

the intended benefits of privacy disclosures. 
Methodology

38

 Participants are recruited using crowdsourcing services

 Amazon Mechanical Turk, Prolific Academic, etc.

 Study presented as a study on “ethical behavior”

 Manipulation: simple privacy disclosures (notices)

 Text and Graphical Notices

 Behavioral Measure: Sharing of sensitive information

 “Have you ever looked at pornographic material”

 “Have you ever cheated on a partner”
Experimental Design

39

Low Protections

High Protections

Low Protections

High Protections

Round 1 Round 2

Increasing 

Protections

Decreasing 

Protections

Low Protections

High Protections

Low Protections

High Protections
Privacy Disclosures

40

The analysis for this study requires that 

your responses are stored using a 

randomly assigned ID. All other 

information that could potentially be used 

to identify you (email, zip code, etc.) will 

be stored separately from your responses. 

As such, your responses to the following set 

of questions cannot be directly linked back 

to you.

The analysis for this study requires 

that your responses are stored 

using your email. As such, your 

responses to the following set of 

questions may be directly linked 

back to you.

High Protection

Low Protection
Behavioral Measures

41
Results

42

 Relative increase in protection results in a 7% increase in 

the sharing of sensitive information

 Relative decrease in protection results in a 8-10% 

decrease in sharing of sensitive information

 Objective differences (High vs. Low) resulted in a 5% 

difference in sharing of sensitive information, but only in 

first round
Advantages of Methodology

43

 Actual sharing of sensitive information as opposed to 

hypothetical behavior or scenarios

 Cost-efficient to run so it is easy to tweak and replicate

 Minimal amount of deception

 Privacy consideration are not explicitly primed

 False information (lying) is a feature of the design, not a 

limitation

 Does not require developing an IT artifact (e.g. a new 

mobile app or plug-in)
Disadvantages of Methodology

44

 Behaviors of individuals may not perfectly reflect real world 

actions

 Difficult to manipulate differences in objective risk 

 Participants may assume some level of riskless-ness in an academic 

study

 Difficult to study long-term effects

 May still trigger some suspicion about purpose of study

 Ethical behavior may not translate well to other privacy 

decision contexts
Conclusions

45

 Framing of privacy disclosures can have a significant 

impact on sharing of sensitive information

 Online experiments and crowdsourcing platforms can 

provide a replicable and reliable methodology for 

evaluating the impact of privacy disclosures

 This approach may not be a great fit if real-world 

behavior is highly desirable or longitudinal studies are 

useful.
Full Paper

46

Adjerid, I., Peer, E., & Acquisti, A. (2016). Beyond the 

privacy paradox: Objective versus relative risk in privacy 

decision making. Available at SSRN 2765097.
Putting Disclosures to the Test September 15, 2016

Ginger Zhe Jin

Director

Bureau of Economics, FTC
Impact of Disclosure 

on Economic Behavior

Ginger Zhe Jin

Director, Bureau of Economics

Federal Trade Commission

This presentation is based on my own research and the economic literature of quality disclosure and certification. The 

research presented was funded in part by the National Science Foundation, the Net Institute, the Alfred P. Sloan 

Foundation, and the National Bureau of Economic Research. Views expressed are my own, not necessarily those of the 

Federal Trade Commission or any of its Commissioners. 
Consumers

Certifiers Sellers

I will focus on quality disclosure that is truthful.

• could be made by firms or a third party;

• could be mandatory or voluntary.

Dranove and Jin (2010) “Quality Disclosure and Certification: Theory and Practice”, Journal of Economic Literature.
Consumers

Certifiers Sellers

How do consumers respond to disclosure?

• Pay attention

• Comprehend

• Sort & match

How do sellers respond to disclosure?

• Comply 

• Adjust price

• Adjust quality

• Entry/exit

• Game the system
Does disclosure improve consumer choice?

A positive example

• Public perception in NYC (18m after):

• 90% consumer approval

• 81% have seen cards

• Among those that have seen the 

cards, 88% consider it in dining 

decisions 

• Revenue in LA county (1y after):

– A grade: + 5.7%

– B grade: + 0.7%

– C grade: – 1.0%

– Industry revenue increases by 

3.3% ($250 million / year) 

Citations: 

• Wong et al. (2015) “Impact of a Letter-Grade 

Program on Restaurant Sanitary Conditions 

and Diner Behavior in New York City”, 

American Journal of Public Health.

• Jin and Leslie (2003) “The Effects of 

Information on Product Quality: Evidence from 

Restaurant Hygiene Grade Cards” Quarterly 

Journal of Economics.
Does disclosure improve consumer choice?

A counter example

Bruce Schneier: “they don’t tell people what 

they can do — they just make people afraid.”

Homeland Security Department on replacement: 

“The goal is to replace a system that 

communicates nothing ... with a partnership 

approach ...”

(NY Times 11/24/2010) 
Does disclosure improve consumer choice?

Another counter example

Lacko and Pappalardo (2004): “The Effect of 

Mortgage Broker Compensation Disclosures on 

Consumers and Competition: A Controlled 

Experiment” FTC BE staff report. 

Lab test using:

• Mortgage cost disclosures 

• With and without HUD-proposed 

mortgage broker compensation 

disclosure

Tested effect of disclosure on:

• Accuracy of consumer cost 

comparisons

• Consumer loan choice (hypothetical)

500+ consumers
Does disclosure improve consumer choice?

Another counter example

Lacko and Pappalardo find: HUD disclosure caused:

• Consumer confusion about which 

loan was less expensive 

• Mistaken loan choices leading a 

significant proportion to choose 

more expensive loans

• Bias against mortgage brokers which 

would put brokers at a competitive 

disadvantage, leading to possibly 

higher cost for consumers
Does disclosure change 

consumer behavior?

Consumers

(+) more informed choice

(0) no response

(-) wrong impression, wrong choice 
Does disclosure change 

seller behavior? Sellers
Does voluntary disclosure lead all firms to disclose?

Theory predicts (almost) yes ... because of unraveling 

Maricopa, AZ adopted voluntary restaurant grade card 

in Oct. 2011

Overall 

disclosure 

rate = 57.6%

Bederson et al. (2016) “Incomplete disclosure: evidence of signaling and 

counter-signaling” working paper. 
Sender: 

• Learn the true number 

{1,2,3,4,5} secretly

• Decide to disclose it or not

Receiver: 

• Observe message from the sender

• “The number I received is xxx” or 

• (blank)

• Guess the true number

58

Key conflict:

• Sender wants the highest guess

• Receiver wants to guess correctly

• With economic incentives

• cannot lie

Similar test in a lab
Sender’s report rate

59

Receiver’s guess 

conditional on blank report

Jin, Luca and Martin “Is No News (Perceived as) Bad News?” NBER working paper w21099.
Does disclosure lead to price changes?

For low quality

For high quality

Sort and match

Disclosure does 

not necessarily 

improve every 

consumer’s 

welfare

For high quality 

when there is 

price or capacity 

constraint
Does disclosure improve quality?

A positive example: restaurant grade cards

• % of A restaurants increases significantly

• Significant public health improvements: 

– Food-borne illness hospitalizations drop in LA county, relative to other 

parts of California (at 1y and 3y marks)

– Sanitary conditions improve in NYC (at the 18m and 2y marks)

– Salmonella infections decline in NYC relative to rest of NY, NJ and CT (at 

the 18m mark)

Citations: 

• NYC health department: “Restaurant Grading in New York City at 18 Months”, accessed on http://www1.nyc.gov.

• Wong et al. (2015) “Impact of a Letter-Grade Program on Restaurant Sanitary Conditions and Diner Behavior in New York City”, 

American Journal of Public Health, January 2015.

• Jin and Leslie (2003) “The Effects of Information on Product Quality: Evidence from Restaurant Hygiene Grade Cards” Quarterly Journal 

of Economics.

• Simon et al. (2005) “Impact of Restaurant Hygiene Grade Cards on Foodborne Disease Hospitalizations in Los Angeles County” Journal 

of Environmental Health.
Does disclosure improve quality?

A counter example

NY and PA mandated report 

cards on physician and 

hospital cardiac surgery 

mortality rates (1991, 1993)

Survey evidence: 

• 63% of cardiac surgeons reported accepting 

only healthier patients due to report cards. 

• 59% of cardiologists reported that report 

cards made it more difficult to place severely 

ill candidates for CABG. 

Empirical evidence:

• Cherry picking healthier patients

• Higher medical expenditure and worse health 

outcomes, particularly for sicker patients

Dranove et al. (2003) “Is More Information Better? The Effects of “Report Cards” on Health Care Providers” Journal of Political Economy.
Truthful quality disclosure is a double-edged sword

• Consumer behavior after disclosure

– May make more informed decision

– May sort and match

– May take wrong or no action due to unclear, incomprehensible, duplicative 

disclosure

• Seller behavior after disclosure

– May or may not disclose

– May adjust price according to disclosed quality

– May or may not improve quality

– May game the system

– May enter, stay or exit
Further Remarks

• Remaining questions

– What to include or exclude from report card?

• Sample size, risk adjustment, weighting, mean reversion 

– How does disclosure regime affect the certifier’s incentive?

• Incentives to be truthful and thorough, financial stake, competition 

• Methodology

– Surveys

– Observational studies

– Lab experiments

– Field experiments
Putting Disclosures to the Test September 15, 2016

Adair Morse

Finance Group, Haas School of Business

UC Berkeley
Engagement with the Private Sector in Testing

Adair Morse

Associate Professor of Finance,

University of California, Berkeley

FTC Workshop: Putting Disclosures to the Test, September 15, 2016
“Information Disclosure, Cognitive Biases and Payday 

Borrowers” Bertrand & Morse 2011 Journal of Finance

Topic: Even if payday loans are priced fairly and non-predatory, one has to 

wonder whether cognitive limitations or biases by some borrowers 

explain the use of payday loans

Idea (not just for this setting) : Mandate disclosure that is

– Better informed as to what mistakes are being made

– Better targeted to de-bias potential cognitive biases causing these 

mistakes

Field experiment at national chain of payday stores

• Can we impact future borrowing with debiasing disclosure. 
Information Treatment 1

Potential problem : People may not internalize APR 

because focus in store is the dollar fee structure on 

the wall. 
Treatment: Reinforce understanding of APR by 

presenting it next to other (smaller) APRs.
Information Treatment 2

Potential Problem: People fail to add up cost of single 

decision over time 
Treatment 2: Present additive dollar costs of 

payday loan fees into future
Results

• De-biasing failure to add up DOLLARS over time reduces future borrowing by 

10%. Not APR treatment

▫ How: People saved more in the interim

 Why I like that result: Economists forget that people are very constrained 

and can’t make decisions in rates, but rather live month-to-month in dollar 

terms

 But people do not go through exercise of thinking about the adding-up

• Paper advocates for 

▫ Understanding the specific cognitive biases that may lead to suboptimal 

decision-making 

▫ And subsequently designing some correcting or “de-biasing” information 

disclosure
Challenges of implementation (a sample of headaches)

• Implementation challenges:

▫ Training store clerks to be uniform!

• Randomization details matter:

▫ Stores not comparable, cannot randomize implementation by store.

▫ Cannot randomize by customer, impossible for clerk to keep track

▫ Randomize by day of the week, but need distribution across days of the 

week, because borrowers on different days of the week not random

• Estimation details matter:

▫ Observations by store may not be independent (same shocks faced by 

location)
Why did the Lender agree to do this study?

Why do any companies want to do testing in partnership with 

unbiased academics or government researchers?

(1) Discussions with private sector about objectives must be done upfront

▫ Objectives are not to show that their product is great. They understand this.

▫ But they have a pre-determined view of what an unbiased approach will 

show. If you tell them that it may not show that. Then they want a veto right. 

▫ Researchers must plan ahead.

 I say: “The reason you are talking to me is because I have credibility for 

producing unbiased research. If you go to a research organization with an 

agenda, the credibility of the study will be questioned. You decide which 

you want.”
Why did the Lender agree to do this study?

Why do any companies want to do testing in partnership with 

unbiased academics or government researchers?

(2) Essential to understand incentives

▫ The payday lender understood that I might find that disclosure reduces 

demand for their product because people acted differently in the interim to 

save for paying back the loan.

 But maybe people would default less

 And, besides, they were facing only negative media from researchers with 

a bias to show them to look bad

 They needed to take a risk on unbiased research
Why did the Lender agree to do this study?

Why do any companies want to do testing in partnership with 

unbiased academics or government researchers?

(2) Essential to understand incentives

▫ Other incentives I encounter

 Companies want to have research to genuinely evolve products to make 

people satisfied

 Fine line: Some companies want to have research to evolve to cater to 

behavioral biases or lapses to make as much profits

 Companies simply trying to learn from the engagement with a research 

team on how to think about testing and what skill sets they need to 

acquire
Final thought

• In consumer finance (and other fields), we are starting to learn about 

heterogeneities in people’s use of products or information

▫ Next slide (not covered in this presentations) has some examples

• Need to take next step:

▫ Implement methods to test designs for “pareto” policy or product 

improvements across heterogeneity of people

▫ I.e.: Make disclosure changes or regulator-governed product changes help 

some people with certain characteristics without hurting others

▫ **** Requires understanding the heterogeneities (in use of a product and 

in understanding disclosure) and then designing remedies 
Next generation: Use the literature on people’s use of 

borrowing to improve product design

• Studies of why people get into trouble

▫ Smoothing issues/making ends meet: Stephens(`03), Parsons van Wesep(`13), Leary Wang(`16)

▫ Preferences: Laibson (1997), Meier Sprenger (2010), Kuchler (2012)

▫ Neglect: Berman, Tran, Lynch, Zauberman (2015)

▫ Aging: Agrawal, Driscoll, Gabaix, Laibson (2009)

▫ Cognition/Focus: Morse Bertrand (2011), Stango Zinman (2011), etc.

• Studies of marginal use of income (helicopter drop studies)

▫ Johnson, Parker Souleles (2006;2013 w McClelland); Agrawal, Liu, Souleles (`07); Bertrand 

Morse (`09)

• Studies of consumer loan contract form

▫ 1980s literature Stiglitz Weiss, Hertzberg, Lieberman, Paravisini(`15); Carter, Skiba, Sydnor (`13)
Putting Disclosures to the Test September 15, 2016

Impact on Decision Making and Behavior

Moderator: Janis Pappalardo

Asst. Director for Consumer Protection

Bureau of Economics, FTC

Lillian Ablon

Information Scientist

RAND Corporation

Idris Adjerid

Mendoza College of Business

University of Notre Dame

Ginger Zhe Jin

Director

Bureau of Economics, FTC

Adair Morse

Finance Group, Haas School of Business

UC Berkeley

Putting Disclosures to the Test September 15, 2016

Case studies

Moderator: 

Hampton Newsome

Division of Enforcement, FTC

Colin Campbell

Dept. of Marketing and 

Entrepreneurship

Kent State University

Sarah J. Farnsworth

Vice President, Scientific Affairs

PEGUS Research, Inc.

Manoj Hastak

Deptartment of Marketing

Kogod School of Business

American University

Heidi Johnson

Office of Research

Consumer Financial Protection Bureau
Putting Disclosures to the Test September 15, 2016

Colin Campbell

Dept. of Marketing and Entrepreneurship

Kent State University
When Disclosure May Not Be Enough: 

Social Media Native Advertising and 

Multiple Ad Recognition Cues

Colin Campbell and Larry Marks 

Kent State University
Motivation

• Native advertising on the rise 

• Industry research suggests ad recognition may be 

as low as 41% (Mane and Rubel 2014)

• Much of social media advertising is native 

• Social media commonly viewed on mobile devices 

(Cohen 2016; Lella and Lipsman 2014; Tadena 2014)

We Focus on Social Media Native Advertising

• Native advertisements (like advertorials/

infomercials) generally match form & style of nonad content

• Social media = majority of native ad spend 

• Social media content = fun & entertaining

• Consumers engage in peripheral processing 

(Hoffman and Novak 2013; Schulze, Scholer, Skiera 2014)
Academic Understanding of Native Still Developing

• Persuasion knowledge activation and awareness of 

persuasive intent = more critical processing (Campbell and 

Kirmani 2000; Kirmani and Zhu 2007) 

• Existing research focuses on article-style native 

advertising (Wojdynski and Evans, 2015) 

• Overt disclosures such as “Advertising” or 

“Sponsor Content” resulted in increased ad 

recognition rates of 12% and 13%
Multiple Recognition Cues

• Ad position (van Reijmersdal, Neijens, 

and Smit, 2009)

• Central region gets more 

attention (Leonhardt, Catlin, and Pirouz 

2015; Tatler 2007)

• Brand familiarity (Keller 1993; Kent 

and Allen 1994)
Study 1

• 2 (brand familiarity: high or low) x 2 (ad position: 

in-stream or sidebar) design 

• Participants (N = 165, Mage = 36, 57% female)

• Facebook-style mockup site used 

• All mockups used “Suggested Post”
Study 1 - Stimuli
Study 1 Results

Ad Recognition

3

3.75

4.5

5.25

6

In-stream Sidebar

Familiar Brand

Unfamiliar Brand
Study 2

• 2 (brand familiarity: high or low) x 2 (image 

professionalism: high or low) x 5 (disclosure 

type)

• 723 participants (Mage = 35, 51% female)

• Examine ads alone - mirrors mobile experience

Study 2 - Stimuli
Study 2 Results

Ad Recognition 

Increases with... Advertisement Sponsored Post

Suggested 

Post

Promoted by 

[Brand] No Disclosure

Professional Image ✔ ✔

Professional Image 

when paired with a

Familiar Brand

✔ ✔
Ad Recognition

3

4.25

5.5

6.75

8

Unfamiliar Familiar 

Professional

Unprofessional

Study 2: No Disclosure
Threshold Effects

• Disclosure will not have an effect when: 

• no other ad recognition cues are present, or 

• multiple ad recognition cues are present 

• Disclosure will have an effect when a single ad 

recognition cue is present 
Study 2: All Conditions

Ad Recognition

3

4.25

5.5

6.75

8

Advertisement Sponsored Post Suggested Post Promoted by [Brand] No Disclosure

Unfam & Unprof

Unfam & Prof

Fam & Unprof

Fam & Prof
Study 2: All Conditions

Ad Recognition

3

4.25

5.5

6.75

8

Advertisement Sponsored Post Suggested Post Promoted by [Brand] No Disclosure

Unfam & Unprof

Unfam & Prof

Fam & Unprof

Fam & Prof

a,b,c

a b

c
Study 2: All Conditions

Ad Recognition

3

4.25

5.5

6.75

8

Advertisement Sponsored Post Suggested Post Promoted by [Brand] No Disclosure

Unfam & Unprof

Unfam & Prof

Fam & Unprof

Fam & Prof

a,b,c

a b

c

e,f,g

e

f g
Study 2: All Conditions

Ad Recognition

3

4.25

5.5

6.75

8

Advertisement Sponsored Post Suggested Post Promoted by [Brand] No Disclosure

Unfam & Unprof

Unfam & Prof

Fam & Unprof

Fam & Prof

a,b,c

a b

c

e,f,g

e

f g
Results

• Multiple ad recognition cues are needed to affect 

ad recognition 

• Only “Promoted by [Brand]” was effective 

• Consumers can identify native advertising in a 

social media setting
Discussion

• Consumers do not seem to be processing social 

media content deeply 

• Current disclosures may need strengthening 

• Supports FTC perspective that disclosure isn’t the 

only ad recognition factor 

• Disclosure finding contradicts enforcement guide 

language recommendations
Thank You
Putting Disclosures to the Test September 15, 2016

Sarah J. Farnsworth

Vice President, Scientific Affairs

PEGUS Research, Inc.
www.pegus.com

Methods for Testing Consumer 

Comprehension of Product Labeling: 

Implications for Putting 

Disclosures to the Test 

Sarah Farnsworth, Ph.D.

Vice President, Scientific Affairs

PEGUS Research, Inc.
www.pegus.com

BACKGROUND

 An important factor that impacts the effectiveness of disclosures is 

whether or not consumers comprehend them.

 We propose that the model currently used to test OTC drug 

product labeling may be relevant to evaluating other product 

information, such as disclosures. 

 Today I will discuss some studies used in the approval of an OTC 

medicine by the Division of Nonprescription Drug Products 

(DNDP) at the FDA as an illustration of this model.
www.pegus.com

INTRODUCTION TO PEGUS RESEARCH

PEGUS Research designs and conducts studies to determine if consumers:

 Adequately comprehend the information on product labels 

(Label Comprehension Studies)

 Can use the information to make a correct decision if the product is 

appropriate for their use based on their own medical history 

(Self-Selection Studies)

 Use the product safely in a simulated OTC setting by following the 

label directions and warnings (Actual Use Studies)

The results of these studies are submitted to DNDP at the FDA for review 

decisions on approval of a new OTC product.
www.pegus.com

LABEL COMPREHENSION STUDIES (LCS)

 OTC labels must be “...likely to be read and 

understood by the ordinary individual, including 

individuals of low comprehension, under customary 

conditions of purchase and use” (21 CFR 330.10 

(a)(4)(v))

 Consumer understanding of OTC labels is 

demonstrated by conducting an LCS. 

 LCS utilize one-on-one standardized interviews 

with a general population of consumers to collect 

data on comprehension. 

 FDA issued a Guidance for Industry1 on conducting 

LCS to help standardize the conduct of these 

studies. 

1. Guidance for Industry: Label Comprehension Studies for Nonprescription Products. US Dept of HHS, CDER, August 2010
www.pegus.com

LABEL COMPREHENSION STUDIES (LCS)

 Content and structure of drug product labeling is governed by 

regulations

 Wording (and other package elements) should be developed and 

optimized through a series of iterative qualitative and quantitative 

comprehension studies 

 Testing is prioritized to focus on messages with the greatest clinical

consequence associated with a consumer failing to understand each 

label direction or warning.

 Messages with the greatest clinical consequences are deemed primary 

endpoints for the study, and assigned a target performance threshold. 
www.pegus.com

OTC DRUG FACTS LABEL 

COMPREHENSION TESTING MODEL

1. Create content (stimuli)

2. Early qualitative research to refine language, formatting, etc.

3. Identify key messages (endpoints) and ultimate performance 

targets (i.e. how good is good enough?)

4. Develop assessment questions and scoring criteria

5. Pilot test the content, questions and scoring

6. Refine

7. Test and refine the content, question and score again (iterative)

8. Evaluate final results against targets
www.pegus.com

CASE STUDY: NASACORT ALLERGY 24HR
www.pegus.com

CASE STUDY – NASACORT ALLERGY 24HR

 A series of LCS conducted in 2011 and 2012 to develop labeling that 

was adequately understood by the general population.

 Qualitative and pilot studies were first conducted to refine the label 

language and data collection tools in an iterative fashion.

 Study protocols describe study design, endpoints, statistical analysis 

plan, and other key elements.
www.pegus.com

STUDY DESIGN

 Pivotal studies for FDA submission were conducted in two phases. 

 Both the outside package (Drug Facts Label) and package insert were tested in 

each phase, with participants randomized to view one or the other to limit 

influence or bias.

 Sample size was large enough to provide a reliable measure of comprehension. 

886 consumers reviewed the product carton, and 734 reviewed the package 

insert. 

 Very few exclusion criteria were applied to ensure a general population of 

subjects were represented, not just those who suffer from the symptoms or 

disease the drug is intended to treat. 

 Participants (≥16 yrs of age) were recruited in 15 retail (mall) sites across the 

USA to ensure diversity and a representative sample. 
www.pegus.com

STUDY DESIGN

 Data collected in one-on-one standardized interviews, with data entered 

real-time in an internet-based electronic data capture system. 

 Each key message on the package had an associated question(s) to measure 

the proportion of subjects who comprehended the direction or warning. 

 Questions had a pre-specified answer key, so that correct responses were 

determined a priori. 

 FDA usually requests that 20-30% of study participants qualify as low literacy. 

Approximately 30% of the entire sample qualified as low literacy as defined 

by The Rapid Estimate of Adult Literacy in Medicine (REALM) Test2, a 

validated instrument.

2. Murphy PW, Davis TC, Long SW, Jackson RH, Decker BC. (1993). Rapid Estimate of Adult Literacy in Medicine 

(REALM): A quick reading test for patients. J of Reading 37(2): 124-130.
www.pegus.com

DRUG FACTS LABEL TESTED
www.pegus.com

EXAMPLES OF COMPREHENSION QUESTIONS

 Scenario questions are typically used, as they require a higher 

degree of assimilating package information into a made-up real-life 

situation. 

 Questions are read out loud by the interviewer, and are openended and neutral to avoid leading the participant to a correct 

response. 

 Silent, pre-coded answer options are programmed for the 

interviewer.

 
www.pegus.com

Message Tested on Label / Endpoint: 
www.pegus.com

1. Julie has a daughter who is one-year old. Her daughter has upper 

respiratory allergies, and Julie is thinking about buying this medicine for 

her to use. Is it okay or not okay for the daughter to use this product?

Do not read answer alternatives

Check only one box

1a. Please explain your answer

Do not read answer alternatives

Check all that apply 

 OK/ Yes 

 Not OK/ No 

 Don’t know / Not sure 

Do not use (if under 2)

She is too young / not old enough

Talk to a doctor

Other: _______________________________

Don’t know / Not sure

Correct: Box 2 in Q1 and Box 1 or Box 2 in Q1-a
www.pegus.com
www.pegus.com

2.Bill used the product last night and wakes up with a rash 

on his chest. What, if anything, does the label say about 

this? 

Do not read answer alternatives

Check only one box

 Stop use

 Ask a doctor

 Seek Medical help

 Seek Medical help right away

 Nothing 

 Other: _________________________________

 Don’t know / Not sure

Correct: Box 1 and Box 2 is checked, or Box 4 is checked 

Acceptable: Box 1 or Box 2 or Box 3 is checked
www.pegus.com

DATA ANALYSIS AND PRESENTATION OF RESULTS 

 Proportions of subjects who provided a 

correct or acceptable response were 

calculated with a corresponding 95% 

Confidence Interval for each key 

communication message. 

 Data were analyzed and presented by normal 

and low literacy subgroups in statistical tables 

and a study report. 

 While not applicable to this case study, various 

other subgroup comparisons may be made 

depending on the nature of the product and 

study sample.
www.pegus.com

OUTCOME

 Statistical tables, study report, and raw data were 

submitted to the FDA 

 Data were presented to the FDA and an Advisory 

Committee in an all-day meeting in which the committee 

votes for or against approval

 FDA approved the product for OTC sale in 2013
www.pegus.com

LCS ADAPTATION FOR PRODUCT DISCLOSURES

This model could be adapted to test product disclosure statements. 

Possible adaptations include: 

 Randomize subjects to view different modalities of disclosure 

statements in order to test the impact of various forms of 

delivery, phrasing, or formatting on comprehension or other 

outcomes. 

 Those randomized to a television or radio commercial would view 

or hear the commercial, and then proceed to a standardized 

interview with comprehension questions.

 Those randomized to a website or printed advertisements could 

proceed with a very similar standardized test of the information as 

is used for the current LCS model for testing OTC package 

information.
www.pegus.com

CONCLUSIONS

 Label comprehension studies provide evidence of consumer comprehension of OTC 

product labeling of package information 

 Similar studies could be conducted to provide FTC with information about 

comprehension of product disclosures. 

 These studies would need to be adapted to differences between OTC products and 

product disclosure statements, as product disclosures come in various methods (e.g., 

television, radio, websites, print). 

 Other consumer behavioral research could also be relevant in this arena, such as 

adapting self-selection study designs to evaluating the effect disclosure statements 

may have on impacting consumer decision-making behavior.
Putting Disclosures to the Test September 15, 2016

Manoj Hastak

Dept. of Marketing, Kogod School of Business

American University
"Assessing the Efficacy of Qualifying 

Disclosures in Advertising: 

Methodological Considerations"

Manoj Hastak, Ph.D.

Professor of Marketing

American University

Presented at the FTC Workshop “Putting Disclosures to the Test”

September 15, 2016. Some of this research was conducted for and 

funded by the FTC
Personal Background

• Long time association with the FTC

• Lead and/or participated in dozens of FTCsponsored research projects

• Focus today on a couple of these projects as 

well as some ongoing (non-FTC) research on 

disclosures
Three Methodological issues Re 

Assessing Disclosure Effectiveness

• Assessing Ad Communication vs. Believability

• Probing for consumer interpretation of 

disclosure intent

• Using eye-tracking data 
Study #1: FTC Study on “Up To” 

Claims in Advertising




Study Overview

135

• Mall Intercept, 344 respondents

• Print ad for “Bristol Windows”

• Three ad treatments

• Cell sizes = 114 - 115

 
Original Ad 
“Cleansed” Ad
“Disclosure” Ad

Results: Ad Communication and 

Believability

% Saying “half” or more

Up To 47% 47% Discl.

Based on ad, how many can expect 

to save about 47%?

48% 40% 46%

In personal opinion, how many will 

save about 47%?

43% 45% 46%

In personal opinion, what would be 

the average savings?

40% 38% 37%
Study #2: “Up To” Claims –

Replication and Extension

• With Nelson Amaral (American University)

– Do findings replicate with a different ad and target 

market?

– Concurrent “think aloud” protocols

– Eye-tracking measures

– Student subjects, lab study 

– Three ad treatments (n=135)

– Preliminary results
LIMITED TIME OFFER FOR 

RAVPOWER® PORTABLE CHAGERS

Buy 1

Get 1

FREE

CHARGE YOUR PHONE 43% FASTER THAN 

A STANDARD WALL CHARGER.
Buy 1

Get 1

FREE

CHARGE YOUR PHONE UP TO 43% FASTER

THAN A STANDARD WALL CHARGER.

LIMITED TIME OFFER FOR 

RAVPOWER® PORTABLE CHAGERS
Buy 1

Get 1

FREE

CHARGE YOUR PHONE UP TO 43%* FASTER

THAN A STANDARD WALL CHARGER.

*On average the RavPower Charger charges 17% faster than other portable chargers

LIMITED TIME OFFER FOR 

RAVPOWER® PORTABLE CHAGERS
Results: Ad Communication and 

Believability

% Saying “half” or more

Up To 43% 43% Discl.

Based on ad, how many can expect 

to achieve charge about 43% faster?

72% 85% 37%

In personal opinion, how many will 

achieve charge about 43% faster?

39% 47% 46%

In personal opinion, what would be 

the average savings?

19% 22% 15%
Results: Eye Tracking Data

Measure

Up To 43% 43% Discl.

Disclosure dwell count 1.67 2.00 5.08

Disclosure dwell duration 2.22 1.86 7.13
Study 3: FTC Dietary Supplement 

Study

• Probing for Consumer Interpretation of Disclosure 

Intent

• Modified Promotion Booklet for Vitamin O

• Cover Page + 3 Pages of Consumer testimonials

• Mall intercept, 200 respondents

• 6 treatments


Results: Interpretation of Disclosure 

Intent

Protect 

Company

Inform 

Consumers

Other

Base: Respondents who 

said there was a 

disclosure in booklet

45.2% 45.2% 16.1%

Base: Respondents who 

accurately recalled 

disclosure

45.7% 54.3% 8.7%
Final Thoughts

• Value of multiple measures

• Probing for disclosure comprehension as well 

as intent

• Value of replication
Putting Disclosures to the Test September 15, 2016

Heidi Johnson

Office of Research

Consumer Financial Protection Bureau
Disclosure research in the lab

Federal Trade Commission Putting Disclosures to the Test Workshop | September 15, 2016

Heidi Johnson, Consumer Financial Protection Bureau
Disclosure research at CFPB

154
155

Disclosure priority question areas

• Relationship between certain 

dimensions of disclosure and 

consumer response

Efficacy

Measurement • How we learn about what works

Market effects • Firm behavior
156

Priority area: Efficacy

• Relationship between certain dimensions of Efficacy disclosure and consumer response

Dimensions of disclosure

 Context

 Content

 Form

 Stages of efficacy

 Attention

 Understanding

 Action
157

Disclosure research approach

Sources of research

 Rulemaking-related

 Consumer-facing tools

 Collaborations with industry

 Project Catalyst

 1032(e) disclosure waiver 

authority

 Foundational research

Methodologies

 Qualitative testing

 Quantitative testing

 Field trials

 Administrative data analysis

 Laboratory studies

 Through contracts with several 

universities and other institutions
Laboratory study:

Context and attention

158
159

Lab experiment on disclosure

 Studying in the lab enables us to:

 Isolate effects in a controlled environment

 Apply findings to future disclosures

 What affects attention to disclosures?

 Examine two factors

 Design: Two versions of the disclosure

 Context: Reading in isolation or in the presence of a researcher
160

Lab experiment on disclosure

 PIs: Dustin Beckett, PhD and Alycia Chin, PhD

 Conducted at the end of an economics experiment

 N = 192

 Gettysburg College students

 Groups of 12

 Provided information about the study and privacy rights to all 

participants 

 Environment randomized at the session level

 Form design randomized at the participant level
161

Measure of attention

“Researchers conducting this study may be interested in contacting you 

regarding additional research studies in the next year. These future studies will 

provide compensation of approximately $35/hour. Please initial anywhere on 

the bottom of this form if you would like us to contact you for these studies. 

Doing so will not affect any aspect of your participation today, including 

payment or privacy.” 

(emphasis added)

 Did they initial?
Factor 1:

Design

162
163

Forms: Two designs
164

Forms: Two designs

Top Bottom
165

Instructions on the form

 Participants asked to sign and affirm they have carefully reviewed the 

information

 Applying concept from Shu et al. (2012) on reporting accuracy to 

engagement with a disclosure 
166

Forms: Two designs

Top Bottom
Factor 2:

Environment

167
168

Gettysburg College laboratory setting
169

Environment: Reading in isolation
170

Environment: Researcher present
Results

171
172

Results: Context influences attention

Form Design: χ

2

(1) = 1.391, p = .238 

Environment: χ

2

(1) = 23.841, p < .001
173

Discussion

 The form design tested did not significantly influence attention

 Context can significantly influence attention

 35% who viewed disclosure on their own initialed compared to 

5% who viewed disclosure in the presence of a researcher

 Opportunity to further increase rates of attention

 Signatures may not indicate attention to other form elements

 Setting specific outcome measures important
Putting Disclosures to the Test September 15, 2016

Case studies

Moderator: 

Hampton Newsome

Division of Enforcement, FTC

Colin Campbell

Dept. of Marketing and 

Entrepreneurship

Kent State University

Sarah J. Farnsworth

Vice President, Scientific Affairs

PEGUS Research, Inc.

Manoj Hastak

Deptartment of Marketing

Kogod School of Business

American University

Heidi Johnson

Office of Research

Consumer Financial Protection Bureau
Putting Disclosures to the Test September 15, 2016

Afternoon break

The next session begins at 4:30 pm