Putting Disclosures to the Test September 15, 2016

Your attention please!

Moderator:

Michael Ostheimer

Division of Advertising Practices, FTC

Nathaniel J. Evans

Department of Advertising & Public 

Relations

University of Georgia

Mariea Grubbs Hoy

School of Advertising & Public Relations

University of Tennessee

David Hyman

Epstein Program in Health Law & Policy

University of Illinois

Rebecca Balebako

Information Scientist

RAND Corporation
Putting Disclosures to the Test September 15, 2016

Nathaniel J. Evans

Department of Advertising & Public Relations

University of Georgia
Disclosure Modality in Advergames: 

Effects and Implications for Policy 

Nathaniel J Evans, Department of Advertising & Public 

Relations, University of Georgia

Evans, Nathaniel J. and Mariea Grubbs Hoy (2016), “Parents Presumed Persuasion 

Knowledge of Children’s Advergames: The Influence of Advertising Disclosure 

Modality and Cognitive Load,” Journal of Current Issues and Research in 

Advertising, 37(2), 146-164.

*The research presented is funded in part by the American Academy of Advertising
Why Disclosures in Advergames?

• Low level of disclosure prevalence

• Children don’t recognize them as advertising

• Parents have limited understanding 

• Advergames require “cognitive effort” to play

– Mental resources devoted to gameplay 

– Affects advertising recognition (persuasion knowledge)
Clear and Conspicuous Standards (CCS)

• What is the role of disclosures in ensuring parents understand 

the “nature” of child-directed advergames?

• Modality

• Type Size

• Contrast

• Single Background

• Presentation Rate

• Distraction

• Proximity

• Consider the Audience

• Single vs. Dual modality

• Dual (simultaneous 

disclosure presentation in 

auditory and visual forms)

• Enhances consumer 

message processing

• Does this work when 

playing an advergame? 
What Did We Do?

• Examined the effect of disclosure modality on parents’ 

persuasion knowledge (ad recognition) of child-directed 

advergames.

• Between subjects online experiment hosted by Research Now

• Panel of 202 parents with children between 7-11

• Played a Pop Tarts “Toasty Turvy” Advergame for 3 minutes

• Random assignment to one of 3 disclosure conditions 

manipulated within the advergame

– None/Single modality (text crawl)/Dual modality (text crawl + voice over)
No Disclosure Single Modality Disclosure Dual Modality Disclosure

Hey Kids! This Game is an Advertisement. Hey Kids! This Game is an Advertisement.

+ 

Voice Over (repeat 10s)

• Disclosure language pretested for reading level: Flesch-Kincaid Grade Level = 4.3
What Did We Find?

7.5

8

8.5

9

9.5

NO DISCLOSURE SINGLE MODALITY (TEXT) DUAL MODALITY (TEXT + 

SOUND)

ADVERTISING DISCLOSURE CONDITIONS

Persuasion Knowledge

Predicted
Implications

• Superiority of single modality over dual modality

• Advergames naturally result in increased cognitive load

• The auditory disclosure “competed” with the sound of the game

• Dotcom (2013) disclosure guideline recommendation: 

– “use of disclosures that align with the modality of the environment in 

which they appear”

– LED TO A REDUCTION IN ADVERTISING RECOGNITION FOR OUR 

ADVERGAME
Applications

• What lessons did we learn about disclosure 

effectiveness?

• How can we apply insights about 

disclosure modality to other formats that 

combine advertising and gaming?
Applications

• What is the predominant modality of the environment?
Applications

• What cognitive systems are at use when playing? 

• Avoid “competing modality” to increase 

consumer understanding
Putting Disclosures to the Test September 15, 2016

Mariea Grubbs Hoy 

School of Advertising & Public Relations

University of Tennessee
Do Consumers Read Disclosures: 

Field of Dreams or Impossible Dream?

Mariea Grubbs Hoy, PhD

School of Advertising & Public Relations

Hoy, Mariea Grubbs and Abbey Blake 

Levenshus (2016 online), “A Mixed-Methods 

Approach to Assessing Actual Risk Readership 

on Branded Drug Websites,” Journal of Risk 

Research.
If you make it clear and conspicuous...

they will notice and read it.

School of Advertising & Public Relations
If you make it clear and conspicuous...

they will notice and read it.

School of Advertising & Public Relations
Do consumers read (drug risk) disclosures?

They self-report that they do:

• 40% read half of more 

• 73% read half or more if really interested in advertised 

drug

• 25% agree “I always read the small print in magazine/ 

newspaper pharmaceutical ads.”

Sources: FDA 1999; Menon et al. 2003, AARP 2010, Experian 2005-2014

School of Advertising & Public Relations
What’s social desirability got to do with it?

• Western societal norms 

encourage and expect 

information seeking.

• Rather than ASKING if they’ve 

read the disclosure, is there a 

better way to determine if they 

REALLY read the disclosure?

School of Advertising & Public Relations
School of Advertising & Public Relations

Eye-Tracking While 

Free-Viewing Website

&

Online Survey

& 

Retroactive Think Aloud 

Interview
Benefits

Risks

Novel Risks

The FDA 

provided 

the website 

stimuli and 

survey.

Risks On Homepage
Risks on Homepage w/Signal to Risks

Homepage with link to Risks Page
Eye-Tracking Measures

Fixation: where the person looked

Duration: how long the person 

looked

Fixation Sequence: order in 

which they looked

School of Advertising & Public Relations

Numbers = fixations of .2 seconds 

or longer

“Reading”
What we found

• Self-Report: 80% of participants 

claimed to have read half or more 

of the drug risk disclosure

• Reality: Eye-tracking data 

revealed limited to no reading of 

drug risk disclosure

School of Advertising & Public Relations
School of Advertising & Public Relations

Participant 2: Just looking around

No fixation (of .2 seconds or longer)

Participant 3: Reading

Numerous fixation points
Participant 3: Risk Page 

Notice the information they missed: 

novel risks

Participant 6: Reading benefits 

but just glancing at risks.
WHAT WERE THEY THINKING?

Why do they seek out, 

ignore or avoid 

drug risk information?
What were they looking for?

School of Advertising & Public Relations
Why didn’t they read the risk disclosure?

• Social Desirability?

• Information Avoidance?

• Optimism Bias?

• Perceived Familiarity? Of particular 

concern because of novel risks.

School of Advertising & Public Relations
Is disclosure reading the impossible dream?

• Consumers will likely self-report higher readership 

than actual readership

• Consumers are focused on benefits

• Identify why they aren’t reading

• Present (drug) risks before benefits

• Create a sense of “unfamiliarity” if there is novel 

information

• Signal novel information

School of Advertising & Public Relations

Putting Disclosures to the Test September 15, 2016

David Hyman

Epstein Program in Health Law and Policy

University of Illinois
David A. Hyman

University of Illinois

David Franklyn

University of San Francisco

Putting Disclosures to the Test:

(What) Do Consumers Notice?
Our Studies

 Series of studies (and articles) evaluating consumer 

knowledge/understanding of:

• Search Results Page (“SRP”) architecture (i.e., layout of paid and 

unpaid content);

• “Native” advertising;

• Labels on SRP and on native advertising.
Basic Methodology

 Online surveys of ≅1k respondents (per study). 

• Mostly U.S., but a few U.K. studies. 

 Conduct simulated searches and/or show static images 

of SRPs and examples of native advertising. 

 Examine:

• Knowledge of which regions are paid v. unpaid;

• Comprehension and/or recollection of labels;

• Perceptions: 

– Is it clear and conspicuous what is paid?

– What do particular labels signify? 

 Exploit changes in SRP architecture and labeling

 Manipulate SRP architecture and labeling.
SRP Examples
Native Ads Examples
What do Labels Signify to Consumers?

Label

Ad/paid 

content

Unpaid 

content

Don't 

know 

Paid Ad 89% 4% 6%

Paid Content 87% 5% 8%

This content was paid for 

by 86% 6% 8%

Paid Post 83% 7% 10%

Ad 81% 7% 12%

Sponsored 79% 11% 10%

Sponsored Content 76% 12% 12%

Sponsored Post 76% 13% 11%

Brand Voice 64% 16% 20%

Brand Publisher 61% 19% 20%

Presented By 60% 20% 20%

Partnered Content 57% 19% 24%

Partner 57% 17% 26%

Written By 23% 52% 25%
“Which of these labels have you seen in the past 

month?” (Jan. 2012)

Comments

Sponsored Links 55% Last used by 

Google in 2010

Sponsored Results 49% Yahoo only

Ads 46% Google 

Commercial Ads 33% Never used

Not noticed any labels 22% N/A

Exploit variation and changes in labeling. 

Use controls!
Respondents Have Difficulty Determining 

Whether Native Ads are Ads

37%

81%

49%

12%

14%

7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Native	Ad Regular	Ad

Ad/Paid Unpaid Don't	know
What if we “tweak” the label 

of a native ad?
Effect of “Tweaking the Native Ad 

Label”

40%

56%

44%

33%

16%

11%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Control Treated

Ad/paid	content Unpaid	content Don't	know	
Specific Studies

 Trademarks as Keywords: Much Ado About Something? 

26 HARVARD J. L. & TECHNOLOGY 481-543 (2013) 

 Search Bias and the Limits of Antitrust: An Empirical 

Perspective on Remedies, 55 JURIMETRICS 339-380 

(2015)

 Going Native: Can Consumers Recognize Native 

Advertising? (forthcoming, 2017)

 Search Bias and the Limits of Antitrust Revisited: 

Dominance and Its Discontents (work in progress)
Putting Disclosures to the Test September 15, 2016

Rebecca Balebako

Information Scientist

RAND Corporation


45

The Impact of Timing on the Salience of 

Smartphone App Privacy Notices

Rebecca Balebako (RAND Corporation), 

Florian Schaub (Carnegie Mellon University)

Idris Adjerid (Notre Dame University),

Alessandro Acquisti (Carnegie Mellon University)

Lorrie Faith Cranor (Carnegie Mellon University)

5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and 

Mobile Devices (SPSM 2015)
What makes a privacy notice effective?

 The notice should have information people 

care about.

 A privacy notice should be salient.

 The ability to remember a notice is a 

measure of salience.

46
Smartphone permission notices are ignored
Does timing matter? Which option is best?

 Smartphone apps can display privacy notices at 

many points

 In the app store

 During install

 Before app use 

 During app use

 After app use

48
Does timing matter? Which option is best?

 Smartphone apps can display privacy notices at 

many points

 In the app store

 During install (not tested)

 Before app use 

 During app use

 After app use

49
Timing does matter

 Smartphone apps can display privacy notices at 

many points

 In the app store

 During install (not tested)

 Before app use 

 During app use

 After app use

50

THE WORST
Simple app quiz on American inventors

51
The privacy notice

52
Both web surveys and a field experiment

 Web Survey (277 Mturk participants)

 Participants played a virtual app online

 Field Experiment (126 participants)

 Participants downloaded and played an app quiz

53
Field study participant recruitment

 Two pools of university participants with funded phones

 Notre Dame University (n=29)

 Phone-lab at University of Buffalo (n=37)

 One pool of participants for multiple studies

 Carnegie Mellon University and community (n=42)

 Online boards open to public

 Craigslist and reddit (n=18)

54
All participants completed following steps

1. Completed consent form and demographic questions

2. Installed and played the app

3. Experienced a distractor or delay

 Web survey: questions about privacy preferences

 Field experiment: 24 hours

4. Answered memory questions about the app

5. Evaluated the notice

55
Rate of Recall for Notice –Web Survey

56

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Rate of correct recalls

Not shown App store Before use During use After use
Rate of Recall for Notice – Field Study

0%

5%

10%

15%

20%

25%

30%

35%

40%

Not shown App store Before use During use After use

57

Rate of correct recall


Thank you

The Impact of Timing on the Salience of

Smartphone App Privacy Notices

Rebecca Balebako: balebako@rand.org

58
Putting Disclosures to the Test September 15, 2016

Your attention please!

Moderator:

Michael Ostheimer

Division of Advertising Practices, FTC

Nathaniel J. Evans

Department of Advertising & Public 

Relations

University of Georgia

Mariea Grubbs Hoy

School of Advertising & Public Relations

University of Tennessee

David Hyman

Epstein Program in Health Law & Policy

University of Illinois

Rebecca Balebako

Information Scientist

RAND Corporation

Putting Disclosures to the Test September 15, 2016

Comprehension

Moderator: Ryan Mehm

Division of Privacy and Identity 

Protection, FTC

Daniel Goldstein

Principal Researcher

Microsoft Research Lab – New York 

City

Elizabeth Howlett

Department of Marketing

University of Arkansas, Fayetteville

Susan Kleimann

President

Kleimann Communication Group

Joel R. Reidenberg

Center on Law and Information Policy

Fordham University
Putting Disclosures to the Test September 15, 2016

Daniel Goldstein

Principal Researcher

Microsoft Research Lab – New York City
Presenting unfamiliar numbers to laypeople

(and checking comprehension)

Dan Goldstein & Jake Hofman

Principal Researcher

Microsoft

Original research in this presentation was funded by Microsoft and Columbia University's Center for the Decision 
Written and graphical techniques

• Written

• Perspective sentences

• Frequency formats

• Graphical

• Icon arrays

• Simulations
How many times larger is a trillion than a 

million?

• One thousand times

• Ten thousand times

• One hundred thousand times

• One million times

• Ten million times

• Don’t know
How many times larger is a trillion than a 

million?

• One thousand times – 18%

• Ten thousand times – 12%

• One hundred thousand times – 21%

• One million times – 21%

• Ten million times – 17%

• Don’t know – 12%
How big is 100 million acres?

• As big as Rhode Island

• As big as Connecticut

• As big as South Carolina

• As big as Illinois

• As big as Utah

• As big as California
Perspectives

Sentences that begin “To put this in perspective” + a specific syntax

(Barrio, Goldstein, & Hofman, 2016).


“The storm killed thousands of people in Honduras, left one million homeless 

and destroyed what was left of a declining banana industry, once the country’s 

lifeblood, as well as other vital crops”

To put this into perspective ...
“The storm killed thousands of people in Honduras, left one million homeless 

and destroyed what was left of a declining banana industry, once the country’s 

lifeblood, as well as other vital crops”

To put this into perspective ...

One million people is about 12% of the population of Honduras
Perspectives improve comprehension

in empirical tests

• Recall

• Estimation

• Error Detection

• Long-term recall

• See Barrio, Goldstein, & Hofman, 2016
Possibilities for disclosures

To put 250 calories into 

perspective

• 11% of daily calories

• 1/3 of a meal

• 50 minutes of walking

• 31 cups shredded lettuce

Bleich, S. N., Herring, B. J., 

Flagg, D. D., & Gary-Webb, T. L. 

(2012). 
Possibilities for disclosures

Front-end load of 5.25%

• “If you invested $50,000 

in this fund, you would 

pay $2,625 of that as a 

fee.”

APR

• Open question

Empirical testing needed
Frequency formats

• Representing risks as frequencies

• Proportions and percentages not in use before French Revolution

• Older texts express chance in natural frequencies

• Increases correct solutions to tough Bayesian problems (compared 

to percentages and proportions). (Kurzenhauser & Hertwig, 2006; 

Hoffrage & Gigerenezer, 1998).

• Lead to more accurate statistics in elicitation tasks in which the 

ground truth is known (Goldstein & Rothschild, 2014)
Icon Arrays

• Sedlmeier & Gigerenezer, 2001 • http://www.iconarray.com/
Simulations

• People understand forecasts and portfolio returns better when 

simulated (Kaufmann, Weber, Haisley, 2013; Hogarth & Soyer, 2011).
Simulations

Goldstein, Johnson, & Sharpe, 2003
Putting Disclosures to the Test September 15, 2016

Elizabeth Howlett

Department of Marketing

University of Arkansas, Fayetteville
Consumers’ Understanding 

of Information Disclosures 

on Product Packaging 

Federal Trade Commission 

Constitution Center 

400 7th St. SW

Washington, DC 20024

Elizabeth Howlett

University of Arkansas, Fayetteville 
Research Stream

• Research supported in part by Healthy Eating Research, a 

program of the Robert Woods Johnson Foundation

• Colleagues involved in this research include 

• Scot Burton, Distinguished Professor and Tyson Chair in Food and 

Consumer Products Retailing, Department of Marketing, University of 

Arkansas, Fayetteville.

• Christopher Newman, Assistant Professor, Department of Marketing, 

University of Mississippi.
The product package is a crowded place
Nutrition Facts
Front-of-package Nutrition Labeling Programs 

What is the number one health concern of parents today regarding their 

children? 

Childhood Obesity 

not smoking, drinking, or drug use (C.S. Mott Children’s Hospital National Poll on 

Children’s Health 2010)
Constituencies

• Scholars/Theory: Are contrasting types of FOP icons 

processed differently under varying decision tasks?

• Food Marketers/Managers: Higher proportion of healthy food 

sales = superior sales growth, returns to shareholders, 

operating profits, and company reputations (Hudson Institute 2011)

• Consumers/Public Policy Makers: Can marketers create an 

environment in which consumers can make accurate 

judgments about product healthfulness? 
Primary Research Question

• How does front of package (FOP) information affect consumer 

evaluations at the shelf? 

• Processing Fluency (e.g., Jacoby & Dallas 1981; Schwarz 2004; Labroo & Lee 2004; Novemsky

et al. 2007; Hong & Sternthal 2010)

• Perceptual - the ease of processing perceptual features of a stimulus such as 

modality and shape (e.g., Jacoby & Kelley 1987)

• Conceptual - the ease of processing the meaning of a stimulus (e.g., Whittlesea

1993)
Conceptual Framework

• Comparative vs. Non-Comparative Processing

comparative non-comparative

Influences attitudes, intentions, and behavior differently

(Oakley et al. 2008; Olsen 2002 Hsee and Leclerc 1998; Nowlis and Simonson 1997)
Fop Labeling System Dichotomy

OBJECTIVE EVALUATIVE
FOP Labeling System
Perceived Conceptual Fluency

Scale Items : Endpoints: strongly disagree/strongly agree 

• Given the information on the front of the package, it is easy to determine how healthy 

the product is...

• Given the information on the front of the package, it is clear whether the product is 

high or low in its level of nutritiousness...

• I feel confident about whether this product is a healthy or unhealthy choice based on 

the information on the front of the package...

• It is easy to understand whether this product is a healthy or unhealthy choice given 

the information shown on the front of the package... 

modified from Fang, Singh, and Ahluwalia 2007; Lee and Aaker 2004; Moorman 1990 
• Please consider the nutrition level of the food product shown. Do you 

believe that the food product is:

not at all nutritious highly nutritious

1 2 3 4 5 6 7

very unhealthy very healthy

1 2 3 4 5 6 7

Perceived healthfulness
Purchase Intentions 

• Assuming you were interested in purchasing this type of food, how likely 

are you to buy this specific item given the information shown on the 

package?

 very unlikely very likely

1 2 3 4 5 6 7 

 not probable very probable

1 2 3 4 5 6 7

 definitely would not definitely would

1 2 3 4 5 6 7
Study 1: Design And Procedures 

• n = 600 

• Online experiment

• 2 (Objective Icon: Present vs. Absent) x 2 (Evaluative Icon: 

Present vs. Absent) Between Subjects Design

• Randomly assigned to 1 of 4 conditions

• Non-Comparative processing
Results: Non-Comparative Processing 
Study 2: Design And Procedures 

• n = 200 adults

• Lab experiment

• 2 (Objective Icon: Present vs. Absent) x 2 (Evaluative Icon: 

Present vs. Absent) Between Subjects Design

• Randomly assigned to 1 of 4 conditions

• Comparative processing
Study 2
Research Lab 
Results: Comparative Processing

Conceptual 

fluency 
Results: Comparative Processing 

Objective Icon

Perceived 

healthfulness
Results: Comparative Processing 

Evaluative Icon

Perceived 

healthfulness
Results: Comparative Processing 

Purchase intention
Conceptual Model
Conclusions

• The type of processing mode used by consumers influences the 

effectiveness of information disclosures. 

VS.
Thank you for your attention and interest.
Putting Disclosures to the Test September 15, 2016

Susan Kleimann

President

Kleimann Communication Group
Going Beyond Words: 

Assessing Comprehension 

at a Deeper Level

September 15, 2016

Susan Kleimann, Ph.D. 

Kleimann Communication Group
The Project – Funded by the Consumer 

Financial Protection Bureau

Mandate

The Bureau shall 

propose...model disclosures 

that combine the disclosures 

required under [TILA] and 

[RESPA] into a single, 

integrated disclosure for 

mortgage loan transactions 

covered by those laws.

 Dodd Frank Act §1032(f)

Result

 A Loan Estimate disclosure 

replaces the Good Faith 

Estimate and the TILA 

disclosure

 A Closing Disclosure 

replaces the HUD-1 and 

the TILA disclosure

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
The 3 goals of the project

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16

Comprehension

•understand the basic terms of a loan and its costs

•Understand immediate costs and costs over time 

Comprehension

•understand the basic terms of a loan and its costs

•Understand immediate costs and costs over time 

Comparison 

•compare one Loan Estimate with another

•compare a Loan Estimate with the Closing Disclosure

Comparison 

•compare one Loan Estimate with another

•compare a Loan Estimate with the Closing Disclosure

Choice

•choose the best loan for their situation with the Loan Estimate

•see differences between the Loan Estimate and Closing 

Disclosure to decide whether to close

Choice

•choose the best loan for their situation with the Loan Estimate

•see differences between the Loan Estimate and Closing 

Disclosure to decide whether to close
Loan Estimate –

3 parts

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Part1: Loan terms

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Part 2. Affordability over long term

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Part 3: Affordability over short term

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
A 3-stage testing phase

 Testing was done in English and Spanish with industry and 

consumer participants

 Stage 1: evolving the Loan Estimate: 5 rounds of qualitative 

testing

 Stage 2: developing and testing Closing Disclosure AND making 

sure it worked with the Loan Estimate: 5 rounds of qualitative 

testing 

 Stage 3: ensuring it worked for modifications after quantitative 

test: 3 rounds for Spanish; 3 rounds for refinancing version; 2 

rounds for closing costs

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Testing characteristics

 Of the protocol

1. Introduce a task

 Buying a house – here is 

Loan Estimate from one 

bank

2. Think aloud 

3. Here’s a second offer

 Which do you choose?

 Why?

 No right or wrong answers

4. Standard detail questions

 Of the participants

1. Consumers

 Mixed gender, age, 

education, race, income

 Experienced and 

inexperienced buyers

 Geographic variations

2. Industry

 Lenders, brokers, others

 Large and small

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16

Comparison of part 1

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16

Knowledge

 Recall of 

Information

 Listing

 Naming

 Finding

“It is a loan that has an interest that 

is not set, meaning it’s not always 

going to be 3% or 4%, it’s going to be 

adjusting over time..”

“...adjustable rate...means that you 

are going to get a low rate usually in 

the beginning and it is going to stay 

consistent for five years...So in Year 6 

and 7 through 30 there are different 

numbers given.”

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Comprehension

 Translating 

 Summarizing

 Discussing

“The only thing on this one is that it doesn’t 

have any principal—meaning for five years you 

pay, but you haven’t paid anything for principal. 

Whereas the other ones, you have paid some 

amount. Meaning that everything is going 

towards the interest on the mortgage. In five 

years, nothing has gone towards your loan...”

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Application & Analysis

 Using & applying

 Experimenting

 Identifying 

patterns

 Using problem 

solving

“When looking...I do see a higher interest 

rate on the Laurel loan, but I also see what 

appears to be a lesser monthly figure for 

Years 1 through 5...because I initially noticed 

that, at the beginning, this loan is slightly 

different. It’s a five year, interest-only loan 

which is reflected in the lesser amount for the 

first 60 months.”

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Synthesis & Evaluation

 Imagining

 Inferring

 Assessing

 Evaluating

“I’m looking at this and you notice a higher 

interest rate and then you see a lower 

payment. You’re like— hmmmm—but if you 

pay attention to the fact that your payment is 

going to go up by $315 in six years, that’s a 

lot of money. That’s kind of a scary process to 

be in anyway because after six years in a 

house you might need to be putting on a roof, 

you might need to be doing your plumbing

and not looking forward to paying an extra 

$300 a month in your payment.”

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Synthesis & Evaluation

 Imagining

 Inferring

 Assessing

 Evaluating

“It was only 10% down...because I could 

then preserve as much of my money as 

possible. But at the same time I would still 

take that with a bit of caution, because that’s 

a good thing now, but maybe, obviously over 

the 30-year period, that’s 10% more or 

roughly $31,500 that I’m financing and 

maybe if I do have the funds, I should be 

putting more down.”

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Synthesis & Evaluation

 Imagining

 Inferring

 Assessing

 Evaluating

“So the bottom line is you have to make this 

decision on...the amount of money it will cost 

you to take a loan out: Are you in a position 

where you want to pay more money, $5,500 

up front [Pecan] or the $633 [Poplar] and 

you’re going to sacrifice a little bit on interest 

rate...”

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
What did the Quant Study tell us about 

our success?

 2 x 2 x 2 x 2 between 

subjects factorial design

 Four factors

 Disclosure type: current vs. 

proposed

 Loan type: Fixed vs. 

adjustable rate

 Difficulty: easier vs. more 

challenging loans

 Consumer: experienced vs. 

inexperienced

 858 participants in 20 locations

 5 parts

1. Review 2 offers; choose 1; explain why

2. Compare terms 

3. Answer questions about 1 of the offers

4. Compare initial with final disclosure and 

answer questions about final disclosure

5. Rate the initial disclosure & final 

disclosure. 

 If bias in phrasing, advantage to current 

disclosures

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
What did the Quant Study tell us about 

our success?

 Proposed disclosures 

performed better on

 Aggregate measures *

 Tasks 2-5 *

 All concept areas * (e.g., Interest 

rate, escrow account, loan amount, monthly 

payments, mortgage insurance, closing costs)

 Regardless of experience 

level, loan type, and loan 

complexity *

* statistically significant

 Participants with the 

proposed disclosures 

listed more comments to 

explain their choice

 Current disclosures 

performed statistically 

significant better in one 

place: a vocabulary issue—

settlement costs vs. 

closing costs

 Result: Modified the Loan 

Estimate and Closing 

Disclosure

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Statistically Significant Findings

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16

Comprehension

• Current: 70.0% correct 

• Proposed: 80.5% correct

Comprehension

• Current: 70.0% correct 

• Proposed: 80.5% correct

Comparison (Initial to Closing)

• Current: 54.9% correct

• Proposed: 69.0% correct

Comparison (Initial to Closing)

• Current: 54.9% correct

• Proposed: 69.0% correct

Choice

• Current: 57.3% correct

• Proposed: 81.4% correct

Choice

• Current: 57.3% correct

• Proposed: 81.4% correct
Final Takeaways

 Comprehension is more 

than understanding words, 

but rather understanding 

the implication and 

impact 

 Comparison requires the 

ability to see the salient 

information and mark 

differences and 

similarities

 Choice requires 

integrating information 

and keying in on what 

makes most sense for you

 For complex information 

with major consequences, 

you need to be able to 

show that consumers can 

comprehend the 

implications, not merely 

the words

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
More information

 Copies of reports available at: 

http://www.consumerfinance.gov/know-before-youowe/compare/

 Contact at: skleimann@kleimann.com

 Kleimann Communication Group

 www.Kleimann.com

Susan Kleimann, Ph.D./Putting Disclosures to the Test/FTC/9.15.16
Putting Disclosures to the Test September 15, 2016

Joel R. Reidenberg

Center on Law and Information Policy

Fordham University
USABLE PRIVACY POLICY PROJECT 131

Usable Privacy Policy Project

FTC WORKSHOP 

PUTTING DISCLOSURES TO THE TEST

Ambiguity in Privacy Policies

Joel R. Reidenberg

Stanley D. and Nikki Waxberg Chair in Law

Fordham University School of Law

New York, NY

jreidenberg@law.Fordham.edu

http://law.Fordham.edu/clip

 Co-authors:

Jaspreet Bhatia (CMU), Travis D. Breaux (CMU) & Thomas B. Norton (Fordham CLIP)
USABLE PRIVACY POLICY PROJECT 132

Forthcoming publications

This research was funded in part by National Science Foundation Awards #1330596 

and #1330214 and by National Security Agency Award #141333

• Ambiguity in Privacy Policies and the Impact of 

Regulation, 45 JOURNAL OF LEGAL STUDIES --

(forthcoming 2016)

• A Theory of Vagueness and Privacy Risk Perception,

IEEE 24th International Requirements Engineering 

Conference (RE'16), Sep 2016
USABLE PRIVACY POLICY PROJECT 133

Research Goals

• Develop a theory for the measurement and comparison 

of vague and ambiguous terms in a privacy policy

• Test whether regulation improves the clarity of privacy 

policies

• Test how vagueness affects users’ perceptions of risk 

and willingness to share personal information
USABLE PRIVACY POLICY PROJECT 134

Categories of Ambiguity

Categories of Vague Terms

Category Description

Condition Action(s) to be performed are dependent on a 

variable or unclear trigger

Generalization Action(s)/Information Types are vaguely 

abstracted with unclear conditions

Modality 

(including modal 

verbs)

Vague likelihood of action(s) or ambiguous 

possibility of action or event

Numeric 

quantifier Vague quantifier of action/information type
USABLE PRIVACY POLICY PROJECT 135

Annotated Example
USABLE PRIVACY POLICY PROJECT 136

Grounded analysis
USABLE PRIVACY POLICY PROJECT 137

Taxonomy from Grounded Analysis

Category Examples of Vague Terms

Condition depending, necessary, appropriate, inappropriate, as 

needed

Generality generally, mostly, widely, general, commonly,

usually, normally, typically, largely

Modality may, might, can, could, would, likely, possible,

possibly, unsure, often

Numeric

Quantifier

anyone, certain, everyone, numerous, some, most,

few, much, many, various
USABLE PRIVACY POLICY PROJECT 138

Vagueness Lattice

We may collect...

We may generally 

collect...

We may collect... 

as needed

We may generally 

collect... as needed

We may collect 

some...

We may collect 

some... as needed

We may generally collect 

some... as needed
USABLE PRIVACY POLICY PROJECT 139

Paired Comparison Study

For each numbered question, please read each pair of statements, and 

identify which of the two statements best represents a more clear 

description of the company's treatment of personal information.

We share your personal information as needed.

We generally may share some of your personal information.

Herbert A. David, The Method of Paired Comparisons, Oxford, 1988
USABLE PRIVACY POLICY PROJECT 140

Bradley-Terry Coefficients

0.0 

0.5 

1.0 

1.5 

2.0 

2.5 

3.0 

3.5 

4.0 

4.5 

0 2 4 6 8 10 12 14 16 

CN 

C CM 

CMN

CG CGN MN

N CGMN M CGM 

GN 

GMN G 

GM 

• Source: J. Bhatia & T. Breaux, Technical Report: Automated Measurement of Privacy Policy Ambiguity (work-in-progress) 
USABLE PRIVACY POLICY PROJECT 141

Bradley Terry Model

Modality Category Survey

 

likely

may

can

would

might

could

possibly

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

2.2

Bradley Terry Coefficients

Relative vagueness of modality vague terms
USABLE PRIVACY POLICY PROJECT 142

Scoring Model and Benchmarks

 ∑ (BTC A-I) Regulatory Models

V = ----------------- Model Privacy Form Policies

 ∑ (A-I) Safe Harbor Certified Policies

V=vagueness

BTC= Bradley-Terry coefficient

A-I= Action-information pair
USABLE PRIVACY POLICY PROJECT 143

Vagueness, perceived risk and willingness 

to share

Factorial Vignettes 

Multilevel Modeling

Independent 

variable 1

Independent 

variable 2

Dependent 

variable

Subject to 

subject 

variability

+ -

+

-
USABLE PRIVACY POLICY PROJECT 144

Vagueness and Risk

 Vignette Factors and Levels

Template for Vignette Generation
USABLE PRIVACY POLICY PROJECT 145

Vagueness and Risk

Condition Generaliza

tion

Modality Numeric 

Quantifier

Extremely 

Unwilling

Extremely 

Willing

Unwilling

Willing
USABLE PRIVACY POLICY PROJECT 146

Applications to Improve Clarity

• Linguistic guidelines: minimize/avoid 

combinations with generalization terms and, if 

using terms, favor those with lower BT 

coefficients 

• Reporting framework: public reporting of scores 

to encourage ratchet effect
USABLE PRIVACY POLICY PROJECT 147

Possible Broader Applications 

• Consumer contracts such as EULAs

• Boilerplate contracts

• Next steps: domain specific taxonomy, policy 

annotations, establishment of BT coefficients 
USABLE PRIVACY POLICY PROJECT 148

Conclusion

For more information and copies of the papers:

http://www.usableprivacy.org

@jreidenberg
Putting Disclosures to the Test September 15, 2016

Comprehension

Moderator: Ryan Mehm

Division of Privacy and Identity 

Protection, FTC

Daniel Goldstein

Principal Researcher

Microsoft Research Lab – New York 

City

Elizabeth Howlett

Department of Marketing

University of Arkansas, Fayetteville

Susan Kleimann

President

Kleimann Communication Group

Joel R. Reidenberg

Center on Law and Information Policy

Fordham University
Putting Disclosures to the Test September 15, 2016

Lunch break

The next session begins at 2 pm