FTC Putting Disclosures to the Test Workshop

September 15, 2016

Segment 1

Transcript

LORRIE CRANOR: Good morning, everybody. We're going to get started in a minute. Good 

morning, and welcome to the FTC's Putting Disclosures to the Test Workshop. I'm going to start 

with a simple disclosure. I'm Lorrie Cranor, Chief Technologist of the Federal Trade 

Commission, and these remarks do, in fact, reflect my views.

I am delighted to welcome everyone who has joined us in person, as well as all of you who are 

joining us online on the webcast. I'm extremely excited about our workshop today. The topic of 

our workshop, putting disclosures to the test, is one that I care deeply about. For over a decade, I 

have conducted research with my students to evaluate privacy disclosures.

We've realized that many of the issues we've encountered are not unique to privacy disclosures, 

and that there is much benefit in studying the lessons learned from researchers in other areas. 

We've also learned just how important it is to run studies to evaluate disclosures. Many times, we 

ourselves have been surprised when our intuition about a disclosure does not match what we find 

when we put the disclosure to the test in a research study. We'll hear stories like this from some 

of our other panelists today.

With the help of a really great team of FTC staff, we've put together a truly amazing group of 

speakers today. Some of our speakers are researchers whose papers I have been reading for 

years, and it is such an honor to meet them and talk with them about their research. Before we 

get started with our workshop, I've been asked to make a few disclosures.

Please be aware that if you leave the Constitution Center building for any reason during the 

workshop, you will have to go back through security screening again when you return. Most of 

you have received a lanyard with a plastic FTC Event Security badge. We reuse these for 

multiple events, so when you leave for the day, please return your badge to event staff. If an 

emergency occurs that requires you to leave the conference center but remain in the building, 

follow the instructions provided over the building PA System.

If an emergency occurs that requires the evacuation of the building, an alarm will sound. 

Everyone should leave the building in an orderly manner through the main 7th Street exit. After 

leaving the building, turn left and proceed down 7th street and across E Street to the FTC 

Emergency Assembly Area. Remain in the assembly area until instructed to return to the 

building.

Please be advised that this event will be photographed, webcast, and recorded. By participating

in this event, you are agreeing that your image and anything you say or submit may be posted 

indefinitely at FTC.gov or on one of the Commission's publicly available social media sites. 

Very important-- restrooms are located in the hallway just outside the conference room. The 

cafeteria is located down the hall, and will be open during our morning break and for lunch.
We encourage you to join us for lunch in the cafeteria so we can continue the workshop 

discussions over lunch, and so you won't have to go back through security. Food and drink are 

not permitted in the auditorium. If you're interested in joining some of us for a happy hour after 

the last session, please pick up a handout with more information. And finally, if you are tweeting 

about this event, please use hashtag #FTCdisclosures.

OK, that was a lot to disclose. Hopefully you were all paying attention because some of the 

researchers we have with us today may want to evaluate your comprehension and measure the 

impact on your decision making and behavior. For those of you who already looked at the 

agenda, these topics should sound familiar. After a brief introduction, we are going to start off 

with a presentation on cognitive models of how people process disclosures. Then we will hear 

about methods used to evaluate disclosure effectiveness, and some trade offs and traps in using 

them.

Next, the Your Attention Please panel will explore methods used to evaluate whether and when 

people are noticing, reading, or paying attention to disclosures, like the one I just made. Later, 

the comprehension panel will focus on evaluating whether people understand the information 

conveyed in disclosures. After lunch, panelists will discuss studies on the impact that disclosures 

have on consumers' decision making and behavior.

The next panel will feature case studies from four researchers. The final panel will present 

studies that evaluate new approaches, or new applications of existing approaches, to disclosure 

design and presentations that suggest ways to make disclosures more efficient and effective. 

Now, to get us started, it is my honor to introduce FTC chairwoman Edith Ramirez.

EDITH RAMIREZ: Thank you, Lorrie. And good morning, everyone, and welcome. We're 

really delighted to have you all here for this workshop on disclosures and the importance of 

testing disclosures. As you might imagine, this is a topic that's of tremendous importance to us 

here at the FTC, and it's one that we've been thinking about and studying practically from the 

moment that the agency opened its doors more than 100 years ago.

In our consumer protection work, one of our primary tasks is to ensure that consumers have 

access to truthful and accurate information to enable them to make informed decisions in the 

marketplace, and one way that businesses have to communicate information is through 

disclosures. Disclosure is a broad term that encompasses a range of statements. Here at the FTC, 

when we think about disclosures, we generally focus on the disclosure of information that affects 

consumer welfare.

In some cases, disclosures are necessary to limit or qualify others' advertising or marketing 

statements in order to prevent deception. Other disclosures are required to inform consumers of 

the risks or dangers from using certain products. Still others make consumers aware of choices 

they may have. For instance, some of the disclosures required by FCC rules include energy 

consumption labels on appliances, fuel labels at the gas pump, and price lists for funeral services. 

These help inform consumers when they're choosing between different products or services. In 

the privacy arena, we encourage companies to disclose their data practices so that consumers can 
make choices that allow them to exercise greater control over the way their personal information 

is used. 

It's clear that disclosures play an important role in protecting consumers. That's why, here at the 

FTC, we require that necessary disclosures be clear and conspicuous. We want consumers to see 

or hear the disclosure, to understand it, and to use the information conveyed to make informed 

decisions. But we also recognize that accomplishing all three of these objectives can sometimes 

be challenging. We're here today to discuss how businesses can test and evaluate their 

disclosures to ensure that they're effective. 

Now, to set the stage for today's discussion, I'd like to spend a few minutes describing some of 

the agency's recent efforts to help the business community provide meaningful and useful 

disclosures to consumers, particularly in the face of technological changes that have transformed 

the marketplace. The basic legal principles we apply remain the same, but there's no question 

that context matters. 

In the early 2000s, the FTC issued guidance on dot-com disclosures. This guidance explains 

what Clear and Conspicuous means in the context of online advertising and sales, explaining 

how the four P's at the core of our Clear and Conspicuous standard-- prominence, presentation, 

placement, and proximity-- apply to the online world. We updated that guidance in 2013 to 

address technological developments like the use of smartphones, tablets, and apps. In addressing 

the challenges that new and emerging technologies can pose to effective disclosures, we've also 

learned from research in the marketing field about the need not only for Clear and Conspicuous, 

but also Just in Time disclosures. As we discussed in a 2013 report on mobile privacy 

disclosures, we recommend that mobile platforms provide Just in Time notices and obtain 

affirmative express consent before allowing apps to access sensitive contact like geolocation 

information. 

We've also provided specific guidance about social media marketing disclosure issues. Our 

Endorsement Guides have long required the disclosure of material relationships between an 

advertiser and an endorser. In 2009, we noted the need for bloggers and celebrities to disclose 

material connections when promoting products on social networks. Last year, we provided 

recommendations about how to make material relationship disclosures in connection with social 

media contests, online review programs, and claims by expert endorsers outside of traditional 

ads. 

We've also sought to highlight the importance of disclosures in these areas with an active 

enforcement agenda. Recently, for example, we settled charges against Warner Brothers Home 

Entertainment alleging that it failed to adequately disclose that it paid YouTube influencers 

thousands of dollars to post positive video reviews of the company's games. Among other things, 

we emphasized that it's not enough to include disclosures in easy to miss description boxes of 

YouTube videos rather than in the videos themselves. 

Additionally, with the rise of new forms of nontraditional advertising, last year we provided 

detailed guidance on disclosures in native ads. These are ads that are made to look like news, 

entertainment, or other editorial content. We emphasize that native advertising disclosures 
should appear where consumers will look. They should be in front of or above a headline. And if 

an image is the focal point of an ad, then the disclosure should be on top of that image. This 

guidance, along with our enforcement actions, policy statements, rules, and reports explain how 

we think about disclosure issues and how companies can stay on the right side of the law. 

Let me now turn to the importance of testing the effectiveness of disclosures. The FTC has long 

benefited from empirical testing of disclosures and used it to inform our enforcement and policy 

efforts. In the last 15 years, for instance, with the help of the economists in our Bureau of 

Economics and our marketing experts in our Bureau of Consumer Protection, we've tested 

different types of disclosures-- financial privacy notices, testimonial disclosures, mortgage 

disclosures, disclosures related to health claims for foods, green marketing disclosures, energy 

label disclosures, Up To claims, homeopathic disclosures, and organic disclosures. I think there 

are a few more, but you get the idea. In addition to our own research and testing, we pay 

attention to the research of academics and experts in marketing, behavioral economics, cognitive 

science, computer science, and other relevant fields. 

So this brings me to the reason that we're here today. Our focus in this workshop is not on what 

has to be disclosed to consumers, or even on the most effective methods of disclosing 

information. Rather, our focus is on how to evaluate whether disclosures are effective. For 

example, we know that ineffective disclosures can overwhelm, confuse, or even distract 

consumers from making informed choices. This is due to the number of factors that can impact 

the effectiveness of disclosures, including whether consumers direct their attention towards 

them, understand them, and are able to use that information in their decision making. 

In a recent case against Practice Fusion, we made clear that the needed disclosures should grab a 

consumer's attention, or as we said in our order, disclosures should be difficult to miss. In that 

case, we alleged that the failure to provide a clear and conspicuous disclosure that patient 

feedback would be posted online resulted in the inadvertent revelation of personal and sensitive 

health information by hundreds of patients. 

In light of recent research on how consumers process information and behave, it's also important 

to pay attention to ensuring that disclosures provide useful information that can translate into 

consumer action. Now, there are also issues that we have to consider. Some critics question the 

effectiveness of disclosures altogether in certain areas. Privacy is one example. We disagree with 

those critics, not only because we believe that targeted, understandable, and timely disclosures 

can be useful to consumers in many circumstances, but also because we recognize that 

disclosures can serve different functions, among them to convey important information to 

consumer advocates and regulators as well. 

But I can assure you that we're committed to asking hard questions about the effectiveness of 

disclosures, and deepening our understanding of the role that disclosures play in protecting 

consumers and enhanced consumer welfare. Now before I close, I also wanted to mention an 

important and related conference that our Bureau of Economics is co-hosting tomorrow with the 

journal Marketing Science. Among other things, we'd like to see marketing experts conduct 

potentially high impact research in the area of consumer protection and regulation. All of the 
papers that will be presented at that conference are going to be posted on our website, and I 

encourage all of you to take a look. 

So let me conclude by emphasizing that, with this workshop, our aim is to highlight the 

importance of empirical analysis of disclosures, and to encourage marketers, businesses, and 

other organizations to test their own disclosures and to learn from researchers. So finally, I'd like 

to thank our speakers for being here today and sharing their expertise with us, and also the FTC 

staff who organized this workshop, especially Mike Ostheimer and Ryan [INAUDIBLE] from 

our Bureau of Consumer Protection, and of course our Chief Technologist, Lorrie Cranor. So 

thank you very much for joining us. I hope that you'll enjoy what I know will be an informative 

and robust discussion. Thank you very much. 

LORRIE CRANOR: All right. Thank you, Edith, and we are going to move on to our first 

session where we are going to talk about the cognitive models that explain how people process 

disclosures. And here with us is Wogalter. 

MICHAEL WOGALTER: Thank you, Lorrie. I'm very happy to be here, and most of my talk is 

going to be about a cognitive model. But I thought I'd give you some background about me and 

where I'm coming from and what I've been working on the last few years. And basically, I come 

from the human factors area, sometimes called ergonomics. Most of the world calls it 

ergonomics, but here we usually call it cognitive ergonomics, and it deals with mental workload, 

most of that research. 

And one definition that you can give to it is that it's a discipline that's concerned with the design 

of things such as products, equipment, tasks, built environments, etc., etc., based on people's 

abilities and limitations. And when I say that, some things we could do really well. We are very 

good at recognition memory. We're good at flexibility with limited information. But we're also 

not so good and have limitations on other things, such as rapid calculations or recall memory, 

etc. And so what we would like to do is make use of the good abilities and avoid the bad abilities

for the purpose of increasing productivity, safety, and comfort. And another definition is, they 

used to be called person-machine interaction, but you could say person-thing or persontechnology or whatever, interfaces. 

My background is, I've been doing research on warnings and risk communication for a long time, 

and I take the area of warnings as being very broad. They are communications to prevent injury 

or loss in a very general sense. And so a lot of people would call it maybe dealing with safety 

information, hazard information, risk information, but it also would be disclosures, and I've done 

some research here and there on informed consent forms, credit card terms, and those online 

acceptance of terms. I left a bibliography-- I don't know if it will be distributed-- but basically, 

the idea is that you want to tell the negatives, not just the positives, and doing it effectively. And 

my research has been dealing with factors that influence the effectiveness of warnings or 

disclosures, safety information, both negative and positive. 

Some of my research have been in these areas-- consumer product warnings, including onproduct labels accompanying inserts or sheets, tags, product manuals. It could be also signage for 

environmental hazards, directions, or other kinds of information, direct to consumer advertising 
or over-the-counter labels, and also on symbols and graphics and pictograms and pictorials, a lot 

of different names. Icons as well, and also visual and auditory presentation, as well as other 

modalities-- odor and vibration. And it has concerned both print and video and the internet. 

Well, that ends my little background of where I'm coming from. Most of my talk will be dealing 

with a model, a framework, called Communication Human Information Processing Model. 

Sometimes it's called C Hip, sometimes it's called CHIP, for short. And it's, as I said, a cognitive 

processing framework, and it combines a little bit of communication theory-- that is the source, 

the channel, and the receiver-- and also, at the receiver, there includes the human information 

processing stages. And I think the utility of the model is that it organizes a lot of haphazard 

constructs and research, and you could pigeonhole it in certain places. 

So it breaks it up into stages, and it's a linear process. And generally I'm going to talk about the 

flow down, from the source down to behavior, but you'll see on the right side there's some 

feedback loops, and a few times I'm going to be talking about that. But basically what goes on in 

this model is that you want to have the information flow from one stage to the next to be 

successful, but there are bottlenecks, potential bottlenecks. And I'm going to go through the 

entire model, briefly in some places, and show some of the bottlenecks and some of the factors 

that influence those stages. 

The source I probably have the least amount of information on, and you may know more about 

that than me, but what I consider it to be is the entity that transmits the information. And they 

have to make decisions on what to transmit, what goes out. And some of that could be due to 

laws, it could be due to industry standards, it could be from hazard analysis and other types of 

things. So they have to make that decision, what goes out. And an important thing that I bring up 

a couple of other times, they have to be sort of complete on what goes out. They have to make 

sure that the concepts are what needs to be put out. 

Now there is some research on source effects. There is some in the social persuasion literature on 

the characteristics of the source, such as expertness or trustworthiness, likeability or similarity. I 

would consider those to be more in the belief stage as opposed to the source stage. It's how 

people feel about it. But there is some research, for example, adding things like saying Surgeon 

General seems to help the credibility, or government seems to help the credibility, and some 

other things, if they are expert in that area. So they put out the information, and the next stage 

here is the channel. 

Now, they could put out the information a variety of different ways. It could be visual, it could 

be auditory. As far as media, it could be a print version, video, TV, radio, internet, label signs, 

posters, placards, etc. Owners' manuals, inserts, billboards, voice warnings, these days. And each 

one of them have different characteristics, how even people come upon getting them are 

different, oftentimes. 

And some of these things reach different people and different groups of people. Generally, what 

the research says is that giving them in more ways-- more different ways-- are better. It adds 

redundancy. Clearly if something is just print, a blind person. That's the easy example. But 

different people will get different things. Not everybody opens the bill for cable TV. 
Then the third stage of the communication theory is the receiver, and that's on the right side here. 

And what this CHIP model has is that inside the receiver is the human, and they process 

information. And of course people are different. There's different demographics, different person 

variables. And I'm not going to go into too much about personalities and demographics here, but 

I will say that a lot of people say, well, you need to design the communications based on the 

lowest common denominator of the population. And that's probably true for, let's say, the public. 

But there's going to be different communications, and probably acceptable and appropriate 

communications, for trained individuals. If this is for trained medical personnel, you don't want 

to start giving just single syllable words. It's just not going to work. So you've got to consider the 

different populations and their characteristics, and in the future-- and I think there's a session that 

deals with it-- you'll be able to maybe tailor the communications to the characteristics of the 

person, or what they're personally interested in. Certainly they're doing that for advertising. 

One other aspect about the receiver-- and a later version of the CHIP model has it as a separate 

stage, and I'll show that later-- is delivery. And it's not really shown here, but it's between the 

channel and attention. And the idea here is that the source may have given out the information, 

gone through several channels, but did it actually arrive to the recipient. I've had many instances 

that I've seen in some my consulting work. Yeah, they made out brochures, but it never really got 

to the people. So one way of checking out whether it's delivered is to find a sample that's

supposed to have gotten it and see if they actually did receive it. 

So now we get, really, to the characteristics of the receiver and the stages there. And here I have 

attention, and attention was just mentioned a couple of times, and I break up attention into two 

stages, and the subsequent model does have them as two separate stages. One is the switch stage 

and one is the maintenance stage, and the way I could describe it as the switch is where you're 

attracted to some stimuli, that is, the movement toward something that is salient, prominent, 

conspicuous. So it's switching from something else to what you want them to be looking at. 

And some of the ways to assess that is to look at looking behavior. You can watch people and 

see where they're looking. There's plenty of movement equipment these days, and maybe you 

could look at whether saccades move to the stimuli. Also, in a lot of cognitive psychology 

research, they do reaction time, response time, where the idea is, if they respond faster, it's easier 

to find. It's more conspicuous in that particular environment. 

You could do post-exposure stuff, where you expose them to the information and then sometime 

later, ask some questions. And the idea is that, well, if they remember it, they must have seen it. 

None of these are absolutely perfect methods, but that's one way. And of course another way is 

just subjective evaluation, perhaps using ratings. Rate how conspicuous it is. 

Some of the features that benefit this tension switch is whether it's large, high contrast-- and by 

high contrast I mean a great difference between the print and the background; foregroundbackground types of things like black and white. Color will help. Location. You don't want to 

have a disclosure embedded multi pages into it when really, they're not going to get to that. It 

should be placed in the visual field. And another way is to reduce the competing stimuli, if that's 
possible. And then symbols will also help. Even if they're not even understandable, actually 

graphics may draw attention. 

Then the other stage of attention, once you've drawn people's attention to something because it's 

salient, you want them to hang around and to read and examine it. One of the major ways that I 

think sometimes gets lost is it needs to be legible, where you can distinguish the parts of the 

words, parts of the letters, and so forth. Commonly you see a lot of things with all capital letters, 

for example, and you may or may not know that a lot of people say that, well, that's not that 

good. 

Mixed case is better because there's more distinguishable characteristics in mixed case. It's 

harder to tell the blocky letters of all caps. All caps is good for once in a while, limited use, but 

there's other ways to make things more legible, and that's being large, high contrast. Just like the 

other attention switch. 

Also brevity, low density. You don't want dense text. They're not going to hang around and read 

that. Structured format, headings, bullet points, and that sort of thing-- organizing it. And putting 

the most relevant information first. 

For example, one of the classic things is for extension ladders. They have something like 700 

words on extension ladders, and there's many hazards. One of them is that it slips out. But if you 

look at what's on those ladders, one of the first things they give is keep your shoes clean, and 

that's not nearly that important. You start reading it, and you're saying, I already to keep my 

shoes clean, or I don't care. And you don't continue reading it. 

Or UL, the owner's manuals. They almost always require that you have these warnings for 

electrical things, but it's similar for every electrical thing, and the only unique things are later in 

the owner's manual. So they really should put the most important, the highest hazards, things that 

people don't know much about, up top. That's oftentimes not done. 

Some of the ways to measure attention maintenance is dwell times, eye fixations. Some of these 

are ambiguous measures. You don't know if they're hanging around just because they don't 

understand it. So you've got to use multiple methods. For legibility, there's a whole host of 

measures. One of them is how far away they can see something, or you could cover it up, 

obscure it in certain ways. 

You could do participant evaluations, whether it's readability, that type of thing. And also, the 

prioritization evaluation. The way we do it is have things on cards and have them sort them 

under different evaluations such as importance or hazard level and so forth. 

One issue in attention is habituation. You probably all know a little bit about that. Basically 

when you see things over and over and over again, you don't give attention to it that much 

anymore, unless it's very important to do so. Novel things are more salient. They tend to be more

salient than things that have been around and you've seen it repeatedly. 
So this is a problem of standardization. I know in the warnings literature, there is a standard for 

product warnings from the American National Standards Institute, and they've prescribed certain 

signal word panels and certain kinds of layouts and so forth. And it's pretty good. It's somewhat 

based on research-- not always-- but the problem is that if you keep on seeing the same signs, the 

same stimuli, it may no longer capture attention. So you may need to throw in some change, if 

that's possible. 

So we're now at comprehension. Now, here's the deal. The source could have put out the 

message and gone through one or more channels, the person has attended to it, switched attention 

and is holding attention and reading it, but it doesn't mean that they understood it. They may 

even know the individual words and what they mean, but do they really understand the content 

and the concepts that you want to give? 

And a common way to measure understanding is-- it's just too easy and it's oftentimes wrong, but 

it's convenient, it sometimes could help-- is to use readability formulas that the word processors 

go. It could be so wrong because they use certain things like length of words, how long the

sentences are, and the frequency in the language. And sometimes they can give you rather bogus 

information on how easy it is to read, but it could be helpful. 

A better way would be to show the information-- give the information-- to people and have them 

examine it, and then afterwards, or even while it's there, is to ask open ended questions. Did they 

understand the concepts they're supposed to understand? And you could also do cognitive 

interviews where you successively give more and more probing, to pull out what they do know. 

Because sometimes people just write something in for the open ended questions, but it's not their 

complete understanding that they've relayed on paper, so doing a cognitive interview could pull 

out more of what they really know. 

Some features that benefit generally-- and these are generalizations-- simpler terms, high 

frequency in the language, has content, reflecting the intended contents. And in the warnings 

literature, it says that the message ought to have the nature of the risk and the hazard, instructions 

of what to do or not do, and consequences, what bad could happen to you. And so, just to give 

you a sign example, a no diving sign, the nature of the risk is shallow water. Instructions-- do not 

dive. And then the consequences-- you may be permanently paralyzed, because that's a common 

neck injury. 

A lot of times, some of this stuff is left out. They won't tell the consequences. As a matter of fact, 

a lot of times, I see in warnings, they just give the instructions. Do not do this. But people don't 

know why. And so locally, in their own head, they may rationalize, well, it's for that reason and 

I'm not going to have that problem. That's the reason you need to-- as I say later on-- be explicit 

as much as possible. You can't deliver massive amounts of information to them, but at least tell 

them that something is more than maybe hazardous to your health. If you know it's going to 

cause liver disease or liver failure, you ought to say that. 

Avoid ambiguity or misinterpretation. You probably know many instances about that. One of the 

things with low birth weight. A lot of people say, I want low birth weight, for the women. No, 
it's for the babies. And so there's a variety of things that you need to find out about 

misinterpretations, potential ones, that may have not occurred to you. 

Some other things-- consider the target audience factors. The skills, as I was mentioning, 

whether you've got sophisticated population that you're delivering it to, whether they have 

cognitive impairment. And certainly a lot of this stuff depends, if you're sending it out to the 

general public, it's a wide range of people. I have in here development, that you ought to check 

whether the content that you're giving out is all of the necessary contents that is-- and I think 

that's what the source needs to determine, whether it has everything that is needed. 

And in my field, what they would do is usability tests. That's what a lot of companies do when 

they are making manuals or instructions or other things, they bring in people and watch them do 

it. In this case, you could ask them whether they understand it, what do they understand, and then 

you could fix it and then give it to some more subjects or more participants and see if that helps, 

and do that a few cycles. 

I mention here about symbols because a lot of people are going towards symbols. Because there's 

a lot of people who speak different languages, who are less skilled in reading, they may want to 

use symbols or pictograms and so forth. And I mention it here is that the ANSI standard-- the AN-S-I, American National Standards Institute-- has a standard on symbols and what makes it an 

acceptable symbol for warnings. And it's a performance standard. 

What they say is that you need a certain amount of subjects, generally 50 to do this test, but you 

could have more-- that a symbol is acceptable to use if 85% understand it with no more than 5% 

critical confusions. And critical confusions are, they define as opposite answers, but it's any bad 

answer that could lead into problems. So it's good to throw that in, that it's just not wrong 

answers. These are really bad answers. So it's something that maybe could be applied to written 

material. OK 

Now we're at attitudes and beliefs. So they may attend to it. They may understand it. But they 

may not believe it. They may not believe the message, and so the processing stops there. It's 

much easier to process information if it concurs with your existing beliefs. It's much more 

difficult when they don't believe it, and the problem is when they think that something's lower 

risk then it really is. Or they may believe that, oh yeah, the government wouldn't let that product 

on the market, or they wouldn't allow that to happen. And we know that government can't do 

everything. 

And so what you need to do-- at least what I've been saying-- is that you need to have something 

that's salient to capture attention, and then persuade them, at least as much as possible, to try to 

overcome their erroneous beliefs. And some of the concepts that are out there is perceptions of 

hazard or risk, and what we've been finding in my former lab is that people to tend to pay more 

attention or be influenced more by severity of how badly they may have the loss, and not so 

much the probabilities, mainly because they think it's near zero and it's not going to happen to 

them. 
But one way to maybe handle some of the-- and some people do it in the warnings literature-- is 

try to match the message risk level to the actual risk level. So you see in warnings, they'll have a 

Danger, which is supposed to be, at least according to ANSI, to be the highest level of risk, 

danger. And then Caution is somewhere lower, and Note is supposed to be no hazard, no risk. 

And so that may help in getting people to reach the right level of perceived risk. 

Another factor is familiarity. If people have seen something over and over again, they may not 

look at it again, or they may believe that they're already knowledgeable about that particular 

subject. And if they do, they're not going to read the information. And this is an example of 

where beliefs influence an earlier stage. So if you think something is, I already know that, you're 

not going to attend to it, so that's the feedback loops. Another one is perceived relevance. Is it 

relevant to me or someone else? If they think it's for someone else, then forget about it. And the 

ways that you can assess it is self-report participant evaluations and rating scales. 

And then down to motivation. And so processing could go all the way down, but get to 

motivation. And this is where they may believe the message, but they don't have the motivation 

to carry it out. And it could be a variety of things-- cost of compliance, is it effortful to carry out? 

Is it going to take a lot of time? That's going to prevent the process from continuing. And some 

other factors-- people tend to follow what people do or not do, social modeling. There's research 

on that. Other factors-- time stress, mental workload, busyness interferes at this stage, but also in 

other stages. 

And so finally, behavior. This is where they're actually seeing whether people are complying or 

adhering to what the message is, or doing or not doing something appropriate to be safe. And a 

good measure of effectiveness is actually to see whether they're carrying out what the disclosure 

warning says. And there's a variety of empirical methods to do this. There's direct methods where 

you can overtly see someone carrying out the behavior. There's indirect methods where you 

could see some aspect, some outcome that's related to it. And what you can see is that it takes a 

whole lot of things to come together so that behavior is changed. 

So a summary slide. What this is is a framework, a model. It shows linear processing coming 

down. That's what I've primarily dealt with, but you could have feedback loops that influence 

earlier stages, and it describes why familiarity may cause someone not to look at a warning or a 

disclosure, and helps to organize a bunch of diverse research. 

And it also could enable some more specific fixes. I've seen situations where they think that, oh 

yeah, we need to make it more conspicuous. Well, it turns out they did see the warning, but they 

didn't bother to read it. You needed to fix the maintenance part of it, or the comprehension part 

of it. So it could save money and a lot of work if you could track down what the problem is. 

And this is the current version. I'm not going to go through it here, but this is in my 2006 book, 

and it includes separating the two attentions, it's got delivery, it has environmental stimuli and so 

forth, and memory in there. And that's it. 
LORRIE CRANOR: All right. Thank you very much. Our next session is going to focus on 

evaluation procedures and methods for evaluating disclosures. We'll have two speakers in this 

session, and we will start with Ilana Westerman. 

ILANA WESTERMAN: Well, thank you very much for having me here today. So I assume I hit 

the green button here. So just a little bit of background before I get started. I'm Ilana Westerman. 

I'm from Create With Context, and we're a user experience design consultancy. We do a lot of 

work with big companies, helping them design digital products. 

But we also, in 2008, started our digital trust initiative. And what that is is really something that 

we put out into the public forum, something that we're very passionate about, trying to help both 

smaller companies and developers become more transparent around their practices. So I'm going 

to go over, today, a little bit about some of the different methodologies that we use when we're 

trying to figure out how best to convey to people what's happening, especially in a digital space. 

But before I really dive into the methodologies, I want to give you guys a little bit of context, a 

little bit of background, because it's really important what the goal is of a discipline when they 

look at what evaluation methods you use. So we come from the design background, and what's 

really important in the design background is we have a goal of actually doing something. We're 

trying to solve a problem. 

We're not just trying to understand, like a scientist would. We're not just trying to create art, like 

an artist would. We're actually trying to solve a problem for other people. And why that's so 

important is that, because we're not creating something for ourselves, we have to understand 

before we start. We have to understand who it is that we're creating for, we have to understand 

what they know, we have to understand what they want, and then we also are very much always 

driving to a solution. 

And so if you look at our overall design methodology that we follow, you'll see it's kind of in a 

funnel shape. And the reason it's in a funnel shape is we're really driving towards getting 

something out there. So we want the notice to be out there and we want the notice to be effective. 

But we always start off at the very beginning, in the foundation phase, is we've got to 

understand. And so we have to know who it is that we're designing for. We also have to know 

their background, we have to know what their expectations are, and just like Michael just spoke 

about, what they care about. So that's a really important part of our process. 

So we tend to start with a lot of exploratory methodologies at the very beginning to understand, 

but as we understand and we start to actually develop solutions, we need to test those. And we're 

never right. Never, ever, ever is the first design the right design. A lot of times there's 200 to 300 

different iterations before we get it, because we think we know, but it's a very humbling process 

design is you don't. 

So now I'm going to go into more details on the actual methodologies. And I know that this is a 

real simplistic model, but I think it's still just really important to think about these methodologies 

and these terms. So you can boil them all down to two different ways that we gather our data. 
One way is we ask people. And there's a variety of ways of how you ask people that's important, 

but it still can be a very effective model, where you ask people what they think. 

The problem about asking people is, can they actually answer you? So if I ask you how many 

steps you take a day, if you don't have a pedometer, can you actually effectively tell me? Do you 

know how many steps you take a day? So that's a question that you shouldn't ask people. Also, if 

you ask them a question like, do you feed your kids vegetables every week, are people going to 

say yes? Probably. But do they really? Maybe not. So you have to think about the questions that 

you're asking and can people actually answer them. 

Then we move into the observational methodologies, and we love these methodologies because it 

has less of an impact. When you ask somebody something, you can plant a seed, but when you're 

just observing and they're doing something naturally, that's great data. But there are risks there 

too. 

So you'll see things where people say, well, nobody clicked on that link, and that means they 

don't care. And you say, well, you can say nobody clicked on the link. That you can say. You 

could observe that. But to say that no one cared, you don't know. You don't know why they 

didn't click on that link. So we just have to always be careful, and there's a lot of times people 

take that leap from what they see and what they think the reason is why. 

And then we move into experimentation, and that is when we really have to control the situation 

to get to the answer. So we want to have a controlled experiment where everybody is going 

through the same stimuli and we're doing things in the same way, in a controlled manner, so that 

we can actually test something. 

So I'm not going to go through all of these. The purpose of this slide is to show you that there's a 

lot of methodologies. In the UX world, we are agnostic. We will pull methodologies from many 

different disciplines. So we might pull from medical fields like eye tracking, and we might also 

pull from cultural anthropology, like ethnography. So really, we're looking across many different 

fields of science, and pulling what matters, what makes sense to answer the questions that we 

have. 

But ultimately what we find is there's no one silver bullet. There's no one methodology that's the 

perfect methodology, and typically it's a combination. Every method has pros and cons, so really 

what we want to do is, as much as possible, combine different methodologies so we can get 

different perspectives on the problem. But we're designers, and we're still trying to get to 

solutions. So typically there is a time and money factor here that comes into play, and we don't 

get perfection. So we do as much as we can to try to get things right, but we do want to definitely 

get things out there into the public domain. 

So I thought what would be helpful if I'm going to go through two different case studies and 

actually show you, in practice, what the methodology is we use and how they actually work and 

what came out of it. And so the first case study is really going to be talking about the first part of 

the funnel, that exploratory phase when we don't know what we don't know, when we're trying to 
understand. So before we even start to design, we want to understand what is our context that 

we're designing for? 

So this was a piece of work that we did when we were really trying to figure out, for shoppers 

who are in stores with their mobile devices, when information was being collected on their 

mobile devices, how could we inform them of this? So first, we combined methodologies, and 

we did an ask, observe, and experiment. We did in-depth interviews, we did recall and 

recognition tests, and we did in-context observations. 

So the first thing we did is we asked people. We said to them, do you realize this is happening? 

We also asked people to read articles. This is a New York Times article, and we asked them, 

what are your thoughts there? Now there's a risk. When you ask somebody something like this, if 

they didn't know before, the fact that you've even asked, you could run into an issue where 

you've planted a seed. So we were aware of this when we did this experiment, but I'm going to 

show you some video clips of people's reactions. And while their faces are blurred for privacy, 

you can't see the facial expressions, we could. And when you see people's facial expressions and 

when you hear their voice, you see that no, they did not know that this was happening before. 

They weren't aware of this. 

And why that's important to us as designers is that that's the point we have to start at. If people 

know something's happening, then what we want to do is just inform them. If not, we have to 

explain to them what's happening and inform them. So we have to know, where are we starting 

with our design? So I think this is just going to start, or do I have to press it again? 

INTERVIEWER: What are your thoughts about it? 

SUBJECT 1: Well, I would think that'd be interesting. First of all, I'd want to know how they do 

it, and second of all, I would want to know what exactly are they looking for and what 

information can they get off of your phone? So, that part I just-- I don't get that part. 

ILANA WESTERMAN: So that was when we just asked people, do you expect this is 

happening? You could hear in her voice that no, she doesn't expect that this is happening, so we 

knew that we had to start from more of an explanation perspective. This is another example of 

someone who read that article and their reaction, and you can hear in her voice the surprise. 

INTERVIEWER: I just want to know a little bit of your thoughts about the article. 

SUBJECT 2: That's crazy that they can do that, that even if you don't sign into their Wi-Fi, they 

can track you. 

INTERVIEWER: And did you expect that stores were doing that? 

SUBJECT 2: No, not really. 

ILANA WESTERMAN: And so you can hear just the word choice she used, that's crazy. The 

other thing is, you'll hear the researcher there did ask very open ended questions, just like 
Michael talked about. So things like, tell me your thoughts. So even though you are asking 

questions, we are asking them in a very open ended way. 

So the next thing we did was we want to say, OK, low hanging fruit, we want to get a solution 

out there. Let's go ahead and figure out if we can put notice on people's phones. It makes sense.

That's where the information is being collected from. So before we started, we went into stores, 

and we wanted to understand how people were using their mobiles in the store so we could 

figure out what is our context that we're designing for, and when are we going to provide the 

notice, when are they looking at their phones, things like that. And so this is a great observational 

study. We didn't have to recruit people into a research study. We just went in and observed 

people using phones. 

What happened, though, was unfortunately, people weren't using their phones. And so, that was 

the first thing was we have found that only 11% of people in retail stores actually had their 

phones out. So then we asked ourselves, well, why? What are we going to do about this? What's 

going on here? And we started to realize, well, people can't. From a usability perspective, they 

don't have the capability to use their phones because their hands weren't free. So you're in a store,

you're pushing a cart, you're picking up clothes, you're doing something. 

So we counted. So we counted the number-- basically, the ability for people to use their phones 

in stores. So how many free hands do they have? So if you have no hands free, you can't use 

your phone. If you have one hand free-- and if you're like me, you still can't use your phone 

because your hands are too small, so you're limited-- and if you have two hands free, then you 

can. 

But then we always do a comparison, like, is this a good number or a bad number? So we 

compared it to in mall, so people who were sitting in the mall, and you will see that much higher 

numbers of people who had ability had had their phones out in the mall, and that goes to 

attention. We weren't going to be able to get their attention in stores. 

So then to the experiment. And so here, what we're thinking is, OK, phones aren't going to work. 

Even though we tend to do things in the digital environment, let's not forget that paper 

sometimes works quite well. And what about just your plain old sign? Let's just tell people using 

signage. So what we wanted to do here was figure out, what's the best way to convey the 

information to people? What size worked best? What are the characteristics of those signs that 

work well? And so that way, we can make sure we start out with designing the best possible sign. 

So we recruited people to participate in the study, and they didn't know what the study was 

about. We told them it was a secret shopper study. This is a common study that's done in the 

retail world. We asked them to go buy items at five different stores, then they went back and they 

return the items at the stores, and then they came into our lab for a recall test. And so the first 

thing we did was called open recall, so we gave them a floor plan of the actual stores and we had 

them draw out their pathway they took. 

And we did test this to make sure people could accurately draw out their pathway, and they do 

remember how they walk through the store. And then we asked them to put all the signs that they 
saw on the map. And the second part of the study, what we did is we went ahead and showed 

them series of signs, and half those signs were actually in the stores on the pathway they took, 

and half were not. 

And so what we found was kind of disheartening. This goes to attention. Only 8% of the signs 

that were in people's path would they recall in an open ended way. And by the way, we set this 

up kind of best case scenario. We said, go find socks in a large department store, hoping that 

people would need to look at signs to get to the socks, that type of thing. 

But only 8% recalled-- and actually, we said, OK, well, we can kind of see that that's not your 

main goal. You might see something, but it's something that you don't remember when asked. 

But if you're prompted, meaning from a recall perspective, you see the sign and you say, oh 

yeah, I do recall seeing that. But that number was low too. Only 26% of the signs that actually 

were in their path and we showed them the signs did they say, oh yeah, I did see that. 

And the sign that did the worst was in one of the counties, this Attention Consumers sign was at 

all the registers, so they saw this five times. And we had them return an item on purpose, so 

they're returning items, they're standing there at the register for a long time, nobody recalled this 

sign. And so we were like, OK, how are we going to actually compete with this? I mean, you've 

got sale 50% off. If people aren't paying attention to that, I don't know how we're going to get 

their attention. 

The most telling, though, is people were more often wrong than right, meaning that the signs that 

weren't in their pathway, they were more likely to say they were, the signs that were in their 

pathway. That's that 36% number. So now all of a sudden we're like, OK, this is not going to 

work. 

So that's kind of a sad story, it sounds like it, but it really isn't. What it means is as designers, we 

have to start from that point. I'm not going to be able to make people take their phones out, and 

I'm not going to be able to make people look at signs. I can't do that, so I have to come up with 

solutions that are going to fit into that world. I have to design for what's reality that's out there. 

And so we actually do have a number of solutions that we're working on that are working, but in 

this talk, I don't want to get into all that detail. I want to instead switch gears to another case 

study, to show you some of the methodologies we use when we're actually in the design process, 

when we're creating solutions and trying to refine those solutions. So this was when we had 

mobile phones and we wanted people to be aware that information was being collected on their 

phones when they're using their phones. 

And it had been determined that we were going to use icons to do this, and there was a lot of 

upfront work that was done to determine if an icon was the best. I'm not going to get into that 

exploratory phase, but what we were doing here is we were trying to figure out where to put the 

icon, and we were also looking at, can we do things like flashing it so people will pay attention 

to it? But if you flash things too much, obviously it's irritating. This is kind of this fine line. 
And so this is what we did in eye tracking that was mentioned previously. And so eye tracking 

technology is a box that sits below a computer or a phone that can actually determine where the 

eye is looking. So it can actually determine if you're actually focusing, your eyeball's actually 

focusing on a point on the screen, and that's a red dot, and the saccade, which is a line, is where 

your eye is passing but not focusing. By the way, participants don't see this red dot. That would 

not be good for the experiment. I think that the video played before, but the video showed that 

the person actually saw the icon. 

And so first we want to create awareness, because ultimately if people don't see something, it 

doesn't matter how great our icon is. If no one looks at it, who cares? So then, multiple rounds of 

iterative tests and design, and what we wanted to do is figure out, what's the best icon, best way 

to communicate this? And so for this type of study, what we do is we give people a word choice. 

And so we say, what does this icon convey to you? And we also want to make sure people don't 

say things like ugly or distracting. We designers, we don't like hearing those kinds of words, so 

we do want it to be appealing, but we also want it to convey the meaning. 

And then for control, we use location. That's an icon that people know. And not 100%. People 

don't recognize things at 100%. So we're starting to get there with a trust icon, and showing that 

people actually, with no other background than what words would you attribute to this icon, 

people are able to say, yeah, this is about tracking. 

But I just want to, in closing-- and Craig is going to talk about this in more detail-- the pitfalls. 

So as designers, ultimately we always have in the back of the head, maybe we're wrong, for 

making assumptions. But as soon as we get data, we're like, cool, we got it, and we don't 

question it anymore. So bad data is really dangerous for us. 

And this is just one example right here. This is a study done by a Pew in 2012 which reported 

that 57% of people had uninstalled apps because of privacy concerns. And we were like, whoa! 

We've been tracking this for years, and we ran studies where we bring people in, we have them 

read these articles, we get their reactions, they're like, I hate this, I'm going to uninstall 

everything. But then we call them a month later and ask them what's on their phone, and our 

numbers were not at all 57%. 

And so we dug into this, and what's nice about Pew is they actually publish their instrument. A 

lot of stats out there, you can't figure out how they did their study. But the video already played, 

but basically the questions that they asked, the first question was, has your phone ever been 

stolen? Then the next question they asked is, has anybody ever accessed your phone and as your 

privacy ever invaded? Those are the words that were used. A person on your phone. Then they 

asked, do you back up your phone? Have you ever lost data? And the last question they ask is, 

have you ever uninstalled apps due to privacy concerns? 

So what we asked ourselves is, in that context, is that really valid feedback? Was that an order 

effect because they were primed by the earlier questions? And with that, I think Craig's going to 

talk a lot more about these pitfalls. And thank you. Oh, last slide. Just in case anybody has 

questions, feel free to reach out via email. 
LORRIE CRANOR: Thank you. Our next speaker will be Craig Andrews. 

CRAIG ANDREWS: OK. Thank you very much, and it's fantastic to be back in D.C. I worked at 

the Federal Trade Commission just a few years ago, and the FDA more recently. Some of you 

might notice some traps up here. I'm going to talk about some broader issues, some things not to 

get in trouble with as a researcher. Some of these traps up here are for researchers. Are there any 

Metro riders out there? OK. You might notice that picture down below there, and they were 

setting some animal traps, not traps for researchers, but I think it was OK yesterday. 

OK, a quick example. Some companies are doing a good job, I think, in trying to work with 

symbols and cues and other things, because this is a serious topic regarding privacy disclosure 

examples. So, how many of you have downloaded-- it's OK-- Pokemon Go app? OK, that's good. 

[INAUDIBLE] I've heard from people that, why do I have to look at this legalese? I just want to 

click "I agree" and get my app as soon as possible. So that's really the crux of some of these 

issues. It's a very difficult situation, so habitual processes there, behavior, may not breed 

understanding. 

In fact, I found this cartoon when Mike and Lorrie asked me to do. I found this cartoon the very 

same week, and we have a software agreement writer crying in his beer because please, stop just 

clicking agree. Nobody's reading his stuff. So I think that's the difficulty that we have as far as 

trying to nudge or push people at least to restrict some of the privacy information, and I think we 

really could do a better job of that. 

All right. There's a lot of broader issues, because the testing disclosures and warnings that we do 

are not in a vacuum. So I think before you get involved, you definitely want to lay out the 

purpose and objectives. Is it behavior? Is it awareness? Is it comprehension? And so forth. And 

the expected outcomes. 

This is a big one, and I've been involved in a lot of studies over the years. Is there an evaluation 

plan in place? Sometimes not. They're rushing to get information out and there's no plan in place, 

so be careful of that. 

Teams. How many work in teams out here? Most of you. I think keep it simple is very important. 

In fact, one of my colleagues suggested this at the FTC a few years ago-- please talk about the 

difference between interesting and important. So things like 30 different treatment conditions. 

Where I got a phone call one day, jeez, I think we need about 100 coding categories. IVs and 

DVs, independent variables and dependent variables. Are they clearly identified? So just by 

clicking on a link, that doesn't necessarily mean that the disclosure has been viewed. 

This is a huge one-- the baggage that people bring in. They all bring in baggage to the processing 

of disclosures and warnings. What are their motivation, as Michael talked about earlier? Ability. 

This is a huge one, and we've measured this of pharmacy leaflets regarding the literacy rates out 

there. An opportunity to process information, as Ilana just talked about, maybe the phones aren't 

on, or maybe in the pharmacy leaflets, they're not next to the respondents. 
This is huge. Right now I'm looking at the audience. Everybody probably has different 

disciplines where they're coming from that has quite different issues. So for example, maybe it's 

the maximization of information, trying to get out as much information as possible. Other 

disciplines, it might be-- in fact, I was taught this at the FDA-- a single independent variable and 

single outcome. Or maybe it's minimizing legal errors. Or maybe it's trying to minimize any 

restrictions of information out there. So you have to be careful. Sometimes they're not seeing eye 

to eye. Listening is important. 

This is playing off what Michael presented a little bit ago. There are information processing 

steps, and it's funny, when I put this together, I ID'd three areas. And I saw in the panel that that's 

exactly-- there was some validation there, exactly what we're going to be doing today. So I think 

key areas are attention-- for example, recall, recognition, facial imaging, FMRI--

comprehension-- accuracy tests, beliefs-- and the key area, I think, are different decisions and 

behavior, and that can be very, very challenging. 

There are trade-offs with these measures in general. So for awareness, as most of you know, and 

in fact, I'm challenged a little bit here, as we age, recall tends to go down with age. Recognition 

tests are beautiful. In fact, I was revising our text chapter on ad research and there's a lot of cool 

techniques here. The problem is, a lot of times with a multiple choice list, you're trying to get at a 

single item, like a brand that one would recognize. 

Eye tracking-- great on controversial issues. Get into fixations and saccades and gazes. But the 

issue is, where is the understanding? And sometimes on the valence of the direction. Maybe you 

want to look at this further with beliefs and attitudes. 

Comprehension-- a lot of creativity. I love challenging my colleagues with, let's get creative. 

And so you can do things such as accuracy tests on the nutrition labeling stuff we do. You can 

get into maybe, if you were to read four servings of this, would it be greater than or less than 

your daily value for a nutrient? 

Ultimately, decision making behavior is really important, but you have to be careful. There are 

field experiments, and I've been involved in some, where maybe they rushed out without some 

sort of control group. Or there's no control. You can't have control counties or control states if it's 

out throughout the entire nation. Or how about this? You've got a warning or a disclosure out 

there, and then suddenly there is taxation or pricing issues that are varied, so you have other 

confounds. 

Experiments. I've seen this a lot. There was a recent review where they-- this was on graphic 

health warnings-- and some suggested there might be null effects, or no effects at all, when in 

fact, the author said, you know what? A lot of these studies were poorly done, and it's really 

inconclusive. So poor stimuli, exposure issues, wrong sample. I know some of you review papers 

out there for academics or evaluators, and you may have seen some of this. 

A lot of type of data out there. I really prefer-- we do a lot of work on experimentation with 

causality. Does x cause y? That's very important, I think, in a lot of these fields. Internal versus 

external validity. You might focus on causality, but does it mean anything? So can it be 
generalizable to the real world? You have issues with student samples and other things that 

occasionally you see in our field. Cross-sectional vs. longitudinal, and I know in the graphic 

health warnings area this is becoming more important. 

I'm going to get into this in a second on the study designs, and we wrote a little about this-- this 

goes back a few years-- with one of my colleagues on different types of ag control groups, so it 

was a purged ad, a tombstone ad, is it a corrected ad with a disclosure? So some of these have 

been used in copy tests at the FTC. 

OK, class. Sometimes I share this with my class, and there's some dangers out there with 

different studies. So one and two have a little bit of a problem. So if your treatment as a 

disclosure and O is the outcome or observation, you can run into problems on whether or not it 

was, in fact, your disclosure that caused that outcome. So you don't know that for a fact with one 

and two. Three is a little better, where we have control groups. You've got an experimental group 

and a control group with a randomized trial, the only problem with this is that you have to make 

sure that these groups are fairly equivalent. So there's always threats to validity, as a lot of you 

may be aware. 

Sampling issues, wow. Knowledge, literacy issues. Keep your eye on that. Seventh to ninth 

grade on the average reading level in the U.S. Senior citizens, English as a second language. One 

of my colleagues shouted out-- this was a conference call last year-- "Who are these people?" 

And you get a continuum from, maybe you're collecting directly on MTurk and somebody terms 

out, and their baby was crying, and please, please, please include me, to these panels that we've 

used occasionally where you have click throughs. And this is not clicking on a link to go to other 

ad information. These are people, and you know these [INAUDIBLE] lists go straight down. 

And so, it's been running about 15% or so with some of these panels. How do we know that? We 

include Tony the Tiger that was not on the package or in the ad. Did you see Tony anywhere on 

this? And again, sadly, about 10%, 15% of people say yes. 

Probability samples-- that's very difficult. Most times it's quota with screeners. Partners, watch 

out with panel companies. I hope I'm not offending anybody out there, but occasionally this 

happens, where suddenly they run out of sample and then the incidence rates are very different. 

Weighting and propensity scores. Are you matching the U.S. Census data? Power. There was a 

case a number of years ago-- I think [INAUDIBLE]-- where it was an underpowered study 

because the sample size was too low. 

I just put up some of our stimuli. This is a pet peeve of mine that you read through the papers and 

then the stimuli is not that hot. But we tend to really be out there with color. And by the way, on 

these stimuli, these are smaller versions of what would be seen, but you first of all, is it in color? 

Is it large enough? What about the type size, as Mike was talking about? The Clear and 

Conspicuous standard is very good, if you hasn't seen this on dot-com disclosures on 2013 from 

the FTC. 

But one in there I thought I would mention-- this caught my eye. I have some college age and 

teenagers there, and there was a Snapchat case here at the Commission. And some of you may--

people use Snapchat? I guess it's growing as far as age groups. Some of you know that those 
snaps don't always disappear, and this was a major issue with a case. And so I was just curious. I 

went to their privacy disclosure, and I highlighted it up here and I ran a readability test. It was 

11.4 grade level, and kind of moved in in the middle up here. So there's certainly a broad array of 

different types of stimuli. 

OK, questions? I'm going to put a lot of these up there. I know Ilana talked a little bit about 

these. There's a lot of bad things that can happen. Biasing, no screeners. One of my favorites is a 

situation where oh, attitudes or beliefs don't influence behavior. But if you start to look at the 

question, it lacks specificity on those attitudes or beliefs. There's usually funneling from open 

end down to closed end. Manipulation and confound checks, click through checks are really, 

really important, especially if you're collecting data online. Multiple measures. I think Michael 

talked about that earlier. 

And I think the hard part is, take a look at the information processing steps. Where are you? So is 

it behavior without thinking that we have with a lot of the "I agree" statements with disclosure. 

Or it could be the opposite, where you never get to behavior. 

Analysis also can be a thorn difficulty. Obviously it depends on the data. We tend to have-- in 

fact, I have some colleagues that read outside of the discipline, which is very, very important. 

Maybe it's the latest on mediation tests in psychology. You've got a hundred different categories, 

you make a hundred different comparisons. Well, are you adjusting for that? So, for example, 

Bonferroni tests and other things. 

All right. Hopefully researchers out there, you don't have your heads in the sand. We want to 

take those out of the sand. Here are some common problems that I've seen for decades across a 

lot of the research-- poor planning, no objectives, evaluation, treating exposure as awareness. Oh 

jeez, they're clicking on that. They're getting the disclosure. But in fact, not true awareness or 

understanding and so forth. No control groups, bad stimuli, wrong sample. This can happen a lot. 

The graphic health warnings, so maybe you're testing it on people who don't smoke, or maybe 

people that have smoked for 30 years, not adolescents who maybe are experimenting, which 

might be the key group. 

This is a favorite of mine. One of my colleagues mentioned this. It was a meeting with an ad 

agency a number of years ago. One of the comments was, we don't need to do that. We already 

tested these disclosures and warnings three years prior in a different state. He immediately 

shouted out, oh, a sample of one! So we're not the target market here when you get involved in a 

lot of the research. 

I already talked about that one. Too many cooks in the kitchen. If you're on these teams, you 

have to be careful. I think absolutely somebody has got to be calling the shots that are tied to 

your evaluation plan as well as the objectives. OK, here's a broad conclusion. There are a few ifs 

here. In a review of six decades of research, and we had a book out at the FDA a few years ago, 

and in this book chapter I point out if-- there's some big ifs-- If you can account for audience 

characteristics and delivery modes, disclosures can, and I probably should have underlined, can 

be effective communication tools and remedies. 
There's some helpful research out there, if you're interested in this area, that we've written on, 

and I believe that Lorrie and Michael have that if you're interested. So thank you very much. 

LORRIE CRANOR: All right. Thank you very much. We are now into our break. If you'd like to 

purchase refreshments, they're available in the cafeteria down the hall, and please be back in 

ready to start at 11:00.