An Exploration of Consumers’ 

Advertising Recognition 

in the Contexts of Search Engines 

and Native Advertising 

FTC Sta ̃ Report 

December 2017 
An Exploration of Consumers’ 

Advertising Recognition in the 

Contexts of Search Engines and 

Native Advertising 

FTC Staff Report 

December 2017 

FEDERAL TRADE COMMISSION 

Maureen K. Ohlhausen, Acting Chairman 

Terrell McSweeny, Commissioner 
 

 

Contents 

60T Overview60T ............................................................................................................................1 

60T Study Methodology60T ...........................................................................................................2 

60T Disclosure Techniques Explored in the Study60T ................................................................4 

60T Google Desktop6 0T ................................................................................................................................ 5 

60T Google Mobile6 0T ................................................................................................................................. 7 

60T Bing Desktop60T .................................................................................................................................... 9 

60T Bing Mobile6 0T ................................................................................................................................... 10 

60T Gear Patrol Desktop6 0T ....................................................................................................................... 11 

60T Time Mobile60T ................................................................................................................................... 13 

60T Yahoo! Desktop6 0T ............................................................................................................................. 15 

60T Chicago Tribune Mobile60T ................................................................................................................ 17 

60T Qualitative Observations60T ................................................................................................19 

60TThe Aggregate Effects of Improved Disclosures on Assessed Ad Recognition60T .........22 

60T The Relationship Between Improved Ad Recognition and Time Spent Looking at 

the Ads60T ......................................................................................................................24 

60T Methodology Limitations and Areas for Future Research60T .........................................25 

60T Conclusion60T ........................................................................................................................26 

60T Endnotes60T ...........................................................................................................................27 
 

 

60TAppendices 

60T Appendix A: Screening Procedure60T ............................................................................. A-1 

60T Attachment 1 to Appendix A: (First) Screener Questionnaire6 0T ..................................................... A-2 

60T Attachment 2 to Appendix A: (Second) Screener Questionnaire60T ................................................ A-7 

60T Appendix B: Experimental Design Table60T ...................................................................B-1 

60T Appendix C: Main Study Procedure60T .......................................................................... C-1 

60T Attachment 1 to Appendix C: Main Study Questionnaire60T ........................................................... C-2 

60T Attachment 2 to Appendix C: Post-Test Debriefing Interview6 0T .................................................. C-36 

60T Appendix D: Coding of Ad Recognition Data60T ........................................................... D-1 

60T Attachment 1 to Appendix D: Coder Instructions6 0T ........................................................................ D-2 

60T Appendix E: Processing of Eye Tracking Data60T ..........................................................E-1 

60T Appendix F: Regression Analyses of Ad Recognition and Eye Tracking Data60T ......F-1 
 

 

 

 

 

 

 

 

 

 

 

 

Overview1 9F 

The Federal Trade Commission’s Enforcement Policy Statement on Deceptively Formatted 

Advertisements explains that an advertisement is deceptive under Section 5 of the FTC Act if it materially 

misleads consumers as to its commercial nature or source. In other words, consumers should be able to 

recognize an ad as an ad.P1 0F 

2 

P If a separate disclosure is necessary to make that happen, the disclosure should be 

made in a way that ensures consumers can read, process, and understand it. 

Ad recognition may be particularly challenging for consumers in the areas of search advertising and 

“native advertising,” i.e., digital advertising that resembles news, feature articles, product reviews, 

entertainment, or other non-advertising online content. The FTC staff has expressed concern that consumers 

sometimes have difficulty distinguishing regular search results from advertised search results and has provided 

specific guidance to search engines on this issue.P11F 

3 

P The FTC staff also has provided specific guidance to native 

advertisersP 12F 

4 and has provided general guidance on making effective disclosures in digital media.P 13F 5 P 

Only a small number of studies have investigated how consumers perceive these non-traditional forms 

of advertising. Some of these studies are online surveys that explicitly ask about respondents’ understanding of 

an ad’s source or commercial nature.P1 4F 

6 

P Other studies measure changes in participants’ behavior, such as their 

clicking or purchasing behavior or visual attention, but not their understanding of the ads.P1 5F 

7 

This report summarizes exploratory research the FTC staff conducted in 2014-2015 to gain additional 

insights into consumer recognition of native and search ads. In particular, the staff was interested in whether 

making modest changes to the design and wording of disclosures – based on guidance previously issued by the 

FTC staff and usability and web design principles – could improve consumers’ recognition of these ads. The 

study compared participants’ reactions to actual ads (both native and search) as they appeared on a series of 

captured web pages to the reactions of other participants to nearly identical versions of these same web pages 

with ad disclosures modified to improve their prominence, legibility, or clarity. We made a number of 

modifications, including changes to disclosure language, position, text size and color, and to other visual cues 

such as the borders around or background shadings of ads or ad groupings. We tested an equal mix of ads 

viewed on either a desktop computer or a smartphone. 

During the study, we did not ask participants whether they believed any content item on the web pages 

was advertising or to specify its source except with regard to the final series of web pages presented. Instead, 

we primarily evaluated whether participants recognized ads as ads based on indirect measures. After we 

completed the study, coders subjectively scored whether participants recognized content items as advertising 

based upon participants’ spontaneous comments while interacting with the web pages and their responses to 

general questions concerning particular content items.P16F 

8 

P In this study, we also evaluated participants’ eye 

movements as they viewed the ads. 

Overall, the study results suggest that using disclosures that are consistent with FTC staff’s guidance can 

improve the likelihood that consumers will recognize an ad as an ad. Across the different ads we tested – after 

controlling for the influence of other factors such as the type of ad, an individual participant’s tendency to 

recognize or not recognize ads, and the order in which a participant saw the ads – the disclosure modifications 

substantially increased the likelihood that a participant recognized an ad as an ad. The improvements observed 

were substantially similar across search and native ads, as well as between ads viewed on a desktop and on a 

smartphone. Even with the improved disclosures, a significant percentage of participants still did not recognize 

some ads as ads. 

In examining the effect of the modified disclosures on participants’ visual attention, there also was an 

overall reduction in time participants looked at ads. However, our analysis also indicates that any reduction in 

1 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

the amount of time participants looked at an ad was independent of their understanding of the ad’s commercial 

nature. This underscores the importance of testing consumers’ actual understanding of an ad, as opposed to 

exclusively relying on behavioral measures, such as changes in visual attention, to evaluate a disclosure’s 

effectiveness. 

The ads we tested in the study represent only some of the many different variations of search and native 

ads used today, and since we conducted the study, the designs of search engine and media sites have only 

continued to evolve.P1 7F 

9 

P Because we made a number of changes to each ad’s disclosure, we also do not know the 

extent any particular change affected consumers’ ad recognition. Nor does the study, given its limitations, 

establish that consumers were deceived by any one ad we tested. Nevertheless, the results of this study strongly 

suggest that using some commonsense disclosure techniques can reduce the potential for consumers to be 

misled as to the commercial nature of paid search and native ads. 

Study Methodology 

During this exploratory study, we observed, interviewed, and tracked the eye movements of 48 

participantsP1 8F 

10 P as they interacted with eight different sets of web pages or “scenarios” in a usability lab. (See 

Appendix A for the screening protocol and demographic information about participants). Four of these eight 

scenarios were search engine web pages (“search scenarios”) and four were web pages from media sites that 

included native advertising (“native ad scenarios”). In two search and two native ad scenarios, participants 

viewed the web pages on a desktop computer, and in the other two search and other two native ad scenarios, on 

an Android smartphone. Participants saw one of two versions of the web pages: (1) a copy of the actual web 

pages (“original”)P 19F 

11 P or (2) a nearly identical version except that we modified the ad disclosures marking 

particular ads or ad groupings on the web pages (“modified”). Some web pages had more than one ad or ad 

grouping of interest. The study tested participants’ recognition of these particular ads or ad groupings as ads 

(“ad conditions”). Given that we made a number of changes to the disclosures of each ad condition tested, the 

study could not measure the effect of any one disclosure modification on participants’ ad recognition. 

The web pages shown and the particular ad conditions tested in each scenario were as follows: 

Google Desktop: A search results page returned in response to a search for the term “tablet” on google.com, 

as viewed on a desktop computer, and a second page or “click-into page” that displayed after a participant 

clicked a link labeled “Shop for tablet on Google.” The ad conditions examined in this scenario were: (1) the 

“Shop for tablet on Google” box (“Shop Box ads”) on the search results page; (2) ads on the right-hand side of 

the same page (“Right-side ads”); (3) ads above the organic search results on that page (“Top ads”); and (4) the 

ads shown on the click-into page (“Shop Page ads”). 

Google Mobile: A search results page returned in response to a search for the term “carpet cleaner” on 

google.com, as viewed on an Android phone, and a second page that displayed after a participant clicked a link 

labeled “Shop on Google.” The ad conditions examined in this scenario were: (1) the Shop Box and Top ads 

on the search results page, and (2) the Shop Page ads on the click-into page. 

Bing Desktop: A search results page returned in response to a search for the term “air purifier” on bing.com, 

as viewed on a desktop computer. This scenario did not include a click-into page, although participants could 

click on various items on the main page. The ad conditions examined in this scenario were the Top and Rightside ads on the search results page. 

Bing Mobile: A search results page returned in response to a search for the term “vitamin d supplements” on 

bing.com, as viewed on an Android phone. This scenario did not include a click-into page, although 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

participants could click on various items on the main page. The single ad condition examined in this scenario 

was the Top ads on the search results page. 

Gear Patrol Desktop: The home page of the lifestyle magazine Gear Patrol, as viewed on a desktop 

computer, and a second page that displayed after a participant clicked on a review of a sports car included 

among the content items on the home page. The ad conditions examined in this scenario were the car review 

promoting the Subaru WRX as it appeared on the main page and on the click-into page. 

Time Mobile: The health section of Time magazine’s site containing a continuous stream of content items, as 

viewed on an Android phone, and a second page that displayed after a participant clicked on an article about the 

incidence of home burglaries. This article was listed in a Taboola-branded “content recommendation widget,” 

which was nestled within the site’s content stream on the main page. The ad conditions examined in this 

scenario were the feature article promoting an ADT home security system as it appeared on the main page and 

on the click-into page. 

Yahoo Desktop: The home page of Yahoo! containing a stream of content, as viewed on a desktop 

computer, and a second page that displayed after a participant clicked on an article about pain relief in the 

content stream on the home page. The ad conditions examined in this scenario were the feature article 

promoting the pain-relief supplement Instaflex as it appeared on the main page and on the click-into page. 

Chicago Tribune Mobile: An article about a woman’s battle against hair loss appearing on the Chicago 

Tribune’s newspaper site, as viewed on an Android phone. The article displayed after a participant clicked a 

link to the article forwarded in an email from a friend. The email shown to participants was an actual email 

generated by clicking one of the social media buttons on the Chicago Tribune site, which enabled sharing of the 

article via email. This native ad scenario included only the click-into page. The ad condition examined in this 

scenario was the feature article promoting the hair-replacement services of Embassy Studio, a Chicago-area 

salon. 

* * * 

A moderator presented these scenarios to participants in a counterbalanced order, i.e., a different order 

for each participant, which reduced the potential for contaminating the results with learning effects. (See 

Appendix B for the Experimental Design Table.) During each scenario, participants “thought aloud,” i.e., 

verbalized what they were thinking as they looked at and interpreted the web pages shown to them, and 

responded to open-ended questions. (See Appendix C for a description of the study procedure and copies of the 

main study questionnaire and post-study interview questions.) When participants initially saw a web page, they 

looked at the page uninterrupted for several seconds, scrolling around as desired, while eye tracking equipment 

measured their eye movements and fixations on the screen. 

It is important to note that we designed the study to prevent participants from deducing its purpose and, 

as a result, altering their normal behavior while interacting with the web pages. The moderator did not 

explicitly ask participants about whether anything on the web pages was advertising and also specifically asked 

their opinions of regular content or “decoy” items. The moderator asked if participants saw any ads on the web 

pages only for the last scenario presented, after completing the main study. Participants’ responses to questions 

asked after the moderator presented all eight scenarios confirmed that the study was successful in concealing its 

purpose from them. 

We video recorded each participant session. These video recordings included audio of the sessions and 

real-time renderings of participants’ computer mouse movements, eye gazes, and fixations on the screen. Based 

on an examination of these video recordings and transcripts of the sessions, two different coders – an expert 

consultant and either one of two FTC paralegal staff – coded each participant on whether or not he or she 

3 
 

 

 

 

 

 

 

 

recognized particular ads as ads or if it was unclear to the coder. (See Appendix D for a description of the 

coding process.) Eye tracking measures also assessed the amount of time participants fixated on different areas 

on the screen, including on the areas encompassing particular ads and ad groupings. (See Appendix E for a 

description of how these eye tracking data were processed.) In this study, we considered the expert consultant’s 

12 13 coded judgments,P2 0F P eye tracking measures, and various visualizations – in the form of heat mapsP2 1F P and gaze 

plotsP22F 

14 P – depicting where and the order in which participants gazed and fixated on the web pages in 

qualitatively evaluating the performance of the various disclosure techniques tested. 

Disclosure Techniques Explored in the Study 

We made a number of changes to improve the saliency and clarity of disclosures used to distinguish paid 

search ads and native ads from other content on the web pages that participants saw during the study. The 

original and modified versions of the web pages shown to participants were identical except for these disclosure 

modifications. Consistent with FTC staff guidance on making effective disclosures in the context of search 

engines and native advertising, the changes we made focused on factors such as the location, language, text 

color, and size of the ad labels and the prominence of other visual cues used to distinguish ads from other 

content. Although the changes to the disclosures varied given design and format differences among the web 

pages, certain principles guided these changes. We kept the disclosure modifications to a minimum so as to 

achieve increased salience and clarity without significantly altering the overall design and appearance of the 

web pages. Moreover, we made changes with the goal of more closely aligning the ad disclosures with userinterface and web design principles articulated by one of our consultants, including: 

1. Distinctive labeling, highly visible borders, and distinctive background colors can be used to visually 

distinguish different types of content. 

2. Larger and bold fonts make headings, labels, and notices more noticeable. 

3. Text color should contrast strongly with the background. Light text on a dark background is more 

difficult for users to read and process because the effective size and boldness of the text is reduced when 

the dark background visually bleeds into the light text characters. 

4. Organize and order the content on a page, i.e., use visual hierarchy, to ensure that users see ad labels and 

associate them with the correct item. Position and space ad labels so it is clear to which content items 

they relate. If a web page is written in a language that is read left to right, such as English, labels should 

be placed above and/or immediately to the left of their associated content. 

5. When color differences are intended to signal something, ensure that the colors contrast well with each 

other and include redundant cues other than color in case some users cannot see the color differences. 

6. Use terminology and symbols consistently on a site to ensure that users understand their meaning. 

7. For labels, use terms that are familiar and clear to users. Avoid technical or industry-internal jargon. 

Below are descriptions of the specific changes we made to the ad disclosures in each scenario and screen 

shots of the original and modified versions of the web pages that illustrate these changes. 

4 
 

 

Go gle '-"I 1 ______________________ ._._4_ 

Go gle 

Web Shopping News Images Videos 

About 286,000,000 results (0.31 seconds) 

Shop for tablet on Google 

Samsung 

Galaxy Tab 3 ... 

$49.99 

Apple iPad mini Dell Venue 8 

with Wifi... Tablet 

$279.00 $179.99 

Sprint Staples Dell 

9 In store 

iPad mini 

m!I www.apple.ooml • 

with Retina display. Small wonder. Leam more. 

Back to School • iPad Air · iPad mini · Buy now 

More .... Search tools 

Sponsored © 

Microsoft Ematic 7" Tablet 

Surtace Pro 3... 8GB M ... 

$999.00 $54.99 

MicrosoftStore Walmart 

22% price drop 

<D 

7101 Democracy Blvd, Bethesda, MD · (301) 634-9930 

Web ShOpping News Images Videos 

About 286,COO,OO0 results (0.31 seconds) 

IMiJ<D Shop for tablet on Google 

Samsung 

Galaxy Tab 3 ... 

$49.99 

Apple iPad mini 

withWifi ... 

$279.00 

Staples 

Dell Venues 

Tablet 

$179.99 

Sprint Dell 

9 In store 

iPad mini ~ <D www.apple.com/ • 

with Retina display. Small wonder. learn more. 

Back to School - iPad Air - iPad mini - Buy now 

More• 

Microsoft 

Surtace Pro 3 ... 

$999.00 

MicrosoftStore 

7 101 Democracy BIVCl, BetnescJa, MD - (301) 634-9930 

Search tools 

Ematic r Tablet 

8GBM ... 

$54.99 

Walmart 

22% price drop 

Nexus 7 from $229 

www.google.com/nexus ... 

The 7" tablet from Google. Free 

shipping for limited time. Buy now 

High Quality Dell Tablets 

www.dell.com/Tablets • 

4.0 **** rating for dell.com 

Perfect for Work, Perfect for Play. 

Wrth Intel Inside®. Buy One Today! 

Android Tablet PC $39.99 

www.tmart.com/ • 

.. 

• <D 

Nexus 7 from $229 

www.googl~.com/n'1xus • 

The 7" tablet from Google. Free 

shipping for limited time. Buy now 

High Quality Den Tablets 

www.dell.com/Tablets • 

4.0   ratirg for dell.com 

Perteet for Work, Perteet for Play. 

W~h Intel Inside®. Buy One Today! 

Android Tablet PC $39.99 

www.tman.com/ ... 

Google Desktop 

On the initial or “search results page” shown to participants in the Google Desktop scenario, we: 

1. replaced the “Sponsored” label for the Shop Box ads with an “Ads” icon used to mark other ads on the 

page; 

2. for the Shop Box ads, relocated this new label and the information icon from the top right of the box to 

the top left; 

3. darkened the border around the Shop Box ads to better separate them from the regular search results; 

and 

4. changed the text and background color of the “Ad(s)” icons from white-on-yellow to black-on-orange. 

Original 

Modified Version 

5 
 

Go gle 

Electronics 

Audio 

Computers & Devices 

Photo & Optics 

1V & Video 

More 

Clear all fillers 

Germantown, MD 

Change 

Go gle 

Eloctronlcs 

Audio 

Computers & Devices 

Photo & Optics 

TV & Video 

More 

Clear all fillers 

Gennantown. MO 

Change 

tablet 

Web Images 

Tablet Computers 

Most popular 

tablet 

Web Images 

Tablet Computers 

Most popular 

Maps Shopping 0 

Sol1: Default • View: Lisi • My Shortlist (0) 

Amazon Kindle Eice HPX - 16 GB - Wi-Ei Only - Black - wjth Special 

$199.00 from 25+ stores In stock nearby 

** **; 964 product mviews #1 in Tablet Computers 

October 2013 · Amazon · Android · 16 GB · 7 inch · Wi-Fi Only · 11 hour battery · 10.7 ounce · 1920 x 1200 

Kindle Fir& HDX tablets are the firnl tabteta to be powered by the 2.2 GHz quad-com Snapdragon 800 processor, 

making them the first to clock in ... 

Other capacity options: 32 GB ($269) 64 GB ($310) 

Kindle Fire HD 7" 8GB 

$139.99 from 20+ stores In stock nearby 

 1 1,094 product reviews #2 in Tablet Computers 

Maps Shopping Mom• (II 

Sol1: Default • View: Lisi • My Shortlist (0) 

(!) Amazon Kindle Fire HDX - 16 GB - Wi-Ei Qo!Y. - Black - wjth SpecjaL. 

$199.00 from 25+ stores In stock nearby 

** **1 964 product r&views #1 in Tablet Computers 

October 2013 ·Amazon· Android· 16 GB· 7 inch · Wi-Fi Only · 11 hour battery· 10.7 ounce · 1920 x 1200 

Kindle Fire HDX tableta are the first tablets to be powemd by the 2.2 GHz quad-oom Snapdn,gon 800 processor, 

making them the first to clock in .. . 

Other capacity options: 32 GB ($269) 64 GB ($310) 

(!) Kindle Fire HD 7" 8GB 

$139.99 from 20+ stores In stock nearby 

 ' 1,094 product review-s 12 in Tablet Computers 

On the second or “click-into” page shown to participants in this scenario, we: 

1. deleted the “Merchant links are sponsored” label located above the search results and in the top right of 

the page; and 

2. in place of this label, added a combined “Ad” and information icon – in the same format used on the 

initial page – to each result listed on that page. 

Original 

Modified Version 

6 
 

 

 

Google carpet cleaner II Web Shopping Maps Images More • 

Shop on Google -<D 

l S99.99 $89.99 

bissell.com 

S165.00 

Target Walmart 

$75 - 2 Rooms Cleaned - Mention Ad To 

Receive This Offer 

www.drimasters.neV 

Call Now Same Day Service Available \,. Coll 

Stevens Chem Dry MD - Chem-Dry Carpet 

Cleaning System 

www.stevenschemdrymd.oom/ 

$35/Room Call or Book Online Now 

Rug Doctor Carpet Cleaners 

www.rugdoctor.oom/ 

\. can 

Rent or buy carpet cleaning machines and products. 

3ermantown ~ @) 

A Cleaner Car et Cleaner 

\,OIU/lll 

Highland 

@ (ill) 

North Lau1 

&monsville 

Lau Mop data 0201 Googlo 

<D 

Google carpet cleaner 

Web Shopping Maps Images More • 

Ada <D Shop on Google 

$99.99 

Target 

$89.99 

blssen.com 

$165.00 

Walmart 

$75 - 2 Rooms Cleaned - Mention Ad To 

Receive This Offer 

(!) www.drimasters.neV 

Call Now Same Day Service Available \,. Call 

Stevens Chem Dry MD - Chem-Dry Carpet 

Cleaning System 

(!) www.stevenschemdrymd.com/ 

S35/Room Call or Book Online Now 

Rug Doctor Carpet Cleaners 

www.rugdoctor.com/ 

\,. Call 

Rent or buy carpet cleaning machines and products. 

- "-'OIUITll 

~ Hightend :iermantown @) 

Olney @ @I 

(ill) Nor1h La111 

~ 8urtonsville rnutown Lau 

m> Rockvilfe hn.on U ill l,lap<lata 0201 °""9lo 

A Cleaner Ca et Cleaner 

Google Mobile 

The changes made to the web pages in the Google Mobile scenario tracked the changes above in the Google 

Desktop scenario. Specifically, on the search results page shown to participants, we: 

1. replaced the “Sponsored” label for the Shop Box ads with an “Ads” icon used to mark other ads on the 

page; 

2. for the Shop Box ads, relocated this new label and the information icon from the top right to the top left 

of the box; 

3. added tan background shading to better separate the Shop ads from the regular search results; and 

4. changed the text and background color of the “Ad(s)” icons from white-on-yellow to black-on-orange. 

Original Modified Version 

7 
 

 

 

 

Web 

Goog

Shopping 

le carpet 

More • 

cleaner 

--

I 

Sponsored (D 

BISSELL ReadyClean Deep Cleaner 

40N7 - Upright Carpet ... 

$89.00 from 10+ stores 

** **"' (53) 

Hoover SteamVac With Clean Surge 

F5914900 - Upright ... 

$135.00 from 25+ stores 

 ' (943) 

BISSELL Powerlifter PowerBrush Deep 

Cleaning System 1622 ... 

$99. 98 from 25+ storas 

** **"' (60) 

WhiteWing II Steamer Vapor Steam 

Cleaner 

$399.95 from AllergyBuyersClub 

BISSELL QuickSteamer Powerbrush Pet 

Deep Cleaner - 47B2T 

$99.99 from Target 

Carpet & Rug Cleaning Machine 

$1 ,279.00 from Cleanfreak.oom 

Google carpet cleaner 

Web Shopping More • • Rofino 

l 

Ad <D BISSELL ReadyClean Deep 

Cleaner 40N7 - Upright Carpet .. . 

$89.00 from 10+ stores 

 "' (53) 

Ad <D Hoover SteamVac With Clean 

Surge F5914900 - Upright ... 

$135.00 from 25+ stores 

** **" (943) 

Ail <D BISSELL Powerlifter 

PowerBrush Deep Cleaning System 

1622 ... 

$99.98 from 25+ stores 

****"' (60) 

Ail <D WhiteWing II Steamer Vapor 

Steam Cleaner 

$399.95 from AllergySuyersClub 

Ail <D BISSELL QuickSteamer 

Powerbrush Pet Deep Cleaner -

47B2T 

$99.99 from Target 

<D Carpet & Rug Cleaning 

Machine 

$1 279.00 from Cl@anFreak.com 

On the click-into page in this scenario, we: 

1. deleted the “Sponsored” label located above the search results and in the top right of the page; and 

2. in place of this label, added a combined “Ad” and information icon – in the same format used on the 

initial page – to each result listed on that page. 

Original Modified Version 

8 
 

 

l> bing 

l, bing 

WES 'MAGES VlDF.OS MAPS NEWS MORE 2 • 1 ':° So;Ji- 1n .:. 0 

~r purifier 

1.110,000 RESULTS .... -. Ma t9IMd 11D.., purifllr 

Jop 10 Air Purifiers -See Which Made UJe TOJ) 10 List AJr-Pul'IAln-Alnliric:a..com 

Expe,t ~w,wti. F,.. Shffing! 

•-~.com• rated on e.,,g (18 ,..,_., 

Air Pudfiers at Lowe's®"' www.LO't11M9...c:om 

Keep'l'olJrNtAllerven& Du&aFreaWlln Alr Purlftera Ml.ow9'18 

er.nct1-· G.-m Q'*"'9n. ~.Man. Fall(lwfl. FJll'N.,,dmora 

8755 Brwd'I A'l'IIIUI, Climon Dnctionl. (301) 877•7790 

kMeil.com la rated on Bing {7739 ~ ) 

Em.Shi~ Lpwp) A:ldil StoocN 

.l::lish..0wllilY. - Air Purifier • Ultra Quiet -BioGS Air eurmer www.RllbbltAlr.com 

FrM(ln1ppil'lglndUOCM'Y'l'91uffl 

THE Best Air Purifiers - Free ShioQiog & Five Year warranty. www.Roctw"MountalnAl~riflara.com 

~Pur11Rod::yMoutUs!A14 

Ale Pudflers - Wa!mart eom WNW wa!mart.com • AU o.,,.rtrn.,"IS : 

Air purtflrtrs_ Git• I"*- air purifier 0t HEPA air fftlf It a gru1 low pre. Save money 

L.iY9 bettar 'Wlilnwrt.com 

P] 

~ I 

- -

~~-­ 1219.00 1,2,c.s.oa ~~ $2t9,00 S2tt.t9 

-

><ZoLAbt - ~- w-­

Aleo A375 Air Pudfier www.AlenCorp.com/AlenA375 

S,,ipt FtN_..,.24 Houri 60 DlyG1Art1noN •Ufwn.WfllfMt; 

Austin Ne Purifier Sale www.Betterffeatthlnnovatlorm..com 

WEB IMAGES VlDEOS MAPS NEWS lrK>RE '} . • 1' S"J" m ..:. (.1 

aJrpurifiH 

8.1110.0DO RESULTS .., ..... Ads related to air purifier 

IQO 10 Air Purifiers - See Which Made the ToP...lil.Lin. .............. ........_ ~~F,.Shlppingt -~•raed one...g(MteYIN'sJ 

Air _..__ Purifiers at Lowe's®"' 

Keep Your Alr Mergen & Di.191 Free w.ti Air Pur1n.r. al Lowe-... 

fnrw:11. GtnnGuMl.ln. ~. Alien. F410Wtit.Fll._.end ~ 

8765BranchAVWAJ11,ClrlWI ~ (301)8n•77IIO 

~comlara'8d onlk'l3 (7739N!MeWS) 

~Mi! Lqwn) Cnpdi; Sm1Rtt 

l::!lgh.OuaJity - Air Purifier - Ultra Oujet --BioGS Air Purifier -Rataoi\lUt..«M'l'I 

Free&l'llppwlg and90 m,"S return. 

THE Best Ale Purifiers - Free Shlooioo & F1ve Year warrantv"' WftW.RockyMountlilnAkPuirfflen.com 

CllaccNarPul'IIRl:d:yMCIUflla'I AlrCI 

Air Purlfiers - wa1mart com WWW ¥13Jffiaft.com I IJ 0epa""*'11! • 

Nr purine,.. Gell• new air purln.r or HEPA air flt.- at a great ION prkll,.. Sava money. 

UYw blltier Wllmertmm 

p 

Ads related to air purlfler 

~--- Whl!QRa! ~ Vd'fCfl HauseM .l::k:tCl:!ILl:a: 

~ 

121t.oo sn.n 020.00 -14.1.11 

Flllefsfasl.aMrl Global~ Dl'\IDSICn COl'T'I 

Aleo A375 Air Purifier www~nCo,p.caml.tJfflA.375 

Shls» F,_ wltll"I 2~ Hows 1110 O.y ~IM +Llatrrri. Wananty 

AusHn Ale Purifler Sale www.BetterHullhlnnCMltJOM.com 

Bing Desktop 

On the search results page shown to participants in the Bing Desktop scenario, we: 

1. changed the background color of the Top and Right-side ads from light gray to tan so that it contrasted 

better with the background color of other content on the page; and 

2. enlarged the font size of the ad label “Ads related to air purifier.” 

Original 

Modified Version 

9 
 

 

 

 

 

 

vitamin d supplements 

WEB IMAGES VIDEOS NEWS 

Ads related to vitamin d supplements 

Free Vitamist Spray 

http://www. vita mi sis pray.net 

Buy 2 or more Vltamist products, get 1 free. Free shipping 

USA 

Calcium & Vitamin D 

m.Caltrate.com 

Calcium & Vitamin D Are Essential Throughout Life -

Learn More. 

VITAMIN D: Uses, Side Effects, Interactions 

and Warnings ... 

www.webmd.com/vitamlns-supplementslingredientmon ... 

Find patient medical information for VITAMIN Don WebMD 

including its uses, effectilleness, side effects and safety, 

interactions, user ratings and .. 

Best Vitamin D Supplements - Consumer 

Reports 

www.consumerreports.org/cro/magazine/2013/05/most-vl 

Find the best vitamin D supplement with Consumer 

Reports' review of 32 brands. 

Vitamin D - What Should I Know About It? 

altmedicine.about.comlodlhert,supplementguide/aMtam .. . 

What is vitamin D? What are the benefrts of using vitamin 

D? What are the possible side effects? 

vitamin d supplements 

WEB IMAGES VIDEOS NEWS 

Ads related to vitamin d supplements 

Free Vitamist Spray 

http://www.vitamistspray.net 

Buy 2 or more Vitamist products, get 1 free. Free shipping 

USA 

Calcium & Vitamin D 

m.Cattrate.com 

Calcium & Vitamin D Are Essential Throughout Life -

Learn More. 

VITAMIN D: Uses, Side Effects, Interactions 

and Warnings ... 

www.webmd.com/vitamins-supplements/ingredientmon ... 

Find patient medical information for VITAMIN D on WebMD 

including its uses, effectiveness, side effects and safety, 

interactions, user ratings and .. . 

Best Vitamin D Supplements - Consumer 

Reports 

www.consumerreports.org/crolmagazine/2013/05/most-vi ... 

Find the best vitamin D supplement with Consumer 

Reports' review of 32 brands. 

Vitamin D - What Should I Know About It? 

allmedicine.about.com/od/herbsupplementguide/a/vitam ... 

What is vitamin D? What are the benefits of using vitamin 

D? What are the possible side effects? 

Bing Mobile 

On the search results page shown in the Bing Mobile scenario, we: 

1. darkened the green background color of the Top ads and other ads located on the page to increase 

contrast with the regular results on the page; 

2. enlarged the font size of the ad labels; and 

3. increased slightly the space between the label “Ads Related to . . .” and the first item listed below it to 

better signal that the label related to both the Top ads. 

Original Modified Version 

10 
 

 

 

RISE OF THE MACHINES 

POSITIVELY CHARGED: 7 BEST 

ELECTRIC MOTORCYCLES OF 2014 

While electJic cars have Slumbled in the recent paat. motorcycles 

have reaidtly embraced digital proputslon - (wen Harle~·Oavi<fson 

has one in lhe works. With l&ss mass to moliva1e, shorler runtime 

requlremen1s and a complete lad!. o1 pow&r-r0Cbh1g creature 

features. motorcydes are a na1ural fl1 for battery-opere1ed 

powerpfants. Here are the seven besl electlic motorcycles 012014. 

READ M.ORE 

A WORLD RALLY BLUR 

WITH THE 2015 SUBARU WAX, A RALLY 

CAR MATURES 

Since it anived onlo North American shores in 2002 lhe Subaru 

WAX has been making the case lor unadullera,ed run, one IOUI· 

wheel drift al a time. Vetdcles anef oor expectations ol 11\em howev0t, 

have changed a kit over the last 13 model '.,'Bat$. No longer conlenl 

to merefy juke and jive wrth the best 01 them. we r'IOW demand our 

cars have slyhng. mo,e toys than an only ctuld ano sofMOUCh 

evecything. To keep these newly ,elined ralliers,: content in lheir 

OOCkpilS, Subaru sent the WAX to fir'IISl'ling sehool. And. ,, you ask 

us. ii gradualed - WilJ'I hOnots. 

READ MORE 

THIS POST IS PRESENTED B't SUBARU. 

SUSA.RU 

RISE OF THE MACHINES 

POSITIVELY CHARGED: 7 BEST 

ELECTRIC MOTORCYCLES OF 2014 

While electric cars have slumbled in the recent past, mo101cydes 

have readily embraced digilal propulsion - even Harley-Davidson 

has one in lhe WOfks_ With less mass to mo1Na1e. shorter run1ime 

r&quaremenis and a complete lack ol i:,ower•rObbing c,eature, 

fealutM. motor'C)'cles aro a natural h1 for ttattery-opetatod 

pow6rplants. Hoto are'"° MYon boSI Ol&C'lric motorcy<:IO:!i OI 2014, 

AUD MORE 

SUBARU 

A WORLD RALLY BLUR 

WITH THE 2015 SUBARU WAX, A RALLY 

CAR MATURES 

Since II arrived onto North American shores In 2002 the Subaru 

WRX has been rT\iOOng lhe ease for unaclulterated lun, one lourwheel drifl al a time. Vehicles and our expeclations o1 them however. 

have changed a lo1 over the lasl 13 model years.. No longer cont&nt 

to merel'.,' juke, and jive with 1hO best Ol them. we l1¢'W demand our 

ca,s l\aw stytang, n"IOl'e toys lhan at1 ontv cl*! and t;Oft•toueh 

everything. To k.eep lhese nov.ty ,etined ralliot.s C01"1ton1 in 11'1eir 

COCl<p.ls. Subaru Mnl tno WAX 10 linistilng .sd'!ool. And, Ii you ask 

us, rt graduated - wi11l honors. 

READ MORE 

Gear Patrol Desktop 

On the initial or “main page” shown to participants in the Gear Patrol Desktop scenario, we: 

1. applied a tan background color to ads appearing within the stream of content items on the page; 

2. reworded the ad label for these ads from “This post is presented by . . .” to “Advertisement by . . .”; and 

3. relocated this new ad label, along with the brand logo, above the ad to which it relates. 

Original Modified Version 

11 
 

10 BARGAIN THRILLER CARS 

'M'IOIIYl~-b'h 

.,........, 5 BEST RUNNING SHOES 

w.~~Jilll00fWIMrlll)[NI 

-- ~ Ol'FICE-l'R1EJIOLV 

_ MOTORCYCLE GEAR 

Sll'*YOONl'l,11'1Nft.Nl'1Clftll 

THUASOAY. JUL 10, 2014 -

GEAR PATROL 

BEST UNDERWEAR FOR 

ACTIVE GUYS 

5WrOIIIOIN*"4f!IIP 

OP1D0 FEATURES BU'i' IN G GUI D ES K ITS FILMS OPIN I ON SER I ES SEARCH SUBSCRIBE AB O UT 

( PREY I Tho Supo,w- Is Being Po,,_ - •• a Hyt)rid --Cha,vod: 7 a.s1 EJoclliC: Moco<o:ydOs er 201 • I NEXT ) 

A WORLD RALLY BLUR 

WITH THE 2015 SUBARU WRX, A RALLY CAR MATURES 

ii4'11fhh-14•fli VEHICI.E$: ¢AR$ 

10 BAAOAJN THRILLER CAFtS 

Who 

__ MYS l)lifb'll"4Ja II ac.,r thl, , 

Sy MAiT N(UNDORF on 7,7. U Pl'loto by THARANGA RAMANAVAK E 

5 GEST RU NI NO SHOES 

W.!Mffll.ffn9llll~lo--

- ~ OFFICE-FRIENDLY 

_ MOTORCYCLE GEAR 

SlrttttOONrl,fflMtlM10nli 

THUFISOAV, JUL 10, 2014 -

GEAR PATROL 

BEST UNDERWEAR FOR 

ACTTYE GUYS 

S&lyOIIIOlllhe....wnpGP100 Fll:AT URES BUYING G U IDES KITS FILM$ OPINION SERIES Sll:A RCH SUBSCRIBE ABOUT 

( PREY I The Superear Is Being Porl0<1e<1 - as a Hjl)riO 

A WORLD RALLY BLUR 

WITH THE 2015 SUBARU WRX, A RALLY CAR MATURES 

VE HICI.ES : CA.RS 6Y' MA.TT HEUNtlOR f 01"1 7,7,1-4 Pl'IOIO by l'HA.RA.N!GA RAMANAYAKE 

On the page displayed upon clicking the Subaru WRX ad or the “click-into page” in this scenario, we: 

1. reworded the ad label from “Sponsored Post” to “Advertisement;” 

2. enlarged the label’s font size; and 

3. changed its text color from white to black. 

Original 

Modified Version 

12 
 

 

= TIME 

of the amygdala were activated based on 

judgements of trustworthiness or nontrustworthiness. This, the researchers 

conclude, is evidence that our brains 

make judgments of people before we 

even process who they are or what they 

look like. 

Keep that in mind the next time you're 

meeting someone new. No pressure. • 

Sp()f'ISOf'Od Conl nt 

Burglars Aren't Scared of 10 Surpri sing Signs of 

Anything... Stroke Risk 

The Art Of Living Caring.com 

b~Taboda 

EIIIIEII 

= TIME 

of the amygdala were activated based on 

judgements of trustworthiness or nontrustworthiness. This, the researchers 

conclude, is evidence that our brains 

make judgments of people befo re we 

even process who they are or what they 

look like. 

Keep that in mind the next time you're 

meeting someone new. o pressure. • 

PAID CONTENT 

Ad Burglars Aren't 

Scared of Anything ... 

The Art Of Living 

1 O Surprising Signs of 

Stroke Risk 

Caring.com 

by Taboola 

DIEIIIIEII 

Time Mobile 

On the main page shown to participants in the Time Mobile scenario, we: 

1. deleted the “Around the Web” labels above each of the content recommendation widgets inserted into 

the content stream; 

2. reworded each “Sponsored Content” label to “Paid Content”; 

3. centered this new “Paid Content” label above each content recommendation widget on the page; and 

4. applied an “Ad” label to any item listed in the content recommendation widgets that was an ad. 

Original Modified Version 

13 
 

THE ART OF 

Last Updated on 2014-06-09 

Cil!IDG:: "#Tweet 0 

Did I Lock My Front 

Door? 5 Tricks To Stop 

worrying About Your 

Home 

9 

We all know that draining feeling al the pit of your 

stomach. You forgot something, but you are not sure 

what it is. You're an hour from home, stuck in traffic, or 

THE ART OF 

l.851 Updo1o<I on 2014-06-09 

CB G:: w,._1 o 

IADVERTISEMBNT I 

Did I Lock My Front 

Door? 5 Tricks To Stop 

Worrying About Your 

Home 

9 

We all know that draining feeling al the pit of your 

stomach. You forgot something, but you are not sure 

what it is. You're an hour from home stuck in traffic or 

On the click-into page for the ADT ad in this scenario, we added the label “Advertisement” above the headline. 

Original Modified Version 

14 
 

 

 

A Home Mail Answe,s Groups Fhc.kr Tumblr G..1mes. l 1,,,.e Screen Mobile More v • v 

AH0O! C ______ _ ,.,,, Manatee Keeps Watch Over Dog Before Rescue 

A momenl of kindness between two animals was caplt.-ed by a Tempa 

police officer just befOC'8 a colleague rescued a stranded dog from the 

Hill~gh R1..,ec. R~idenl5 in Tempe•s Seminole H4'1ighls 

Good Morning Atnel1ca 

Knee & Shoulder Relief 

Ooctors spoal< out al>OUt snocking 1<nee & 

shooklor rollof. 

lnstafklt. ~ 

Bad, Bad Gir1! Stephanie Beaudoin, 1World's Sexiest 

Crook' Facing 114 Charges, Creates Online Frenzy 

StOjlhanle Beaudoin has boon Judged - by social medla users - tho 

world's sexlost crook. Facing 1 14 theft cha,ges, her l'witlor pie Is al Tho , ...... tr t,nm ASS[: 

My Yahoo a Sign In Ill Ma;1 

Today 

69" 

Tomorrow 

74• 

Quotes Yahoo Finance 

S&PSOO 

NASOAO Composite 

Dow Jor,es lndl!st 

Scores Yahoo Spo~s 

NfL 

1,988.17 

4,522.43 

17,105.38 

Thursday 

,,. 

-0.319' 

-0.129' 

-0.399' 

A Home Mail Answers Groups Flickr Tumblr Games l ive Screen Mobole More v .II v 

YAHOO! "------------------- 11¥¥11 

Ad 

Manatee Keeps Watch Over Dog Before Rescue 

A momen1 of kindness betwNn IWO animals was captur1N1 by a Tampa 

polie& ~.,.., ju&t ""'°"' a cotl-rasctJed a •tranded dOg rrom the 

Hin:slx>mugh Rl'Jer. Residents in Tampa's Seminole Hefgtlt.s 

GoodMonw,gllmerico 

Knee & Shoulder Relief 

Ooclors speak out aoovt ohocking knee & 

shoulder relief • ........ 

Bad, Bad Gir1! Stephanie Beaudoin, 'Worf d's Sexiest 

Crook' Facing 114 Charges, Creates Online Frenzy 

Stopt,anlo Beaudoin has boon /udgod - by social modla """"' - tho 

wortd's sexlost crook. facing 114 thef1 Cllarges. her Tuittor pie Is all 

The ll"lqldiu rtom RSS [• 

My Yahoo s;gn rn Ill Mail 

Today 

69' 

Tomorrow 

74• f 

Quotes Yahoo Rnanc<> 

S&PSOO 

NASOAQ Composite 

Dow Jones Inch.is,-.• 

Enter company/locker 

Scores Yahoo Sport• 

NA. 

1.988.17 

4.522.43 

17,105.38 

Thursday 

71' 

-0.31'ff, 

-0.12'ff, 

-0.39% 

Yahoo! Desktop 

On the main page shown to participants in the Yahoo! Desktop scenario, we: 

1. added a black border around each ad inserted into the content stream on the page; 

2. reworded the ad label for each of these ads from “Sponsored” to “Ad”; and 

3. relocated these new ad labels above the ads to which they relate, instead of below the ads. 

Original 

Modified Version 

15 
(• health ,;;;,i;;. 

Powerful Joint Discovery Revealed by Dr. 

David Katz on CBS "The Doctors" 

~· health 

ADVERTISEMENT BY 

lnstaflfK' 

New research into joint health compounds has finally paid off. On November 

15th, Dr. David Katz presented results from a new Clinical study on CBS's 

"The Doctors". The Clinical study at the North Carolina Research campus 

shows that one joint compound in particular can deliver significant, effective 

joint relief. 

In the double-blind University clinical study, lnstaflex, a specialty joint 

supplement resoundingly beat the control group. Researchers at the North 

Powerful Joint Discovery Revealed by Dr. 

David Katz on CBS "The Doctors" 

New research into joint health compounds has finally paid off. On November 

15th, Dr. David Katz presented results from a new Clinical study on CBS's 

"The Doctors". The Clinical study at the North Carolina Research campus 

shows that one joint compound in particular can deliver significant, effective 

joint relief. 

In the double-blind University clinical study, lnstaflex, a specialty joint 

On the click-into page for the Instaflex ad in this scenario, we: 

1. relocated the ad label “Advertisement by Instaflex” appearing in the banner area at the top-right of the 

web page to immediately above the ad’s headline; and 

2. changed the coloring of this label from white text on a dark-gray background to black text on a white 

background. 

Original 

Modified Version 

16 
 

 

 

= (fhirago <[rilllllll' Visusl ~rows,, 

ERAr,O t'lJF.l ISt-!1':C; Th1s.1ssponsor&dconl8'nl > l13am m~ 

Brand Publishing / 

Embassy Studio 

After hair-loss battle, resolution for 

female alopecia sufferer 

, 

= {fhiragO Qtibt11tl' Vurual ~rowsc 

r:.,rJ pc.s r This is sponsored content. > !!li!!!Ll!!2<_. 

Paid Post/ 

Embassy Studio 

After hair-loss battle, resolution for 

female alopecia sufferer 

Chicago Tribune Mobile 

On the click-into page displayed after participants clicked a link for the Embassy Studio ad in an email in the 

Chicago Tribune Mobile scenario, we: 

1. reworded the label “Brand Publishing” to “Paid Post”; 

2. enlarged the font size of the “This is sponsored content” label; and 

3. reworded the author credit at the end of the article from “Brand Publishing Writer” to “Writer for 

Embassy Studio.” 

Original (Top of Page) Modified Version (Top of Page) 

17 
 

= ([hiCllQO(Lribllltl' V,.,,-ual Bro"se -- - V 

BRA~, D PUEl \SHl~, c Th1t,1$sponsorodcon!onl ) ll'J-O~~ITIQ!Q 

the remam,ng areas. 

"lf someone was really discouraged by all the otller 

methods, 1 would be happy lo how her what it was like," 

he ays. "! can understand why women feel so bad, 

because I do, too." 

The number of NeoGraft procedures needed varies from 

person lo person, but Smith is confident that she won't 

need a third procedure. Patient.S typically see 

improvement within six months, with full result. with.in 

one year. 

-Lisa Schryver, Brand Publishing Writer 

= ([hiClll\0 ([riblllll' Vi.sual Browse - - V 

P,"D pr,s T This ,s sponsored content. > ~oc• 

the rema,mng areas. 

"If someone was really discouraged by all the other 

methods, I would be happy to show her what it was like," 

she says. "I can undersrand why women feel so bad, 

because! do, too." 

The number of eoGraft procedures needed varies from 

person to person, but Smith is confident that she won't 

need a third procedure. Patients typically see 

irnpro,ement within ix month with full re ults within 

one year. 

- Lisa Schryver, Writer for Embassy Studio 

Original (Bottom of Page) Modified (Bottom of Page) 

18 
 

 

 

111 3 

C: 

0 

E 

"C 

C: 

0 

u 

~ 2 

-0 

I.. 

(1) 

.c 

E 

::, 

Z1 

0 

0 5 10 15 20 25 30 35 40 45 

Percentage Point Change 

Qualitative Observations 

Although we cannot determine the effect of any one particular disclosure modification on participants’ 

recognition of advertising, we can make some qualitative observations about the potential effectiveness and 

noticeability of the disclosure techniques explored in the study. Overall, without testing for statistical 

significance or controlling for confounding factors, we found that, across the different ads we tested, the 

observed effects of the disclosure modifications were all in the positive direction. Based on an examination of 

coded judgments of ad recognition, for 13 of the 17 ad conditions tested, the modifications increased the 

likelihood that a participant recognized an ad as an ad in the range of 10 to 45 percentage points. (See Figure 1). 

Figure 1: Distribution of Observed Increases in Ad Recognition 

Each bar represents the number of ad conditions for which there was an increase of a given percentage-point magnitude in participants 

who recognized the ad as an ad. For example, in three ad conditions there was an increase of 15 to 20 percentage points and in four 

ad conditions an increase of 25 to 30 percentage points. 

Because the study had only 48 participants, with only 24 viewing any given ad condition,P23F 

15 P we did not 

have enough statistical power to determine at the level of individual ad conditions whether the disclosure 

modifications had real effects on participants’ recognition of advertising unless the observed differences were 

extremely large. Nevertheless, the following qualitative observations may be helpful to companies in weighing 

which disclosure techniques to use or to researchers in considering potential areas for future study. 

19 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Distinctive Background Shading and a Sufficiently Noticeable Ad 

Label 

In both the Bing Desktop and Bing Mobile search conditions, the changes we made to the original Top 

ad disclosures consisted of enlarging the font size of the “Ads related to ...” labels and making the background 

shading darker. With the revised disclosures, most participants seemed to recognize the Top ads as ads, in both 

the desktop and the mobile versions. When the labels were less prominent and the shading lighter, recognition 

appeared lower. 

Text Color That Sufficiently Contrasts with Background Color 

In the Google Desktop Top ads condition, the primary modification was changing the ad label’s text 

color from white to black. With this modification, many more participants appeared to recognize these ads as 

ads. This may be due to the original label’s relative lack of contrast with the background on a desktop screen. 

In the Yahoo! Desktop condition, a native story was originally marked with a gray on white “Sponsored” label. 

A modified disclosure that increased the contrast of this ad label, together with other improvements, also 

seemed to produce a similar improvement in ad recognition, with most participants recognizing the native story 

as an ad. 

Consistent Labeling for Ads on the Same Page 

We replaced the “Sponsored” label for the Shop Box ads in both the Google Desktop and the Google 

Mobile conditions with the “Ad(s)” icon used to mark other ads on each of those pages. This may have 

contributed to increased ad recognition in both the Google Desktop and Mobile conditions, although it is 

unclear whether this improvement was attributable to the increased prominence of the labeling alone. Eye 

tracking data also revealed a change in where participants looked on the web pages with the modified 

disclosures; many participants spent less time on the Shop Box ads and more time on the Top ads and regular 

search results below those ads. A possible explanation for this change is that on the web pages with the 

modified disclosures, participants were quicker to realize that the Shop Box ads were ads. 

Segregating Ads from Other Content 

Among the various ads assessed in the search conditions, the Right-side ads had relatively high assessed 

ad recognition before the disclosures were modified. After we attempted to improve the disclosures, there was 

no change in Right-side ad recognition for the Google Desktop condition and very little for the Bing Desktop 

condition. A plausible explanation for this is that the Right-side ads, which are clearly segregated from the 

other search results, already tended to be more identifiable as ads so that disclosure modifications could not 

substantially improve ad recognition. 

Ad Labels Located in the Top Right-Hand Corner of a Web Page or an 

Ad Grouping 

In the study, we observed that disclosures located in the top right-hand corner of a web page or ad 

grouping were less likely to be noticed. On the original Google Desktop Shop Page, the disclosure “Merchant 

links are sponsored” appeared in the top right-hand corner of the page. Based on the eye tracking data, few 

participants who saw this page even registered fixations in the area of the disclosure. For the click-into page of 

the Yahoo! Desktop condition, we relocated the “Advertisement by Instaflex” label from the top-right of the 

web page to immediately above the ad’s headline. The eye tracking data showed that while most looked at the 

20 
 

 

 

 

 

 

 

 

 

 

relocated label, many did not look at the same label when it appeared in the top-right hand corner. In the Time 

Mobile main page condition, when the “Sponsored Content” label appeared in the upper right-hand corner of 

the content recommendation widget, many seemed not to recognize the ADT ad as an ad. When the label was 

moved over and centered above the ads, only a few did not recognize the ad as an ad. Although there were 

several other potential contributing factors for this improvement in ad recognition, i.e., an increase in the size 

and contrast of the label and changing “Sponsored Content” to “Paid Content,” the eye tracking data suggests 

that participants were less likely to look at the label in the upper right-hand corner of the widget. On the 

original Time Mobile main page, while only a handful of participants fixated on the “Sponsored Content” label 

in the upper right-hand corner of the widget, most participants attended to the “Around the Web” label located 

in the upper left-hand corner. 

Disclosures That Are Placed Close to an Ad’s Headline 

Even though the modified disclosure “Advertisement by Subaru” on the main page appeared to improve 

ad recognition in the Gear Patrol condition, a sizeable proportion of participants still did not recognize the 

Subaru ad as an ad. The modified Gear Patrol disclosure may not have performed as well because participants 

failed to see the label “Advertisement by Subaru,” which appeared above an image of the featured car rather 

than above the story headline. Heat maps indicate that normal viewing patterns were to attend to the story 

headlines and story blurbs in the middle of the page, not the images on the left. The eye tracking data also 

shows that seven of the eight participants who were coded as not recognizing the Subaru native ad as an ad did 

not appear to look at the “Advertisement by Subaru” label. This suggests that participants tended to read the 

headlines when browsing through the content stream on the web page and that disclosures placed outside of this 

viewing pattern were less likely to be noticed. 

When ad disclosures were located below the story headlines and blurbs, ad recognition also seemed 

lower among participants. Two of the conditions tested – Gear Patrol and Yahoo! – involved native ads in a 

content stream. The labels of these native ads appeared below the story headlines and story blurbs. In the Gear 

Patrol main page condition, when the disclosure “Advertisement by Subaru” was placed in the top left corner of 

the ad (and the ad’s background was changed to tan), more participants seemed to recognize the Subaru ad as an 

ad. As discussed above, ad recognition likely would have been greater had the disclosure been placed closer to 

the ad’s story headline rather than on the left above the picture of a Subaru. In the Yahoo! Desktop condition’s 

main page, even though initial ad recognition was relatively high, when the word “Sponsored” was relocated to 

above the story headline from below the story blurb, together with some other changes, ad recognition 

increased, with only a few participants not recognizing the native story with the modified disclosure as an ad. 

That so many people initially recognized the nature of the Yahoo! native ad may in part be a function of the 

subject and content of the ad. 

Clarity of Disclosure Meaning 

In the Chicago Tribune condition we changed the disclosure label “Brand Publishing” to “Paid Post.” 

This was the only condition for which the modified disclosure was not a variation of “Ad” or “Advertisement.” 

With the modified disclosure, ad recognition appeared to improve somewhat but was still relatively low. Based 

upon the eye tracking data, of those participants who did not recognize the Chicago Tribune article as an ad, 

most appeared to have looked at the modified ad label even if they failed to process it. A plausible explanation 

for this low ad recognition could be that the meaning of the “Paid Post” label was not sufficiently clear in 

meaning to overcome a strong overall impression that the Chicago Tribune, whose brand seemed to have strong 

recognition among participants, rather than an advertiser was the source of the content. 

21 
 

 

 

 

 

 

 

 

Strong Net Impression in Some Native Ad Contexts 

The Gear Patrol and Chicago Tribune conditions appeared to have fewer indicia separate and apart from 

the disclosure that they were advertisements. For both these native ads, assessed ad recognition was low to 

begin with, and seemed to improve very little with the improved disclosures. By contrast, once participants 

began reading the unmodified click-into page in the Time Mobile condition, which was no longer on the Time 

site, a substantial majority recognized the ADT ad as an ad even though it did not have any disclosure. In 

addition, initial ad recognition in the Yahoo! condition’s main page was relatively high and even greater with 

the improved disclosure. One plausible explanation for this is that stronger and clearer disclosures may be 

necessary when native ads more strongly resemble native content or appear in publications with a strong 

journalistic brand. 

Same Disclosure Techniques on a Desktop and Mobile Device 

We made identical changes to the Google Desktop and Mobile conditions with respect to the Top ads. 

We changed the text and background color of the “Ads” icons from white-on-yellow to black-on-orange. While 

this appeared to increase ad recognition for the Google Desktop condition, improving the contrast of the “ad” 

label for the original Google Mobile condition seemed to have little effect on ad recognition. Indeed, ad 

recognition for Top Ads in the original Google Mobile scenario was relatively high as compared to the Desktop 

scenario. It may be that the contrast was already easier to recognize on a mobile device so increased contrast in 

this scenario did not help with ad recognition. 

The Aggregate Effects of Improved Disclosures on 

Assessed Ad Recognition 

Although we could not systematically examine the effects of the disclosure modification on ad 

recognition for each separate ad condition, we have the necessary statistical power to estimate the combined 

effects of the disclosure modifications across all the different ads tested. We used a regression analysis to 

detect changes in the coded judgments of whether participants recognized an ad as an ad or if it was unclear to 

the coder. (See Sections I, II and III of Appendix F for a description of the regression analysis used to estimate 

the effects of the disclosure modifications on ad recognition.) This regression analysis controls for various 

complicating factors that could possibly influence the likelihood of a participant recognizing an ad as an ad, 

including the fact that some ads are more recognizable as ads than other ads, the tendency of an individual 

participant to correctly recognize an ad or to presume that all content is advertising, or the order in which a 

participant saw the ads. 

Applying this analysis, the aggregated effects of the disclosure modifications across the different ads we 

tested are substantial and statistically significant. Indeed, the disclosure modifications increased the likelihood 

that a participant would recognize an ad as an ad by 21 percentage points (from an average ad recognition of 

approximately 47% to an average ad recognition of approximately 68%). These effects are substantially similar 

across the subgroups of search ads and native ads, as well as between ads viewed on a desktop computer and a 

smartphone. (See Figure 2.) 

22 
 

 

 

 

Cl) 

c, 20 C: 

.s:: "' 

(.) 

... 

C: 

·o 

a. 0 Cl) 

Cl 

.l9 

C: 

Cl) 

c.J 

... 

~ -20 

-40-

f f f ·······························f ····················+- ········· 

t+ f 

t 

Overall Search Native Ad 

Ad Condition Subgroup 

Coded Response • Ad -+- No Ad • Unclear 

Figure 2: Estimated Effects of Disclosure Modifications on Ad Recognition 

Figure 2 depicts the 95% confidence intervals for the estimated effects of the disclosure modifications on ad recognition based on the 

statistical model, both for all ad conditions and for the search and native ad conditions separately. The red triangle, blue diamond, and 

green circle respectively depict the effect on the probability that: (1) an ad was recognized as an ad; (2) an ad was not recognized as 

an ad; and (3) it was unclear to the coder. 

We also note that, while improved recognition of the native ads with the modified disclosures derives 

entirely from a corresponding decrease in participants not recognizing them as ads, approximately 75% of the 

improved recognition of search ads comes from a corresponding decrease in ads scored as unclear to the coder. 

(See Figure 3) 

23 
 

 

 

 

 

-

C: 

"' C. 

·c:; 

60-

:e 40-

D.. "' 

~ I 

~ 20 _1 

u. 

0-

All Ads 

Original 

All Ads 

Modified 

Native Ads 

Original 

69 

Native Ads 

Modified 

Search Ads 

Original 

Coded Response • Ad . No Ad . Unclear 

Search Ads 

Modified 

Figure 3: Fraction of Ads Coded as Recognized as an Ad, Not Recognized as an 

Ad, and Unclear 

Figure 3 depicts the percentages of ads that were coded as: (1) recognized as an ad; (2) not recognized as an ad; and (3) unclear to 

the coder, both for all ad conditions and for search and native ad conditions separately. 

Ads coded as unclear could include some instances when a participant had correctly recognized an ad as 

an ad but the coder did not have enough information to make a judgment. Thus, with regard to search ads, it is 

less certain that the increased rate of ad recognition with the improved disclosures was due primarily to more 

participants recognizing ads that they otherwise would not have recognized as such, as opposed to the coder 

having less difficulty assessing participants’ recognition. 

The Relationship Between Improved Ad Recognition and 

Time Spent Looking at the Ads 

We analyzed eye tracking data collected during the study to test whether participants spent less time 

attending to ads with improved disclosures. (See Section IV of Appendix F for a description of the regression 

analysis used to analyze the eye tracking data.) In this analysis, we used eye tracking data for a subset of the ad 

conditions: only those ads that appeared among non-advertising content on a web page.P2 4F 

16 P We applied a 

regression model to estimate the effect of the disclosure modifications across the ad conditions, measuring the 

effect as a percent change between the unmodified and modified ads. We found that the disclosure 

modifications significantly reduced the amount of time that participants spent looking at the ads by 21%. While 

the observed effects were larger for native ads than for search ads (26% reduction versus 21%), observed effects 

were statistically different from zero only across the search ads, for which we had more statistical power to 

detect differences.P2 5F 

17 P 

24 
 

 

 

 

 

10 

Cl) -10 

en 

r:: 

~ 

.c: 

~ -30 

r:: 

Cl) 

~ Cl) 

a. -50 

-70 

-90 

I 

Overall Search Native Ad 

Ad Condition Subgroup 

• Without Ad Recognition Controls -+- With Ad Recognition Controls 

We further analyzed the eye tracking data by controlling for ad recognition in the regression analysis to 

evaluate whether the observed decrease in the amount of time participants fixated on the ads was due to a larger 

proportion of them recognizing the ads as ads. We found that, although participants looked substantially less at 

ads with the modified disclosures, this behavioral effect did not correlate with their recognition of the ads as 

ads. When controlling for the possible influence of ad recognition on the amount of time participants spent 

looking at the ads, we discovered that this behavioral effect did not become smaller but instead was slightly 

larger. Thus, improved ad recognition cannot be driving the reduction in time participants fixated on the ads 

with the modified disclosures. This supports the importance of testing consumers’ actual understanding of 

whether an ad is an ad and not assuming that behavioral measures such as visual attention are necessarily valid 

proxies for consumer understanding. (See Figure 4). 

Figure 4: Estimated Effects on Average Amount of Time Spent Looking at Ads 

With and Without Controlling for Improved Ad Recognition 

Figure 4 depicts the 95% confidence intervals for the estimated effects of the disclosure modifications on time spent looking at ads with 

and without controlling for ad recognition, for all ad conditions and separately for the search and native ad conditions. The red triangles 

depict the results not controlling for ad recognition, while the blue diamond depicts the results controlling for ad recognition. 

Methodology Limitations and Areas for Future Research 

The study was designed to be exploratory in nature and thus has a number of limitations. It does not 

isolate the effects of any one modification or combination of modifications on participants’ recognition of the 

ads as ads. It may be that some of the changes made to the ad disclosures had an outsized influence while 

others had little or no influence on whether participants recognized an ad as an ad. Future research would be 

necessary to isolate the effects of various factors such as disclosure location, language, text color and size, and 

the prominence of background shading or borders on consumers’ recognition of an ad. 

25 
 

 

 

 

 

The study also necessarily depends on subjective judgments of whether or not a participant recognized 

an ad given we did not explicitly ask that question to participants except with regard to the final ad tested. 

Observed differences in ad recognition could be due to more participants recognizing an ad with the improved 

disclosures as an ad or to more participants voicing this belief or giving other clear indicia easily interpreted as 

ad recognition. Another complicating factor in the search engine context is that some participants revealed 

during the study that they consider any search result that promotes a company or its products to be an “ad,” 

regardless of whether the result was paid for or returned organically in response to a query. Thus, accurately 

judging whether participants correctly distinguish paid search ads from regular search results can be more 

difficult without the benefit of direct questions. 

In this study we also did not test participants’ understanding of non-advertising content or traditional ads 

on the web pages. In future research this information could be useful in accounting for the possibility that some 

consumers may have a tendency to interpret all content as commercial in nature. 

It also is important to note that with the exception of just one condition, participants viewed the clickinto pages with modified disclosures after we had exposed them to an improved disclosure on the preceding 

web page. This aspect of the study design makes it more challenging, if not impossible, for us to assess 

accurately the effect of the modified disclosures on participants’ understanding of these click-into pages. To 

overcome this limitation, we would need to test the click-into pages in isolation without a preceding web page 

or test the click-into pages without an improved disclosure on the preceding page. 

Conclusion 

Ads should be identifiable as advertising. Using some of the common sense disclosure techniques 

recommended in FTC staff guidance, as discussed above, can greatly increase the likelihood that consumers 

will recognize an ad as an ad. Minor modifications, including changes to disclosure language, position, text 

size and color, and to other visual cues such as the borders around or background shadings of ads or ad 

groupings, can in combination substantially increase the likelihood that a consumer recognizes an ad as an ad 

and reduce the potential for consumers to be misled as to the commercial nature of paid search and native ads. 

26 
 

 

 

 

 

 

 

 

 

 

 

Endnotes 

1 The primary authors of this FTC staff report are Laura M. Sullivan and Michael F. Ostheimer, senior staff 

attorneys in the FTC’s Division of Advertising Practices. In designing and conducting the study, the FTC staff 

consulted Jeff Johnson, the President and Principal Consultant of the usability firm UI Wizards and Assistant 

Professor in the Department of Computer Science at the University of San Francisco; Manoj Hastak, Professor 

of Marketing in the Kogod School of Business at American University; and Bernard J. Jansen, Professor in the 

Department of Computer Science at the Pennsylvania State University. UserWorks, a user experience firm in 

the Washington, D.C. area, administered the study, including collecting and processing eye tracking data. 

Devesh R. Raval, an economist in the FTC’s Bureau of Economics, provided valuable assistance by conducting 

regression analyses of the study results. Caitlyn Brady, an FTC paralegal, provided substantial assistance with 

the study’s implementation and data validation. Teresa J. Peeler and Christopher Hundycz of the FTC’s 

Division of Consumer and Business Education also assisted with creating the stimuli tested. 

2 FTC Enforcement Policy Statement on Deceptively Formatted Advertisements, 81 Fed. Reg. 22596 (Apr. 18, 

2016). 

3 Press Release, Federal Trade Commission, FTC Consumer Protection Staff Updates Agency’s Guidance to 

Search Engine Industry on the Need to Distinguish Between Advertisements and Search Results (June 25, 

2013), available at www.ftc.gov/news-events/press-releases/2013/06/ftc-consumer-protection-staff-updatesagencys-guidance-search; see also Exemplar letter from Mary K. Engle, Associate Director, Division of 

Advertising Practices, Federal Trade Commission to General Purpose Search Engines (June 24, 2013), 

available at www.ftc.gov/sites/default/files/attachments/press-releases/ftc-consumer-protection-staff-updatesagencys-guidance-search-engine-industryon-need-distinguish/130625searchenginegeneralletter.pdf. 

4 Native Advertising: A Guide for Businesses (business guide document offering guidance on when disclosures 

can prevent consumer deception and how to make effective disclosures in the context of native advertising), 

available at https://www.ftc.gov/tips-advice/business-center/guidance/native-advertising-guide-businesses. 

5 .com Disclosures: How to Make Effective Disclosures in Digital Advertising (2013), available at 

https://www.ftc.gov/tips-advice/business-center/guidance/com-disclosures-how-make-effective-disclosuresdigital. 

6 See, e.g., Bartosz W. Wojdynski, The Deceptiveness of Sponsored News Articles: How Readers Recognize 

and Perceive Native Advertising (2017) (online survey tests the effects disclosure position, language, and other 

characteristics on respondents’ recognition of a news article as advertising), available at 

http://journals.sagepub.com/doi/abs/10.1177/0002764216660140; David A. Hyman, David Franklyn, Calla Lee 

& Mohammed Rahmati, Going Native: Can Consumers Recognize Native Advertising? Does it Matter? (2016) 

(online survey tests respondents’ recognition of different images as ads, including original native ads, a 

modified native ad, traditional ads, regular editorial content, and video commercials), available at 

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2816655; Chris Jay Hoofnagle & Eduard Meleshensky, 

Native Advertising and Endorsement Schema, Source-Based Misleadingness and Omission of Material Facts, 

Technology Sciences (2015) (online survey tests respondents’ understanding of a sponsored blog post, 

including the source of the post); see also Bartosz W. Wojdynski & Nathaniel J. Evans, Going Native: Effects 

of Disclosure Position and Language on the Recognition and Evaluation of Online Native Advertising, 45 J. 

Advertising 157 (2016) (study consisting of an online survey and a smaller eye tracking experiment tests effects 

of disclosure language and position on participants’ recognition of a native ad as an ad and on their likelihood 

of looking at the disclosure). 

27 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

7 See, e.g., Navdeep S. Sahni & Harikesh S. Nair, Native Advertising, Sponsorship Disclosure and Consumer 

Deception: Evidence from Mobile Search-Ad Experiments (2017) (study measures changes in clicking and 

purchasing behavior between participants who use one of three versions of a restaurant search site with different 

ad disclosures of varying saliency), available 

at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2737035&download=yes. 

8 In scoring participants’ recognition of the last ad presented to them, coders also considered participants’ 

answers when asked if they saw any ads on the web pages. 

9 This ever-evolving landscape is evident in a recent review of publisher sites by the company MediaRadar, 

which reported that the percentage of publisher sites that were not clearly labeling native ads dropped to 37% 

from 70% as found in a survey conducted the previous year. Third of Publishers Still Falling Foul of FTC 

Native Ad Guidelines, Mar. 6, 2017, available at https://resources.mediaradar.com/newsroom/third-ofpublishers-still-falling-foul-of-ftc-native-ad-guidelines. Search engine sites also have undergone significant 

changes. See, e.g., Ginny Marvin, UPDATED: A visual history of Google ad labeling in search results, Mar. 3, 

2017, available at http://searchengineland.com/search-ad-labeling-history-google-bing-254332. 

10 Participants were not aware of either the FTC’s role in the study or the study’s purpose, and were paid $125 

for their time. 

11 Between June 2, 2014 and September 4, 2014, we captured and preserved the HTML code of the actual web 

pages tested in the lab. To create modified versions of the preserved web pages, we revised relevant parts of the 

HTML and CSS code in copies of these preserved web pages. The pages rendered during a session in the same 

way as if a participant were normally browsing the Internet. 

12 Coder agreement between the expert consultant and the FTC paralegal staff was 79.2%. We did not, 

however, resolve differences by the usual method of discussion between the coders. Instead, we used the expert 

consultant’s coding for the analyses discussed in this report. Based on independent testing, we found very little 

substantive differences in the results whether we used the expert consultant or FTC paralegal staff’s coded 

judgments. (See Section III of Appendix F for a description of the regression analysis that we applied to 

compare results between using the expert consultant and the FTC paralegal staff’s coding.) 

13 Heat maps are color-coded visualizations of the frequency with which all of the participants seeing a 

condition fixated on different areas of a page. 

14 Gaze plots depict where and in what order a particular participant fixated on a page and the duration of each 

fixation. 

15 Only 12 participants, an even smaller number, saw four of the ad conditions tested in the study. With regard 

to two search scenarios – Google Desktop and Google Mobile – participants viewed the click-into page in the 

first scenario presented to them during a session. 

16 We expected only to observe differences in the amount of time participants looked at the ads with the 

modified disclosures when the ads appeared among non-advertising content on a web page, such as, for 

example, in a content stream or within unpaid search results. Therefore, we excluded eye tracking data for 

those ads appearing on a click-into page from the analysis. 

17 Of the 22 ad conditions included in the eye tracking analysis, only six were native ads. 

28 
 

 

 

 

 

 

 

 

 

 

 

Appendix A: Screening Procedure 

We used a series of screening questions to ensure that participants had experience with search engines 

and the Internet (See Attachments 1 and 2 for screening protocols).P0F 

1 

P To participate in the study, participants 

had to: (1) be age 18 or older; (2) be fluent in English; (3) use both a desktop computer and smartphone at least 

once per week; (4) use search engines to look for information about products and services; and (5) browse 

websites to look for information about products and services. We excluded participants if they or anyone in 

their household worked for a marketing research firm, magazine or newspaper, or search engine or worked for a 

company or store that makes or sells: (1) home security products; (2) hair care products; (3) drugs or dietary 

supplements; (4) advertising; (5) computers; or (6) smartphones. We also excluded participants if they had 

participated in a research study or focus group in the past six months or advertised on their own blog or website. 

Finally, to ensure that the eye tracking equipment would work correctly, participants were excluded if they: (1) 

had certain eye-conditions; (2) wore bifocals, trifocals, or progressive lenses; or (3) wore contact lenses and 

reading glasses simultaneously when using a computer or smartphone.P 1F 

2 

The screening process established age, gender, ethnicity, and frequency of Internet use to ensure 

reasonable diversity in the participant sample. Of the 48 participants included in the study, 26 were female and 

22 were male. Five participants were between 18 and 24 years old; nine were between 25 and 34 years old; 

eight were between 35 and 44 years old; 10 were between 45 and 54 years old; 12 were between 55 and 64 

years old; and four were at least 65 years old. Thirty participants reported their race as White; 13 as Black; two 

as Hispanic; two as Asian; and one as Native American/Pacific Islander. Participants were better educated than 

the United States population, with more than 62% having a college degree or higher. 

1 We recruited for the study on a rolling basis. The screener questionnaire included a series of questions to 

ensure that we had a mix of participants based on frequency of Internet and search engine use. Early in the 

recruitment process, we had to adjust our criteria for occasional usage because it was unreasonably difficult to 

find candidates who used the Internet less than once a day. 

2 Three participants were excluded from the study because of a failure to correctly calibrate the eye tracking 

equipment during their sessions. We also excluded another five participants after we subsequently discovered 

that they had viewed an incorrect web page due to a technical problem. In each case, new participants were 

recruited to replace the excluded participants. 

A-1 
 

 

 

 

 

 

 

 

 

 

 

 

D D 

Attachment 1 to Appendix A: (First) Screener Questionnaire 

Appt. Date & Time: @ 

Directions Sent: Recruiter: Reminder Call: 

Participant Name: _________________ Male Female (Recruit a mix) 

Daytime Phone # _____________Evening Phone # __________________ 

Participant Screener for Federal Trade Commission Search & Sponsored Content 

Usability Test 

Recruitment Objectives 

Data Collection: November 17 through December 3, 2014 

Participants: 48 web and smartphone users 

Location: In-person at UserWorks’ facilities in Silver Spring 

Length: 90 minute sessions 

Incentives: $125 

Recruitment Approach 

UserWorks will recruit participants for in-person sessions from its Washington, DC area database of volunteers. The 

recruitment follows this procedure: 

• People who are registered in the database will be sent a broadcast email announcement, which includes a 

general description of the project and a few preliminary screening questions (see Preliminary Screening 

Questions below). 

• Interested candidates who think they may qualify will respond by email with answers to the preliminary 

screening questions. 

• Based on these responses, our recruiter(s) will select a subset of candidate participants to call. 

• These selected candidates will be administered a more detailed screening questionnaire (see Screener 

Questions below) by telephone. Qualified candidates will be invited to participate in the study and 

scheduled for a specific time slot. 

Email announcement to our participant database will say: 

Email Subject: Participants Needed for a Study about Search and Web Content 

UserWorks, Inc., a user experience design and usability research firm, is recruiting participants for a study to evaluate the 

usability of search and web content. 

The study will take about 90 minutes to complete and will be conducted in-person at UserWorks’ facility in Silver Spring, 

th rd MD between November 17P P through December 3P P, 2014. You will be paid for your participation. 

Who: 

Web and smartphone users 

What: 

An in-person session lasting about 90 minutes 

When: 

Monday, November 17, 2014 through Wednesday, December 3rd, 2014 

Where: 

A-2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

Silver Spring, MD 

If you are interested, please click this link https://www.surveymonkey.com/s/PSGZJXL 

to answer a few preliminary questions. We will then contact selected candidates by phone to go through an additional set 

of questions to determine if you qualify. 

After reviewing the responses from the participants, UserWorks’ recruiter(s) will then call the selected candidates and 

say/ask the following: 

• Hello, my name is _____, and I’m calling from UserWorks. You recently replied to our announcement about an 

upcoming usability study. 

• We are doing a project to learn users’ opinions about the usability of web search and online content. We are trying 

to find out if the information is easy to use and understand. If you are interested in participating, I would like to 

ask you a few questions to see if you qualify. 

• If you qualify, you will be offered [$125] for your participation in the study. 

• The study will involve an interview session lasting about 90 minutes and will take place between Monday, 

th nd November 17P P through Wednesday, December 3P P, 2014 at our office in Silver Spring. 

• May I ask you a few questions? This will take about 5 minutes. All of the information you provide here will be 

treated as confidential. It will not be shared with anyone other than the UserWorks people working on this project 

and our client. The information passed on to the client will be only in aggregate form and will not include 

anything that identifies you as an individual. 

1. When is the last time you participated in a research study or focus group? 

Within the last 6 months [Terminate]

 More than 6 months ago or never 

2. Have you or has anyone in your household ever worked for any of the following types of companies? 

Market research company

 Search engine company, e.g. Yahoo, MSN, AOL, or Google 

Magazine or newspaper, e.g. Time, Newsweek, New York Times, or Huffington Post 

Security products or services, e.g. home security systems, camera surveillance, window 

protection, or car alarms 

Hair care products company, such as color treatments, hair restoration, or laser hair removal

 Drug or dietary supplement company

 Advertising agency or advertising department of a company 

 Car company, e.g. General Motors, Ford, Subaru, or Mercedes Benz

 Computer or mobile device manufacturer 

[Terminate if yes to any of the above answers] 

3a. Do you have a personal website, personal blog, or website/blog for your personal business? 

Yes 

No 

If yes, continue to Q3b. Otherwise skip to Q4. 

3b. Do you engage in advertising on your website/blog? 

Yes [Terminate] 

A-3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

No 

One thing of interest in this study is participants’ eye movements as they view online content. Thus we need to ask several 

questions about your vision. 

4. Do you have any of the following eye conditions? 

Nystagmus (involuntary eye movements, sometimes called “dancing eyes”) [Terminate] 

Strabismus (sometimes called cross-eyed or wall eyed) [Terminate] 

5. Do you wear any of the following corrective lenses when you use a computer or smartphone? 

Glasses 

Bifocals [Terminate] 

Trifocals [Terminate] 

Progressive Lenses (where the magnification changes gradually from top to bottom of the lenses)[Terminate] 

Reading glasses [probably OK, but check as the study progresses] 

Contact Lenses 

Do you use reading glasses in addition to your contacts when working with a computer or 

smartphone? 

Yes [Terminate] 

No 

6. Which of the following do you use to access the Internet, at least occasionally? (check all that apply) 

Recruit only participants who use both desktop/laptop AND smartphone 

Desktop or Laptop computer

 Tablet computer [Terminate if only method to access Internet]

 Smartphone - Brand of smartphone: ____________________________________ 

Other – Please describe: _____________________________________________ 

7. Which of the following best describes how often you access the Web for purposes other than email, social media like 

Facebook, or online games? (Record one response) 

More than three times per day 

Three times per day

 Two times per day 

One time per day 

A few times per week

 About once per week 

Less than once per week [Terminate] 

8. Which of the following activities, if any, do you do when you are online? 

Use search engines to look for information about products and services 

Browse websites to look for information about products and services 

Read news or other articles in online magazines, newspapers, newsfeeds, or social media 

Write a blog, read blogs, or comment on blogs 

Watch videos at a video site, (e.g., YouTube, Vimeo, Netflix) 

Play online games 

Read or post in social networking sites (e.g., Facebook, Meetup, Tumblr, Twitter) 

Read or send e-mail 

A-4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D D D 

D D D 

D D D 

D D D 

D D D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

Other 

All participants must check the first 3 boxes, even if it was a few months ago, otherwise [Terminate]. 

9. Which search engine do you use most on the devices that you use to access the Internet on your 

[Desktop/Laptop, Tablet, Smartphone]? (Record only one mention in each column) 

Desktop Tablet Smartphone 

AOL 

Bing 

Google

 Yahoo! 

Other Which one?____________ 

10. How often do you use search engines like Google, Yahoo!, or Bing to look for information about products 

and services? This might include things like using a search engine to look up a restaurant or local shop, 

identify a store or website where you can buy a product, or to find information about a product itself. Please 

think about all your searches for products or services, even if you are not planning to make a purchase of 

that product or service based on that search. How often do you use search engines? 

Frequent Users: 

More than once per day 

About once per day 

A few times per week 

Occasional Users: 

About once per week 

Once every few weeks 

Monthly 

Once every few months 

Never [Terminate] 

User Category Summary: 

Frequent Internet Users 

• Uses Web at least more than once per day 

• Performs several online activities 

• Uses search engine at least a few times per week 

Occasional Internet Users 

• Uses Web at least once per week, but no more than once per day 

• Performs a few online activities 

• Uses search engine at least once per week 

11. Gender – record without asking: 

Male 

Female 

12. What is your age? 

Recruit a mix for each user category

 18-24

 25-34

 35-44

 45-54 

A-5 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

 55-64

 65 or older 

13. What is the highest level of education you have completed? 

Recruit half college graduate or more for each user category 

Some high school 

High school graduate 

Some college 

College graduate 

Post-graduate 

14. What is your race or ethnicity? Please select all that apply. If “Other,” please describe. 

Recruit a mix for each user category 

Caucasian/White 

African-American/Black [Recruit 1-2 in each user category] 

Hispanic or Latino [Recruit 1-2 in each user category] 

Native American or Alaska Native 

Asian 

Native Hawaiian or Pacific Islander 

Other 

Declined to state 

15. The study session will be video recorded. Only the team working on this project will use the recording and your name 

will not be associated with it or other data in any way. You will be asked to sign an informed consent form. Would 

you be willing to be video recorded? 

Yes 

No [Terminate] 

16. How would you like the directions to UserWorks sent to you? 

Email Address 

Fax Number 

Over Phone 

You are the only person scheduled to participate at [time & date of session]. If you are not able to keep this appointment, 

please call me at 301.431.0500 [or 888.558.7872] as soon as possible. 

During the session we will be using an eye tracking device to measure your eye position and movements while viewing 

the computer screen/smartphone. A camera will view a close-up of your eyes and determine from the reflected light the 

direction that you are looking. A computer will record an indication of where you are looking from moment to moment on 

an image of the screen that you are seeing. There is nothing distracting or intrusive about the process. Nothing touches 

you eye or skin. There is nothing harmful about the process. Women should not wear mascara or false eyelashes because 

these may interfere with the eye tracker measures. 

If you wear glasses to use a computer and/or smartphone please bring those with you to the session. 

A-6
 

 

 

 

 

 

 

 

 

 

 

 

 

 

D D 

Attachment 2 to Appendix A: (Second) Screener Questionnaire 

Appt. Date & Time: @ 

Directions Sent: Recruiter: Reminder Call: 

Participant Name: _________________ Male Female (Recruit a mix) 

Daytime Phone # _____________Evening Phone # __________________ 

Participant Screener for Federal Trade Commission Search & Sponsored Content 

Usability Test 

Recruitment Objectives 

Data Collection: November 17 through December 12, 2014 

Participants: 48 web and smartphone users 

Location: In-person at UserWorks’ facilities in Silver Spring 

Length: 90 minute sessions 

Incentives: $125 

Recruitment Approach 

UserWorks will recruit participants for in-person sessions from its Washington, DC area database of volunteers. The 

recruitment follows this procedure: 

• People who are registered in the database will be sent a broadcast email announcement, which includes a 

general description of the project and a few preliminary screening questions (see Preliminary Screening 

Questions below). 

• Interested candidates who think they may qualify will respond by email with answers to the preliminary 

screening questions. 

• Based on these responses, our recruiter(s) will select a subset of candidate participants to call. 

• These selected candidates will be administered a more detailed screening questionnaire (see Screener 

Questions below) by telephone. Qualified candidates will be invited to participate in the study and 

scheduled for a specific time slot. 

Email announcement to our participant database will say: 

Email Subject: Participants Needed for a Study about Search and Web Content 

UserWorks, Inc., a user experience design and usability research firm, is recruiting participants for a study to evaluate the 

usability of search and web content. 

The study will take about 90 minutes to complete and will be conducted in-person at UserWorks’ facility in Silver Spring, 

th rd MD between November 17P P through December 3P P, 2014. You will be paid for your participation. 

Who: 

Web and smartphone users 

What: 

An in-person session lasting about 90 minutes 

When: 

Monday, November 17, 2014 through Wednesday, December 3rd, 2014 

Where: 

Silver Spring, MD 

If you are interested, please click this link https://www.surveymonkey.com/s/PSGZJXL 

A-7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

to answer a few preliminary questions. We will then contact selected candidates by phone to go through an additional set 

of questions to determine if you qualify. 

After reviewing the responses from the participants, UserWorks’ recruiter(s) will then call the selected candidates and 

say/ask the following: 

• Hello, my name is _____, and I’m calling from UserWorks. You recently replied to our announcement about an 

upcoming usability study. 

• We are doing a project to learn users’ opinions about the usability of web search and online content. We are trying 

to find out if the information is easy to use and understand. If you are interested in participating, I would like to 

ask you a few questions to see if you qualify. 

• If you qualify, you will be offered [$125] for your participation in the study. 

• The study will involve an interview session lasting about 90 minutes and will take place between Monday, 

th th November 17P P through Wednesday, December 12P P, 2014 at our office in Silver Spring. 

• May I ask you a few questions? This will take about 5 minutes. All of the information you provide here will be 

treated as confidential. It will not be shared with anyone other than the UserWorks people working on this project 

and our client. The information passed on to the client will be only in aggregate form and will not include 

anything that identifies you as an individual. 

1. When is the last time you participated in a research study or focus group? 

Within the last 6 months [Terminate]

 More than 6 months ago or never 

2. Have you or has anyone in your household ever worked for any of the following types of companies? 

Market research company

 Search engine company, e.g. Yahoo, MSN, AOL, or Google 

Magazine or newspaper, e.g. Time, Newsweek, New York Times, or Huffington Post 

Security products or services, e.g. home security systems, camera surveillance, window 

protection, or car alarms 

Hair care products company, such as color treatments, hair restoration, or laser hair removal

 Drug or dietary supplement company

 Advertising agency or advertising department of a company 

 Car company, e.g. General Motors, Ford, Subaru, or Mercedes Benz

 Computer or mobile device manufacturer 

[Terminate if yes to any of the above answers] 

3a. Do you have a personal website, personal blog, or website/blog for your personal business? 

Yes 

No 

If yes, continue to Q3b. Otherwise skip to Q4. 

3b. Do you engage in advertising on your website/blog? 

Yes [Terminate] 

No 

A-8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

One thing of interest in this study is participants’ eye movements as they view online content. Thus we need to ask several 

questions about your vision. 

4. Do you have any of the following eye conditions? 

Nystagmus (involuntary eye movements, sometimes called “dancing eyes”) [Terminate] 

Strabismus (sometimes called cross-eyed or wall eyed) [Terminate] 

5. Do you wear any of the following corrective lenses when you use a computer or smartphone? 

Glasses 

Bifocals [Terminate] 

Trifocals [Terminate]

 Progressive Lenses (where the magnification changes gradually from top to bottom of the lenses)[Terminate] 

Reading glasses [probably OK, but check as the study progresses] 

Contact Lenses 

Do you use reading glasses in addition to your contacts when working with a computer or 

smartphone? 

Yes [Terminate] 

No 

6. Which of the following do you use to access the Internet, at least occasionally? (check all that apply) 

Recruit only participants who use both desktop/laptop AND smartphone 

Desktop or Laptop computer

 Tablet computer [Terminate if only method to access Internet]

 Smartphone - Brand of smartphone: ____________________________________ 

Other – Please describe: _____________________________________________ 

7. Which of the following best describes how often on average you access the Web for purposes other than email, social 

media like Facebook, or online games? (Record one response) 

Frequent users: 

More than three times per day 

Three times per day

 Two times per day 

Occasional users: 

One time per day 

A few times per week

 About once per week 

Less than once per week [Terminate] 

8. Which of the following activities, if any, do you do with some regularity when you are online? 

Use search engines to look for information about products and services 

Browse websites to look for information about products and services 

Read news or other articles in online magazines, newspapers, newsfeeds, or social media 

Purchase items online 

Use online banking or other financial services 

Write a blog, read blogs, or comment on blogs 

Watch videos at a video site, (e.g., YouTube, Vimeo, Netflix) 

A-9 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D D D 

D D D 

D D D 

D D D 

D D D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

Play online games 

Read or post in social networking sites (e.g., Facebook, Meetup, Tumblr, Twitter) 

Read or send e-mail 

Other 

All participants must check the first 2 boxes, even if it was a few months ago, otherwise [Terminate]. 

9. Which search engine do you use most on the devices that you use to access the Internet on your 

[Desktop/Laptop, Tablet, Smartphone]? (Record only one mention in each column) 

Desktop Tablet Smartphone 

AOL 

Bing 

Google

 Yahoo!

 Other Which one?____________ 

10. How often do you use search engines like Google, Yahoo!, or Bing to look for information about products 

and services? This might include things like using a search engine to look up a restaurant or local shop, 

identify a store or website where you can buy a product, or to find information about a product itself. Please 

think about all your searches for products or services, even if you are not planning to make a purchase of 

that product or service based on that search. How often on average do you use search engines? 

Frequent Users: 

More than once per day 

About once per day 

A few times per week 

Occasional Users: 

About once per week 

Once every few weeks 

Monthly 

Once every few months 

Never [Terminate] 

User Category Summary: 

Frequent Internet Users are those who meet ALL of the following criteria: 

• Uses Web more than once per day on average 

• Performs more than two of the first five online activities listed above 

• Uses search engines at least a few times per week 

Occasional Internet Users are those who fail to meet one or more of the above Frequent User criteria but who 

otherwise qualify for the study 

11. Gender – record without asking: 

Male 

Female 

12. What is your age? 

Recruit a mix for each user category (i.e., for Frequent and Occasional Users)

 18-24

 25-34 

A-10 
 

 

 

 

 

 

 

 

 

 

 

 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

D 

 35-44

 45-54

 55-64

 65 or older 

13. What is the highest level of education you have completed? 

Recruit half college graduate or more for each user category 

Some high school 

High school graduate 

Some college 

College graduate 

Post-graduate 

14. What is your race or ethnicity? Please select all that apply. If “Other,” please describe. 

Recruit a mix for each user category 

Caucasian/White 

African-American/Black [Recruit at least 1-2 in each user category] 

Hispanic or Latino [Recruit at least 1-2 in each user category] 

Native American or Alaska Native 

Asian 

Native Hawaiian or Pacific Islander 

Other 

Declined to state 

15. The study session will be video recorded. Only the team working on this project will use the recording and your name 

will not be associated with it or other data in any way. You will be asked to sign an informed consent form. Would 

you be willing to be video recorded? 

A-11
 

 

 

 

Appendix B: Experimental Design Table 

Participants were randomly assigned to one of two groups: (1) “Group A,” who first viewed four mobile 

scenarios followed by four desktop scenarios; and (2) “Group B,” who first viewed four desktop scenarios 

followed by four mobile scenarios. We grouped the scenarios in this way rather than intermixing mobile and 

desktop scenarios because the eye tracking equipment had to be positioned and calibrated differently for a 

computer or smartphone screen. 

Abbreviation Key for Study Scenarios 

Scenario Name 

Google Desktop Original 

Ad Type 

Search 

Scenario Code 

S-GDO 

Google Desktop Modified Search S-GDM 

Google Mobile Original Search S-GMO 

Google Mobile Modified Search S-GMM 

Bing Desktop Original Search S-BDO 

Bing Desktop Modified Search S-BDM 

Bing Mobile Original Search S-BMO 

Bing Mobile Modified Search S-BMM 

Gear Patrol Desktop Original Native N-GPDO 

Gear Patrol Desktop Modified Native N-GPDM 

Time Mobile Original Native N-TMO 

Time Mobile Modified Native N-TMM 

Yahoo Desktop Original Native N-YDO 

Yahoo Desktop Modified Native N-YDM 

Chicago Tribune Mobile

Original 

Native N-CTMO 

Chicago Tribune Mobile

Modified 

Native N-CTMM 

Experimental Design Table 

Group A 

Group A 

A-1 

Desktop Scenarios 

S-GDO, S-BDM, N-GPDM, N-YDO 

Mobile Scenarios 

S-GMM, S-BMO, N-TMO, N-CTMM 

A-2 S-GDO, S-BDM, N-YDO, N-GPDM S-GMM, S-BMO, N-CTMM, N-TMO 

A-3 S-GDO, N-GPDM, S-BDM, N-YDO S-GMM, N-TMO, S-BMO, N-CTMM 

A-4 S-GDO, N-GPDM, N-YDO, S-BDM S-GMM, N-TMO, N-CTMM, S-BMO 

A-5 S-GDO, N-YDO, S-BDM, N-GPDM S-GMM, N-CTMM, S-BMO, N-TMO 

A-6 S-GDO, N-YDO, N-GPDM, S-BDM S-GMM, N-CTMM, N-TMO, S-BMO 

A-7 S-BDM, S-GDO, N-YDO, N-GPDM S-BMO, S-GMM, N-CTMM, N-TMO 

A-8 S-BDM, S-GDO, N-GPDM, N-YDO S-BMO, S-GMM, N-TMO, N-CTMM 

A-9 S-BDM, N-GPDM, N-YDO, S-GDO S-BMO, N-TMO, N-CTMM, S-GMM 

A-10 S-BDM, N-GPDM, S-GDO, N-YDO S-BMO, N-TMO, S-GMM, N-CTMM 

A-11 S-BDM, N-YDO, N-GPDM, S-GDO S-BMO, N-CTMM, N-TMO, S-GMM 

A-12 

Group A 

A-13 

S-BDM, N-YDO, S-GDO, N-GPDM 

Mobile Scenarios 

N-TMO, S-GMM, S-BMO, N-CTMM 

S-BMO, N-CTMM, S-GMM, N-TMO 

Desktop Scenarios 

N-GPDM, S-GDO, S-BDM, N-YDO 

A-14 N-TMO, S-GMM, N-CTMM, S-BMO N-GPDM, S-GDO, N-YDO, S-BDM 

A-15 N-TMO, S-BMO, S-GMM, N-CTMM N-GPDM, S-BDM, S-GDO, N-YDO 

A-16 N-TMO, S-BMO, N-CTMM, S-GMM N-GPDM, S-BDM, N-YDO, S-GDO 

B-1 
A-17 N-TMO, N-CTMM, S-GMM, S-BMO N-GPDM, N-YDO, S-GDO, S-BDM 

A-18 N-TMO, N-CTMM, S-BMO, S-GMM N-GPDM, N-YDO, S-BDM, S-GDO 

A-19 N-CTMM, S-GMM, N-TMO, S-BMO N-YDO, S-GDO, N-GPDM, S-BDM 

A-20 N-CTMM, S-GMM, S-BMO, N-TMO N-YDO, S-GDO, S-BDM, N-GPDM 

A-21 N-CTMM, S-BMO, N-TMO, S-GMM N-YDO, S-BDM, N-GPDM, S-GDO 

A-22 N-CTMM, S-BMO, S-GMM, N-TMO N-YDO, S-BDM, S-GDO, N-GPDM 

A-23 N-CTMM, N-TMO, S-BMO, S-GMM N-YDO, N-GPDM, S-BDM, S-GDO 

A-24 N-CTMM, N-TMO, S-GMM, S-BMO N-YDO, N-GPDM, S-GDO, S-BDM 

Group B 

Group B Desktop Scenarios Mobile Scenarios 

B-1 S-GDM, S-BDO, N-GPDO, N-YDM S-GMO, S-BMM, N-TMM, N-CTMO 

B-2 S-GDM, S-BDO, N-YDM, N-GPDO S-GMO, S-BMM, N-CTMO, N-TMM 

B-3 S-GDM, N-GPDO, S-BDO, N-YDM S-GMO, N-TMM, S-BMM, N-CTMO 

B-4 S-GDM, N-GPDO, N-YDM, S-BDO S-GMO, N-TMM, N-CTMO, S-BMM 

B-5 S-GDM, N-YDM, S-BDO, N-GPDO S-GMO, N-CTMO, S-BMM, N-TMM 

B-6 S-GDM, N-YDM, N-GPDO, S-BDO S-GMO, N-CTMO, N-TMM, S-BMM 

B-7 S-BDO, S-GDM, N-YDM, N-GPDO S-BMM, S-GMO, N-CTMO, N-TMM 

B-8 S-BDO, S-GDM, N-GPDO, N-YDM S-BMM, S-GMO, N-TMM, N-CTMO 

B-9 S-BDO, N-GPDO, N-YDM, S-GDM S-BMM, N-TMM, N-CTMO, S-GMO 

B-10 S-BDO, N-GPDO, S-GDM, N-YDM S-BMM, N-TMM, S-GMO, N-CTMO 

B-11 S-BDO, N-YDM, N-GPDO, S-GDM S-BMM, N-CTMO, N-TMM, S-GMO 

B-12 S-BDO, N-YDM, S-GDM, N-GPDO S-BMM, N-CTMO, S-GMO, N-TMM 

Group B Mobile Scenarios Desktop Scenarios 

B-13 N-TMM, S-GMO, S-BMM, N-CTMO N-GPDO, S-GDM, S-BDO, N-YDM 

B-14 N-TMM, S-GMO, N-CTMO, S-BMM N-GPDO, S-GDM, N-YDM, S-BDO 

B-15 N-TMM, S-BMM, S-GMO, N-CTMO N-GPDO, S-BDO, S-GDM, N-YDM 

B-16 N-TMM, S-BMM, N-CTMO, S-GMO N-GPDO, S-BDO, N-YDM, S-GDM 

B-17 N-TMM, N-CTMO, S-GMO, S-BMM N-GPDO, N-YDM, S-GDM, S-BDO 

B-18 N-TMM, N-CTMO, S-BMM, S-GMO N-GPDO, N-YDM, S-BDO, S-GDM 

B-19 N-CTMO, S-GMO, N-TMM, S-BMM N-YDM, S-GDM, N-GPDO, S-BDO 

B-20 N-CTMO, S-GMO, S-BMM, N-TMM N-YDM, S-GDM, S-BDO, N-GPDO 

B-21 N-CTMO, S-BMM, N-TMM, S-GMO N-YDM, S-BDO, N-GPDO, S-GDM 

B-22 N-CTMO, S-BMM, S-GMO, N-TMM N-YDM, S-BDO, S-GDM, N-GPDO 

B-23 N-CTMO, N-TMM, S-BMM, S-GMO N-YDM, N-GPDO, S-BDO, S-GDM 

B-24 N-CTMO, N-TMM, S-GMO, S-BMM N-YDM, N-GPDO, S-GDM, S-BDO 

B-2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix C: Main Study Procedure 

The duration of each participant session was approximately 90 minutes. Prior to presenting the first set 

of web pages, a moderator reviewed logistics with the participants, directed them to complete a pre-test 

interview form, and instructed them to think aloud when interacting with web pages. The moderator next 

calibrated the eye tracking equipment for the first four scenarios to be presented. At the mid-way point of each 

session, the moderator re-positioned and re-calibrated the equipment for the next four scenarios. 

The study presented each set of web pages to participants in the context of a real-life scenario, for 

example, using a search engine to find information about a vitamin that your doctor recommended. (See 

Attachment 1 for a copy of the main study questionnaire, including the real-life scenarios.) Participants read 

each scenario aloud and then pressed a button to display the main page for the scenario. Participants looked at 

this page uninterrupted for several seconds, scrolling around as desired.P2F 

3 

P During this time, the eye tracking 

equipment recorded participants’ eye movements on the screen. Once participants paused or were about to 

click into another page, the moderator directed their attention back to the main page and began to ask probing 

questions. The moderator then directed participants’ attention to specific items on the main page and asked for 

comments. Some of these items were the specific ads examined by the study, while other items were regular 

content that masked the study’s purpose. The moderator also asked participants to click both ads of interest and 

decoy items. Their spontaneous comments and eye movements upon clicking into the new pages were 

recorded.P3F 

4 

P Once participants paused, the moderator asked probing questions about these pages. If a scenario 

was the final one the moderator presented, he also asked participants a series of open- and closed-ended 

questions concerning whether they recognized any ads on the main page. 

After the moderator presented all eight ad scenarios, he interviewed participants about the purpose of the 

study, their normal response to ads in various contexts, and their interpretation of different ad-disclosure terms 

(See Attachment 2 for Post-Test Debriefing questions). 

3 For the one native ad scenario without a main page, the study procedure was slightly different. In this 

scenario, participants only quickly glanced at an email with a link to an item before clicking into the next page. 

4 As noted in the body of this report, two of the four search scenarios examined as part of this study did not 

include a click-into page. Also, with regard to the other two search scenarios, participants saw the click-into 

page for only the first one presented to them during a session. 

C-1 
Attachment 1 to Appendix C: Main Study Questionnaire 

Group A 

C-2 
 

 

 

 

 

 

--

Scenario GDO (Google Desktop Original - Formerly Scenario 1) 

You are at your computer at home. You are interested in purchasing a tablet computer. You go to Google and 

search for tablet. These are the results. Take two minutes to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Google search-results page 

if not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g., “what are you looking at and why,?,” “what would you tend 

not to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST Google Scenario, direct P to Organic result “Tablets CNN CNET.com” (the 2P 

nd P listed 

organic search result). 

If this is the SECOND Google Scenario, direct P to the Shopping Carousel labeled “Sponsored” at the top of the 

page. 

NOTE: P should only be shown the Google Shopping page in ONE scenario. If P clicks into the Google 

shopping page of his/her own volition on the FIRST Google scenario, direct him/her to the ORGANIC search 

result in the SECOND Google scenario. -----

Organic Search Result: 

M directs P to natural result labeled “Tablets CNN CNET.com.” M tells P to imagine he/she is interested in the 

link and decides to explore further but for now do not click on it. [M asks P probe questions, e.g., “tell me what 

you see here?” “is it useful to you? why or why not?”] 

M asks P to click natural search result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

Shopping Carousel (2P 

nd P Google Scenario):

M directs P to “Shop for tablet” link. M tells P to imagine he/she is interested in the link and decides to explore 

further but for now do not click on it [M asks P probe questions, e.g., “tell me what you see here?,””is it useful 

to you? why or why not?”] 

M asks P to click “Shop for tablet” link. Please browse as you normally would. Don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

***Shopping Carousel (Participant Selected): 

C-3 
 

 

M redirects P to “Shop for tablet” link. [P describes link and M asks probe questions, e.g., “why did you click 

here before?” “why didn’t you click in this area (e.g, point to natural search result) or in this area (point to a 

result labelled ad)?”] 

Please browse this page (i.e., Google Shopping Page) as you normally would but don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes page and M asks probe questions, e.g. “what are you looking at and why?,” is this what you 

expected to see? why or why not?” “is this page useful to you? why or why not?” 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Google SERP if not there.] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-4 
 

 

 

 

 

 

 

 

 

 

 

 

--

Scenario BDM (Bing Desktop Modified - Formerly Scenario 3 [modified]) 

You are at your computer at home. You are interested in purchasing an air purifier. You go to Bing and enter 

this search [air purifier]. These are the results that appear. Take two minutes to do what you normally would in 

this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Bing search-results page if 

not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST BING Scenario, direct P to the 2P 

nd P Ad Result located at the top of the page, above the 

organic search results – labeled “Air Purifiers at Lowe’s” 

If this is the SECOND BING Scenario, direct P to the organic search result labeled “The Best Air Purifiers – 

Free Shipping & Five Year Warranty (2P 

nd P Organic result).

NOTE: If P clicks on an ad during the 2P 

nd P Bing scenario, explore that result, then direct P to the specified 

organic result. ------

Naturally-Clicked Result (any result, including second ad result)/First or Second Bing Search Scenario 

If P naturally clicked away from the home page, M directs P to link he/she clicked. [P describes link and M 

asks probe questions, e.g.,“why did you click here before?,” “why didn’t you click in this other area?,”] 

M asks P to re-click on the result. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

First Bing Search Scenario (Second Ad Result) 

If necessary, M directs P to second ad result labeled “Air Purifiers at Lowe’s.” M tells P to imagine he/she is 

interested in the link and decides to explore further but for now do not click on it. [M asks P probe questions, 

e.g., “tell me what you see here?,” “is it useful to you? why or why not?”] 

M asks P to click the result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

Organic Search Result (Either scenario) 

M directs P to the natural search result labeled “THE Best Air Purifiers – Free Shipping & Five Year 

Warranty,” the second result below the shaded ads. M tells P to imagine he/she is interested in the link and 

decides to explore further but for now do not click on it. [M asks P probe questions, e.g., “tell me what you see 

here?,””is it useful to you? why or why not?”] 

C-5 
 

 

M asks P to click natural search result. Don’t click on anything for now. Remember to think aloud. Feel free 

to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Bing SERP if not there] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-6 
 

 

 

 

 

 

 

 

 

--

Scenario GePaDM (Gear Patrol Desktop Modified - Formerly Scenario 5 

[modified]) 

You are at home at your computer. You are interested in buying a car. A friend sent you a link to Gear Patrol, 

an online magazine showcasing new flashy products, because that day’s issue of Gear Patrol focused on cars. 

Take five minutes to review the information and do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Gear Patrol page if not 

there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------Direct P to “Three New Wave Hybrids that emphasize design (and efficiency)” first. 

This is the first Gear Patrol article below the menu. 

After exploring first story, direct P to the Subaru ad titled “A world rally blur:...” 

NOTE: If P selects Subaru item first of his/her own volition, explore that and then direct them to the New 

Wave Hybrid item. 

If P selects another organic item, explore that item then continue with New Wave and Subaru. ------

[M directs P to the item on new-wave hybrids, which leads to a story] 

M tells P to imagine that this item attracts your attention because you are interested in a hybrid car. For now, 

don’t click on it. [M asks P probe questions, e.g., “tell me what you see here?,” “is it useful to you; why or 

why not?”] 

M asks P to click on the link that leads to the next page, and, if necessary, directs P to select “View All” at the 

bottom of the next page. (This presents a full gallery of automobiles) 

Please browse as you normally would. [Capture eye-movements, scrolling, clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to organic car story page if not 

there.] Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If necessary, M directs P to the Subaru item leading to the story about the Subaru WRX car] 

M tells P to imagine that this item also draws your attention and you want to explore it further. For now, don’t 

click on it. [M asks P probe questions, e.g., “tell me what you see here?,”is it useful to you; why or why not?”] 

M asks P to click on the link that leads to Subaru WRX story. 

Please browse as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to Subaru WRX story page if not 

there.] Please describe the page to me. 

C-7 
 

 

 

 

 

 

 

 

 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Subaru WRX item page if not there] 

Please rate this page on each of the following scales. 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

[Return to Gear Patrol main page] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-8 
 

 

 

 

 

 

 

 

--

Scenario YDO (Yahoo! Desktop Original - Formerly Scenario 7) 

You are at home at your computer. You check the news on Yahoo. This is the page you see. Take two minutes 

to act as you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Yahoo News page if not 

there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------Direct P to News item titled “US Officials: Terrorist Group...” This is the first post below the purple bar, 

in the middle of the Yahoo! Page. 

After exploring this story, direct P to the “Instaflex” ad titled “Knee & Shoulder Relief...” 

If P should select Instaflex ad of his/her own volition, explore that page, then direct to a different, organic item. 

If P clicks on something other than the two directed, explore, then continue with US story and Instaflex.------

[M directs P to item on terrorist group nearing an attack, located below the purple bar, in the middle of the 

Yahoo! page, which leads to an AP story on Yahoo! News.] 

M tells P to imagine that this item attracts your attention and you decide to explore it further. For now, don’t 

click on it. [M asks P probe questions, e.g., “tell me what you see here?,” “is it useful to you; why or why 

not?”] 

M asks P to click on the link that leads to the news story. 

Please browse the page as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, 

clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to Yahoo! Health story if not 

there.] Please describe the page to me.[P describes the page and M asks probe questions, e.g., “tell me what 

you see here?”; “is this what you expected to see when you clicked on the link on the previous screen? why or 

why not?,”“is this useful to you; why or why not?”] 

[If necessary, M directs P to the Instaflex ad on the page, which leads to a story discussing pain relief 

research] 

M tells P to imagine that this item attracts your attention because you are interested in pain relief solutions for 

you or a family member. For now, don’t click on it. [M asks P probe questions, e.g., “tell me what you see 

here?” “is it useful to you; why or why not?”] 

C-9 
 

 

 

 

 

 

 

 

 

 

 

 

M asks P to click on the link that leads to the story on research about the ingredients in Instaflex. 

Please browse the page as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, 

clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to Instaflex story if not there.] 

Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Instaflex item page if not there] 

Please rate this page on each of the following scales. 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

[Return to Yahoo! main page] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-10 
 

 

 

 

 

 

 

--

Scenario GMM (Google Mobile Modified - Formerly Scenario 2 [modified]) 

You are waiting at the doctor’s office and decide to do some searching on your mobile phone. You are 

interested in purchasing a carpet cleaner. You go to Google and enter this search [carpet cleaner]. These are the 

results that appear. Take two minutes to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Google search-results page 

if not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST Google Scenario, direct P to Organic result “Tablets CNN CNET.com” (the 2P 

nd P listed 

organic search result). 

If this is the SECOND Google Scenario, direct P to the Shopping Carousel labeled “Sponsored” at the top of the 

page. 

NOTE: P should only be shown the Google Shopping page in ONE scenario. If P clicks into the Google 

shopping page of his/her own volition on the FIRST Google scenario, direct him/her to the ORGANIC search 

result in the SECOND Google scenario. -----

Natural Search Result: 

M directs P to the natural search result labeled “A Carpet Cleaner,” the first result that appears below the map. 

M tells P to imagine he/she is interested in the link and decides to explore further but for now do not click on it. 

[M asks P probe questions, e.g., “tell me what you see here?,””is it useful to you? why or why not?”] 

M asks P to click natural search result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

Shopping Carousel (2P 

nd P Google Scenario):

M directs P to “Shop for tablet” link. M tells P to imagine he/she is interested in the link and decides to explore 

further but for now do not click on it [M asks P probe questions, e.g., “tell me what you see here?,” “is it 

useful to you? why or why not?”] 

M asks P to click “Shop for tablet” link. Please browse as you normally would. Don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

***Shopping Carousel (Participant Selected)/First Google Search Scenario: 

C-11 
 

 

M redirects P to “Shop for tablet” link. [P describes link and M asks probe questions, e.g., “why did you click 

here before?” “why didn’t you click in this area (e.g, point to natural search result) or in this area (point to a 

result labelled ad)?”] 

Please browse this page (i.e., Google Shopping Page) as you normally would but don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes page and M asks probe questions, e.g. “what are you looking at and why?,” is this what you 

expected to see? why or why not?” “is this page useful to you? why or why not?” 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Google SERP if not there.] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-12 
 

 

 

 

 

 

 

 

--

Scenario BMO (Bing Mobile Original - Formerly Scenario 4) 

You are waiting at the car service center and decide to do some searching on your mobile phone. Your doctor 

recommended that should take Vitamin D. You go to Bing and enter this search [vitamin D]. These are the 

results that appear. Take two minutes to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Bing search-results page if 

not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?” “what would you tend 

not to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST BING Scenario, direct P to the 2P 

nd P Ad Result located at the top of the page, above the 

organic search results, labeled “Calcium and Vitamin D.” 

If this is the SECOND BING Scenario, direct P to organic search result labeled “How do I choose the best 

vitamin D supplement? (with pictures)” (3P 

rd P Organic result below the ads)------

Participant Selected Result (any result, including second ad result)/First or Second Bing Search Scenario 

If P naturally clicked away from the home page, M directs P to link he/she clicked. [P describes link and M 

asks probe questions, e.g.,“why did you click here before?,” “why didn’t you click in this other area?,”] 

M asks P to re-click on the result. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

First Bing Search Scenario (Second Ad Result) 

If necessary, M directs P to second ad result labeled “Calcium and Vitamin D.” M tells P to imagine he/she is 

interested in the link and decides to explore further but for now do not click on it. [M asks P probe questions, 

e.g., “tell me what you see here?,” “is it useful to you? why or why not?”] 

M asks P to click the result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

Organic Search Result (Either Scenario) 

M directs P to the natural search result labeled “How Do I Choose the Best Vitamin D Supplement? (with 

pictures),” the natural search result ranked third on the page, below the shaded ads. M tells P to imagine he/she 

is interested in the link and decides to explore further but for now do not click on it. [M asks P probe questions, 

e.g., “tell me what you see here?,””is it useful to you? why or why not?”] 

C-13 
 

 

M asks P to click natural search result. Don’t click on anything for now. Remember to think aloud. Feel free 

to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Bing SERP if not there] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-14 
 

 

 

 

 

 

 

 

--

Scenario TMO (Time Mobile Original - Formerly Scenario 6) 

You are waiting in line at a coffee shop. You’ve been reading an item on Time’s site, so you go to Time.com 

and continue reading it. When you have finished reading the item, continue to interact with Time.com as you 

normally would. 

You finish reading that item, so you decide to browse further on the page. [Capture eye-movements, scrolling, 

clicking] 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Time article page if not 

there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------ Direct P to “Burglars aren’t scared of anything” item (ADT Ad) first, then to story titled “Sierra Leone 

dispatches...” (second article on the page) 

NOTE: If P clicks on ADT ad of his/her own volition, explore the item then direct to a different organic item on 

the main Time page. If P selects a completely different item, explore that item, then move on to ADT and Sierra 

Leone items.------

[If necessary, M directs P to ADT ad on the Time page, which leads to the ADT story.] 

M tells P to imagine that this item attracts your attention because you are concerned about the safety of your 

home. For now, don’t click on it. [M asks P probe questions, e.g., “tell me what you see here?,””is it useful to 

you; why or why not?”] 

M asks P to click on the link that leads to the next captured page (story offering tips on how to keep your home 

safe). 

Please browse as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to ADT Story page if not there.] 

Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[M directs P to organic item on Troops Enforcing the Ebola Quarantine in Sierra Leone, the second Time 

article] 

M tells P to imagine that this item attracts your attention because you are concerned about Ebola and you decide 

to explore further. [M asks P probe questions, e.g., “is this item useful to you? why or why not?,” “is this what 

you expect to see on this page? why or why not?”] 

C-15 
 

 

 

 

 

 

 

 

 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to ADT item page if not there] 

Please rate this page on each of the following scales. 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

[Return to Time main page] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-16 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Scenario CTMM (Chicago Tribune Mobile Modified - Formerly scenario 8 

[modified]) 

You are in a coffee shop. You are using your mobile phone to check your email and you open an email 

message from a friend. She forwarded you an item that she thought would interest you because your sister 

recently started struggling with hair loss. You click on the item and this is what you see. Read this item and 

continue to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. 

M asks P to click on the link in the email. 

Please browse the page as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, 

clicking] 

[when P comes to stop state] Okay, thank you. Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Chicago Tribune page if not there] 

Please rate this page on each of the following scales. 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

C-17 
Do you see any other ads? 

Why do you say that? 

C-18 
Group B 

C-19 
 

 

 

 

 

 

 

--

Scenario GDM (Google Desktop Modified - Formerly Scenario 1 

[modified]) 

You are at your computer at home. You are interested in purchasing a tablet computer. You go to Google and 

search for tablet. These are the results. Take two minutes to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Google search-results page 

if not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g., “what are you looking at and why,?,” “what would you tend 

not to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST Google Scenario, direct P to Organic result “Tablets CNN CNET.com” (the 2P 

nd P listed 

organic search result). 

If this is the SECOND Google Scenario, direct P to the Shopping Carousel labeled “Sponsored” at the top of the 

page. 

NOTE: P should only be shown the Google Shopping page in ONE scenario. If P clicks into the Google 

shopping page of his/her own volition on the FIRST Google scenario, direct him/her to the ORGANIC search 

result in the SECOND Google scenario. -----

Natural Search Result: 

M directs P to natural result labeled “Tablets CNN CNET.com.” M tells P to imagine he/she is interested in the 

link and decides to explore further but for now do not click on it. [M asks P probe questions, e.g., “tell me what 

you see here?” “is it useful to you? why or why not?”] 

M asks P to click natural search result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

Shopping Carousel (2P 

nd P Google scenario):

M directs P to “Shop for tablet” link. M tells P to imagine he/she is interested in the link and decides to explore 

further but for now do not click on it [M asks P probe questions, e.g., “tell me what you see here?,””is it useful 

to you? why or why not?”] 

M asks P to click “Shop for tablet” link. Please browse as you normally would. Don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

***Shopping Carousel (Participant Selected): 

M redirects P to “Shop for tablet” link. [P describes link and M asks probe questions, e.g., “why did you click 

here before?” “why didn’t you click in this area (e.g, point to natural search result) or in this area (point to a 

result labelled ad)?”] 

Please browse this page (i.e., Google Shopping Page) as you normally would but don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes page and M asks probe questions, e.g. “what are you looking at and why?,” is this what you 

expected to see? why or why not?” “is this page useful to you? why or why not?” 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

C-20 
[Return to Google SERP if not there.] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-21 
 

 

 

 

 

 

 

--

Scenario GMO (Google Mobile Original - Formerly Scenario 2) 

You are waiting at the doctor’s office and decide to do some searching on your mobile phone. You are 

interested in purchasing a carpet cleaner. You go to Google and enter this search [carpet cleaner]. These are the 

results that appear. Take two minutes to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Google search-results page 

if not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST Google Scenario, direct P to Organic result “Tablets CNN CNET.com” (the 2P 

nd P listed 

organic search result). 

If this is the SECOND Google Scenario, direct P to the Shopping Carousel labeled “Sponsored” at the top of the 

page. 

NOTE: P should only be shown the Google Shopping page in ONE scenario. If P clicks into the Google 

shopping page of his/her own volition on the FIRST Google scenario, direct him/her to the ORGANIC search 

result in the SECOND Google scenario. -----

Natural Search Result/First or Second Google Search Scenario: 

M directs P to the natural search result labeled “A Carpet Cleaner,” the first result that appears below the map. 

M tells P to imagine he/she is interested in the link and decides to explore further but for now do not click on it. 

[M asks P probe questions, e.g., “tell me what you see here?,””is it useful to you? why or why not?”] 

M asks P to click natural search result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

Shopping Carousel (2P 

nd P Google Scenario):

M directs P to “Shop for tablet” link. M tells P to imagine he/she is interested in the link and decides to explore 

further but for now do not click on it [M asks P probe questions, e.g., “tell me what you see here?,” “is it 

useful to you? why or why not?”] 

M asks P to click “Shop for tablet” link. Please browse as you normally would. Don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes the page and M asks probe questions, e.g., “what are you looking at and why?,” “is this what you 

expected to see?; why or why not?” “is this useful to you? why or why not?”] 

Shopping Carousel (Participant Selected): 

M redirects P to “Shop for tablet” link. [P describes link and M asks probe questions, e.g., “why did you click 

here before?” “why didn’t you click in this area (e.g, point to natural search result) or in this area (point to a 

result labelled ad)?”] 

Please browse this page (i.e., Google Shopping Page) as you normally would but don’t click on anything for 

now. Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, 

clicking] 

[P describes page and M asks probe questions, e.g. “what are you looking at and why?,” is this what you 

expected to see? why or why not?” “is this page useful to you? why or why not?” 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

C-22 
[Return to Google SERP if not there.] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-23 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

--

Scenario BDO (Bing Desktop Original - Formerly Scenario 3) 

You are at your computer at home. You are interested in purchasing an air purifier. You go to Bing and enter 

this search [air purifier]. These are the results that appear. Take two minutes to do what you normally would in 

this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Bing search-results page if 

not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST BING Scenario, direct P to the 2P 

nd P Ad Result located at the top of the page, above the 

organic search results – labeled “Air Purifiers at Lowe’s” 

If this is the SECOND BING Scenario, direct P to the organic search result labeled “The Best Air Purifiers – 

Free Shipping & Five Year Warranty (2P 

nd P Organic result).

NOTE: If P clicks on an ad during the 2P 

nd P Bing scenario, explore that result, then direct P to the specified 

organic result. ------

Naturally-Clicked Result (any result, including second ad result)/First or Second Bing Search Scenario 

If P naturally clicked away from the home page, M directs P to link he/she clicked. [P describes link and M 

asks probe questions, e.g.,“why did you click here before?,” “why didn’t you click in this other area?,”] 

M asks P to re-click on the result. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

First Bing Search Scenario (Second Ad Result) 

If necessary, M directs P to second ad result labeled “Air Purifiers at Lowe’s.” M tells P to imagine he/she is 

interested in the link and decides to explore further but for now do not click on it. [M asks P probe questions, 

e.g., “tell me what you see here?,” “is it useful to you? why or why not?”] 

M asks P to click the result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

Organic Search Result (Either scenario) 

M directs P to the natural search result labeled “THE Best Air Purifiers – Free Shipping & Five Year 

Warranty,” the second result below the shaded ads. M tells P to imagine he/she is interested in the link and 

decides to explore further but for now do not click on it. [M asks P probe questions, e.g., “tell me what you see 

here?,””is it useful to you? why or why not?”] 

M asks P to click natural search result. Don’t click on anything for now. Remember to think aloud. Feel free 

to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Bing SERP if not there] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

C-24 
Do you see any other ads? 

Why do you say that? 

C-25 
 

 

 

 

 

 

 

 

 

 

 

 

 

--

Scenario BMM (Bing Mobile Modified - Formerly Scenario 4 [modified]) 

You are waiting at the car service center and decide to do some searching on your mobile phone. Your doctor 

recommended that should take Vitamin D. You go to Bing and enter this search [vitamin D]. These are the 

results that appear. Take two minutes to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Bing search-results page if 

not there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?” “what would you tend 

not to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------If this is the FIRST BING Scenario, direct P to the 2P 

nd P Ad Result located at the top of the page, above the 

organic search results, labeled “Calcium and Vitamin D.” 

If this is the SECOND BING Scenario, direct P to organic search result labeled “How do I choose the best 

vitamin D supplement? (with pictures)” (3P 

rd P Organic result below the ads)------

Participant Selected Result (any result, including second ad result)/First or Second Bing Search Scenario 

If P naturally clicked away from the home page, M directs P to link he/she clicked. [P describes link and M 

asks probe questions, e.g.,“why did you click here before?,” “why didn’t you click in this other area?,”] 

M asks P to re-click on the result. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

First Bing Search Scenario (Second Ad Result) 

If necessary, M directs P to second ad result labeled “Calcium and Vitamin D.” M tells P to imagine he/she is 

interested in the link and decides to explore further but for now do not click on it. [M asks P probe questions, 

e.g., “tell me what you see here?,” “is it useful to you? why or why not?”] 

M asks P to click the result. Please browse as you normally would. Don’t click on anything for now. 

Remember to think aloud. Feel free to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

Organic Search Result (Either Scenario) 

M directs P to the natural search result labeled “How Do I Choose the Best Vitamin D Supplement? (with 

pictures),” the natural search result ranked third on the page, below the shaded ads. M tells P to imagine he/she 

is interested in the link and decides to explore further but for now do not click on it. [M asks P probe questions, 

e.g., “tell me what you see here?,””is it useful to you? why or why not?”] 

M asks P to click natural search result. Don’t click on anything for now. Remember to think aloud. Feel free 

to scroll down the page. [Capture eye-movements, scrolling, clicking] 

[P describes the page and M asks probe questions, e.g., “is this what you expected to see?; why or why not?” 

“is this useful to you? why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Bing SERP if not there] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

C-26 
Why do you say that? 

C-27 
 

 

 

 

 

 

 

 

 

 

 

--

Scenario GePaDO (Gear Patrol Desktop Original - Formerly Scenario 5) 

You are at home at your computer. You are interested in buying a car. A friend sent you a link to Gear Patrol, 

an online magazine showcasing new flashy products, because that day’s issue of Gear Patrol focused on cars. 

Take five minutes to review the information and do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Gear Patrol page if not 

there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------Direct P to “Three New Wave Hybrids that emphasize design (and efficiency)” first. 

This is the first Gear Patrol article below the menu. 

After exploring first story, direct P to the Subaru ad titled “A world rally blur:...” 

NOTE: If P selects Subaru item first of his/her own volition, explore that and then direct them to the New 

Wave Hybrid item. 

If P selects another organic item, explore that item then continue with New Wave and Subaru. ------

[M directs P to the item on new-wave hybrids, which leads to a story] 

M tells P to imagine that this item attracts your attention because you are interested in a hybrid car. For now, 

don’t click on it. [M asks P probe questions, e.g., “tell me what you see here?,” “is it useful to you; why or 

why not?”] 

M asks P to click on the link that leads to the next page, and, if necessary, directs P to select “View All” at the 

bottom of the next page. (This presents a full gallery of automobiles) 

Please browse as you normally would. [Capture eye-movements, scrolling, clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to organic car story page if not 

there.] Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If necessary, M directs P to the Subaru item leading to the story about the Subaru WRX car] 

M tells P to imagine that this item also draws your attention and you want to explore it further. For now, don’t 

click on it. [M asks P probe questions, e.g., “tell me what you see here?,”is it useful to you; why or why not?”] 

M asks P to click on the link that leads to Subaru WRX story. 

Please browse as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to Subaru WRX story page if not 

there.] Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Subaru WRX item page if not there] 

Please rate this page on each of the following scales. 

C-28 
 

 

 

 

 

 

 

 

 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

[Return to Gear Patrol main page] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-29 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

--

Scenario TMM (Time Mobile Modified - Formerly Scenario 6 [modified]) 

You are waiting in line at a coffee shop. You’ve been reading an item on Time’s site, so you go to Time.com 

and continue reading it. When you have finished reading the item, continue to interact with Time.com as you 

normally would. 

You finish reading that item, so you decide to browse further on the page. [Capture eye-movements, scrolling, 

clicking] 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Time article page if not 

there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------ Direct P to “Burglars aren’t scared of anything” item (ADT Ad) first, then to story titled “Sierra Leone 

dispatches...” (second article on the page) 

NOTE: If P clicks on ADT ad of his/her own volition, explore the item then direct to a different organic item on 

the main Time page. If P selects a completely different item, explore that item, then move on to ADT and Sierra 

Leone items.------

[If necessary, M directs P to ADT ad on the Time page, which leads to the ADT story.] 

M tells P to imagine that this item attracts your attention because you are concerned about the safety of your 

home. For now, don’t click on it. [M asks P probe questions, e.g., “tell me what you see here?,””is it useful to 

you; why or why not?”] 

M asks P to click on the link that leads to the next captured page (story offering tips on how to keep your home 

safe). 

Please browse as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to ADT Story page if not there.] 

Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[M directs P to organic item on Troops Enforcing the Ebola Quarantine in Sierra Leone, the second Time 

article] 

M tells P to imagine that this item attracts your attention because you are concerned about Ebola and you decide 

to explore further. [M asks P probe questions, e.g., “is this item useful to you? why or why not?,” “is this what 

you expect to see on this page? why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to ADT item page if not there] 

Please rate this page on each of the following scales. 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

C-30 
 

 

 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

[Return to Time main page] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-31 
 

 

 

 

 

 

 

 

 

 

--

Scenario YDM (Yahoo! Desktop Modified - Formerly Scenario 7 

[modified]) 

You are at home at your computer. You check the news on Yahoo. This is the page you see. Take two minutes 

to act as you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. [Return to Yahoo News page if not 

there.] Please describe the page to me. 

[P describes page; M asks probe questions, e.g.,“what are you looking at and why,?””what would you tend not 

to pay attention to and why?”] [Note: Probe questions should not directly mention paid vs. unpaid results.] 

------Direct P to News item titled “US Officials: Terrorist Group...” This is the first post below the purple bar, 

in the middle of the Yahoo! Page. 

After exploring this story, direct P to the “Instaflex” ad titled “Knee & Shoulder Relief...” 

If P should select Instaflex ad of his/her own volition, explore that page, then direct to a different, organic item. 

If P clicks on something other than the two directed, explore, then continue with US story and Instaflex.------

[M directs P to item on terrorist group nearing an attack, located below the purple bar, in the middle of the 

Yahoo! page, which leads to an AP story on Yahoo! News.] 

M tells P to imagine that this item attracts your attention and you decide to explore it further. For now, don’t 

click on it. [M asks P probe questions, e.g., “tell me what you see here?,” “is it useful to you; why or why 

not?”] 

M asks P to click on the link that leads to the news story. 

Please browse the page as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, 

clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to Yahoo! Health story if not 

there.] Please describe the page to me.[P describes the page and M asks probe questions, e.g., “tell me what 

you see here?”; “is this what you expected to see when you clicked on the link on the previous screen? why or 

why not?,”“is this useful to you; why or why not?”] 

[If necessary, M directs P to the Instaflex ad on the page, which leads to a story discussing pain relief 

research] 

M tells P to imagine that this item attracts your attention because you are interested in pain relief solutions for 

you or a family member. For now, don’t click on it. [M asks P probe questions, e.g., “tell me what you see 

here?” “is it useful to you; why or why not?”] 

M asks P to click on the link that leads to the story on research about the ingredients in Instaflex. 

Please browse the page as you normally would. Remember to think aloud. [Capture eye-movements, scrolling, 

clicking] 

[when P clicks away from screen or time expires] Okay, thank you. [Return to Instaflex story if not there.] 

Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Instaflex item page if not there] 

Please rate this page on each of the following scales. 

C-32 
 

 

 

 

 

 

 

 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

[Return to Yahoo! main page] 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-33 
 

 

 

 

 

 

 

 

 

 

 

 

 

Scenario CTMO (Chicago Tribune Mobile Original - Formerly Scenario 8) 

You are in a coffee shop. You are using your mobile phone to check your email and you open an email 

message from a friend. She forwarded you an item that she thought would interest you because your sister 

recently started struggling with hair loss. You click on the item and this is what you see. Read this item and 

continue to do what you normally would in this situation. 

[when P clicks away from homepage or time expires] Okay, thank you. 

M asks P to click on the link in the email. 

Please browse the page as you normally would. [Capture eye-movements, scrolling, clicking] 

[when P comes to stop state] Okay, thank you. Please describe the page to me. 

[P describes the page and M asks probe questions, e.g., “tell me what you see here?”; “is this what you 

expected to see when you clicked on the link on the previous screen? why or why not?,”“is this useful to you; 

why or why not?”] 

[If final scenario presented to P, ask P additional probe questions below; otherwise close browser and open 

file for next scenario.] 

[Return to Chicago Tribune page if not there] 

Please rate this page on each of the following scales. 

Very Informative Not Informative 

1 2 3 4 5 6 7 

Very Reliable Not Reliable 

1 2 3 4 5 6 7 

Very Believable Not Believable 

1 2 3 4 5 6 7 

Very Useful Not Useful 

1 2 3 4 5 6 7 

Do you see any ads on this page? 

Why do you say that? [Ask question for each ad identified] 

Do you see any other ads? 

Why do you say that? 

C-34 
 

 

_________________________________ 

FTC Search & Sponsored Content Study 

Pre-Test Interview 

Participant Initials:________ Participant Group & #:________ Date:______________ 

1. How often do you access the Web? Several times a day___ Once a day___ 

A few times/week___ Once a week___ Less than once/week___ Never___ 

2. Devices used to access the Internet? Desktop computer___ Laptop computer___ 

Tablet computer___ Smart phone___ Other_____________________________ 

3. Where do you work and, if for a company, what does that company do? 

Participants should use Web at least once/week, via desktop or laptop computer and tablet or smartphone. 

They should not work in any of the industries listed in the recruitment screener. 

If participant doesn’t match study requirements (i.e., they somehow slipped through screening process), 

abort session. 

C-35 
 

 

 

 

 

_______________________________________________________________ 

______________________________________________________________ 

____________________________________________________________________ 

____________________________________________________________________ 

____________________________________________________________________ 

____________________________________________________________________ 

____________________________________________________________________ 

Attachment 2 to Appendix C: Post-Test Debriefing Interview 

FTC Search & Sponsored Content Study 

Post-Test Debriefing Interview 

Participant Initials:________ Participant Group & #:________ Date:______________ 

1. How do you usually respond to ads in search results? Always ignore them___ 

Always look at them ___ Sometimes ignore, sometimes look at them ___ 

2. How do you usually respond to ads placed in or near articles or stories on the Internet? Always ignore 

them___ Always look at them ___ Sometimes ignore, sometimes look at them ___ 

3. How do you usually respond to ads mixed into recent posts? Always ignore them___ Always look at 

them ___ Sometimes ignore, sometimes look at them ___ 

4. Are you usually able to distinguish ads from non-ads? 

5. Have you ever discovered that something you thought was regular information was an ad? How did you 

feel about that? ______________________________________________________________ 

6. How would you want things marked so that you can see clearly that they are ads? 

7. If something is marked with one of the following words what does that tell you? 

Ad ______________________________ 

Advertisement _____________________ 

Sponsored_________________________ 

Paid______________________________ 

Presented by_______________________ 

Additional Thoughts and Comments 

C-36 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix D: Coding of Ad Recognition Data 

Based on an examination of the video recordings and verbatim comments of participants for each 

participant, two different coders – an expert consultant on the study and either one of two FTC paralegal staff 

who otherwise were not involved in the studyP4F 

5 

P – coded each of the 34 (17 original and 17 modified) ad 

conditions tested as to whether: (1) the participant recognized the ad as an ad (“recognized as ad”); (2) the 

participant did not recognize it as an ad (“not recognized as ad”); or (3) the coder could not tell if the participant 

recognized the ad as an ad or did not recognize it as an ad (“unclear”).P5F 

6 

P The FTC paralegal staff coded either 

the Group A or Group B participant sessions so that they did not see both the original and modified version of 

the same web pages. The expert consultant, however, coded all the sessions. It is important to note that the 

study participants were not asked directly if an item was an ad; the coders considered participants’ behavior and 

comments as they viewed the ad conditions to make a judgment (See Attachment 1 for criteria used by coders). 

Ads coded as unclear could mean that a coder did not have enough information to make a judgment or that a 

participant was uncertain whether the ad was an ad. 

Coder agreement between the expert consultant and the FTC paralegal staff was 79.2%. We did not 

resolve differences by the conventional method of discussion between the coders and instead deferred to the 

judgment of the expert consultant. Differences between the coders primarily were due to the greater likelihood 

of the FTC paralegal staff coding that it was unclear whether or not a participant recognized an ad. We 

analyzed the ad recognition data in two ways – by defaulting entirely to the coding of the expert consultant and 

by defaulting to the coding of the FTC paralegals – and found very little substantive differences in the results. 

The probability that an item was recognized as an ad was only slightly lower when we based our analyses on the 

FTC staff data. (See Section III of Appendix F for a description of the regression analysis using the FTC staff 

ad recognition coding.) The analyses discussed in this report utilized the expert consultant’s coding. 

5 The FTC paralegals were given detailed instructions and practiced with video recordings of a few pilot 

sessions before they reviewed and coded the study session videos and verbatim comments. 

6 One participant did not actually see one of the ad conditions due to human error, and thus we excluded that 

data point from our analyses. 

D-1 
 

 

 

 

Attachment 1 to Appendix D: Coder Instructions 

Usability Study 

Coder Instructions 

Background 

As part of this study, 48 participants were taken through 8 different digital scenarios or conditions. Four 

of these conditions were on desktop and four were on a mobile device. The conditions either related to search 

engines (categorized as “search”) or to a variety of publisher sites, such as Yahoo! (categorized as “native”). 

Specifically, the 8 conditions are as follows: 

• Search-Google Desktop (S-GD) 

• Search-Google Mobile (S-GM) 

• Search-Bing Desktop (S-BD) 

• Search-Bing-Mobile (S-BM) 

• Native-Gear Patrol Desktop (N-GPD) 

• Native-Time Mobile (N-TM) 

• Native-Yahoo! Desktop (N-YD) 

• Native-Chicago Tribune Mobile (N-CTM) 

Each session was videotaped and transcribed, and special equipment also captured where participants 

looked on the various digital pages shown to them. For each session, there are 8 separate recordings in total, 

one for each condition presented. Each recording has a corresponding transcript, located in the same folder. 

Each participant was presented the conditions in a different order. For each participant, the conditions 

are numbered in the order in which they were presented. 

One coder will review recordings for participants 1 through 24, and a second coder will review 

recordings for participants 25 through 48. If you cannot hear what the participant says in a recording, you also 

can consult the corresponding transcript. 

The recording of the final condition (# 8 of 8) includes a number of “post-study” questions to which 

participants respond. For these final conditions, you only need to listen until the point when participants 

respond to a question about what ads are on the page. The next question asks how participants usually respond 

to ads in or near search results; you need not listen to recording beyond this point, as it is not relevant to your 

coding. 

Areas-of-Interest/Classification Table 

An area-of-interest is a specified item or grouping of items on a particular page shown to a participant. 

Each condition is designed to test participants’ interaction with one or two particular pages, although 

participants may be shown or voluntarily click-into other pages during a condition. For each page that a 

condition is intended to test, there are one or more specified areas-of-interest. 

For example, the Search-Google Desktop (S-GD) condition tests the main Google search page and a 

click-into page (Google Shopping page). [Show visual]. The main Google Page has three areas-of-interest: the 

shop box at the top of the page, ads under the box, and ads on the right-hand side of the page. The click-into 

page (Google Shopping page) has two areas-of-interest: the main list of results and ads located at the bottom of 

the page. 

D-2 
 

 

 

 

 

 

 

 

 

 

 

 

Another example is the Native-Yahoo! Desktop (N-YD) condition, which also tests two pages: the main 

Yahoo! page and a click-into page or article page. [Show visual]. The area-of-interest on the main Yahoo! page 

is an item consisting of the headline “Knee & Shoulder Relief: Doctors speak out about shocking knee & 

shoulder relief” accompanied by an image of a skeleton. The entire click-into page for this link is a second 

area-of-interest. The provided spreadsheet lists the area-of-interest for each of the conditions. [Walk-through 

areas of interest listed in the Classification Table]. 

Many conditions also can include a “decoy” item, which are listed below. 

0B Condition 1B Decoy Item 

Search-Google Desktop (S-GD) 

nd Result named “Tablets CNN CNET.com” (the 2P P 

listed regular search result) 

Search-Google Mobile (S-GM) 

st Result named “A Cleaner Carpet Cleaner” (the 1P P 

listed regular search result) 

Search-Bing Desktop (S-BD) Result named “Top Air Purifier Reviews / Best Air 

rd Purifier – Consumer... (3P P regular search result 

listed) 

Search-Bing-Mobile (S-BM) 

Result named “How do I choose the best vitamin 

th D supplement? (with pictures)” (4P P listed regular 

result below the ads) 

Native-Gear Patrol Desktop (N-GPD) 

Item titled “Three New Wave Hybrids that 

st emphasize design (and efficiency)” (1P P article 

below the menu) 

Native-Time Mobile (N-TM) 

nd Item titled “Sierra Leone dispatches...” (2P P article 

on the page) 

Native-Yahoo! Desktop (N-YD) 

st Item titled “US Officials: Terrorist Group...” (1P P 

post below the purple bar, in the middle of the 

Yahoo! Page.) 

Native-Chicago Tribune Mobile (N-CTM) No Decoy 

Coding Instructions 

Different participants saw different versions of the same pages to be tested. Some pages may look more 

familiar and other pages less familiar to you. In coding the areas-of-interest, only consider what the participant 

actually says and does. 

In a spreadsheet, create a separate tab for each participant, labeled with the participant number (i.e., 1, 2, 

3, etc.). For each participant, classify the specified area-of-interest based on the following: 

“Seen as ad” 

“Not seen as ad” 

“Unclear/no indication” 

Once you make a classification, find the appropriate condition and area-of-interest in the provided 

classification table and insert an “X” into the appropriate column (i.e., “Seen as ad; “Not seen as ad; or 

“Unclear/no indication.” If you classify an area-of-interest as “Seen as ad” or “Not seen as ad,” note what 

caused you to make that classification in the column “Reason” for that area-of-interest. 

D-3 
 

 

 

 

 

 

 

 

[Review ways in which participants may verbalize or show they notice an area-of-interest as an ad or do not 

notice it as an ad.] 

In some cases, you may not be able to tell whether a participant recognized an area-of-interest as an ad. 

If that happens, code that area-of-interest “Unclear/no indication.” For example, in the search conditions 

(Search-Google Desktop/Mobile (S-GD/GM) and Search-Bing Desktop/Mobile (S-BD/BM), participants may 

not say anything about an area-of-interest or only say things about its format (such as the images/pictures, price 

information, or color) or the product or store it promotes. Similarly, in native conditions (Native-Gear Patrol 

Desktop (N-GPD), Native-Yahoo! Desktop (N-YD), Native-Time Mobile (N-TM) and Native-Chicago Tribune 

Mobile (N-CTM), participants may talk about their interest in the product discussed in an area-of-interest but 

give no indication whether they notice the item-of-interest as an ad. 

If it happens that a participant (due to moderator or technical error, or not scrolling down the page), was 

not shown an area-of-interest, code it as “Unclear/no indication” but also note that fact in the column “AOI Not 

Seen.” 

In addition, different from a classification of “Unclear/no indication,” if a participant only noticed an 

area-of-interest as an ad during post-test questioning, code the area of interest as “Not seen as ad” and also mark 

an “X” in the column “Seen as ad only in post-test” for that area-of-interest. However, if the moderator told a 

participant that something is an ad, that does not count as the participant noticing an area-of-interest as an ad in 

the post-test questioning. 

If a participant identifies an area-of-interest as an ad, it does not matter if they figured that out based on 

the item’s label or its content. In either situation, classify the area-of-interest as “Seen as ad.” 

In classifying an area-of-interest, consider not only what the participant says or does during the 

condition that shows the particular area-of-of-interest but also any other relevant information learned from what 

the participant says or does during other conditions or the post-test questioning. For example, a participant may 

say something during a later condition or the post-test questioning that reveals he or she actually noticed an 

area-of-interest as an ad earlier, whereas it may have been unclear during the particular condition in which that 

area-of-interest was shown. If you relied on information revealed outside of the particular condition during 

which the area-of-interest was shown to make a classification, be sure to note that in the “Reasons” column. 

A situation where a participant may reveal in a later condition that he or she actually noticed an area-ofinterest as an ad previously differs from the situation described above in which a participant says or does 

something that suggests he or she only realized an-area-of-interest was an ad during the post-study questioning 

(and thus did not recognize that area-of-interest as an ad during the condition.). As instructed earlier, you 

should code that area-of-interest as “not seen as ad” and also mark an “X” in in the column “Seen as ad only in 

post-test.” 

The fact that a participant’s eyes scanned an ad-label does not necessarily mean that the participant 

really noticed or processed the label. Human vision does not process everything our eyes scan. However, if the 

eyes lingered on a label and there is other corroborating evidence, e.g., the participant mentions skipping the 

items under that label, that can be taken as strongly suggesting that the participant noticed the label. 

In the search conditions (Search-Google Desktop/Mobile (S-GD/GM) and Search-Bing Desktop/Mobile 

(S-BD/BM), just saying that an area-of-interest (search-result-item) links to a company selling something does 

not necessarily mean that the participant recognizes that area-of-interest as an ad. Participants often say the 

same thing about search-result-items that are not paid search results. 

D-4 
 

 

However, in the native conditions (Native-Gear Patrol Desktop (N-GPD), Native-Yahoo! Desktop (NYD), Native-Time Mobile (N-TM) and Native-Chicago Tribune Mobile (N-CTM), saying an area-of-interest 

probably links to a sales pitch is good enough to classify the participant as recognizing that item as an ad. 

Also, in the native conditions, it can happen that participants say the area-of-interest on the first page is 

an ad or links to an ad but when participants click into the next page they change their mind and say the second 

area-of-interest is not really an ad. The reverse situation also can happen. Participants may not notice the area 

of interest as an ad on the first page but after clicking into the next page say that it is an ad. 

Review pilot recordings; ask coder to classify and go over reasoning [at least one in which participant: notices 

area-of-interest as an ad, does not notice as an ad; and it is unclear whether the participant does or not][sic] 

D-5 
 

 

 

Appendix E: Processing of Eye Tracking Data 

While participants interacted with the web pages, we also collected time-sequenced eye tracking data. 

We only processed these data for specific time segments when participants interacted with the web pages 

without interruption. These time segments included the first few seconds after: (1) the main web page was 

displayed; (2) the moderator directed a participant’s attention to particular ads; and (3) the click-into page was 

displayed. 

For each time segment, we defined “areas of interest” on the relevant pages, which contained the 

particular ads examined by the study. We then mapped these areas of interest (“AOIs”) to the eye tracking data 

to generate statistics, i.e., “AOI statistics.” Among other things, the AOI statistics measured the total amount of 

time that a participant gazed within each such area or “total visit duration.” The time-segmented eye tracking 

data also were used to generate gaze plots and heat maps. 

The process of mapping the AOIs to the eye tracking data was a quasi-manual process for ad conditions 

that participants viewed on a smartphone screen, whereas this mapping process was fully automated for the 

desktop conditions. 

E-1 
 

 

 

 

 

 

 

 

 

Appendix F: Regression Analyses of Ad Recognition 

and Eye Tracking Data 

I. The Regression Model Used to Assess Changes in Ad Recognition 

We used a multinomial logit model to estimate the change in ad recognition across the ad conditions due 

to the disclosure modifications.P6F 

7 

P The model enabled each ad condition and each participant to have a unique 

effect on the probability that a participant was in one of the three possible outcome groups, i.e., recognized the 

ad as an ad; did not recognize the ad as an ad; or it was unclear to the observer. For participant i, ad condition j, 

and category k, we modeled the latent value for each category as: 

mm Uijk = ββkkDD + ααiikk + γγjjkk + ∑mm=1:8 δδkk

mmOOiijj , 

where DD is an indicator for whether the participant saw the original or modified condition, ααiikk are participant – 

category effects, γγjjkk are ad condition – category effects, and OOiijj

mm is an indicator variable that participant i saw ad 

condition j in order m. Here, ββkk is the mean effect of the modifications across all ad conditions. The effects of 

all of these variables were allowed to vary across categories, so the probabilities of ad, no ad, or unclear could 

vary by participant, ad condition, and ad condition order, as well as because of the modifications. 

To account for the fact that the same participants, and same websites, are present in the data multiple 

times, we clustered standard errors at the participant and website levels, allowing for arbitrary correlation 

between responses for the same website and between responses for the same participant. We report the 

marginal effects of the modifications. 

In additional analyses, we examined how the effects of the modifications varied. We interacted the 

modifications with whether the ad condition was a search or native ad, and whether the participant viewed the 

ad on a mobile device or desktop. 

II. The Disclosure Modifications Significantly Improved Ad Recognition 

Overall, the disclosure modifications increased the probability that a participant recognized an ad by 21 

percentage points, with the 95% confidence interval ranging from 15 to 27 percentage points. Because the 

bottom of the confidence interval is greater than zero, these effects are statistically significant. See Figure 1 

below. In addition, given that in the original ad conditions the average rate of ad recognition was approximately 

47%, an increase of 21 percentage points translates into a 45% improvement in ad recognition, which is 

substantial. 

7 See A. Colin Cameron & Pravin K. Trivedi, Microeconometrics: Methods and Applications, ch. 15 (2005). 

F-1 
 

 

 

 

 

 

 

 

 

Cl) 

e> 20 

C: 

.s:::. "' 

u 

-

C: 

·o 

a. 0 

Cl) 

C) 

~ C: 

Cl) 

~ a, -20 

a. 

-40 

f 

Overall 

f 

f 

Search Native Ad 

Ad Condition Subgroup 

Coded Response -A Ad -+- No Ad • Unclear 

Figure 1 

Confidence Intervals for Estimated Effects of Disclosure Modifications on Ad Recognition 

Figure 1 depicts the 95% confidence intervals for the estimated effects of the disclosure modifications on ad recognition based 

on the statistical model, both for all conditions and for the search and native ad conditions separately. The red triangle, blue 

diamond, and green circle respectively depict the effect on the probability that: (1) an ad was recognized as an ad; (2) an ad 

was not recognized as an ad; and (3) it was unclear to the observer. 

We also found substantial improvements in ad recognition when we separately examined the search and 

native ad condition groups. See Figure 1 above. In the search ad conditions, we found that the probability that 

participants recognized the ads as ads increased by 19 percentage points (95% confidence interval of 11 to 27 

points.) The effects of the disclosure modifications in the native ad conditions was a 23-percentage-point 

increase (95% confidence interval of 14 to 32 points). These results were significantly different from zero. 

With regard to the native ad conditions, improved ad recognition derives entirely from a corresponding decrease 

in ads that were coded as not recognized as an ad, whereas, with regard to the search ad conditions, 

approximately three-fourths of it derives from a corresponding decrease in ads that were coded as unclear. 

The effects on ad recognition were substantially the same in the desktop and mobile ad conditions. Ad 

recognition increased by 21 percentage points in the desktop ad conditions (95% confidence interval of 13 to 28 

percentage points) and by 22 percentage points in the mobile conditions (95% confidence interval of 13 to 31 

percentage points.) See Figure 2 below. These results were significantly different from zero. 

F-2 
 

 

 

 

 

(I) 

e> 20 

C: 

~ 

.c: 

(.) 

-

C: 

I I 

·o I i o ------------------I--------------------------1----------------

i-20 ! f 

-40 

Desktop Mobile 

Ad Condition Subgroup 

Coded Response • Ad -+- No Ad • Unclear 

Figure 2 

Confidence Intervals for Estimated Effects of Disclosure Modifications on Ad Recognition 

for Desktop and Mobile Ad Subgroups 

Figure 2 depicts the 95% confidence intervals for the estimated effect of the disclosure modifications on ad recognition based 

on the statistical model for the desktop and mobile ad conditions. The red triangle, blue diamond, and green circle 

respectively depict the effect on the probability that: (1) an ad was recognized as an ad; (2) an ad was not recognized as an 

ad; and (3) it was unclear to the observer. 

III. Ad Recognition Analysis Using the FTC Paralegal Staff’s Coding Results 

As explained in Appendix D, two different coders, an expert consultant and either one of two FTC 

paralegal staff, independently coded whether or not participants recognized each ad condition as an ad. We 

used the expert consultant’s coding to conduct the ad recognition analysis discussed in the body of the report. 

The two coders for each ad condition were in agreement 79% of the time across ad conditions and participants. 

Thus, to examine the robustness of the ad recognition results, we also ran the regression using the alternative 

coding of the FTC paralegal staff. We found that when we used the FTC paralegal staff’s coding, the effects of 

the modified disclosures on the probability of participants recognizing an ad as an ad only fell slightly, from 21 

percentage points to 18.5 percentage points. See Figure 3 below. 

F-3 
 

 

 

 

 

Cl) 

e> 20 

C: 

.c: "' 

(J 

-C: 

·o 

a.. 0 --------------------------------------------------------------- (1) 

C) 

-

"' C: 

Cl) 

~ 

Cl) -20 

a.. 

-40 

Paralegel Coding Expert Coding 

Ad Condition Subgroup 

Coded Response • Ad -+- No Ad • Unclear 

40 

Figure 3 

Confidence Intervals for Estimated Ad Recognition Effects of Disclosure 

Baseline and Alternative Coding Schemes 

Figure 3 depicts the 95% confidence intervals for the estimated effects of the disclosure modifications on ad recognition based 

on the statistical model, using both the baseline coding data and the alternative data derived from the FTC paralegals’ coding. 

The red triangle, blue diamond, and green circle respectively depict the effect on the probability that: (1) an ad was 

recognized as an ad; (2) an ad was not recognized as an ad; and (3) it was unclear to the observer. 

IV. Eye Tracking Data Analyses 

We used a Poisson regression model to estimate the effects of the disclosure modifications on the length 

of time participants looked at particular ads.P7F 

8 

P Unlike a standard Poisson regression, however, we clustered 

standard errors at the participant and website levels. The model allowed us to estimate the average percent 

change in each variable due to the modifications. It measured changes in percentage terms, not in seconds. 

The statistical model was: 

mm nn log E(Yij) = ββDD + ααii + γγjj + ∑mm=1:8 δδmmOOiijj + ∑nn=1:3 ψψnnAAiijj, 

where E�Yij� is the expected value of the total visit duration Y for individual i and ad condition j. In addition, D 

is an indicator for whether the participant saw the original or modified condition, ααii are participant effects, γγjj nn are ad condition effects, OO is iijj

mm is an indicator variable that participant i saw ad condition j in order m, and AAiijj

8 See A. Colin Cameron & Pravin K. Trivedi, Microeconometrics: Methods and Applications, ch. 20 (2005). 

F-4 
 

 

 

 

 

 

 

 

 

 

 

an indicator for whether individual i had ad recognition code n for ad condition j. We report results for 

specifications both excluding and including the ad recognition variables. 

We examined 22 ad conditions, only those that appear on the main page,P8F 

9 

P to estimate changes in visual 

attention. Sixteen of the 22 ad conditions were search ads and six were native ads, and thus it is important to 

note that we had less power to test for statistical significance when we examined the native ad conditions. 

We found that the disclosure modifications significantly reduced the amount of time that participants 

spent looking at the ad conditions we tested. Overall, the disclosure modifications decreased the time 

participants fixated on these ads by 21%, with the 95% confidence interval ranging from -38% to -5%. See the 

red triangle confidence intervals in Figure 5 below. Examining the search and native ad condition groups 

separately, we also found that the effects were large in both groups. The amount of time participants fixated on 

the ads decreased by 21% in the search ad conditions (95% confidence interval ranging from -38% to -3%); it 

decreased by 26% in the native ad conditions (95% confidence interval ranging from -87% to 19%). The 

effects of the disclosure modifications is significantly different from zero when analyzed across all ad 

conditions and across the search ad conditions, but not when analyzed across the native ad conditions. 

We detected differences in the effects between the desktop and mobile ad conditions. Although in both 

condition groups participants looked less at the modified ads, the amount of time decreased by 25% in the 

desktop ad conditions (95% confidence interval ranging from -45% to -4%) and by 16% in the mobile ad 

conditions (95% confidence interval ranging from -45% to 12%). See the red triangle confidence intervals in 

Figure 5 below. Only the results for the desktop ad conditions were significantly different from zero. 

9 We did not expect to observe the same effect on visual attention with ads appearing on the click-into pages, 

given that, unlike the main pages which contain a mix of advertising and no-advertising content, the primary 

content on the click-into page is the actual ad or ad grouping that we were testing. Thus, we restricted our eye 

tracking analysis to ad conditions on the main pages. 

F-5 
 

 

10 

(I) -10 

C) 

C: 

.r. "' 

~ -30 

C: 

(I) 

~ (I) 

a. -50 

-70 

-90 

Desktop 

I 

Mobile Overall Search Native Ad 

Ad Condition Subgroup 

A Without Ad Recognition Controls + With Ad Recognition Controls 

30 

Figure 4 

Confidence Intervals for Estimated Effects of Disclosure Modifications on Average Amount of Time 

Spent Looking at Ads 

Figure 4 depicts the 95% confidence intervals for the estimated effects of the disclosure modifications on time spent looking at 

ads based on the statistical model, for all conditions and separately for the search, native ad, desktop, and mobile conditions. 

The red triangles depict the results not controlling for ad recognition, while the blue diamonds depict the results controlling for 

ad recognition. 

We further analyzed the eye tracking data by controlling for ad recognition in the regression analysis to 

evaluate whether the observed decrease in the amount of time participants fixated on the ad conditions was due 

to a larger proportion of them recognizing the ads as ads. If the effects on visual attention were primarily due to 

increased ad recognition, when we controlled for ad recognition we should have found a significant reduction in 

the effects of the disclosure modifications. When we applied this regression model, we did not find a reduction 

in the effects of the disclosure modifications but instead that the effects were slightly larger (21% reduction in 

effects without controlling for ad recognition and 25% reduction with controlling for ad recognition.) See 

Figure 5 below. Thus, the decreased time that participants fixated on the ad conditions does not seem due to an 

increased rate of ad recognition. 

F-6