• • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • •Understanding Contextual
Relevance and Efﬁciency
A Comparison of Contextual Intelligence Vendors
and Behavioral TargetingIntroduction
Digital advertising has evolved over the past decade to rely on targetingusers programmatically based on previous browsing activity. This common
practice of purchasing speciﬁc audiences or behavioral segments relies on
the web browser storing cookies for individual users. Given privacy-related
regulation (e.g., GDPR and CCPA) which enable users to more easily opt outof cookies, and decisions from large players like Apple and Google to phaseout third party cookies, advertisers are looking for different solutions.
Contextual Intelligence is another way of programmatically purchasing
digital advertising based on appropriate categories of relevance. For
example, a tech advertiser can choose to purchase advertising directly on
CNET or the technology section of the New York Times by purchasing
advertising from those sites directly at a high CPM. Alternatively, they canleverage Contextual Intelligence technologies to ﬁnd technology related
pages across thousands of websites at a much lower CPM.
In recent years, a number of companies have developed in the ContextualIntelligence space. Given the proliferation of Contextual Intelligence
vendors, it is challenging for advertisers to gauge the effectiveness of eachvendor without running a test campaign. We partnered with Dentsu Aegis
Network to run a rigorous experimental design with four live campaigns inorder to better understand the overall effectiveness of Contextual
Intelligence targeting and to benchmark four of the top Contextual
Intelligence vendors.

Background
What is Contextual Intelligence?
Vendors are able to analyze the data on a given webpage todetermine whether the content is relevant to a speciﬁc advertiser (e.g.,technology content for a tech brand, makeup or beauty content for abeauty brand, etc). In addition to ensuring that the content on thepage is relevant to the advertiser, these vendors also ensure that thecontent on the page is “brand safe” - ensuring that brands’ creative
messages do not appear adjacent to inappropriate content. Relevant
ads are served based on this real-time determination of the pages’
context.

P2GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I EuropeBehavioral Targeting
Behavioral targeting, on the other hand, leverages an individual’s past
online behavior, using cookies, to target across the web, regardless ofcontent, safety or relevance. For example, if a user visits CNET to lookat laptops, that same user could be served an ad for a laptop in asubsequent web surﬁng session even if she is on a baking site orreading her local news.

Study Objectives:
The goal of this research was to examine and compare behavioral
and contextual targeting methods in an effort to cost effectively
reach target audiences in relevant, brand safe environments.
Objective A: Evaluate cost efﬁciency of Contextual Intelligencetargeting vs. behavioral targeting
Objective B: Compare the accuracy of Contextual Intelligence
vendors in placing ads within relevant content

Methodology:
We partnered with Dentsu Aegis Network to test our hypotheses
using live campaigns. Four of Dentsu’s clients representing different
industries participated in this test: A major technology company,
Sephora, a big box retailer and a direct to consumer retailer. We ran
live campaigns for each of these four advertisers in May 2020 acrossfour Contextual Intelligence vendors, and we compared efﬁciency ofcontextual to behavioral targeting.
1 million impressions served for 2 weeks in May 2020, using thesame eligible inventory

P3GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

Campaign Set-Up: Each of the ﬁve campaigns (one behaviorally
targeted campaign and four campaigns each targeted by four
different Contextual Intelligence vendors) were set-up identicallyand were served 200K impressions each. The three other
Contextual Intelligence vendors campaigns ran through Xandr
Invest, and the behavioral targeting and GumGum Verity™
campaigns ran through GumGum’s platform.
Creative & Performance: The same ad unit (728 x 90) was run
across all ad lines, and no optimizations were conducted during
the campaign.
Cost: Each of the four Contextual Intelligence vendors inventory
was set to bid at a $3 CPM, plus any additional data costs.
Although GumGum Verity™ is not yet priced for externalmarketplaces, we added a data fee to keep the CPMs comparable
across Contextual vendors. For the behavioral targeting ad line,
the CPMs varied.
Brand Safety: The eligible inventory (across all vendors) waslimited to what GumGum Verity™ deemed ‘Safe’ inventory.
Again, the same pool of inventory was leveraged for all ﬁve adlines.
Oversight: An independent research consultant and digital media
pioneer (Dr. Michele Madansky) oversaw the design,
implementation, reporting, and analysis.
Campaign Metrics were captured using the following tools:
Part A - Cost Efﬁciency: Nielsen Digital Ad Ratings Reporting,
Xandr’s Buying Platform, Xandr Invest and GumGum, MOAT.
Part B - Content Relevance: Appen, a third party vendor
specializing in human annotation of randomly selected URLsP4GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

Visualizing the Approach for
Each Campaign
1M impressions were served for 2 weeks in May 2020.
A consultant, Dr. Michele Madansky, oversaw design,
implementation, reporting, and analysis.
Behavioral

Campaigns ran for 2 weeks andsetup identically. Constants:

Collected advertiser
inputs:• Campaign targetingparameters (demo,audience, contextual)• Creative (728x90)• White/blacklist, if
requiredContextual

GumGum Verity ad line
(200K impressions)
Vendor A ad line
(200K impressions)
Vendor B ad line
(200K impressions)

Brand safe GumGum Inventory

Behavioral ad line
(200K impressions)• Same creative• Same eligible inventory• No optimizations• $3 CPM, plus cost of data

Vendor C ad line
(200K impressions)

Reporting & Analysis
Part A• Human annotation of randomly
selected 400 URLs per ad line. 3
annotators per page, and 4th if
initial 3 made inconsistent
relevance determination
Part B• Nielsen Digital Ad Ratings
Reporting• Xandr• GumGum• MOAT

Campaign Set-Up:
Each advertiser provided us with a description of their intendedaudience (e.g., IT Decision Makers in companies with 100+ employees).
In addition, they provided us with contextually relevant categories fortheir campaign (e.g., computer networking, laptops, business IT). These
were the parameters that we utilized to set up their campaigns throughXandr Invest or GumGum.
In addition, the advertisers provided us with typical demographic
characteristics of their target audience (e.g., P25-54, M35+). This
information was not used during the campaign for targeting, but wasused after the campaign to help us understand how effective thetechnology was at reaching their desired demographics.

P5GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

It is important to note that the speciﬁcity with which one can target
contextually relevant content varies signiﬁcantly across the Contextual
Intelligence vendors. Two of the Contextual Intelligence vendors had
only broad categories to choose from. In contrast, GumGum Verity™ andone of the other vendor had many more granular classiﬁcations withincategories and aligned with the IAB 2.0 taxonomy.
As an example, when targeting technology content, GumGum Verity™
has nine categories to choose from, including: Technology & Computing,Business IT, Computer Networking, Desktops, Information and Network
Security, Laptops, Operating Systems, Antivirus Software, PC Support.
One of the other Contextual Intelligence vendors only has one option forthe overall technology category.
Measuring Content Relevance:
We hired a third party human annotation company, Appen, to measure
whether the content on which the ads were served was relevant to thecategory that had been pre-speciﬁed by the advertiser.
Step 1) Annotation company, Appen was sent n = 400 random urls frompages where ads ran from each Contextual Intelligence vendor as well
as from the behavioral targeted ad line (n = 2,000 urls total/campaign).
Note: conﬁdence interval for results of +/- 1%.
Step 2) Human annotators were instructed to evaluate whether a page
contained content that pertains to the appropriate category“Is this article related to any/all of these categories?”
Computer Networking, Operating Systems, Desktops, Information andNetwork Security, Laptops, Business IT.
Step 3) Three humans evaluated each page to determine contextual
relevance (yes, no).
If all three annotations were not the same, additional humans evaluatedpage until there was a consensus of 3 annotators. It’s also worth
pointing out that the annotation company did not know which vendors
were represented by each tab of the spreadsheet.

P6GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I EuropeOutput: Each page was designated as contextually relevant (Yes) ornot contextually relevant to the campaign (No). These data points
were aggregated across each ad line and advertiser.

Key Findings:
Part A: Relative cost efﬁciency of behavioral targeting vs. contextualtargeting
In order to evaluate the cost efﬁciency of Contextual Intelligence
vendors and behavioral targeting, we looked at three different
measures of efﬁciency. And while we recognize that each of these
metrics may not be how advertisers evaluate their campaigns, thesethree metrics do allow us to fairly and accurately measure efﬁciencyacross Contextual Intelligence vendors and behavioral targeting.
#1 - In-Demo eCPM: Cost per thousand in-demo impressions#2 - CPC: Cost per click#3 - vCPM: Cost per thousand that were viewable under the MRC
standard (50% of pixels in-view for at least 1 continuous second)
Cost Efﬁciency: Cost per In-demo Impression (eCPM)
Although we understand that demographic targeting may not be how
many advertisers plan their campaigns, we did want to answer thequestion of whether the ads reached their intended targets. We did
not use any demographic targets when setting up the contextual
targeted and behaviorally targeted campaigns. However, in order toassess how well the campaigns mapped to the advertisers’ primarydemographic, we analyzed the percentage of in-demo impressions
from Nielsen Digital Ad Ratings Reporting. This allowed us to assess
whether the campaigns reached their intended audiences in a
uniform way. Taking into account CPMs, Contextual Intelligence
Vendor impressions cost 29% less than behaviorally targeted
impressions, and GumGum Verity™ impressions cost 36% less than
behaviorally targeted impressions.

P7GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

$6.85
$4.86

$4.36

Cost Efﬁciency: Cost per Click (CPC)
GumGum’s platform auto optimizes for clickthrough rates. Since the othercontextual vendors’ impressions were not automatically optimized for CTR
given study limitations, we only compared behavioral targeted ads (whichran through GumGum’s platform) to GumGum’s Verity™ Contextual
Intelligence Platform. Even with the auto optimization in place, GumGum’sVerity™’s cost per click was 48% lower than behaviorally targeted ads.

CPC
$2.00

$1.50

$1.53
$1.00

$0.80
$0.50

$0.00
Behavioral

P8GumGum VerityGumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

Cost Efﬁciency: Cost per Viewable Impression (vCPM)
The GumGum platform also auto optimizes for viewability. As a result, weare also comparing behaviorally targeted ads vs. GumGum Verity™ torepresent contextual targeting. We analyzed the percentage of viewable
impressions as deﬁned by the MRC (50% of pixels in-view for at least 1
continuous second). After taking into account CPMs for each ad line,
Verity™’s cost per viewable impression was 41% less than behaviorally
targeted ads.

$5.85

$3.44

For all three metrics, eCPM, CPC and vCPM, Contextual targeting wasmore cost efﬁcient than behaviorally targeted ads.
Part B: Accuracy of Contextual Intelligence vendors
Relevant Page Percentage: The percentage of pages that are relevantto the contextual targeting goals of the campaign. Based on themethodology we described above using 3+ human annotators, wefound that:
71% of pages GumGum Verity™ selected were relevant, outperforming
Contextual Intelligence vendors overall by 1.7X.

P9GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

71%

47%

42%

27%

25%

Results Summary
Part A
Contextual targeting was more efﬁcient than behavioral targeting onaverage across campaigns when looking at three different measures forcost efﬁciency (CPC, vCPM and in-demo eCPM)
Higher CPMs for behaviorally targeted ads certainly factor into thisincreased efﬁciency, but we also hypothesize that increased relevancealso improves campaign engagement.
Part B
GumGum Verity™ had the highest percentage of relevant pages across all
four Contextual Intelligence vendors.

P10

GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

About GumGum Verity™:
GumGum Verity™’s performance is fueled by a decade of collecting
labeled data, training and reﬁning our algorithms.
Advanced Machine Learning: Verity™’s Machine learning allows
computers to understand a web page like a human would. Reading
between the lines, Verity™ comprehends the holistic and subtle meaningof all text and imagery in an article. Verity™ continuously improves withmore training data, making predictions and decisions without being
explicitly programmed to do so. Some other Contextual Intelligence
vendors rely exclusively on keywords. This technology does not allow thealgorithms to adapt as well as machine learning.
Full-Page Analysis using Computer Vision + Natural Language
Processing: GumGum Verity™ utilizes computer vision to understandimages and natural language processing to understand text, bothleveraging deep neural networks trained on its proprietary data sets.
Combining GumGum's decade of experience in computer vision, with thebest NLP, no other vendor can match the accuracy of Verity™.

About GumGum
GumGum is a global technology and media company specializing inContextual Intelligence. For over a decade, GumGum has applied provenmachine learning expertise to extract value from digital content for theadvertising and sports industries. For advertisers, GumGum offers a fullsuite of pioneering future-proof solutions. The company’s contextual
advertising engine, Verity™, comprehends the meaning of text, images
and video online, allowing marketers to safely and precisely place ads
where people are most engaged. Combining contextual targeting andbrand safety intelligence with proprietary high-impact ad formats,
GumGum’s advertising solutions deliver industry leading efﬁciency,accuracy and performance.
For more information, contact: contextualadvertising@gumgum.com

P11

GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe

About Dentsu Aegis Network
Part of Dentsu Inc., Dentsu Aegis Network is made up of ten global
network brands - Carat, Dentsu, dentsu X, iProspect, Isobar,
dentsumcgarrybowen, Merkle, MKTG, Posterscopeand Vizeum andsupported by its specialist/multi-market brands. Dentsu Aegis Network
is innovating the way brands are built for its clients through its
best-in-class expertise and capabilities in media, digital and creative
communications services.
Offering a distinctive and innovative range of products and services,
Dentsu Aegis Network is headquartered in London and operates in 145
countries worldwide with more than 40,000 dedicated specialists.
www.dentsuaegisnetwork.com

P11

GumGum HQ I 1314 7th St, Santa Monica, CA 90401 I (844) 522-7270 I gumgum.com
US I Canada I UK I Japan I Europe