T HES TAT EOF
DE E PFA K E S
L ANDSC APE, T HRE ATS , AND IMPAC T

September 2019www.deeptracelabs.com

September 2019©2019 Deeptrace
Contact: info@deeptracelabs.com
Authors: Henry Ajder, Giorgio Patrini, Francesco Cavalli & Laurence Cullen
Graphic design: Eleanor WinterCover image: Joel Filipe
Cite: The State of Deepfakes: Landscape, Threats, and Impact, Henry Ajder, Giorgio Patrini,
Francesco Cavalli, and Laurence Cullen, September 2019.

About Deeptrace
Deeptrace is an Amsterdam-based cybersecurity company providing
deep learning and computer vision technologies for the detection
and online monitoring of synthetic media. Our mission is to protect
individuals and organizations from the damaging impacts of AIgenerated synthetic media.
Deeptrace also publishes Tracer, a curated weekly newsletter covering
key developments with deepfakes, synthetic media, and emerging
cyber threats.

Acknowledgements
We would like to thank the following people for providing valuable
feedback and quotes for this report: Curtis Barnes, Tom Barraclough,Danielle Citron, Jack Clark, Adam Dodge, Sam Gregory, Gabi Ivens,
Hamish Ivey-Law, Jason Mancuso, Samantha North, Rodolfo Rosini,Paul Scharre, Gaurav Tanwar, Leroy Terrelonge, and Eleanor Winter.

Table of Contents
The State of Deepfakes: An Overview

1

The Commodification of Deepfakes

3

Deepfake Pornography

6

Politics & Cybersecurity

9

Concluding Remarks

15

Appendix16

DEEPTRACE

Foreword
The rise of synthetic media and deepfakes isforcing us towards an important and unsettling
realization: our historical belief that video
and audio are reliable records of reality is nolonger tenable.
I choose the word “historical” as this belief isa coincidence of how technology has evolved.
Still today, we trust a phone call from a friend
or a video clip featuring a known politician,simply based on the recognition of their
voices and faces. Previously, no commonly
available technology could have syntheticallycreated this media with comparable realism,
so we treated it as authentic by definition.
With the development of synthetic media anddeepfakes, this is no longer the case. Every
digital communication channel our society
is built upon, whether that be audio, video, oreven text, is at risk of being subverted.
Since its foundation in 2018, Deeptrace has
been dedicated to researching deepfakes’
evolving capabilities and threats, providing
crucial intelligence for enhancing our detection
technology. In this report we share the insightsfrom our most comprehensive mapping of
the deepfake landscape to date, revealing
deepfakes’ real-world impact. In doing so, we
provide an expert overview of the current stateof deepfakes, cutting through some of thehyperbole surrounding the topic. The findings
we present here are grounded in independently
sourced data, accompanied by insights fromleading experts in this area.
Our research revealed that the deepfake
phenomenon is growing rapidly online,with the number of deepfake videos almost
doubling over the last seven months to 14,678.
This increase is supported by the growing
commodification of tools and services thatlower the barrier for non-experts to create
deepfakes.
Perhaps
unsurprisingly,
weobserved a significant contribution to thecreation and use of synthetic media tools fromweb users in China and South Korea, despite
the totality of our sources coming from theEnglish-speaking Internet.
Another key trend we identified is theprominence of non-consensual deepfake
pornography, which accounted for 96% of

www.deeptracelabs.com

the total deepfake videos online. We also
found that the top four websites dedicated to
deepfake pornography received more than 134
million views on videos targeting hundreds offemale celebrities worldwide. This significant
viewership demonstrates a market for websites
creating and hosting deepfake pornography, atrend that will continue to grow unless decisive
action is taken.
Deepfakes are also making a significant impact
on the political sphere. Two landmark cases
from Gabon and Malaysia that received minimal
Western media coverage saw deepfakes linked
to an alleged government cover-up and apolitical smear campaign. One of these cases
was related to an attempted military coup,
while the other continues to threaten a highprofile politician with imprisonment. Seen
together, these examples are possibly themost powerful indications of how deepfakes
are already destabilizing political processes.
Without defensive countermeasures, theintegrity of democracies around the world are
at risk.
Outside of politics, the weaponization of
deepfakes and synthetic media is influencing
the cybersecurity landscape, enhancing
traditional cyber threats and enabling entirely
new attack vectors. Notably, 2019 saw reports ofcases where synthetic voice audio and imagesof non-existent, synthetic people were used toenhance social engineering against businesses
and governments.

THE STATE OF DEEPFAKES – Foreword
Deepfakes are here to stay, and their impact isalready being felt on a global scale. We hope
this report stimulates further discussion on
the topic, and emphasizes the importanceof developing a range of countermeasures toprotect individuals and organizations from theharmful applications of deepfakes.
Giorgio Patrini
Founder, CEO, and Chief Scientist

DEEPTRACEThe State of Deepfakes: An Overview
Since emerging in late 2017 the phenomenon of deepfakes has developed rapidly, both in termsof technological sophistication and societal impact. In this section, we provide an overview of
the current state of deepfakes by analyzing their proliferation and content.1

Total number of deepfake videos online

14,678percentage of deepfake
videos online bypornographic andnon-pornographic
content96%

Deepfake pornography accounts for asignificant majority of deepfake videos online,
even as other forms of non-pornographic
deepfakes have gained popularity.

4%

Total number of video views across top four
dedicated deepfake pornography websites

134,364,438
1

Our findings revealed that the total numberof deepfake videos online is rapidly increasing,with this measurement representing analmost 100% increase based on our previous
measurement (7,964) taken in December 2018.

Despite being a relatively new phenomenon(the earliest of these websites was registeredin February 2018), deepfake pornography has
already attracted a large viewership on the
top four dedicated deepfake pornography
websites alone.

The appendix details how we collected the data presented in this reportwww.deeptracelabs.com

1

THE STATE OF DEEPFAKES – The State of Deepfakes: An Overview

Demographic breakdown of deepfake videos from top
five deepfake pornography websites and top 14 deepfake
YouTube channels
We analyzed the gender, nationality, and profession of subjects in deepfake videos from the top
5 deepfake pornography websites, as well as the top 14 deepfake YouTube channels that host
non-pornographic deepfake videos.

Genderfemale39%
female100%

Deepfakepornography
websitesotherIndia 13%
3%
CanadaDeepfake6%
pornographyUKwebsites
12%
South Korea
25%

news & media
1%

Deepfakepornography
websites

entertainment
99%

2

male
61%

Deepfake pornography is a phenomenon
that exclusively targets and harms women.
In contrast, the non-pornographic deepfake
videos we analyzed on YouTube contained amajority of male subjects.

other
5%

Nationality

YouTubechannels

South
Korea
2%
Canada
4%
USA
41%

UK
14%

Taiwan2%

YouTubechannelsUSA73%

business owner
politician 2%
12%news &
media
5%
YouTubechannels

entertainment81%

We found that over 90% of deepfake videos on
YouTube featured Western subjects. However,
non-Western subjects featured in almost a thirdof videos on deepfake pornography websites,with South Korean K-pop singers making up aquarter of the subjects targeted. This indicatesthat deepfake pornography is an increasingly
global phenomenon.

Profession
All but 1% of the subjects featured in deepfake
pornography videos were actresses andmusicians working in the entertainment
sector. However, subjects featuring in YouTube
deepfake videos came from a more diverse
range of professions, notably including
politicians and corporate figures.

DEEPTRACEThe Commodification of Deepfakes“Based on the rate of AI progress, we can expect deepfakes to become better, cheaper,and easier to make over a relatively short period of time. Governments should invest
in developing technology assessment and measurement capabilities to help themkeep pace with broader AI development, and to help them better prepare for the
impacts of technologies like this.” 2Jack Clark, Policy Director of OpenAI, and author of Import AI
The term deepfake was first coined by the Reddit user u/deepfakes, who created a Reddit forumof the same name on November 2nd 2017. This forum was dedicated to the creation and useof deep learning software for synthetically faceswapping female celebrities into pornographic
videos. Since Reddit’s removal of /r/Deepfakes on February 7th 2018, deepfakes have become
increasingly commodified as new deepfake forums, tools, and services have emerged.

The growth of GANs
Synthetic audiovisual media can be generated
with a variety of deep learning techniques.
Currently, the most popular of these techniques
is the Generative Adversarial Network (GAN) due
to its flexible applications and realistic outputs.
The graph below measures the research
output on GANs from their invention in 2014.

This provides an indirect indication of how fast
GAN quality is improving and the rate thatnew capabilities for real-world applications arebeing developed. In the context of this section,it also indicates how this research is being
incorporated into technology accessible to awider audience outside academia. 3

Academic papers mentioning GANS in
title/abstract published per year on the arXiv 31207

papers published

1250
1000

932

750
469

500
250
0

3

9

2014

2015

74
2016

2017

year
2
3

2018

2019
(annualized)

Clark, Jack, private correspondence, September 23 2019arXiv is an online archive of scientific and mathematical research papers maintained by Cornell University.

www.deeptracelabs.com

3THE STATE OF DEEPFAKES – The Commodification of Deepfakes

Deepfake Creation Communities & Forums
Deepfake creation communities and forums are a key driving force behind the increasing
accessibility of deepfakes and deepfake creation software. Many of these creation communities
and forums provide an entry point for people interested in creating deepfakes, and facilitate
collaboration between more experienced creators.

20
20

deepfake creation community
websites and forums
deepfake creation community
websites and forums
non-unique members(from sources that disclosed
membership numbers)

non-unique members(from sources that disclosed
membership numbers)

95,791
95,791

We found the majority of creation communities
and forums were located on deepfake
pornography websites and established forumbased websites including Reddit, 4chan, 8chan,
and Voat. Some of these websites such as 4chan
and 8chan are notorious for hosting illegal andunethical activity.
13 deepfake creation communities and forums
disclosed their total membership, with theseaccounting for almost 100,000 members
without the other seven whose membershipwas not accessible.

Github Stars
Open-source code repositories are the backbone of modern machine learning projects, anddeepfake creation tools are no exception. The faceswap source code from the anonymous /r/
Deepfakes creator was donated to the open-source community and uploaded on Github. Since
then, development has been driven by many project forks, with programmers contributing to
improved quality, efficiency, and usability of these new code libraries.
The chart below measures the popularityof the two best maintained deep learning
faceswap projects (project names omitted) by“Github stars”. Both projects show an organic

faceswap project A

growth in popularity since their initial upload,with faceswap project A’s 20,000 stars rivaling
the popularity of other industrial-level opensource projects.

faceswap project B

25000

Github stars

20000
15000
10000
5000

20
1

71
20 2
18
-0
20 1
18
-0
20 2
18
-0
20 3
18
-0
20 4
18
-0
20 5
18
-0
20 6
18
-0
20 7
18
-0
20 8
18
-0
20 9
18
-1
20 0
18
-2
20 1
18
-1
20 2
19
-0
20 1
19
-0
20 2
19
-0
20 3
19
-0
20 4
19
-0
20 5
19
-0
20 6
19
-0
7

0

4

DEEPTRACE

Deepfake Computer Apps & Services
Beyond research and code libraries, commodification is also about the growing accessibilityof deepfake creation tools for a non-technical audience. Here we outline the three types ofdeepfake tools we identified during our research, with each providing a different approach tothe creation and distribution of deepfakes.

Computer Apps
Deepfake computer apps are downloadable
tools or graphical user interfaces (GUIs) thatare used for creating deepfakes. Most of theseapps provide ‘faceswapping’ capabilities, but
we also identified one app for synthetic voicecloning. Many of these apps require knowledgeof programming and a powerful graphics
processor to operate effectively, making them
unsuitable for amateurs. However, several
detailed tutorials have been created thatprovide step-by-step guides for using the mostpopular deepfake apps, and recent updates
have improved the accessibility of several GUIs.

Service Portals
Service portals function as online businesses forgenerating and selling custom deepfakes. In
most cases, this involves an individual uploading
photos or videos of their chosen subjects to theonline platform and receiving the deepfake
once it has been generated. On one service
portal we identified, this required 250 photos of
the target subject and two days of processing
to generate the deepfake. The prices of these
services vary depending on the quality andduration of the video requested, but can cost as
little as $2.99 per deepfake video generated.

Marketplace Services
Marketplace services refer to individual deepfake
creators who advertise their services on forums
and online marketplaces. Our research foundmarketplace services ranging from bespoke
faceswap videos for $30 to custom voice cloning
for $10 per 50 words generated.

www.deeptracelabs.com

5THE STATE OF DEEPFAKES – Deepfake Pornography

Deepfake Pornography“Deepfake technology is being weaponized against women by inserting their faces
into porn. It is terrifying, embarrassing, demeaning, and silencing. Deepfake sex
videos say to individuals that their bodies are not their own and can make it difficult
to stay online, get or keep a job, and feel safe.”4
Danielle Citron, Professor of Law, Boston University, and author of Hate Crimes in Cyberspace
Deepfake pornography is by far the most prevalent kind of deepfake currently being created andcirculated. The damaging effects of deepfake pornography have already affected a significant
number of women, including celebrities and private individuals. In this section, we outline thedeepfake pornography ecosystem and present a case study on the development of the deepfake
pornography creation app DeepNude.

The rise of deepfake pornography

8

We measured the online presence of deepfake
pornography by tracking the number of videos
that have been uploaded, and the number of
websites that hosted these videos.

of the top 10 pornography
websites host deepfakes

9

dedicated deepfake
pornography websites

mainsteam
pornography
websites
6%

4

6

deepfakepornography
videos by hostwebsite typededicated
deepfakepornography
websites
94%

Citron, Danielle, private correspondence, August 16 2019.

Our
research
revealedthat
deepfakepornography has a significant online presence
across several different websites. These websites
fell into two categories: dedicated deepfake
pornography websites and mainstream
pornography websites.
We found that the deepfake pornography
ecosystem is almost entirely supported bydedicated deepfake pornography websites,
which host 13,254 of the total videos we
discovered.
By
contrast,
mainstream
pornography websites only hosted 802 videos.
The number of these dedicated deepfake
pornography websites suggests that deepfake
pornography could represent a growing
business opportunity, with all these websites
featuring some form of advertising.

DEEPTRACE

10 individuals most frequently targeted by deepfake pornography
We chose not to publish the names of these
women given the sensitive nature of the data, butrather specify their nationality and profession.
Of the 10 most frequently targeted individuals
we identified, the majority were actresses from

Western countries. However, the second andthird most frequently targeted individuals, aswell as the most frequently viewed individual,
were South Korean K-pop singers.

millions of views

video features
Britishactress

5.8South Korean
musician

3.1

134

Americanactress

126

1.6

Britishactress118

1.4

Australianactress

113

Americanactress

106

Israeliactress

105

Americanactress

104

South Korean
musician

103

4.3

2.4
1.9
1.1
4.4

www.deeptracelabs.com

5167South Korean
musician

5.9

10

257

0

0

100

200

300

7

THE STATE OF DEEPFAKES – Deepfake Pornography

A Case Study: DeepNude
DeepNude is a computer app that enables
users to ‘strip’ photos of clothed women. The
app uses deep learning image translation
algorithms that have been tuned tosynthetically remove clothes from images ofwomen, and generate naked parts of theirbody that were previously covered. Thesealgorithms cannot perform similar translations
on images of men, having been specificallytrained on images of women. DeepNude’s
release illustrates the dangers of automatedtools for creating deepfake pornography
becoming increasingly accessible.

LaunchResurrection

DeepNude’s website launched on June 23rd
2019, featuring a demo and download options
for Windows and Linux computer apps. The app
could be downloaded for free, but generated
images were obscured by a large watermark.
To remove this watermark, users had to pay$50 for the licensed version. Images generated
with this licensed version still contained asmall watermark designating the image as“fake”. Each image took the app 30 seconds
to generate.

Despite the creators of DeepNude taking theofficial website offline, the software continues tobe independently repackaged and distributed
through various online channels, such as opensource repositories and torrenting websites.
In addition, two new service portals opened
offering an allegedly improved version of
DeepNude, with charges ranging from $1 perphoto to $20 for a month’s unlimited access.
Likely having recognized the app’s businesspotential, the original creators put DeepNude
up for sale on July 19th for $30,000 via an online
business marketplace, where it sold to an
anonymous buyer.

Takedown
DeepNude saw a surge in interest following
critical coverage by journalist Samantha Colefor the online publication, Motherboard on June
27th5. This overwhelmed DeepNude’s servers
with traffic and download requests, causing
the website to go offline less than 24 hours
after Cole’s article was published. According
to DeepNude, the website received 545,162
visits and 95,464 active users in June, with the
majority of this activity likely occurring in this 24
hour period. The creators subsequently statedthey had “greatly underestimated [the volume
of download] requests” and that they would notrelease any further versions of the app, claimingthat “the world is not ready for DeepNude”.

58

Losing control
The moment DeepNude was made available
to download it was out of the creators’ control,and is now highly difficult to remove fromcirculation. The software will likely continue tospread and mutate like a virus, making a popular
tool for creating non-consensual deepfake
pornography of women easily accessible anddifficult to counter.

Cole, Samantha. “This App Uses Deepfake Technology Solely to Objectify Women” Motherboard. June 27 2019.

DEEPTRACE

Politics & Cybersecurity“We’ve already seen so-called “shallowfakes” circulated online with the attempt todistort political discourse or delegitimize politicians, and even selective editing has
been used to falsely represent how a politically-charged event occurred. High-quality
AI-manipulated video ups the stakes considerably. It is only a matter of time before
deepfakes are used in an attempt to manipulate elections.” 6Paul Scharre, Director of Technology & National Security, CNAS

Much of the media attention surrounding deepfakes has focused on their potential to undermine
democratic processes and enhance cyberattacks against individuals and businesses. In thissection, we outline several cases where deepfakes and the related phenomenon of shallowfakes
have had a disruptive impact on the political sphere. We also detail cases where synthetic images
and synthetic voice audio have allegedly been used to enhance social engineering and fraud.

Buzzfeed News & Monkeypaw Studios (2018)

6

Scharre, Paul, private correspondence, August 8 2019.

www.deeptracelabs.com

9THE STATE OF DEEPFAKES – Politics & Cybersecurity

The destabilizing political impact of deepfakes
We have provided technical consultation on two cases where deepfakes have played a significant rolein creating or exacerbating political crises. Both are powerful examples of how growing awarenessof deepfakes is already damaging political discourse, and undermining the perceived objectivity ofrecorded video in politically charged contexts.

Gabon: Accelerating politicalunrestIn late 2018 there was intense speculation about
the health of the Gabonese President Ali Bongo,
who had been absent from public life for several
months. In an attempt to end the speculation,
the government released a video of Bongodelivering a traditional New Year’s address.
However, Bongo’s unusual appearance in thevideo led many on social media, including
Gabonese
politician
Bruno
Moubamba,
to declare that the video was a deepfake,confirming their suspicion that the government
was covering up Bongo’s ill health or death. A
week after the video’s release amid growing
unrest, members of Gabon’s military launched
an attempted coup against the government.
In a video announcing the coup, the military
mentioned the video’s odd appearance as
proof that something was wrong with the
President.7 Subsequent forensic analysis

Mother Jones (2019)

did not find signs that the video had been
manipulated, and Bongo has since made publicappearances following what the government
stated was a severe stroke in August 2018.

Malaysia: Sex scandal

The Straits Times (2019)7
8

10

A political scandal emerged in June 2019
surrounding a sex tape allegedly featuring
the Malaysian Minister of Economic Affairs
Azmin Ali and a rival minister’s male aide.8
Same-sex sexual activity is illegal in Malaysia,
with politicians having previously been
controversially imprisoned on sodomy charges.
While the aide claimed the video was real
and was subsequently arrested, Ali and his
supporters, including the Malaysian Prime
Minister, argued that the video was a realistic
deepfake made to sabotage his political career.
However, international experts could not findany signs that the video had been manipulated.
As of mid-August 2019, it is still unclear whether
Ali will face criminal charges over the video.

Breland, Ali. “The Bizarre and Terrifying Case of the “Deepfake” Video that Helped Bring an African Nation to the
Brink” Mother Jones. March 15, 2019.
Blakkarly, Jani. “A gay sex tape is threatening to end the political careers of two men in Malaysia” SBS News. June 17
2019.

DEEPTRACE

Shallowfakes
The term Shallowfakes was first coined by Sam Gregory of the human rights organization WITNESS,
and refers to videos that have been manipulated with basic editing tools or intentionally placed outof context.9 The Washington Post’s guide to manipulated video provides three specific categories forthese kinds of shallowfake manipulations: 10›› Missing context – misrepresenting a video or isolating it from its original context;›› Deceptive editing – omitting passages of a video or inserting new ones;›› Malicious transformation – doctoring or manipulating a video’s content.
Shallowfakes are typically designed to exploit an individual’s cognitive biases which can result indamage to a target person’s reputation even if the fake is of a low quality.

Nancy Pelosi: Manipulated voiceaudio
A recent high profile shallowfake involved amanipulated video of US Speaker of the Houseand Democrat Congresswoman Nancy Pelosi.11
In the video that was shared on May 23rd 2019,
Pelosi’s speech had been slowed down, making
it sound like she was slurring her words. The
edited version of the video went viral on social
media and was retweeted by the official Twitter
account of US President Trump, receiving over
6.3m views as of July 31st 2019.12 On a popular
Facebook page, the video received over 2.2m
views in the 48 hours following its initial upload,with commenters calling Pelosi “drunk” and a“babbling mess”.13

New York Times (2019)

Jim Acosta: Manipulated body
movement

The Washington Post (2018)

9
10
11
12
13

Shallowfakes have also been falsely cited as
evidence to justify controversial political actions.
In this case, CNN’s correspondent Jim Acosta
had his White House press pass revoked on
November 7th 2018, following an interaction
where an intern attempted to take a microphonefrom his hand after a tense exchange with the
President. In edited footage of the event shared

Gregory, Sam. “Deepfakes will challenge public trust in what’s real. Here’s how to defuse them.” Defusing Disinfo.
February 19, 2019.
Ajaka, Nadine, Samuels, Elyse, and Kessler, Glen. “Seeing Isn’t Believing The Fact Checker’s guide to manipulated
video” The Washington Post. June 25 2019.
Mervosh, Sarah. “Distorted Videos of Nancy Pelosi Spread on Facebook and Twitter, Helped by Trump” The New YorkTimes. May 24 2019.
Monje Jr, Carlos. “Twitter letter to Chairman Schiff”. Twitter July 31 2019.
Harwell, Drew. “‘Sexist’ videos edited to make Nancy Pelosi look drunk go viral, with Trump’s help” The Independent.
May 24 2019.

www.deeptracelabs.com

11THE STATE OF DEEPFAKES – Politics & Cybersecurityby the White House Press Secretary, the video
was sped up, making it appear that Acosta had
aggressively blocked the intern from taking the
microphone from him with his arm.14 This wascited as “inappropriate behavior” and the sole
reason for his press pass being revoked. However,reporting by The Washington Post’s Drew
Harwell and other news organizations revealed
that the video had been manipulated, leading
to Acosta’s press pass being reinstated roughly
two weeks after the incident first occurred.

Political Satire & Art
The impact of deepfakes on the political sphere is not solely restricted to malicious and deceptive
disinformation campaigns, but also includes the creation of new forms of political satire and art. One
famous early example of this use of deepfakes was seen with Buzzfeed’s and director Jordan Peele’s“Deepfake Obama PSA”, a deepfake of Barack Obama that warned of the dangers posed by thetechnology.15 As awareness of deepfakes has grown, so have efforts to use them as part of politicaland social commentary.

SpectreSpectre is a project by the art collective
Brandalism that involves several deepfake
videos of prominent public figures, including
Mark Zuckerberg, Kim Kardashian, andDonald Trump.16 Each deepfake syntheticallysynchronized the subject’s lip movements
to match a voice actor imitating their voice,speaking about how the mass collection of
personal data and other technologies havebrought them power or wealth.
Upon Spectre’s release, the deepfakes had an
immediate viral impact across a variety of social
media platforms. However, the project did notpresent the deepfakes as real, but as an artistic
device to provoke discussion of political andsocial issues.

14
15
16

12

Harwell, Drew. “White House shares doctored video to support punishment of journalist Jim Acosta” The Washington Post. November 8 2018.
Newman, Craig. “How To Spot A Deepfake Like The Barack Obama–Jordan Peele Video” Buzzfeed News. April 17 2019.
Posters, Bill. “Gallery: Spectre launches (Press Release)”. Bill Posters. May 29 2019.

DEEPTRACE

Cybersecurity
Aside from the politics, malicious applications of deepfakes and synthetic media are also changing
the cybersecurity landscape by enhancing traditional cyberthreats and enabling entirely new attack
vectors. In 2019 we have seen several cases where deepfakes have been deployed to enhance social
engineering and fraud.

Enhancing fake digital identities: fraud, infiltration and espionage
We observed two cases where realistic synthetic photos of non-existent people were used on fakesocial media profiles, in an attempt to deceive other users and extract information.

Deceiving Tesla short sellers
In March 2019, accounts posing as a senior
Bloomberg journalist “Maisy Kinsley” connectedwith over 195 people on LinkedIn, and followeda large number of Tesla short sellers on Twitter. 17
Several of these short sellers claimed that theaccount had contacted them in an attempt
to extract personal information. However,Bloomberg stated they did not employ anyone
called Maisy Kinsley, and the account’s profile
picture contained visual anomalies consistent
with synthetically generated images. Bothaccounts were subsequently removed by
LinkedIn and Twitter.

Espionage
A LinkedIn account posing as “Katie Jones”,
a researcher from a top US think tank, wasbelieved by experts to be part of a foreign
spying operation.18 While the account only had
52 connections at the time of its removal by
LinkedIn in June 2019, it had connected withmembers of staff of several government officials
between March and April 2019. Expert analysis
identified several visual anomalies that indicatedthat the image was synthetically generated.

17
18

Fleishman, Glenn. “How to spot the realistic fake people creeping into your timelines”. Fast Company. April 30 2019.
Satter, Raphael.”Experts: Spy used AI-generated face to connect with targets” AP News. June 13 2019.

www.deeptracelabs.com

13

THE STATE OF DEEPFAKES – Politics & Cybersecurity

Synthetic voice impersonation and fraud
There have been several reported cases where synthetic voice audio has allegedly been used todefraud companies.19 While no concrete evidence has been provided to support claims that theaudio was synthetic, the cases illustrate how synthetic voice cloning could be used to enhanceexisting fraud practices against businesses and individuals.

CEO impersonation
A report by French insurance company Euler
Hermes stated that one of their clients, an
unnamed UK energy firm, was the victim ofa fraud attack using synthetic voice audio in
March 2019. According to Euler Hermes, thecybercriminals used synthetic voice audio thatimpersonated the CEO of the firm’s German
parent company, accurately replicating his
accent and speaking style. Recognizing thevoice and believing it to be real, the CEO of the

19

14

British firm complied with a request to wire
$243,000 to a Hungarian supplier. These funds
were then moved from the Hungarian accounts
to several other locations. Euler Hermes covered
these losses as part of the company’s fraud
insurance policy. No further evidence such asrecordings of the call have been released to provethe voice audio was synthetic, and at the time ofwriting the perpetrators have not been identified.

“Fake Voices ‘help cyber-crooks steal cash’” BBC News. July 8 2019.

DEEPTRACE

Concluding Remarks
Deepfakes pose a range of threats, many of which are no longer
theoretical. In this report we have identified five key conclusions on
the current state of deepfakes:››

Deepfake creation technologies and tools are being
commodified through a growing number of communities,
computer apps, and services;››

The online presence of deepfake videos is rapidly
expanding, with the vast majority of these videos involving
pornographic content;››

Deepfake pornography is a global phenomenon supported
by a significant viewership across several dedicatedwebsites, with women exclusively being targeted;››

Awareness of deepfakes alone is destabilizing politicalprocesses by undermining the perceived objectivity ofvideos featuring politicians and public figures;››

Deepfakes are providing cybercriminals with new
sophisticated capabilities to enhance social engineeringand fraud.

These conclusions are drawn from our analysis of the deepfake
landscape as it currently lies. However, the speed of the developments
surrounding deepfakes means this landscape is constantly shifting,with rapidly materializing threats resulting in increased scale andimpact. It is essential that we are prepared to face these new challenges.
Now is the time to act.

www.deeptracelabs.com

15

THE STATE OF DEEPFAKES – Appendix

Appendix
The quantitative data collection detailed here took place between June 1st 2019
and July 31st 2019. Examples and case studies were collected up until August
31st 2019.

Data Sensitivity
Open-source research on sensitive data is a delicate matter that we take very
seriously. Providing methodological transparency on some of our sources would
involve publicizing tools, services, and websites that are legally ambiguous or have
explicitly unethical applications. Accordingly, we have decided to censor the names
and URLs of these sources, particularly when related to deepfake pornography.

Mainstream pornography websites
Sourceswww.Aelieve.com, 10 most popular mainstream pornography websites in terms ofuser traffic.

Methodology
We identified the 10 most popular porn websites in terms of web traffic, as indicatedby Aelieve.com. We then used the internal site search engine with the keyword“deepfake(s)” and recorded the number of videos returned. One of these websites
had disabled searches for the keyword “deepfake”, but we were still able to detect
several deepfake videos on this website by conducting a manual search.

Dedicated deepfake pornography websites
Sources
Google search, redirect links from deepfake communities and forums.

Methodology
We identified dedicated deepfake pornography websites using Google searchwith the key phrases “deepfake porn portal” and “deepfake porn website”, alongwith redirect links we found on deepfake forums that were not indexed through
Google searches. All dedicated deepfake pornography websites we identified solely
contained fake videos (as well as the occasional fake image). Two of these websites
also contained some videos that were possibly created without the use of generativealgorithms or other AI-based tools, and therefore could be shallowfakes. Since thesites didn’t provide any guidance on how this content was created, and in one casethe content was not fully visible without buying premium access, we have decided
to count these videos.
16DEEPTRACE

Excluding these two websites, we selected the top five in terms of content volume
for additional data analysis. We used the web scraper Beautiful Soup to identify every
hosted video and developed proprietary tools to extract data such as title, views,
and likes. For one of these five websites information on views was not available.
Accordingly, the statistics on views are reported for the top four only.
The compiled video metadata gave us results such as total viewership numbers but
told us little about the people targeted by deepfake pornography videos. In somecases, we were able to extract the exact names of people present in videos fromthe page HTML. Unfortunately, in many cases, all we had was a title which usuallycontained the target’s name as part of a broader description of the video. In cases
where only a title was available, we used a combination of NLP processing, manual
editing, and validation to extract the people named in the video titles. For this, weused the python library SpaCy, taking advantage of its named entity recognition
capabilities. We were, therefore, able to determine the names of people targeted inalmost every deepfake video we tracked.
Having identified the names of targeted individuals we used the Wikidata API
to extract demographic information on each individual, including their gender,
nationality, and profession. Wikidata offers a plain text query function that canidentify well-known people even with alterations and errors in the spelling of theirnames as they appear in video titles.

Non-pornographic video hosting websites/channels
SourcesYouTube, Vimeo, LiveLeak, Dailymotion. Reddit was not taken into consideration as
almost all deepfakes on Reddit are hosted on YouTube.

Methodology
Searching for the keyword “deepfake” on YouTube leads to inflated results, where
search results output content on deepfakes but not actual deepfake videos. To avoid
this pitfall we focused our search on outputs from dedicated deepfake channels
or creators who exclusively or largely publish deepfake videos. The other sources
host a much smaller volume of deepfakes videos, which we were able to find andcount by a keyword “deepfake” search. Each video was manually checked to see if it
contained deepfakes or deepfake style alterations.
For further analysis of YouTube videos, we downloaded each videos’ metadata. Post
metadata extraction analysis based on videos’ titles was identical to the analysis
performed on pornographic deepfake metadata as described above.

Forums and creation communities
Sources
Google search, deepfake pornography websites, Reddit, Telegram, 4Chan, 8Chan,
Voat

www.deeptracelabs.com

17

THE STATE OF DEEPFAKES – Appendix

Methodology
We used the terms “forum” and “creation community” to refer to online communities
where people can share, discuss, and find information on deepfakes, with a focus
on the creation process. We conducted Google searches for “deepfake forums” and“deepfake communities”, as well as specific searches on deepfake pornography
websites and forum-based websites. From here, we calculated the number ofregistered users of 13 of the 20 deepfake forums and creation communities we
identified, based on publicly accessible data on the related websites and channels.

Deepfakes computer apps
Sources
Google search, Github

Methodology
We use the term “deepfake computer app” to refer to downloadable software forcreating deepfakes that are independently operated by the user. This can be further
broken down into open-source code repositories and graphical user interfaces. We
identified deepfake apps and GUIs with Google search using the keywords “deepfakeapps” and “deepfake tools”, and through Github’s repository search engine. A voicecloning GUI was found on Github by searching for “voice cloning”.

Deepfake services portals
Sources
Google search

Methodology
We use the term “deepfake service portal” to refer to an online business thatgenerates and sells custom deepfakes. We identified these deepfake service portals
with Google search, using the keywords “deepfake service” and “deepfake tools”. Our
analysis of each service portal focused on the kinds of deepfake creation services
being offered, and the prices that accompanied these services.

Marketplace services
Sources
Google search, website search18

DEEPTRACE

Methodology
We used the term marketplace services to refer to individual deepfake creators thatadvertise their services on forums and online marketplaces. We identified thesemarketplace services using Google search with the phrase “deepfake services”,“deepfake tools” or “synthetic voices”, along with the search engines on three of the
top online marketplaces and freelance work portals using the keywords “deepfakes”
and “deepfake services”.

GithubSources
Github Archive stored on Google BigQuery

Methodology
The visual in the report shows the number of stars for two faceswap GitHub
repositories over time. We used the following repositories used to collect this data:
SELECT project,
YEAR(star_date) as yearly,
MONTH(star_date) as monthly,
SUM(daily_stars) as monthly_stars
FROM (
SELECT repo.name as project, DATE(created_at) as star_date, COUNT(*) as daily_
stars
FROM TABLE_DATE_RANGE([githubarchive:day.], TIMESTAMP(“20171001”), TIMESTAMP(“20190731”))
WHERE repo.name IN (“alew3/faceit_live”) AND type = ‘WatchEvent’
GROUP BY project, star_date)
GROUP BY project, yearly, monthly ORDER BY project, yearly, monthly

Papers mentioning GANs in the title/abstract
Sourceswww.arXiv.org is an online archive of research articles in the fields of physics,
mathematics, computer science, quantitative biology, quantitative finance,
statistics, electrical engineering and systems science, and economics. arXiv is owned
and operated by Cornell University. See more information on arXiv.org.

Methodology
We used arXiv’s search engine https://arxiv.org/find with the keyword “GAN” or“generative adversarial network” to identify the total papers containing these
keywords in their titles or abstracts for each year. The annualized 2019 figure wasderived from 704 papers released between January 1st 2019 and July 29th 2019, by
multiplication by 12/7.

www.deeptracelabs.com

19THE STATE OF DEEPFAKES – Appendix

DeepNude
Sources
DeepNude websites (original and repackaged versions), articles (names omitted)

Methodology
The DeepNude business sale price and June 2019 website viewing figures/user
numbers were directly sourced from a listing on the online business marketplace,which is no longer accessible at the time of this report’s publication. We accessed
this data on July 18th 2019. The details of the original app’s pricing and functionality
were sourced from DeepNude’s website, which is also no longer accessible at thetime of this report’s publication. We accessed this data on June 27th 2019.

20

@deeptracelabs

Tracer Newsletter/deeptracelabs

info@deeptracelabs.com/deeptracelabs

www.deeptracelabs.com