Experiments in Detecting
Persuasion Techniques in the News
122Seunghak Yu Giovanni Da San Martino Preslav Nakov
1MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA
2Qatar Computing Research Institute, HBKU, Qatar
seunghak@csail.mit.edu {gmartino, pnakov}@hbku.edu.qa
Computational Propaganda

An evaluation measure for Task FLC needs to be defined. We use a variant of the standard F1 (andPrecision, P, and Recall, R) taking into account partially overlapping spans:
1 X
1 X
P (S, T ) =C(s, t, |s|),R(S, T ) =C(s, t, |t|),|S||T |• Propaganda has been widely used since the advent of mass medias ∈ S,t∈Ts ∈ S,t∈T

where|(s ∩ t)|C(s, t, h) =δ (l(s), l(t))
hhere δ(a, b) = 1 if a = b, and 0 otherwise.
• However, Internet and social media “have allowed cross-border computational propaganda by foreign states or even private organizations” (Bolsover and Howard, Big Data 2017)• Can we automatically detect the use of propaganda?
Can we make America (and the world) aware again?

Tt1 (c=1)

t2 (c=2)

t3 (c=3)

s1 (c=1)s2 (c=2) s3(c=2)

s4 (c=4)

Example of gold annotation (top)and the predictions of a supervised model (bottom) in a document represented as a sequence ofcharacters. The class of each fragment is shown in parentheses.

• Current approaches provide document-level predictions– rely on gold labels based on distant supervision → noisy– lack model explainability

S

Propaganda Techniques

Models• Multi-Granularity Network: It drives the higher-granularity task (FLC, g2) on the basis of thelower-granularity information (SLC, g1) through a trainable gate f :• Propaganda is conveyed through a series of rhetorical and psychologicaltechniquesog2 = f (og1 ) ∗ og2
and we used a weighted sum of losses with a hyper-parameter α
LJ = Lg1 ∗ α + Lg2 ∗ (1 − α)TokenLabel 1
Lg ...

Token
Label 2...

Token
Label N...

T1

T2

TN1

Figure 1: Bandwagon: join in because everyone
else is taking the same action.

CLS

Greta Thunberg: “We are in
the middle of the sixth mass extinction, with more than 200species getting extinct everyday.”Figure 2: Name calling.

Sentence Token
Label
Label 1...
Lg2

CLS

T1TokenLabel 2...

Token
Label N...

T2TN

BERT

BERT(a) BERT(b) BERT-Joint

Sentence Token
Label 1LabelToken
Label 2

Token
Label N

Sentence Token
Label 1
Label...

Lg ...
1,2

Token
Label N...
...

...

Token
Label 2og

1...

wgwg

1

1ffog

og

Figure 3: Appeal to fear.

2

2...

Propaganda Techniques Corpus

CLS

T1

450 news articles from 48 sources (21,230 sentences, 350K tokens) annotated at the fragment levelwith 18 propaganda techniques.

...

T2TN

...

CLS

T1

...

T2TN

BERT

BERT

(c) BERT-Granu(d) Multi-Granularity Network

Figure 4: The architecture of the baseline models (a-c), and of our proposed multi-granularity network (d).

Experiments
ModelP

Spans
R

BERT
39.57 36.42
Joint
39.26 35.48
Granu 43.08 33.98
Multi-Granularity
ReLU 43.29 34.74
Sigmoid 44.12 35.01

F1

FLC Task
PR
F1

SLC Task
PR
F137.90 21.48 21.39 21.39 63.20 53.16 57.74
37.25 20.11 19.74 19.92 62.84 55.46 58.91
37.93 23.85 20.14 21.80 62.80 55.24 58.76
38.28 23.98 20.33 21.82 60.41 61.58 60.98
38.98 24.42 21.05 38.98 62.27 59.56 60.71

Table 1: Evaluation of the models for Spans, FLC and SLC tasks. The proposed models improve over the baselines.

Annotation Process• Phase 1: two annotators, ai and aj , independently annotated the same article• Phase 2: ai and aj discussed with a consolidator c1
all instances to come up with a final annotation.
The table shows γ inter-annotator agreement for spans
only and spans + labels between two annotators and one
annotator and one consolidator.

Annotations spans (γs) +labels (γsl )
a1
a3

a2
a4

0.30
0.34

0.24
0.28

a1
a2
a3
a4

c1
c1
c2
c2

0.58
0.74
0.76
0.42

0.54
0.72
0.74
0.39

Tasks and Evaluation Measure• FLC: detect the text fragments in which a propaganda technique is used and identify the technique.
Spans is a lighter version of the task in which only the span has to be identified.
• SLC detect the sentences that contain one or more propaganda techniques (binary task).

Conclusion and Future Work• Our fine-grained task can complement document-level judgments, both to come out with an aggregated decision and to explain why it has been flagged as potentially propagandistic.
• We plan to build an online platform to annotate propaganda techniques and expand the corpus.

What We Are Up To• SemEval 2020 Task 11 on Fine Grained Propaganda Detection:
https://propaganda.qcri.org/semeval2020-task11• Our Propaganda Analysis Project (where you can find this paper):https://propaganda.qcri.org• The Tanbih Project, which aims to limit the effect of “fake news”, propaganda and media bias bymaking users aware of what they are reading: http://tanbih.qcri.org