Discourse Structures Guided Fine-grained Propaganda Identification
Authors: Yuanyuan Lei
Ruihong Huang
Venue: EMNLP 2023 Main (2023)
Keywords: misinformation
propaganda
discourse structure

Abstract: Propaganda is a form of deceptive narratives that instigate or mislead the public, usually with a political purpose. In this paper, we aim to identify propaganda in political news at two fine-grained levels: sentence-level and token-level. We observe that propaganda content is more likely to be embedded in sentences that attribute causality or assert contrast to nearby sentences, as well as seen in opinionated evaluation, speculation and discussions of future expectation. Hence, we propose to incorporate both local and global discourse structures for propaganda discovery and construct two teacher models for identifying PDTB-style discourse relations between nearby sentences and common discourse roles of sentences in a news article respectively. We further devise two methods to incorporate the two types of discourse structures for propaganda identification by either using teacher predicted probabilities as additional features or soliciting guidance in a knowledge distillation framework. Experiments on the benchmark dataset demonstrate that leveraging guidance from discourse structures can significantly improve both precision and recall of propaganda content identification.

Discourse Structures Guided Fine-grained Propaganda Identification
Yuanyuan Lei and Ruihong Huang
Department of Computer Science and Engineering Texas A&M University, College Station, TX (yuanyuan, huangrh)@tamu.cr edu
Abstract
2017). Despite its misleading nature and harmful effects, propaganda can be pervasive in political news, and has the potential to reach very large au- diences (Glowacki et al., 2018; Tardaguila et al., 2018). Hence, developing intelligent models to identify propaganda in political news is important and necessary.
Propaganda is form of deceptive narratives that instigate or mislead the public, usually with a political purpose. In this paper, we aim to identify propaganda in political news two fine-gra areeggraiedd levels: sentence-level and token- level. We observe that propaganda content is more likely to be embedded in sentences that attribute causality or assert contrast to nearby sentences, well as seen in opinionated eval- uation, speculation and discussions of future expectation. Hence, we propose to incorpo- rate both local and global discourse structures for propaganda discovery and construct two teacher models for identifying PDTB-style dis- course relations between nearby sentences and common discourse roles of sentences a news article respectively. We further devise two methods to incorporate the two types of dis- course structures for propaganda identification by either using teacher predicted probabilities as additional features or soliciting guidance in a knowledge distillation framework. Experi- ments on the benchmark dataset demonstrate that leveraging guidance from discourse struc- tures significantly improve both precision and recall of propaganda content identifica- tion.!
Instead of detecting propaganda the level of articles (Horne et al., 2018; De Sarkar et al., 2018; Rashkin al., 2017; Rubin et al., 2016), this paper focuses on identifying propaganda fine-grained levels: sentence-level and token-level. Through extracting the sentence or fragment imbued with propaganda, our aim is to accurately locate the pro- pagandistic content and thereby provide detailed interpretable explanations. Propaganda content not only presents unverified or even false information. but also employs variety of argumentation strate- gies convince the readers (Yu et al., 2021). There- fore, identifying propaganda fine-graid fine-grained levels still remains a difficult and challenging task, and requires profound understanding of broader context in an article (Da San Martino et al., 2019b).
First, we observe that propaganda can be injected into sentences that attribute causality or assert con- trast to nearby sentences. Take the article in Figure I as an example, the second sentence (S2) makes an illogical and misleading deduction from its pre- ceding sentence: This suggested giving advice on how to prevent Jihadist attacks now against com- munity standards. Propaganda content such as S2 usually disseminate the misinformation by lever- aging causal relations, either by inferring baseless reasons or deducting speculative consequences. In addition, propaganda content can also utilize con- trast relation to raise doubt or challenge credibility. For example, the last sentence (S4) casts doubts towards its previous sentence: why would Face- book remove such a page and never explain why? Through the strategy of contrasting, the author aims to undermine the credibility of Facebook and
1 Introduction
Propaganda refers to type of misleading and de- ceptive information used to promote or publicize a certain political point of view (Lasswell, 1927; Hen- derson, 1943; Stanley, 2015; Rashkin al., 201 7). This information is often manipulated in a strate- gic way to shape societal beliefs (Rashkin et al., 2017: Barron-Cedeno et al., 2019). Propaganda can be harmful to both individuals and society as a whole, such as disseminating false information, inciting people's 's perceptions, leading to conflicts, perpetuating prejudices, impeding democracy pro- cess etc. (Bernays, 2005; Stanley, 2015; Little,
'The code and data link: https//gitua..comyuuauuaaatt nlp/propaganda _aanp_mmmppp_pp 2023

---

Example Arikle 1: scniences denoesstriingg keally cansal comparison discourse relations are more likely to cary propaganda
indox
S1
17th Annivenaryy
Facebook informed was suspended days
Stepe Successfilly
Counter lihad, violated thair "comuunity shandards.
suggested, course, faat giving
to provent nootber 9/11, Jibadist agrinst America, now agrinst
Cousal
Farabook's 's "oommumiy standards'.
53
Counter Coolition's (CIC) Facebook page. human rights group that dedicated protecting America Sharia oppression which Jamie selped run, was removed years with absohnely no explanation.
question why Faccbuok nemowe page,
cxplain
Comparison
Figure I: An example article containing propaganda sentences. Propaganda sentences are highlighted in red. S2 is propaganda sentence showing deduction. S4 is propaganda sentence proposing challenge doubt. Their discourse relations with nearby sentence are shown in right column.
Exumple propaganda contens more Ekely embedded opimionated evaluations expectation speculations
Index
sentences
ciscourse role
SI
Immigrations Cusoms inforcement arresedd than 150 illegals Francisoo Area this week
52
was reported that about half of snaned fagiives with convictions for assultand battey, erimes aguinst childres, and wesyons charges. that
Current Contest
The Democratic mayor Ouiland, Libby
conducting
Bay
Context
54
ICE Deputy Director Thomas Iloman said stalement Tucaday, "The Oakland mayur's docision publicine her supiccions about ICE opentiios moreased my officers
Exalation
cnminal making clear that reokles decision
her political agenda"
85
"Unlike the politicians who undeemine ICE's criccal mission, officers will continue fulfill their swom protect public safety."
Expectation
Figure 2: Another example article containing propaganda sentences, and the corresponding news discourse role of each sentence. Propaganda sentences are highlighted in red. Propaganda content is more likely to he embedded into opinionated evaluations and expectation speculations.
thereby incite public protest. Accordingly, we pro- pose that understanding the discourse relations of sentence with its nearby sentences in the local con- text can enable discovery of propaganda contents.
Motivated by the above observations, we pro- pose incorporate both local and global discourse structures for propaganda identification. Specifi- cally, we establish two teacher models to recognize PDTB-style discourse relations between a sentence and its nearby sentences (Prasad et al., 2008), as well as identify one of eight common news dis- course roles for each sentence based upon news discourse structure (Choubey et al., 2020). We further devise two approaches effectively incor- porate the two types of discourse structures for pro- paganda identification. The first approach concate- nates the predicted probabilities from two teacher models as additional features. The second approach develops a more sophisticated knowledge distilla- tion framework, where we design response- response-based based distillation loss to mimic the prediction behavior of teacher models, well distillation loss to seek guidance from the embed- dings generated by teacher models. The response- based and feature relation-based distillation mutu- ally complement each other, acquiring an enhanced guidance from discourse structures. Experiments on the benchmark dataset demonstrate the effec- tiveness of our approaches for leveraging discourse structures, with both precision and recall improved. The ablation study validates the necessity and syn-
Morcover, we observe that propaganda content deceptive narratives is more likely to be em- bedded into opinionated evaluations or expectation speculations. contrast, sentences describing fac- tual occurrences are less likely to carry propaganda. Take the article in Figure as an example, the first three sentences, either reporting the main event or describing the current context triggered by the main event, all provide the readers with factual occur- rences devoid of deceptive content. However, in the succeeding sentence (S4), the author includes a quotation to express emotional assessments: this reckless decision was based on her political agenda. Propaganda sentences such as S4 always convince the readers and influence their mindset by inserting opinionated evaluations. Furthermore, the author speculates future expectations in the next sentence (S5) that ICE officers will continue protect public safety. Propaganda sentences such $5 usually promise a bright yet unprovable future with the aim of gaining trust and support. Therefore, we propose that understanding the discourse role of sentence in telling a news story can help reveal propaganda.
feature relation-basee

---

Comparison
Contingency
Temporal
Expansion
Tottl
propaganda
(35.66)
(40.56)
(18.18)
(32.13)
(30.48)
benign
184 (64.34)
214 (59.44)
(81.82)
712 (67.87) 1414 (69.52)
Table 1: The number (ratio) of propaganda and benign sentences that have each of the four discourse relations with nearby sentences. The ratio values higher than the overall ratio in the rightmost column are shown in bold.
ergy between local and global discourse structures.
basic language model. The sentence pair embed- ding is the concatenation of hidden states the sentences start tokens 
. A two-layer neural net- work is built on top of the pair embedding to predict discourse relations into comparison, contingency, temporal, or expansion. The model is trained on PDTB 2.0 data (Prasad et al., 2008) that annotates both explicit and implicit relations between adja- cent sentences. Considering propaganda sentences can be connected with nearby sentences with without discourse connectives explicitly shown, we utilize both explicit and implicit discourse relations data for training.
2 Discourse Structures
In this section, we explain the details for the two discourse structures: discourse relation based on PDTB-style relations, and discourse role that draws upon news discourse structure. We also perform a statistical analysis to verify our empirical observa- tions, and introduce the procedure of constructing teacher models for both discourse structures.
2.1 Discourse Relations
2.1.1 PDTB Discourse Structure
The Penn Discourse Treehank (PDTB) discourse structure (Prasad et al., 2008) interprets the dis- course relation between adjacent sentences in news articles into four types: 1). Comparison highlights prominent differences between arguments, and represents the relation of contrasting or concession. 2). Contingency indicates two arguments causally influence each other, and represents cause-and- effect or conditional relationship. 3). Temporal captures the temporal or chronological relationship between two arguments, such as precedence, suc- cession, or simultaneously. 4). Expansion covers relations of elaborating additional details, provid- ing explanations, restating narratives.
Given pair of sentences from the propaganda article, the local discourse relation teacher model generates the predicted probability four relations between i-th sentence and nearby sentence as:
ploeal (Plaeal ,P
plaeal plaoul plaed) (1)
2.2 Discourse Role
2.2.1
News Discourse Structure
The discourse structure (Choubey et al., 2020) categorizes the discourse role of each sentence in news article into three broad types and eight sub- types: 1). main event contents contain two sub- types, Main event (M1) and Consequence (M2), and cover sentences that describe the main event and their immediate consequences which are often found inseparable from main events. 2). context- informing contents have two subtypes, Previous Event (C1) and Current Context (C2), and cover sentences that explain the context of the main event. including recent events and general circumstances. 3). additional supportive contents have four sub- types, describing past events that precede the main event in months and years (Historical Event (D1)) or unverifiable fictional situations (Ancedotal Event (D2)), or opinionated contents including reactions from immediate participants, experts, known per- sonalities as well as journalists or news sources (Evaluation (D3)), except speculations and pro- jected consequences referred as Expectation (D4).
2.1.2 Statistical Analysis
To validate the correlation between propaganda and discourse relations, we also conduct statistical analysis on the validation set propaganda dataset (Da San Martino et al., 2019b), where we run the model of classifying discourse relations. Table shows the ratio of propaganda sentences that have each of the four discourse relations with nearby sentences. The numerical analysis confirms our observation: sentences that exhibit contingency and comparison relations with adjacent sentences are more prone to containing propaganda, whereas sentences that narrate events a chronological order significantly contain less propaganda.
2.1.3 Teacher Model for Discourse Relation
We train the teacher model for discourse relations by using Longformer (Beltagy et al., 2020) as the

---

propagunda
66 (28.08)
(none)
(19.56)
71 (17.07)
(33.12)
(12.00)
335 (42.84)
(36.18)
(30.48)
benign
169 (71.2)
(nonc) 37 (80.44)
345 (82.93)
103 (66.88) (88.00)
447 (57.16) (63.82)
1414 (69.52)
Table 2: The number (ratio) of propaganda and benign sentences under each of the eight news discourse role types. The rightmost column shows the overall number (ratio). The ratio values higher than the overall ratio are shown bold. MI: Main Event, M2: Consequence, CI: Previous Context, C2: Current Context, DI: Historical Event, D2: Anecdotal Event, D3: Evaluation, D4: Expectation
2.2.2 Statistical Analysis
Considering the news articles are typically long. we utilized Longformer (Beltagy et al., 2020) as the basic language model to encode the entire article. Given a candidate propaganda article consisting of sentences, sentence embeddings (81, $2, 8n) are initialized as the hidden state at sentence start tokens 
. The i-th sentence contains m tokens, and its tokens embeddings (Wa, W12, Wm..
To verify the correlation between propaganda and news discourse structure, we perform a statisti- cal analysis on the validation set of propaganda dataset (Da San Martino et al., 2019b), where we run the model of profiling news discourse structure (Choubey and Huang, 2021). Table 2 presents the ratio of propaganda sentences across the eight news discourse roles. The numerics validate our obser- vations: propaganda is more likely to be embedded into sentences expressing opinions or evaluations (D3), speculating future expectations (D4), or fab- ricating historical background (D1). Conversely, sentences describing factual occurrences, such as reporting main event (MI) or informing context (C1, C2) are less likely to carry propaganda.
3.1 Feature Concatenation Model
The feature concatenation model directly concate- nates the predicted probabilities generated by the two teacher models as additional features, since they contain the discourse structures information. The updated feature vectors for i-th sentence and its j-th token in the two fine-grained tasks are:
$ = L Pioccal patoball
2.2.3 Teacher Model for Discourse Role
wiy wy + placal θ poinnat
We follow the same framework in the current state- of-art model of profiling news discourse structure (Choubey and Huang, 2021), where an actor-critic model developed that selects between the stan- dard REINFORCE (Williams, 1992) algorithm or imitation learning for training actor. Additionally, we replace the ELMo word embeddings (Peters et al., 2018) with Longformer language model (Beltagy et al., 2020), which generates contextu- alized embeddings for long documents based on transformer (Vaswani et al., 2017) and provides fur- ther improvements to the current state-of-the-ard
where @ denotes feature concatenation, Placal and polobol are probabilities of discourse relations and discourse roles predicted by two teacher models. Additionally, a two-layer classification head is built on top of the updated embedding to make pre- diction. The cross-entropy loss is used for training.
3.2 Knowledge Distillation Model
The knowledge distillation model constructs addi- tional learning layers to learn local discourse lation and global discourse role respectively. By optimizing the response-based distillation loss mimic the prediction behaviors of teacher, and the feature relation-basee distillation loss to learn from the embeddings generated by the teachers, the dis- course structures information can be distilled into the task of propaganda identification.
Given a candidate propaganda article consisting n sentences, the global discourse role teacher model generates the predicted probability of eight discourse roles for i-th sentence as:
pglabal (Palabal pglobal pglobul
3.2.1 Learning Layers
3 Fine-grained Propaganda Identification
Three types of learning layers are built on top of sentence 8; or token embedding way: propaganda learning layer, student discourse relation learning layer, and student discourse role learning layer.
In order to incorporate the two types of discourse structures into propaganda identification, we fur- ther devise two methods: a feature concatenation model and a knowledge distillation model. Figure3 illustrates the framework of knowledge distillation.
The propaganda learning layer is to learn the main task of propaganda identification at either

---

apputatide mp
Goba Disormn Role
Teacter Madel
000
000
Sa (P. 7a)
rosponae--saa lowa
subeddan
prddcted probahiltien for right discrume roles
for gleball discourse role
relaties-bssed distilationn for globall dicoure rule
Decrured Role Laaring
st 00., Ou Shik
Global discousser role leaming
O0O
Ou.,,
Propaganda leamning layers
propaganda leaming Josa
51: proggandd herign 82: propagadds herign
<14
OOO
Local discouss rebrion lerrnng
St: propagndaa berrign
0v. s2 Oz
ambedding
semeaze Bi-LSTM
fenture relaiin-asso distillation lows for local discourse relation
Local Disccuese Reiation Learingg
cospociso based distillcion loss for local discourse relation
orgut artide
1s:: (hy
52:
Pal
O O
amotee mp
Local Discouse Belation Yeadher Medi
prodicee protabiiitcs Soxe diccuune relations
umboddinn
Figure 3: An illustration of propaganda identification guided by discourse structures via knowledge distillation
sentence level or token level:
3.2.2
Response-based Distillation
Qurape = softmar (W,, b1) b2) Q.7 = sof softmar (W:( waj + bi) + b2)
The response-based distillation loss (Hinton et al., 2015) is designed to minimize the discrepancy be- tween the learned outcome of student layers and the predicted probability generated by the teacher models. By guiding the student layers to mimic the prediction behaviors of teachers, the knowledge of discourse relation and discourse role from the teachers can be distilled into the model.
(4)
where Quropa and Qu are the predicted proba- bility of i-th sentence and its j-th token containing propaganda. W,, W2, bj, b2 are trainable parame- ters. The cross entropy loss used for training:
LosB proopa
puropa log (
Specifically, the Kullback-Leibler (KL) diver- gence loss is employed for measuring the distance between the learned probability of student layers and referenced probability from teacher models:
LosStoken propa
peropa log( (gp(po
where perope and poropa are human annotated pro- paganda label for i-th sentence and its j-th token.
The student discourse relation learning layer is built on top of the concatenation of i-th sentence embedding 8; and its adjacent sentence embedding 8; to learn the discourse relation between them from the teacher model:
plocol log
plocal
Lo8Sveaponae- local
Qloca!
(8)
pplotal
Lossresponse-glaae global polabal log Qutobal
glocal (Qigcal Qlacol ..... glacal
where plocal and polobal are response from the teachers, and are referenced as learning target. Q!ocal and gloobal are learned outcomes of student discourse relation layers and student discourse role layers. The response-basee distillation loss penal- izes the performance gap between teacher models and student layers, and forces student layers be updated with discourse structures knowledge.
(6)
softmas(Wo @ s1-1) + bb) + bu)
where W5. Ws. by, by are trainable parameters in the student discourse relation layer, Qlocol is the learned outcome of predicting discourse relations.
The student discourse role learning layer is built on top of the sentence embedding st. to learn its discourse role information from the teacher model: golobal = (Qulooall Q Qolobal) sof (aa bs) b.)
3.2.3 Feature Relation-based Distillation
where W3. W4, by, by are trainable parameters in the student discourse role layer, and gatobal is its learned outcome of predicting eight discourse roles.
The feature relation-based distillation loss is de- signed to seek guidance from the teacher-generated sentence embeddings which also contain discourse

---

Comparison
Contingency
Temporal
Expansion
Macro
Precision
85.75
80.06
86.42
82.17
83.60
Recall
83.73
74.93
90.32
85.38
83.59
Fl-score
84.73
77.41
88.33
83.75
83.55
Table 3: Performance of the PDTB discourse relations model (local discourse relation teacher) on PDTB 2.0 dataset.
M1
M2
C1
C2
D1
D2
D3
D4
Macro
Precision
55.56
37.88
43.72
67.21
66.67
62.69
75.22
62.15
63.23
Recall
59.78
32.47
33.10
64.06
85.22
69.54
69.75
69.63
64.36
Fl-score
57.59
34.97
37.68
65.60
74.81
65.94
72.38
65.68
63.49
Table 4: Performance of the news discourse structure model (global discourse role teacher) NewsDiscourse dataset. MI: Main Event, M2: Consequence, C1: Previous Context, C2: Current Context, D1: Historical Event, D2: Anecdotal Event, D3: Evaluation, D4: Expectation
structures knowledge. However, sentence embed- ding itself has no absolute meaning and instead relies on its spatial relations with other contexts. Thus, rather than directly minimizing the euclidean distance between teacher-generated and student- learned features, we follow (Park et al., 2019) to guide the student layers to learn the spatial relations between sentences found in the teacher models.
The overall learning objective for identifying pro- paganda at sentence and token level are:
Los8 =
Los 8.088mm prapp a + LOS8global + LosStoca!
L088token = 1.08Rtoken-prope + 1088global + Lo88tocca
(14)
Specifically, let glocal and of glodel denotes the i-th sentence embedding trained by the two teachers. The spatial matrix of the teachers are computed:
Experiments
4.1 Dataset
Acquiring human-annotated labels at fine-grained levels challenging and expensive, leading to limited resource of available datasets. In our sub- sequent experiments, we utilized the propaganda dataset published by (Da San Martino et al., 2019b) that provides human-annotacee labels for propa- ganda contents. We adhere to the same train / dev / test splitting in the released dataset. This propa- ganda dataset also used in the NLP4IF-2019 challenge (Da San Martino et al., 2019a), which featured tasks involving sentence-level identifica- tion and token-level classification. In this paper, we specifically concentrate on propaganda identifi- cation at both the sentence and token levels.
M cosine(s, (ocull tocul .8k )loua)) Malobbll cosine(s))oos global sglobal)
(10)
where Migeal and is tween i-th and k-th sentence in the teachers. Also, the spatial matrix of student-learned features is: (1I) The feature relation- based distillation loss is the mean squared error (MSE) loss between spatial matrix of teacher models and student layers:
Matlobll are spatial relation be-
Mis cs.ine(s..ax)
Σ (Milgat Maa)²
LosSyelation local
(12)
EMMgtobal M
LosSyelation - global
To summarize, the response-based distillation and feature relation-based distillation mutually complement each other, with the former informed by teacher-predicted probabilities and the latter guided by saacerrggenraan embeddings.
4.2 Teacher Models
The teacher model for discourse relation is trained on PDTB 2.0 dataset (Prasad et al., 2008). Follow- ing its official suggestion, sections 2-21, sections 22 & 24 and section 23 are used for train / dev test set respectively. Table 3 displays the classifi- cation performance for the four discourse relations. On the other hand, the teacher model for discourse role trained on News Discourse dataset (Choubey et al., 2020). The performance of classifying the eight news discourse roles is presented in Table
3.2.4 Learning Objective
The total distillation loss for local discourse rela- tion and global discourse role are:
Lo8Siocal LosS + Lo88yelatoon-iece
Los8global Lose peeeerrrepneeglaae LosBrelation-ghee -glaattttoonnnnaalt
(13)

---

Sentence-level Recall
Token-level
Precision
Precision
Recall
Baseline Models
all-propaganda
24.86
100.00
39.82
10.41
100.00
18.86
chatgpt
58.26
34.72
43.51
13.37
19.31
15.80
chatgpt 5-shot
56.42
37.34
44.94
14.68
20.84
17.22
chatgpt discourse structures prompt
57.93
38.61
46.34
15.82
21.96
18.39
(Da San Martino et al., 2019b)
63,20
53.16
57.74
39.57
36.42
37.90
(Da San Martino et al., 2019a)
60.28
66.48
63.23
(Fadel et al., 2019)
61.39
(Vlad et al., 2019)
59.95
57.47
58.68
longformer
60.32
60.50
60.41
34.60
39.81
37.03
Feature Concatenation Models
local discourse relation
61.72
62.09
61.90
35.38
41.92
38.37
+ global discourse role
61.50
63.58
62.52
36.39
41.28
38.68
+ both discourse structures
62.71
64.08
63.38
36.62
42.27
39.25
Knowledge Distillation Models
local discourse relation
60.40
66.17
63.15
35.18
43.49
38.90
global discourse role
61.88
66.86
64.27
37.65
43.32
40.28
both discourse structures (full model)
61.22
69.75
65.21
37.22
46.86
41.48
Table 5: Performance of sentence-level and token-level propaganda identification guided by discourse structures. Precision, Recall, and Fl of the propaganda class are shown. The model with the best performance is bold.
4.3 Baseline Models
(Vlad et al., 2019): capsule model architec- ture designed for sentence-level task
We include the following baselines for comparison:
all-propaganda: naive baseline that pre- diets all sentences tokens into propaganda
we build a baseline that follows the same framework and is equivalent to our developed model without discourse structures
longformer:
chatgpt: an instruction prompt (A.1) is de- signed for the large language model ChatGPT to automatically generate predicted labels for sentence tokens in the same test set
4.4 Experimental Setting
The model takes the entire news article as in- put, and predicts the label for each sentence or token into propaganda benign. The Adam W (Loshchilov and Hutter, 2019) is used as the opti- mizer. The maximum length of input is set to 4096. The number of training epochs is 6. The learning rate adjusted by linear scheduler. The weight decay set to le-2. Precision, Recall, and Fl of propaganda class used as evaluation metric.
chatgpt 5-shot: we add five examples of propaganda sentences and five examples of non-propaganda sentences into the prompt
chatgpt + discourse structures prompt: we add the local discourse relation and global dis- course role of each sentence into the prompt
(Da San Martino al., 2019a): we present the best performance achieved by the rank one team the NLP4IF-2019 challenge, where the model was also trained on extensive cor- pora including Wikipedia and BookCorpus
4.5 Experimental Results
Table shows the performance of sentence-level and token-level propaganda identification. Comparing feature concatenation models with the longformer baseline, we observe that integrat- ing discourse relations discourse roles as addi- tional features brings consistent improvements for precision and recall, both the sentence and ken level tasks. This underscores that these two types of discourse structures can provide beneficial insights for identifying propaganda contents.
(Da San Martino et al., 2019b): where both sentence and token level propaganda identifi- cation tasks are performed
(Fadel et al., 2019): pretrained ensemble learn- ing is employed for sentence-level task

---

senkeooe
dscourse relation
predicton
sugested, of course, giving prevert sooher 911, and quainst now spaiast Facebook's community smodards".
Plompaganta beseline- P(onpagania
sentence
dscourse role
prediction
Direcin Honas in Tacalay, Oalland maya''s decisoo publisize suspiciona about was her political agroda"
(impppanna beseline) F(poopyaada codel) 0.91
Evaluation
opentiion lncreased my offiorss crmuaal clices making
Figure 4: Examples of our method succeed in solving false negative emor. Both the red sentences contain propaganda.
Comparing knowledge distillation models with the longformer baseline, it is evident that distilling the knowledge of discourse relations and discourse roles leads to a notable increase in recall by 9.25% and significant enhancement in Fl score by 4.8%. Furthermore, in comparison to the previous best performance reported in (Da San Martino al.. 2019a), our knowledge distillation model exhibits superior performance in both precision and recall, ultimately achieving state-of-the -otth----he--rt results.
Precision
Recall
F1
longformer
60.32
60.50
60.41
response-bane based
61.64
67.96
64.65
relation- based
60.40
66.77
63.42
both (full model)
61.22
69.75
65.21
Table 6: Ablation study of the two types of distillation
losses: response-basee and feature relation-based based.. Take sentence-level propaganda identification
example.
4.8 Effect of the Four Local Discourse Relations
Comparing knowledge distillation models with feature concatenation models, we can see that dis- tilling the knowledge from teacher models demon- strates stronger ability to incorporate two types of discourse structures, surpassing the approach of simply adding extra features.
In addition, we study the effect of the four local discourse relations in Table 7. The results indicate that removing any one of the four discourse rela- tions leads to performance drop compared to the full model, as expected, the influence of expansion relations is relatively less compared to the other three types of relations.
Comparing our full model with the large lan- guage model ChatGPT, there still remains noticable performance gap, especially the recall. Also, the gap is even larger in terms of token-level task. Pro- viding ChatGPT with extra examples or discourse structures information in the prompt can boost the performance little bit, but it still remains inferior to our developed method.
Precision
Recall
Fl
longformer the full model
60.32
60.50
60.41
61.22
69.75
65.21
comparison
60.91
67.76
64.15
contingency
60.89
67.56
64.06
temporal 61.03
68.16
64.38
expansion 61.19
68.65
64.70
4.6 Ablation Study
The ablation study of local discourse relation and global discourse role is also shown in Table 5. Both the two types of discourse structures play an essen- tial role identifying propaganda content, both the sentence and token level tasks. Incorporating the two discourse structures together can further boost recall and achieves the best performance.
Table 7: Effect of removing each one of the four local discourse relations from the full model. Take sentence- level propaganda identification as an example.
4.9 Qualitative Analysis
Figure 4 presents examples of solving false nega- tive error through the integration of discourse struc- tures. The first propaganda sentence is inaccurately predicted as benign by the longformer baseline. However, by incorporating the local causal dis- course relation into the model, the prediction is corrected to propaganda. Likewise, the second propaganda sentence initially misclassified false negative by the baseline model. However, by leveraging the knowledge from the teacher model that this sentence plays an evaluation role in the article, the model successfully rectifies this error.
4.7 Effect of the Two Distillation Losses
Moreover, we examine the effect of two types of distillation losses in Table 6. Both response- based distillation and feature relation-based distillation yield substantial improvements. This demonstrates that learning from teacher-pree preehrrrpeedeted probabilities and teacher generated embeddings mutually com- plement each other, acquiring an enhanced guid- ance from discourse structures.

---

5 Related Work
opinionated content incite or persuade the public (Da San Martino al.. 2019b), or presents un- verified speculations, projections and deceptions (Miller and Robinson, 2019; Brennen, 2017). In- deed, in the current media landscape, ideologi- cally biased media sources and propaganda me- dia sources are often labeled separately, for exam- ple, Media Bias/Fact Check² distinguishes idcolog- ically biased sources, conspiracy theory sources, questionable sources which includes major propa- ganda sources, and couple other categories. Ideol- ogy bias and propaganda are studied separately as well in the NLP community (Barron-Cedena et al., 2019; Liu et al., 2022), and each task features their own benchmark datasets (Fan et al., 2019; Baly et al., 2020; Da San Martino et al., 2019b) with documents retrieved from different media sources.
Propaganda attracted research interests for years. Prior work focus on detecting propaganda at article- et al.,
level (Rashkin et al., 2017; 2019). The first work on fine-grained propaganda analysis was introduced by (Da San Martino et al.. 2019b,a). A shared challenge focusing on token- level tasks was launched by (Da San Martino et al.. 2020). Several approaches have been developed for propaganda analysis, such as (Vlad al., 2019) designed an unified neural network, (Fadel al., 2019) utilized pretrained ensemble learning, (Dim- itrov et al., 2021) trained a multimodal model mix- ing textual and visual features, and (Vijayaraghavan and Vosoughi, 2022) employed multi-view repre- sentations. In this paper, we focus on identifying propaganda in news articles at both sentence-level and token-level, leveraging discourse structures.
Conclusion
Misinformation Detection was also studied for years, such as fake news (Pérez-Rosas et al., 2018;
This paper aims to identify propaganda at sentence- level and token-level. We propose to incorporate two types of discourse structures into propaganda identification: local discourse relation and global discourse role. We further design a feature concate- nation model and a knowledge distillation model to leverage the guidance from discourse structures.
Oshikawa et al., 2020), rumor (Wei et al., 2021;
Li et al., 2019), political bias (Baly et al., 2020; Chen al., 2020), and logical fallacy (Jin al., 2022; Alhindi et al., 2022). Although propaganda may intersect with fake news, political bias, and logical fallacies, however, they are all distinct phe- nomena and tasks. Fake news and rumor always hallucinate untruthful information. Political bias refers to selectively reporting verified facts while leaving readers to arrive at their own conclusions. Logical fallacies focus on errors in reasoning and argumentations to reach an invalid conclusion. In contrast, propaganda presents unverified specula- tion or projections in the same tone as facts, and employs a variety of persuasion strategies to con- vince the readers, with the purpose to manipulate public beliefs to a predetermined conclusion. Media Bias. In the most broad sense, propaganda news articles a type of biased news reports. How- ever, media bias often refers to ideological bias these days (Kiesel et al., 2019; Fan et al., 2019; Lei and Huang, 2022), and ideological bias is of- ten expressed in a subtle way or under a neutral tone (van den Berg and Markert, 2020; Lei et al., 2022) by selectively including certain facts to sub- tly shift public opinions (Fan et al., 2019). In con- trast, propaganda is not limited to hyper-partisan cases and can be applied to influence public be- liefs in a way that aligns with the interests of the propagandist (Stanley, 2015; Rashkin et al., 2017). Propaganda often contains intensely emotional or
Limitations
This paper specifically concentrates on the identi- fication of propaganda as a specific form of mis- information. There still exists various other forms of misinformation. such as fake news, conspiracy theories, and more. While the designed discourse structures method has demonstrated its usefulness in identifying propaganda, its effectiveness for other types of misinformation remains unknown.
Ethics Statement
This paper focuses the detection of propaganda, which falls within the broader category of misin- formation and disinformation. The release of code and models should be utilized for the purpose of combating misinformation and not for spreading further misinformation.
A Acknowledgments
We would like to thank the anonymous reviewers for their valuable feedback and input. We grate- fully acknowledge support from National Science
Pht //mdii abiasfactcheck.com

---

Foundation via the awards IIS-1942918 and IIS- 2127746.
Proceedings of the Second Workshop on Natural Lan- guage Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 162-170, Hong Kong, China, Association for Computational Linguistics.
References
Giovanni Da San Martino, Alberto Barrón-Cedein Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. Pruceed- ings of the Fourteenth Workshop on Semantic Evalu- ation, pages 1377-1414, Barcelona (online). Interna- tional Committee for Computational Linguistics.
Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, and Smaranda Muresan. 2022. Multitask instruction- based prompting for fallacy recognition. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages $172-8187 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedefio, Rostislav Petrov, and Preslav Nakov. 2019b. Fine- tine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCC)) 5636-5646, Hong Kong, China, Association for Com- putational Linguistics.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Empirical Methods Natural Language Processing (EMNLP), pages 1982-49911 Online. Association for Computa- tional Linguistics.
Proceedings of the 2020 Conference
pages
Alberto Barron- Barron-Cedeno, Giovanni Da San Martino, Is- raa Jaradat, and Preslav Nakov, 2019. Proppy: A system to unmask propaganda in online news. In Proceedings of the 33rd AAAI Conference on Arti- ficial Intelligenee, pages 9847-9848, Honolulu, HI, USA. AAAI Press.
Sohan De Sarkar, Fan Yang, and Arjun Mukherjee. 2018. Attending sentences to detect satirical fake news. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3371-3380 Santa Fe. New Mexico, USA. Association for Computa- tional Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The dog-ddoumennt transformer.
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detect- ing propaganda techniques in memes. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 6603-6617, Online. Association for Computational Linguistics.
Edward L Bernays. 2005. Propaganda. Ig publishing.
Bonnie Brennen. 2017. Making sense of lies, deceptive propaganda, and fake news. Journal of Media Ethics, 179-)117--111
Wei-Fan Chen, Khalid AI Khatib, Henning Wachsmuth, and Benno Stein. 2020. Analyzing political bias and unfairness in news articles at different levels of granularity. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 149-154, Online. Association for Computational Linguistics.
Ali Fadel, Tbraheem Tuffaha, and Mahmoud Al-Ayyoub. 2019. Pretrained ensemble learning for fine-grained propaganda detection. In Proceedings of the Second Workshop on Natural Language Processing for In- ternet Freedom: Censorship, Disinformation, and Propaganda, pages 139 142, Hong Kong, China. As- sociation for Computational Linguistics.
Prafulla Kumar Choubey and Ruihong Huang. 2021. Profiling news discourse structure using explicit subtopic structures guided critics. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1594-1605. Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pra- fulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. arXiv preprint arXiv: 1909.026700
Monika Glowacki, Vidya Narayanan, Sam Maynard, Gustavo Hirsch, Bence Kollanyi. Lisa-Maria Neud- ert, Phil Howard, Thomas Lederer, and Vlad Barash. 2018. News and political information consumption in mexico: Mapping the 2018 mexican presidential election on twitter and facebook. Technical Report COMPROP DATA MEMO 2018.2, Oxford Univer- sity, Oxford, UK
Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang. and Lu Wang. 2020. Discourse as function of event: Profiling discourse structure in news articles around the main event. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5374-5386, Online. Association for Computational Linguistics.
Giovanni Da San Martino, Alberto Barrón-Cedeñia and Preslav Nakov. 2019a. Findings the NLP4IF-2019 shared task on fine- greingained propaganda detection. In
Edgar H Henderson. 1943. Toward definition of propa- ganda. The Journal of Social Psychology, 18(1):71- 87.

---

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015, Distilling the knowledge in a neural network.
David Miller and Piers Robinson. 2019. Propaganda. politics and deception. The Palgrave handbook of deceptive communication, pages 969-988.
Benjamin D. Home, William Dron, Sara Khedr, and Sibel Adali, 2018. Sampling the news producers: large news and feature data set for the study of the complex media landscape. In Proceedings of the International AAA! Conference on Web and Social Media (ICWSM), Stanford, CA.
Ray Oshikawa, Jing Qian, and William Yang Wang. 2020. A survey on natural language processing for fake news detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6086-6093, Marseille, France. European Lan- guage Resources Association.
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 7180-7199 7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. 2019. Relational knowledge distillation.
Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre. and Rada Mihalcea. 2018. Automatic de- tection of fake news. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 3391-3401, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. Semeval- 2019 task 4: Hyperpartisan news detection. In Pro- ceedings of the 13th International Workshop on Se- mantic Evaluation, pages
Matthew E. Peters, Mark Neumann. Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemyeer 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans. Louisiana. Association for Computa- tional Linguistics.
Harold D Lasswell. 1927. The theory of political propaganda. American Political Science Review, 21(3):627-631.
Yuanyuan Lei and Ruihong Huang. 2022. Few-shot (dis)agreement identification in online discussions with regularized and augmented meta-learning In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 5581-5593, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008 The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC' 08), Marrakech, Morocco. European Language Resources Association (ELRA).
Yuanyuan Lei, Ruihong Huang, Lu Wang. and Nick Beauchamp. 2022. Sentence-level media bias analy- sis informed by discourse structures. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10040-10050, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and po- litical fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2931-2937, Copenhagen. Denmark. Association for Computational Linguis- tics.
Quanzhi Li, Qiong Zhang, Luo Si, and Yingchi Liu. 2019. Rumor detection on social media: Datasets, methods and opportunities. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 66-75, Hong Kong, China. Association for Computational Linguistics.
Victoria Rubin, Niall Conroy, Yimin Chen, and Sarah Comwell. 2016. Fake news truth? using satirical cues to detect potentially misleading news. In Pro- ceedings of the Second Workshop on Computational Approaches Deception Delection, pages 7-17, San Diego, California. Association for Computational Linguistics.
Andrew T Little. 2017. Propaganda and credulity. Games and Economic Behavior, 102:224-322.
Jason Stanley, 2015. How propaganda works. In How propaganda works. Princeton University Press.
Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POLI- TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. Find- ings of the Association for Computational Linguis- ties: NAACL 2022, pages 3354-1374, Seattle, United States. Association for Computational Linguistics.
Cristina Tardaguila, Fabricio Benevenuto, and Pablo Ortellado. 2018. Fake news is poisoning brazilian politics. whatsapp can stop it. The New York Times.
Esther van den Berg and Katja Markert, 2020. Context in informational bias detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6315-6326.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
A
Appendix
A.1
ChatGPT Prompt
The designed instruction prompt for sentence-level propaganda identification task is: "Propaganda is form of misinformation deceptive narratives that incite or mislead the public, usually with political purpose. Please reply Yes if the following sentence contains propaganda content, else reply No. Sentence: "xxx". Answer:"
Prashanth Vjjyaraghavan and Soroush Vosoughi. 2022. TWEETSPIN: Fine-grained propaganda detection in social media using multi-view representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3433-3448, Seattle, United States. Association for Computational Linguistics.
The designed instruction prompt for token-level propaganda identification task is: "Propaganda is form of misinformation deceptive narratives that incite or mislead the public, usually with political purpose. Please extract the word in the following sentences that contains propaganda content. Please mimic the following output style. Example: "Of course, no "mistake" had occurred, the ban has been lifted only because of the wide publicity that we engaged in.". Words: wide, publicity. Sentence: "xxx". Words:"
George-Alexandru Vlad, Mircea-Adrian Tanase, Cris- tian Onose, and Dumitru-Clente Clementin Cercel. 2019. Sentence-lever propaganda detection in news articles with transfer learning and model. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 148-154, Hong Kong, China. Association for Com- putational Linguistics.
BiLSTM-capsule
Lingwei Wei, Dou Hu, Wei Zhou, Zhaojuan Yue, and Songlin Hu. 2021. Towards propagation uncertainty: Edge-enhanced Bayesian graph convolutional net- Proceedings of
works for rumor detection. 59th Annual Meeting of the Association for Compu- tational Linguistics and the Iith International Joint Conference on Natural Language Processing (Vol- une 1: Long Papers), pages 3845-3854, Online. As- sociation for Computational Linguistics.
Ronald Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256.
Seunghak Yu, Giovanni Da San Martino, Mitra Mo- htarami, James Glass, and Preslav Nakov. 2021. In- terpretable propaganda detection news articles. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), 1, pages 1597-1605, Held Online, IN- COMA Ltd.


Reviews:
[{"review_id": "DcsUWzvcJb", "reviewer": "Reviewer_sTTb", "summary": "", "questions": "1) Could you elaborate more on the missing document level evaluation. Is it the norm for the task to evaluate at sentence level  and token level?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions! Here are our answers (A) to your questions (Q):\n\n**Q1:** All of the empirical evaluation is performed on one dataset. This makes it hard to judge the generalizability of the approach. But, it is understandable given the difficulty in annotating data for such a task.\n\n**A1:** To the best of our knowledge, the propaganda dataset (*Da San Martino et al. 2019*) used in the paper is the only dataset that annotates propaganda at both sentence-level and token-level.\n\n\n>[1] Da San Martino, Giovanni, et al. \"Fine-grained analysis of propaganda in news article.\" Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.\n\n\n\n**Q2:** The chatgpt baseline is very rudimentary. Few-shot approach isn’t tested . Also, including the discourse relation information in the prompts  (probably in a Chain-of-Thought style approach) might yield good results. This will only add to the paper’s evaluation. But, it is extraneous to their line of evaluation as presented.\n\n**A2:** \n- We follow your suggestion to add another two baselines, and report the results for sentence-level and token-level propaganda identification in the table below.\n    - *chatgpt 5-shot baseline:* we add five examples of propaganda sentences and five examples of non-propaganda sentences into the prompt.\n    - *chatgpt + discourse structures prompt:* we add the local discourse relation and global discourse role of each sentence into the prompt.\n\n- The experimental results demonstrate that **chatgpt with extra examples or discourse structures information in the prompt can perform better, but still inferior to our developed method.**\n\n\n|          | Sentence-level | Token-level |\n| -------- | -------------- | ----------- |\n| | Precision / Recall / F1 | Precision / Recall / F1 |\n| chatgpt  | 58.26 / 34.72 / 43.51 | 13.37 / 19.31 / 15.80 |\n| *chatgpt 5-shot* | 56.42 / 37.34 / 44.94 | 14.68 / 20.84 / 17.22 |\n| *chatgpt + discourse structures prompt* | 57.93 / 38.61 / 46.34 | 15.82 / 21.96 / 18.39 |\n| longformer (baseline) | 60.32 / 60.50 / 60.41 | 34.60 / 39.81 / 37.03 |\n| knowledge distillation model (our method) | **61.22** / **69.75** / **65.21** | **37.22** / **46.86** / **41.48** |\n\n\n\n**Q3:** In addition to sentence level and token level performance, it would have been interesting to see document level evaluation of propaganda as well. It seems like a natural setting for the task which is missing in their evaluation. Could you elaborate more on the missing document level evaluation. Is it the norm for the task to evaluate at sentence level and token level?\n\n**A3:**\n- Compared to sentence and token level task, **document-level propaganda identification is a relatively easy task.** Prior work detects propaganda at document-level, which is a relatively easy task. The work in *Barrón-Cedeno et al. 2019* shows that identifying propaganda document can achieve 97.21 F1 score on TSHP-17 dataset (*Rashkin et al. 2017*), and 82.89 F1 score on QProp dataset (*Barrón-Cedeno et al. 2019*). In addition, the dataset we used consists of primarily news articles from propagandistic media sources (82.5% of article), and was designed for fine-grained propaganda identification.\n- **The propaganda research in recent years has focused on fine-grained level.** The works in recent years focus more on research propaganda at fine-grained level, such as sentence-level (*Da San Martino et al. 2019; Vlad et al. 2019; Fadel et al. 2019*), or token-level (*Da San Martino et al. 2020*). Fine-grained propaganda identification can underscore the propagandistic content and thereby provide detailed interpretable explanations for propaganda document.\n\n\n\n>[1] Barrón-Cedeno, Alberto, et al. \"Proppy: Organizing the news based on their propagandistic content.\" Information Processing & Management 56.5 (2019): 1849-1864.\\\n>[2] Rashkin, Hannah, et al. \"Truth of varying shades: Analyzing language in fake news and political fact-checking.\" Proceedings of the 2017 conference on empirical methods in natural language processing. 2017.\\\n>[3] Barrón-Cedeno, Alberto, et al. \"Proppy: A system to unmask propaganda in online news.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\\\n>[4] Da San Martino, Giovanni, et al. \"Fine-grained analysis of propaganda in news article.\" Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.\\\n>[5] Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). International Committee for Computational Linguistics.\\\n>[6] Vlad, George-Alexandru, et al. \"Sentence-level propaganda detection in news articles with transfer learning and BERT-BiLSTM-capsule model.\" Proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda. 2019.\\\n>[7] Fadel, Ali, Ibraheem Tuffaha, and Mahmoud Al-Ayyoub. \"Pretrained ensemble learning for fine-grained propaganda detection.\" Proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1) The paper does a good job of evaluating their hypothesis using both statistical and empirical evaluation. Ablation study further shows the complementary nature of the proposed approaches and hence makes the technical and conceptual contribution of the paper substantial for the task. As I haven’t worked directly on the task of propaganda detection, I am unable to confidently claim the novelty but it is novel to the best of my knowledge on this task.\n2) Results support their hypothesis and the baselines cover most of the existing state of the art models.\n3) The task is well motivated and very relevant to current social scenario. Examples given in the paper make it very easy to understand the main hypothesis and the approach clearly.\n4) The paper is well-written and very easy to follow and understand.", "reasons_to_reject": "Weaknesses:\n1) All of the empirical evaluation is performed on one dataset. This makes it hard to judge the generalizability of the approach. But, it is understandable given the difficulty in annotating data for such a task.\n2) The chat-gpt baseline is very rudimentary. Few-shot approach isn’t tested. Also, including the discourse relation information in the prompts (probably in a Chain-of-Thought style approach) might yield good results. This will only add to the paper’s evaluation. But, it is extraneous to their line of evaluation as presented.\n3) In addition to sentence level and token level performance, it would have been interesting to see document level evaluation of propaganda as well. It seems like a natural setting for the task which is missing in their evaluation.", "questions_to_authors": "1) Could you elaborate more on the missing document level evaluation. Is it the norm for the task to evaluate at sentence level  and token level?", "paper_topic_and_main_contributions": "Summary:\n\nThis paper addresses the task of propaganda detections. They posit that modeling local and global discourse relations would help in performing better on the classification task of propaganda detection. To test their hypothesis, they first perform statistical analysis on the dev set. Then, they design neural classifiers based on LongFormer to train the models for learning PDTB relations between sentences as well as discourse roles of the sentences according to a pre-defined ontology. Then, they design two models that leverage these predictions to improve on the task of sentence level and token level propaganda prediction. In the first model, they concatenate the probability outputs from the discourse relation models. In the second model, they force the propaganda detection model to mimic the outputs of the discourse relation models while learning propaganda detection in a multi-task setting. They compare the performance of this model to existing baselines and show that it performs better than the baselines on propaganda detection. Then, they perform ablation study to observe that the two types of models complement each other in performance.\n\nOverall, the paper investigates their hypothesis in a robust manner and show that the empirical evidence supports their hypothesis. They also provide examples and statistical analysis to motivate their approach. The paper could be improved by including additional datasets (hard to find for this task) and using the discourse relations and few-shot in-context learning paradigm better (by adding the discourse relation prediction to prompts and evaluating whether it improves the performance in that paradigm as well). But, these are extraneous to their line of evaluation and hence are not necessary for the main message of the paper.\n\nContributions:\n1) Interesting investigation of usefulness of modeling discourse relations for propaganda detections.\n2) Statistical analysis and empirical evaluation to back their hypothesis.\n3) Empirical results and ablation study that demonstrate the effectiveness of the proposed models and their assembly.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "IleSfcgJ0Z", "reviewer": "Reviewer_xm9p", "summary": "", "questions": "- I suggest to the authors replace “embued” with “embodied” or “embed”. \n\n- Please, could the authors provide a reference for the Propaganda definition? (e.x. Lines 028-030). \n\n- As I said before, the authors need to show clearly the difference between media bias detection and propaganda identification tasks in news political articles. For example, all examples presented by authors may be classified as media bias according to the AllSides guideline, which describes 16 different types of media bias <see https://www.allsides.com/media-bias/how-to-spot-types-of-media-bias>.  Please, could the authors clearly present the distinction between both tasks?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable reviews! Here are our answers (A) to your questions (Q):\n\n**Q1:** My main concern related to this paper consists of the propaganda identification task definition. The authors did not provide a reliable conceptualization of the propaganda task, as well as a reference for the definition of propaganda.\n\n**A1:** We have seen a pretty consistent definition of propaganda in prior works. Usually, propaganda is a type of misleading and deceptive narratives that are typically disseminated and used to influence and manipulate public opinion, attitudes, beliefs, or behaviors in a way that aligns with the interests of the propagandist or the organization disseminating the information. \n\nAnd we will add the following references *Lasswell 1927*, *Henderson 1943*, *Bernays 2005*, *Stanley 2015*, *Rashkin et al. 2017*, *Barrón-Cedeno et al. 2019* for defining propaganda.\n\n\n\n>[1] Lasswell, Harold D. \"The theory of political propaganda.\" American Political Science Review 21.3 (1927): 627-631.\\\n>[2] Henderson, Edgar H. \"Toward a definition of propaganda.\" The Journal of Social Psychology 18.1 (1943): 71-87.\\\n>[3] Bernays, Edward L. Propaganda. Ig publishing, 2005.\\\n>[4] Stanley, Jason. How propaganda works. Princeton University Press, 2015.\\\n>[5] Barrón-Cedeno, Alberto, et al. \"Proppy: Organizing the news based on their propagandistic content.\" Information Processing & Management 56.5 (2019): 1849-1864.\\\n>[6] Rashkin, Hannah, et al. \"Truth of varying shades: Analyzing language in fake news and political fact-checking.\" Proceedings of the 2017 conference on empirical methods in natural language processing. 2017.\n\n\n\n**Q2:** The authors need to show clearly the difference between media bias detection and propaganda identification tasks in news political articles. Could the authors clearly present the distinction between both tasks?\n\n\n**A2:** Thank you for your insightful comment! Media bias often refers to ideological bias these days (*Kiesel et al 2019*). Propaganda is not limited to hyperpartisan cases and its language can be more emotionally charged and may describe unverified speculations or deceptions (*Stanley 2015*). Further, many common techniques of propaganda as described in (*Da San Martino et al. 2019*) are seldom seen in bias news, such as flag waving, causal oversimplification, black-and-white fallacy and whataboutism (*Fan et al. 2019*). In short, based on our analysis, we noted two main differences between propaganda and media bias, which have also been discussed in prior work:\n- **Emotional vs. Neutral tone: Propaganda is emotional and opinionated to persuade the public, while media bias tends to express in a neutral tone.** Propaganda intensely inserts emotional or opinionated content to incite or persuade the public (*Da San Martino et al 2019*), while media bias is more often expressed in a neutral and rational way (*van den Berg and Katja 2020*).\n- **Baseless vs. Factual: Propaganda usually employs unproved and baseless allegations, while media bias selectively reports verified facts.** Propaganda usually presents unverified speculations, projections, or deceptions (*Miller and Piers 2019*; *Brennen 2017*) and detecting propaganda requires the discovery of different persuasion techniques (*Da San Martino et al 2019*; *Gupta et al 2019*; *Da San Martino et al 2020*). Media bias refers more to selective bias or framing bias, that selectively includes or omits certain facts to sway readers’ stances (*Fan et al. 2019*).\n\nIndeed, in the current media landscape, **propaganda media sources and biased media sources are labeled separately by media outlet and studied separately in the NLP community**. For instance, the Media Bias/Fact Check website organized media sources to ideologically left/right leaning sources, conspiracy sources, questionable sources which includes major propaganda sources, and a couple other categories. Looking into the existing datasets the community has created, bias news articles and propaganda news articles were retrieved from different media sources (*Barrón-Cedeno et al. 2019*; *Liu et al. 2022*).\n\n\n>[1] Miller, David, and Piers Robinson. \"Propaganda, politics and deception.\" The Palgrave handbook of deceptive communication (2019): 969-988.\\\n>[2] Brennen, Bonnie. \"Making sense of lies, deceptive propaganda, and fake news.\" Journal of Media Ethics 32.3 (2017): 179-181.\\\n>[3] Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In Plain Sight: Media Bias Through the Lens of Factual Reporting. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6343–6349, Hong Kong, China. Association for Computational Linguistics.\\\n>[4] Da San Martino, Giovanni, et al. \"Fine-grained analysis of propaganda in news article.\" Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019.\\\n>[5] van den Berg, Esther, and Katja Markert. \"Context in informational bias detection.\" Proceedings of the 28th International Conference on Computational Linguistics. 2020.\\\n>[6] Kiesel, Johannes, et al. \"Semeval-2019 task 4: Hyperpartisan news detection.\" Proceedings of the 13th International Workshop on Semantic Evaluation. 2019.\\\n>[7] Stanley, Jason. How propaganda works. Princeton University Press, 2015.\\\n>[8] Barrón-Cedeno, Alberto, et al. \"Proppy: A system to unmask propaganda in online news.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\\\n>[9] Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019. Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 162–170, Hong Kong, China. Association for Computational Linguistics.\\\n>[10] Pankaj Gupta, Khushbu Saxena, Usama Yaseen, Thomas Runkler, and Hinrich Schütze. 2019. Neural Architectures for Fine-Grained Propaganda Detection in News. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 92–97, Hong Kong, China. Association for Computational Linguistics.\\\n>[11] Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). International Committee for Computational Linguistics.\\\n>[12] Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1354–1374, Seattle, United States. Association for Computational Linguistics.\n\n\n\n\n\n\n\n**Q3:** Finally, there is a wide range of state-of-the-art propaganda identification references missing in this paper.\n\n**A3:** Thanks for your suggested papers! We have included the previous state-of-art model that achieved the best performance as one of the baselines (*Da San Martino et al 2019*). The model in the suggested papers actually are inferior to this previous state-of-art model baseline. We will follow your suggestion to add them in the paper.\n\n\n>[1] Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019. Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 162–170, Hong Kong, China. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and well-presented\n\n- The discourse framework (PDTB) used in this study is reliable and widely used by NLP literature. Furthermore, taking into consideration the nature of this task (propaganda identification), discourse structure is definitely a relevant strategy.\n\n- The experimental settings seem correct.", "reasons_to_reject": "- My main concern related to this paper consists of the propaganda identification task definition. All examples presented by authors in Figures 1 and 2 consist of examples related to the Media Bias Identification task. For example, according to a reliable guideline proposed by AllSides <https://www.allsides.com/unbiased-balanced-news>, there are 16 different types of media bias that may be found in news articles. And all examples presented by authors are defined by this guideline as media bias. Hence, I am afraid that this kind of content could be classified as biased, therefore it seems not propaganda in news political articles.\n\n- Corroborating my concerns described above, the authors did not provide a reliable conceptualization of the propaganda task, as well as a reference for the definition of propaganda. For example, in lines 028 and 030, there is not any reference for the given definition of propaganda. Therefore, the authors seem unsure about the main task of this paper. \n\n- Finally, there is a wide range of state-of-the-art propaganda identification references missing in this paper.", "questions_to_authors": "- I suggest to the authors replace “embued” with “embodied” or “embed”. \n\n- Please, could the authors provide a reference for the Propaganda definition? (e.x. Lines 028-030). \n\n- As I said before, the authors need to show clearly the difference between media bias detection and propaganda identification tasks in news political articles. For example, all examples presented by authors may be classified as media bias according to the AllSides guideline, which describes 16 different types of media bias <see https://www.allsides.com/media-bias/how-to-spot-types-of-media-bias>.  Please, could the authors clearly present the distinction between both tasks?", "paper_topic_and_main_contributions": "In this paper, the authors propose a study on the impact of discourse structure to improve propaganda identification in political news article. As an advance, they propose a fine-grained approach based on sentence-level identification and token-level identification.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I suggest to the authors replace “embued” with “embodied” or “embed”. \n\n- I encourage the authors to replace “intelligence models” with “expert models” in line 042."}, "extra_text": {"Missing_References": "- Interpretable Propaganda Detection in News Articles <https://aclanthology.org/2021.ranlp-1.179/>\n\n- A Survey on Computational Propaganda Detection <https://www.ijcai.org/proceedings/2020/672>\n\n- Predicting Factuality of Reporting and Bias of News Media Sources <https://aclanthology.org/D18-1389/>\n\n- Predicting Sentence-Level Factuality of News and Bias of Media Outlets <Proceedings of RANLP 2023>\n\n- Sentence-level Media Bias Analysis Informed by Discourse Structures <https://aclanthology.org/2022.emnlp-main.682/>"}}, {"review_id": "jeVvA4lXAN", "reviewer": "Reviewer_qWvx", "summary": "", "questions": "NA.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable questions! Here are our answers (A) to your questions (Q):\n\n**Q1:** I am highly skeptical about the reported performance of the discourse relation classification (Table 3). The authors claim over 80% accuracy for all 1st level sense classification in PDTB 2.0, whereas the state-of-the-art performance is at most 70%, with many relations performing below 60%. Notably, [1] and [2] specifically address this topic, making it unlikely for this paper, where discourse relation classification is only a sub-component, to outperform them. This raises concerns about potential implementation bugs or incorrect evaluation methods.\n\n**A1:** We are glad to clarify this misunderstanding.\n- PDTB 2.0 dataset distinguishes two types of discourse relations: explicit relations where the relation of two sentences are signified by explicit discourse connectives, and implicit relations where discourse connectives are omitted. At the coarse-grained level with the four discourse relations, the F1 score on explicit discourse relation classification can reach 90% or higher. Therefore, recent research on PDTB discourse relation classification primarily focuses on implicit relations. Both of the two papers your pointed to work on implicit relations as their titles indicate this as well. But, as stated in section 2.1.3 (line 178-181), **both explicit and implicit relations in the PDTB 2.0 dataset are used for training our discourse relation model.**\n- To further alleviate your concern, we report the performance of our discourse relation model on explicit and implicit relations separately in the table below:\n\n\n|  | Precision | Recall | F1-score |\n| -------- | -------- | -------- | -------- |\n| Explicit relations | 96.62 | 96.49 | 96.55 |\n| Implicit relations | 61.50 | 60.60 | 61.05 |\n| Both explicit and implicit (reported in the paper) | 83.60 | 83.59 | 83.55 |\n\n\n[1] A Side-by-side Comparision of Transformers for English **Implicit** Discourse Relation Classification https://aclanthology.org/2023.codi-1.2.pdf \\\n[2] Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level **Implicit** Discourse Relation Recognition https://aclanthology.org/2023.acl-long.357.pdf\n\n\n\n\n**Q2:** There appears to be a lack of synergy between learning global and local discourse structure. The ablation analysis in Table 6 is superficial and fails to explain the reasons behind this synergy. The absence of even a simple example to shed light on this issue is concerning. The paper attempts to link this observation with global discourse roles, but the analysis lacks evidence of the \"cross feature\" between local and global features that leads to propaganda.\n\n\n**A2:** We address your question as follows:\n- **The ablation study in Table 5 shows that both local and global discourse structures are essential for identifying propaganda.** We did conduct the ablation study of local discourse relation and global discourse role in Table 5 and illustrate in section 4.6 (line 466-472). We actually finished two types of ablation study in the paper: the first is studying the effect of local and global discourse structures (Table 5); the second is studying the effect of two knowledge distillation strategies (Table 6).\n- **The statistical analysis in Table 1 and Table 2 proves that both local and global discourse structures provide auxiliary information for identifying propaganda.** Statistical analysis in Table 1 (section 2.1.2) and Table 2 (section 2.2.2) confirm that different local discourse relations and global discourse roles present different likelihoods of carrying propaganda, and thus can provide auxiliary information. These statistical analysis provide evidence for us to incorporate both local and global discourse structures into propaganda discovery.\n- **We follow your suggestion to add an analysis example to show the synergy effect of local and global discourse structures.** For the propaganda sentence S2 in the example article 1 of Figure 1: \"*This suggested of course that giving advice on how to prevent another 9/11 is now against Facebook's community standards.*\", the probability of correctly classifying as propaganda of baseline model, adding local discourse relation, add global discourse role, as well as adding them both are shown in the table below. Incorporating either local or global discourse structure can increase the probability confidence and solve false negative error, while integrating them both yields the best confidence.\n\n\n\n| Model | Discourse Structures Information | Prediction |\n| -------- | -------- | -------- |\n| baseline | None | P (propaganda \\| baseline) = 0.33 |\n|+ local discourse relation | local discourse relation is *causal* | P (propaganda \\| + local) = 0.77 |\n|+ global discourse role | global discourse role is *evaluation* | P (propaganda \\| + global) = 0.64 |\n| + both (full model) | both local and global | P (propaganda \\| full model) = 0.86 |\n\n\n\n\n**Q3:** The overall theoretical foundation of the paper requires further development. While the observation that Contrast / Contingency relations are often used for propaganda is noteworthy, it falls short of providing a holistic theory to support this claim...It seems that a discourse relation classifier might not be necessary, and instead, a contrast / contingency detector should be considered, as there is no discussion on expansion / temporal relations...\n\n**A3:** We clarify your concern as follows:\n- In our introduction, we highlighted using Contrast / Contingency discourse relations for propaganda detection as our data analysis shows strong positive correlations between these two relations and propaganda, consistent with some relevant discussions in the literature (*Horz and Korhan 2022, Petee and Alexis 2020, Jowett and Victoria 2018, Shabo 2008, Ecker et al. 2022*). But **we actually analyzed all the four types of discourse relations as shown in Table 1, and noted in section 2.1.2** (line 161-169) that in addition to contingency and comparison relations, temporal relations are clearly useful for propaganda detection as well which however are negatively correlated with propaganda: “Table 1 shows the ratio of propaganda sentences that have each of the four discourse relations with nearby sentences. The numerical analysis confirms our observation: sentences that exhibit contingency and comparison relations with adjacent sentences are more prone to containing propaganda, whereas sentences that narrate events in a chronological order significantly contain less propaganda.”\n- **To further alleviate your concern, we did an extra ablation study focusing on the four relations, which demonstrates the usefulness of all the four discourse relations.** The ablation is on the sentence-level propaganda identification task, the following table shows that removing each one of the four relations leads to a performance drop compared to the full model, as expected, the influence of expansion relations is relatively less compared to the other three relations. The results prove the necessity and effectiveness of all the four relations.\n\n\n|  | Precision | Recall | F1-score |\n| -------- | -------- | -------- | -------- |\n| longformer (baseline) | 60.32 | 60.50 | 60.41 |\n| knowledge distillation model (full model) | **61.22** | **69.75** | **65.21** |\n| *comparison* relation removed | 60.91 | 67.76 | 64.15 |\n| *contingency* relation removed | 60.89 | 67.56 | 64.06 |\n| *temporal* relation removed | 61.03 | 68.16 | 64.38 |\n| *expansion* relation removed | 61.19 | 68.65 | 64.70 |\n\n\n\n>[1] Horz, Carlo, and Korhan Kocak. \"How to keep citizens disengaged: Propaganda and causal misperceptions.\" (2022).\\\n>[2] Petee, Maia, and Alexis Palmer. \"UNTLing at SemEval-2020 Task 11: Detection of Propaganda Techniques in English News Articles.\" Proceedings of the Fourteenth Workshop on Semantic Evaluation. 2020.\\\n>[3] Jowett, Garth S., and Victoria O'donnell. Propaganda & persuasion. Sage publications, 2018.\\\n>[4] Shabo, Magedah. Techniques of propaganda and persuasion. Prestwick House Inc, 2008.\\\n>[5] Ecker, Ullrich KH, et al. \"The psychological drivers of misinformation belief and its resistance to correction.\" Nature Reviews Psychology 1.1 (2022): 13-29.\n\n\n\n\n\n**Q4:** The claim for token-level analysis is weak. Despite the authors' assertion that token-level information is included, the paper lacks evidence pinpointing specific lexical choices or phrases that contribute to propaganda.\n\n**A4:** We are glad to provide some representative examples for token-level propaganda identification, where the propagandistic spans are shown in italic:\n- Example 1: “Vichy journalism,” a term which now fits so much of the mainstream media - *it collaborates in the same way that the Vichy government in France collaborated with the Nazis.*\n- Example 2: His opinion is: “Take it seriously, but with a large grain of salt.” *Which is just Allen’s more nuanced way of saying: “Don’t believe it.”*\n- Example 3: It describes the tsunami of vindictive personal abuse that has been heaped upon Julian from well-known journalists, many claiming liberal credentials. The Guardian, *which used to consider itself the most enlightened newspaper in the country*, has probably been the worst.\n\nOverall, propagandistic spans are usually baseless, speculative, opinionated content with persuasion techniques to convince or mislead the readers. We will add this analysis to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper makes a valuable contribution to the Linguistic side of NLP by uncovering the insight that certain discourse relations can lead to propaganda.\n- The experiments are substantial, but there are implementation and theoretical issues that need to be addressed.", "reasons_to_reject": "Weaknesses (W) of the paper:\n\nW1: The claim for token-level analysis is weak. Despite the authors' assertion that token-level information is included, the paper lacks evidence pinpointing specific lexical choices or phrases that contribute to propaganda.\n\nW2: I am highly skeptical about the reported performance of the discourse relation classification (Table 3). The authors claim over 80% accuracy for all 1st level sense classification in PDTB 2.0, whereas the state-of-the-art performance is at most 70%, with many relations performing below 60%. Notably, [1] and [2] specifically address this topic, making it unlikely for this paper, where discourse relation classification is only a sub-component, to outperform them. This raises concerns about potential implementation bugs or incorrect evaluation methods.\n\nW3: There appears to be a lack of synergy between learning global and local discourse structure. The ablation analysis in Table 6 is superficial and fails to explain the reasons behind this synergy. The absence of even a simple example to shed light on this issue is concerning.\n\nW4: The overall theoretical foundation of the paper requires further development. While the observation that Contrast / Contingency relations are often used for propaganda is noteworthy, it falls short of providing a holistic theory to support this claim. The paper attempts to link this observation with global discourse roles, but the analysis lacks evidence of the \"cross feature\" between local and global features that leads to propaganda. As a result, it seems that a discourse relation classifier might not be necessary, and instead, a contrast / contingency detector should be considered, as there is no discussion on expansion / temporal relations. The soundness of this aspect of the paper requires significant improvement, leading to a recommendation of a low soundness score for this manuscript, despite the idea is still exciting and worth encouragement.\n\nReferences:\n\n[1] A Side-by-side Comparision of Transformers for English Implicit Discourse Relation Classification  https://aclanthology.org/2023.codi-1.2.pdf \n\n[2] Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition  https://aclanthology.org/2023.acl-long.357.pdf", "questions_to_authors": "NA.", "paper_topic_and_main_contributions": "This paper explores using local discourse relations and global discourse roles to improve propaganda detection.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA.", "Typos_Grammar_Style_and_Presentation_Improvements": "NA."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a well-presented paper which connects local (sentence, token-level) propaganda signals with global discourse structure, via both statistical analyses and empirical evidence via discourse aware propaganda prediction models.

The contribution on the conceptual (local vs global aspects of propaganda) and empirical (strong evaluation) levels are strong and are likely of broader interest to researchers in the field of propaganda prediction.

One remaining concern with the paper is its contextualization in theory and related work, by (1) incorporating a clearer definition of propaganda and distinguishing it from the phenomenon of "media bias" and (2) a more structured and exhaustive review of relevant related work as per the detailed suggestions by reviewer xm9p.
Decision: Accept-Main