# Improving the Debate: Interface Elements that Enhance Civility and Relevance in Online News Comments

Emilie Bossens, Elias Storms, David Geerts

<figure><img src="image_1.png" /></figure>

## To cite this version:

Emilie Bossens, Elias Storms, David Geerts. Improving the Debate: Interface Elements that Enhance Civility and Relevance in Online News Comments. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.433-450, 10.1007/978-3-030-85610-6_25. hal-04215483

HAL Id: hal-04215483

https://inria.hal.science/hal-04215483v1

Submitted on 22 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Improving the Debate: Interface Elements that Enhance Civility and Relevance in Online News Comments

Emilie Bossens[0000-0001-8130-6973], Elias Storms[0000-0002-7613-481X] and David Geerts[0000-0003-3933-9266]

Meaningful Interactions Lab, Mintlab - KU Leuven, Leuven, Belgium
{emilie.bossens, elias.storms, david.geerts}@kuleuven.be

**Abstract.** Online news websites face an increasing amount of hateful reactions of readers, and reactive attempts at enabling more civil discourse, such as moderation, do not prove sufficient. In this study, we propose a proactive approach to counter anti-social behavior by redesigning the comment section's interface. We conducted an exploratory online experiment with 255 participants to determine the impact of including a discussion statement related to the news article's topic in combination with two types of opinion elements. Results reveal a significant positive effect of the novel interfaces on civility and relevance, compared to a traditional comment section (control group). Users evaluated the pragmatic qualities of the traditional comment section significantly more positive compared to the interface with a discussion statement and the continuous opinion element.

**Keywords:** User Interfaces, Commenting, Online News.

## 1 Introduction

Comments posted on news platforms do not always live up to the ideals of rational reasoning and civility [7]. Often, contributions are irrelevant, uncivil or misleading [13, 18]. Consequently, many journalistic outlets resort in reactive approaches to counter those unwelcome low-quality comments, such as deleting comments, blocking commenters or, more drastically, closing comment sections. Although moderation interventions do have an impact, doing it properly is highly challenging, resource-intense and the results are not always satisfactory [38]. Alternatively, a proactive approach attempts to counter anti-social behavior by supporting civil and respectful discourse in the comment section itself.

In this contribution we explore the potential of such a proactive approach by redesigning the comment section's interface to stimulate healthy conversations. First, we address framing and negativity bias by
narrowing down the focus of the debate and proposing a well-defined
frame. Therefore, we include a conversational prompt in the interface,
namely a specific statement to comment on, to nudge or encourage us-
ers to leave more constructive comments and to discourage users to ex-
pand on controversial details. Second, to guide participants through
their commenting process, we add an opinion element below the state-
ment which serves as an anchor from which subjects can start writing.
In this research, we explore how the presence of a statement, whether
or not combined with an anchor, can influence argumentation, civility,
relevance, orthography and length of participants' comments.

Our exploratory online experiment with 255 participants found evi-
dence for a positive impact of the novel interface elements on civility
and relevance. More specifically, when a comment section's interface
contains a discussion statement, with or without an opinion element,
users tend to be more civil and stick more to the central topic compared
to users facing a traditional comment section.

This study contributes to research on online discussions and user in-
terfaces in a news environment in two ways. (1) Based on empirical in-
vestigation, we reveal insights about how specific interface elements
impact the quality of online news comments below controversial news
articles in terms of argumentation, civility, relevance, orthography and
length. (2) This work opens up new directions in enhancing the quality
of the online debate of news outlets by proposing novel interface ele-
ments that can be incorporated in a comment section. Such interface de-
signs could assist newsrooms that maintain comment sections and other
forms of reader engagement, and contribute to the ideal of online dis-
cussion as a space for public deliberation.

## 2 Background and Related Work

2.1 Online Discourse in a News Environment

With the advent of ‘Web 2.0’, our online world has turned into a ‘par-
ticipatory web’ [8]. Such nomenclature refers to, among other things, a
broad participation in discussions in online spaces. Audiences became
more than mere spectators, but could contribute insights and voice
opinions. These developments had an impact on news media and the re-
lationship with their readership, as many journalistic outlets attempted
to invite contributions from readers via polls, message boards and com-
ment sections [33].

As news media play a crucial role in fostering a constructive debate,
these new tools for discussion with and among journalists and their au-
dience can be seen as part of an attempt to create a new public sphere.
On a broader level, then, the integration of commenting platforms di-
rectly below news stories are an attempt to contribute to the ‘delibera-
tive potential’ of rational and civil discussion in a digital public sphere
[42]. Besides these lofty moral ideals, commenting sections and other
forms of audience participation provide journalistic opportunities to en-
gage with the audience [40] or can simply serve as features that draw
and retain audiences to news websites. As it stands, however, comment
sections seem unable to realize either of these goals. While different
themes and newspapers result in variations in the tone of the debate,
reader comments on newspaper websites are often uncivil or outright
hateful [58]. The general toxicity in such discussion spaces means that
maintaining comment sections quickly becomes frustrating and labor
intensive. Especially reactive interventions, such as content moderation,
take up a substantial number of human resources. As a result, many
newspapers have decided to close down their comment sections.

While maintaining a comment section involves considerable effort,
research suggests that fighting against toxic participation can benefit of
a double win: it increases the overall debate quality and lightens the
burden of subsequent moderation. In this context, Fredheim [23] refers
to an ‘inverse’ broken window effect in online comment sections. An
overall increase of comment quality (and thus a metaphorical repair of
broken windows) stimulates positive behavioral change in other users.
An increase in comment quality can spur a virtuous circle. In other
words: increasing comment quality could have effect beyond the com-
ments themselves.

## 2.2 Cognitive Biases and Debiasing Strategies to Overcome them

One type of psychological mechanisms that stand in the way of higher
quality comments are cognitive biases. While interpreting information,
for example reading news articles, our mind tries to simplify infor-
mation processing by using ‘rules of thumb’ to make judgements and
decisions with relative speed [71]. In the case of writing online com-
ments, people decide what to write not just based on rational infor-
mation but also on mental shortcuts, feelings and emotions [1]. Alt-
hough such heuristics can mostly prove effective, when people blindly
rely on filtered information or are not receptive to update their opinion
in light of new evidence, these shortcuts can lead to suboptimal deci-
sions, in the sense that important information is also filtered out [71].

One common bias related to news environments is framing bias. Framing refers to the way information is presented which in turn influences how people orient their thinking about an issue. Even subtle changes in the rhetorical frame of the information presentation, such as prioritizing particular interpretations, can significantly influence people's responses [34, 59].

Another bias especially prominent when reading news is the negativ-
ity bias or the tendency to pay more attention or to react more strongly
to negative stimuli than to positive stimuli [4, 5, 53]. Research has also
shown that people perceive negative information as more salient and
identify those stimuli with greater ease [53]. That said, it is not surpris-
ing that negativity is considered as one of the news values in journal-
istic agenda setting [25, 39].

A way to reduce the negative effects of cognitive biases is by deploy-
ing debiasing techniques. Using certain strategies, individuals could
change their thinking to be less susceptible to biases, debias themselves
and consequently become more open, constructive and tolerant [15,
44]. Prior research explored various debiasing approaches [12] ranging
from simply mitigating the undesired effect of a bias (such as blind re-
views) to putting external constraints to force people to detect and cor-
rect potential biases in reasoning. Here, Correia [14] refers to the notion
of ‘choice architecture’ of Sunstein and Thaler [68], a model that seeks
to promote rational reasoning by setting external constraints on the con-
texts so that individuals are less likely to make decision errors. A con-
cept closely related to this approach is nudging. Hence, debiasing can
take equally subtle forms in the context of online comments, guiding
participants’ attention in a specific direction [20].

In computer-mediated environments, debiasing or nudging can be
achieved by intentionally designing interfaces with the goal of chang-
ing a person's attitude or behavior in a certain way [22]. In the context
of online discussions, a series of studies found evidence for strategies
that can positively affect the commentary such as altering the structure
of the comment section [52, 67], applying a specific rhetorical frame
[34, 53, 59, 67], providing a short summary to highlight relevant facts or to present contrasting information [52], showing visualizations [34], exposure to CAPTCHA's that prime positive emotions [62], using conversational prompts such as posing questions [16, 69, 49], or through word count anchors and partitioned text fields [48].

More specifically, Peacock [52] found that presenting news readers arguments in favor of and opposed to a statement at the top of the comment section, rather than offering some facts, increased both the quantity and quality of the comments. Another recent experiment [48] reports that a word count anchor below a text field, showing the length of the comment, had a significant positive impact on the number of arguments and respect towards or engagement with others' arguments.

Although several studies were successful in debiasing in the context of online news comments, other results were mixed or inconclusive [3, 14, 44, 48]. For example, based on a quasi-experiment, Stroud [66] suggests that when posing concrete questions in comment sections on Facebook incivility may decrease, but also other forms of engagement such as asking genuine questions and providing evidence may be reduced. The author also implied that prompting participants to make yes- or no-judgements may lead to less substantive discussion. In contrast, inviting people to leave comments without a question, such as ‘Tell us your thoughts’, was related to increased incivility and reduced chances of offering evidence. Because these effects were only marginally significant, the researchers called for additional investigations.

## 2.3 Defining Comment Quality

Beyond studying the underlying mechanisms leading to low-quality contributions and the techniques to overcome them, a challenging question arises: What exactly describes a high-quality comment? In other words, the challenge lies in finding agreement on what separates a high-quality comment from a low-quality one [7]. In literature, the quality of online discourse has been approached as a multifaceted and complex construct, and various efforts have been undertaken to formulate a definition and a set of normative criteria [9, 18, 20].

Literature often builds on theories of deliberation and group norms to define comment quality. Stroud [66] for example defined the quality of news comments based on incivility, provision of evidence, relevance,
and asking genuine questions. More recently, Beckert [19] used two in-
dices, one reflecting deliberativeness (including number of arguments,
level of elaboration, information value and comment length) and the
other one reflecting (in)civility (including disrespectful language, polar-
ization, simplification and humor). Fredheim [23] measured discourse
quality by the use of offensive language, length, the number of proper
names (reflecting personalization of the debate) and the comments' or-
thography and structure. Other attributes of comment quality found in
literature are meta-talk [13, 30], engagement with others [7, 9, 13, 20]
and additional perspectives [30].

One paper approached the challenging question by studying profes-
sionally curated comments: “the NYT Picks” [18]. Based on interviews
with moderators, the researchers reported five criteria considered when
identifying high-quality comments on a news site: (1) overall quality
(spelling, grammar, argumentation, literary value), (2) diversity of per-
spectives, (3) constructive dialogue with others, (4) short, funny or unu-
sual comments, and (5) relevant personal stories and experiences. The
moderators emphasized that these five criteria apply to different situa-
tions and that there is no one-size-fits-all model.

For further guidance on which dimensions are critical for identifying
qualitative comments posted in response to news stories, we reviewed
the comment policies of five local online newspapers that have a com-
ment section for their readers. We found that each of these five news
outlets defined guidelines with regard to reasoning, civility, relevance
and orthography (such as punctuation, words in all capitals, language).
Three newspapers included recommendations about comment length.

These criteria for comment quality correspond with the studies men-
tioned above. In the following paragraphs, we turn to literature to fur-
ther reflect on these dimensions. These dimensions will then serve as
criteria that allow us to code the comments in our experiment, thus ena-
bling more detailed analysis.

**Argumentation.** One key characteristic of qualitative commenting is reasoned discussion or supporting one’s claim with explanations and evidence [24, 52, 65, 70, 73]. Simply expressing an opinion or making an assertion without any kind of validation needs to be distinguished from a justified claim [29, 37, 72]. In addition, the source of validation can vary in legitimacy, ranging from non-sensical rhetorical argumentation to substantiated argumentation [70, 72, 75]. Furthermore, argumentation is often considered as an element of deliberative quality
[6,21, 52, 65], a concept that relates to Habermas' framework of the de-
liberative public sphere [24]. According to Engelke [20], users value
deliberative elements such as rational arguments in comment sections,
ensuring the quality of the online discourse. Another study shows that
when participants in a political discussion focused on carefully structur-
ing a valid argument, their contributions were more civil [50]. When
we consider the local newspapers' commenting policies, the guidelines
highlight that claims should be supported with (correct) facts, experi-
ences, expertise, examples, etc.

(In)civility. A second criterion that has been proposed as relevant when describing comment quality is the level of (in)civility. Incivility is a complex construct since uncivil speech can take different forms and so-called uncivil language can be perceived differently depending on individual and group characteristics and on the context [10, 13]. Despite the relativeness of the concept, several scholars have attempted to offer some theoretical clarity. Coe [13] for example describes incivility as “features of discussion that convey an unnecessarily disrespectful tone toward the discussion forum, its participants, or its topics”. Similarly, Ksiazek [42] identifies uncivil comments as “intentionally designed to attack someone or something and, in doing so, incite anger or exasperation through the use of name-calling, character assassination, offensive language, profanity, and/or insulting language”. Uncivil comments on online platforms can increase polarization [2], reduce the willingness of others to participate in the conversation [31] or encourage others to leave uncivil responses as well [10]. Civility, on the other hand, is usually defined as the mere absence of hostility [42, 58], but Herbst [32] offers a more elaborate definition, namely “constructive engagement with others through argument, deliberation and discourse”. Based on the review of the newspapers’ policies, each of them clearly addresses the issue of respectful commenting, reflecting the elements discussed in literature such as no hate, racism, personal attacks, name-calling, offensive language, etc.

**Relevance.** Third, quality comments are relevant and address the topic at hand [7, 30, 65, 70]. The link between a comment's relevance to the news article and the editor's selection of high-quality posts has been confirmed in a study [18] that shows that comments that are more on-topic are significantly more likely to be selected by an editor as “most interesting and thoughtful comments” (“NYT Picks”). Accordingly, the newspapers' policies we reviewed call for comments that do
not stray from the central theme of the respective article. Comments addressing spelling errors, the functioning of the media outlet or irrelevant topics are not desired.

**Orthography.** A dimension associated with civility is the degree to which the comment's orthography is clear and correct. In computer-mediated communication, using provocative punctuation and capitalization of words, for example, are considered yelling [10, 17, 63]. This type of discourse functions as a strategy to evoke the sound and intonation of spoken words and is widely used in hostile and uncivil interactions [10, 17, 26]. Qualitative comments, by contrast, are characterized by correct spelling, grammar and punctuation and are easy to read [51]. This dimension is also reflected in the newspapers' policies, requesting readers to use clear and correct language, and to avoid words in all capitals or excessive punctuation.

**Comment length.** Expressing a compelling, deliberative argument is assumed to require some space and longer messages can represent more (complex) ideas [9, 10, 29, 22]. Therefore, comment length is final dimension worth taking into account. Prior research has found that longer comments are more likely to contain evidence or legitimate questions, but also to be more uncivil, compared to more concise ones [10]. On the other hand, based on the reviewed policies, newspapers ask their users to remain concise and to not write excessively long comments.

## 3 Research Question

In the current paper we explore a proactive approach to comment quality based on debiasing and nudging theory, whereby we try to subtly encourage participants to engage in debate in a more constructive manner via the implementation of specific interface elements. To translate this goal into a specific interface element, the effects of which we can test, our research focus is two-fold.

First, we address framing and negativity bias by narrowing down the focus of the debate and proposing a well-defined frame. We argue that since a journalist often makes multiple statements in a news article, it might be hard for commenters to agree or disagree with the full article. Therefore, we expect that narrowing down the focus of the discussion using a conversational prompt, i.e. request users to comment on a specific question or statement, can encourage users to comment more con-
structively. More specifically, we present a concrete discussion statement related to the news article's topic on which users can comment. Since research has shown that relevant commenting is related to higher civility and reasoning, we expect that including a discussion statement has a positive impact on the comment quality variables.

Second, to nudge commenters to take a stance and justify their opinion with arguments, the discussion statement will be expanded with an opinion element that acts as a reference point or anchor [71]. Given that prior information can serve as a reference point from which people make later decisions [71], we combine the statement with two other interface elements: (1) a way for commenters to indicate their opinion, using either two opinion buttons or an opinion scale, and (2) a text box prompting participants to clarify their opinion. As such, the anchor point guides users in the interaction in the sense that participants can set a reference point for themselves, that is the degree to which they agree or disagree with the statement, from which they can start clarifying their opinion with arguments.

In order to evaluate the quality of the posted comments, we take into account the five quality dimensions which we earlier discussed in the literature review and to which we come back in the measurements section. More specifically, this paper focuses on how the design of the comment section in an online news environment can influence the quality of the comments in terms of argumentation, civility, relevance, orthography and comment length. This leads to the following research question: *What is the impact of the presence of a statement, whether or not combined with a way to indicate an opinion, on the quality of the comments posted?*

# 4 Experimental Design and Method

## 4.1 Data Collection and Participants

To examine our research question, we conducted an online experiment using Qualtrics and asked participants to comment on a news article. Subjects were recruited through paid social media posts, a call in the comment section of a local news website and an item in the newsletter of a research platform. A total of 258 subjects participated in the study. Three participants left a comment containing only ‘yes’ or a punctuation mark and were therefore omitted from the analysis. The remaining
sample consisted of 255 participants (108 males, 113 females and 34 did not disclose their gender) between 16 and 84 years old (M = 47.33, SD = 20.09). Gender, age and education were evenly distributed across the conditions. As an incentive, participants had a chance to win one of the ten gift vouchers of 15 euros. The study was approved by the local Social and Societal Ethics Committee.

## 4.2 Experimental Stimulus

To assess whether the novel interfaces have an impact on the quality of participants' comments, a suitable stimulus, more specifically a news article, was needed. First of all, to limit survey time and to ensure the stimulus was comprehensible for a wide audience, the text had to be short and easy to read. Second, with the aim of having a base level of hateful debate, the content of the news article had to deal with a topic that elicited a high level of heated and polarized reactions. Finally, it was important that the article was published recently so that participants could more easily formulate an opinion with regards to the topic.

The stimulus selected was an article about a local far-right federal MP, Dries Van Langenhove, caught attending a 'lockdown party'. Van Langenhove is the founder of an ultra-conservative student organization and has been a fierce critic of the government's coronavirus policies. Earlier during the Covid-19 lockdown, Van Langenhove publicly criticized citizens going to lockdown parties in a Twitter post, which was included in the article. Consequently, the news article provoked a high number of uncivil comments on social media.

To simulate a real discussion and thus increase ecological validity, we included three comments below the article. These comments were based on Facebook comments and modified to match realistic language and not violate privacy. They varied in tone, quality and language, and were presented on behalf of fictive as well as anonymous user names. In the experimental conditions, the statement presented below the news article and comments was articulated as follows: "A politician must also privately behave according to his or her exemplary role."

## 4.3 Experimental Setup

At the start of the experiment, respondents in all conditions were instructed to read the news article and comments on the next screen, and
to leave a spontaneous comment of their own. With the goal of emulating a real online discussion, participants were led to believe that other people would be able to read their comment too and that these other people might in turn react on their comment. Before heading to the next screen to start the writing task, participants had to fill in their (nick)name or choose the option ‘anonymous’<fnref n="1" />.

Next, participants were randomly assigned to one of the four conditions. All respondents were shown the same news article including three prior comments, but the interface differed according to the condition. The four conditions were: (1) control condition, the news article and comments were followed by the text ‘Leave your comment below’ and a text box, comparable to a traditional comment section; (2) STATopen, under the article and comments a discussion statement was presented, followed by the text ‘Leave your comment about the statement below’ and a text box; (3) STATbuttons, the same statement was shown including two buttons ‘agree’ and ‘disagree’, followed by the text ‘Why do you agree or disagree?’ and a text box; and (4) STATscale, instead of two buttons a scale was included with at the left side ‘disagree’ and at the right ‘agree’. The scale was followed by ‘Why do you agree or disagree?’ and a text box (see ).

<figure><img src="image_4.png" /></figure>

<footnote n="1">¹ The choice to request participants’ name was relevant with the goal of making the experiment as realistic as possible, since many online platforms do not allow anonymous comments. However, during the first days of the pilot study, many respondents left the questionnaire at this stage. Based upon this finding, an ‘anonymous’ option was added, and consequently, the completion rate of the pilot survey increased.</footnote>
<caption>Fig. 1. Condition STATscale</caption>

A pilot-study with a similar set-up revealed that the comment section's layout was a fair representation of commenting on a real news site, that users had the feeling that other participants would read their comments and that they had commented almost the same way as they would do otherwise.

## 4.4 Questionnaire

After submitting their comments, participants completed a series of extra questions. First, with the aim of capturing participants' impression of the comment section, the short version of the User Experience Questionnaire [60] was included. This validated questionnaire contains eight items representing the meta-dimensions pragmatic quality and hedonic quality of the original UEQ [21]. Each dimension consists of four pairs of terms with opposite meanings and participants rate each pair on a scale from 1 (if the alternative on the extreme left is marked) to 7 (if the alternative on the extreme right is marked). The four pairs of pragmatic quality are confusing vs. clear, inefficient vs. efficient, complicated vs. easy, and obstructive vs. supportive. The four pairs of hedonic quality are boring vs. exciting, not interesting vs. interesting, conventional vs. inventive, and usual vs. leading edge. A reliability test showed that both the pragmatic and hedonic quality subscales had high reliabilities, with a Cronbach's alpha of respectively .802 and .749.

Following the UEQ, participants were asked to share how frequently they read, write and vote on comments on online platforms, and to provide basic social and demographic characteristics. Since this study involved the use of deception, an elaborate debriefing was included at the end of the questionnaire.

## 4.5 Operationalization and Measurement

To study variations in the quality of the comments, we carefully constructed a classification system to provide a valid and reliable measurement of comment quality, as suggested by Riffe [54]. As discussed in related work, several schemes for classifying online discourse have been developed by researchers, with both manual and automatic approaches. Since we did not find a coding scheme in existing literature
that reflects all relevant aspects for assessing the quality of an individ-
ual comment in an online news environment, we developed a new
codebook for rating news comments based on a review of multiple
prior studies [1, 8, 10, 12, 28, 29, 30, 37, 42, 51, 58, 70, 76]. As this litera-
ture discusses a wide range of quality criteria, we narrowed down the
relevant dimensions based on the comment policies of five local online
newspapers. We deliberately chose a manual coding approach over an
automatic classification to not overlook textual nuances in comments
[64].

One coder being blind to the condition coded all comments accord-
ing to five criteria. To assess reliability, a second coder categorized a
random set of 20% of all comments. For each variable, both percentage
of agreement and Krippendorff’s alpha were computed and reported
(see Table 1). The true level of reliability falls between those two
measures [54]. As can be seen in Table 1, agreement for the argumenta-
tion criteria is rather low [41]. However, since all comments were
coded by one researcher and since we are interested in the relative dif-
ference between the conditions rather than absolute scores, we are con-
vinced that this is not a major concern for our exploratory study.

**Argumentation.** Argumentation was measured based on the Asser-
tion, Reasoning and Evidence model [35, 46]. The three steps include
“0” = assertion or expressing a simple opinion (“*If you don’t set an ex-
ample as a politician, you’re doing a bad job*”<fnref n="2" />), “1” = reasoning that
supports the assertion, or the ‘because’ part of an argument, whether
simple or complex (“*That’s not something you do. Rules are rules and
you follow them. After all, it is for everyone’s safety*”), and “2” = evi-
dence that supports the reasoning, whether valid or not, such as refer-
ring to an example, expert, personal story, news facts, etc. (“(...) *On the
other hand, all Corona measures violate our fundamental rights
“treaty of human rights”. We live in a rule of law, which means that
the measures are invalid. (...)*”)

**Civility.** For measuring civility, we drew from past research [57, 58].
Civility was coded on three levels: “0” = explicitly offensive, disre-
spectful, threatening, vulgar or hateful language or a personal attack
(“*Fuck him*”), “1” = implicitly unrespectful, forceful or crude com-
ment, or expression of scorn, ridicule, derision or disapproval, without
resorting to hateful language or personal attacks (“*They should put him
'civility' into English.*”)

<footnote n="2">² Original comments were translated into English.</footnote>
in strict quarantine for 2 weeks”), and “2” = none of the uncivil elements and free of any insults (“Anyone can make mistakes.”)

**Relevance.** All comments were reviewed to assess whether they were related to the news article’s topic or not. In line with previous research [30, 65, 52, 66], when (the majority of) a comment was not related to the central theme of the article or when the comment was only one word long (e.g. “hypocrisy”), it was coded as “0”. Comments that were relevant to the news article’s topic were coded as “1” (“It is a pity that he actually does not stick to the rules himself while he profiles himself that way on Twitter ...”).

**Orthography.** A comments’ orthography was measured on a dichotomous scale with code “0” for comments that were not clear, difficult to read, contained at least 2 spelling errors, no or excessive punctuation (e.g. !!! or ???) or words in all capitals. Comments that were clear, easy to read and (largely) correctly written obtained code “1”.

**Comment length.** To quantify comment length, we calculated the number of words per comment and compared this to the overall median. Comments with less than 25 words were given score “0”, whereas comments with 25 or more words were given score “1”.

<table><caption>**Table 1.** Intercoder Reliability Coefficients</caption><thead><tr><td>Coding category</td><td>Percentage of agreement</td><td>Krippendorff's alpha</td></tr></thead><tbody><tr><td>Argumentation</td><td>66.67</td><td>.573</td></tr><tr><td>Civility</td><td>82.35</td><td>.775</td></tr><tr><td>Relevance</td><td>94.12</td><td>.768</td></tr><tr><td>Orthography</td><td>92.16</td><td>.77</td></tr></tbody></table>

## 5 Data Analysis and Results

Participants in our study are already active in online discussions. More than three out of four respondents (78%) reported reading comments on social platforms at least several times a week. 28.1% of participants stated that they regularly write online comments and 41.5% indicated that they frequently engage in voting such as liking, upvoting or down-voting comments.

Before starting the writing task, 71.4% of participants filled in their (nick)name, whereas 28.6% continued anonymously<fnref n="3" />. Across all condi-

<footnote n="3">³ We found no significant impact of anonymity on the comment quality variables.</footnote>
tions, comments averaged 31 words and varied between 1 ("hypoc-
risy") and 181 words in length. In the following paragraphs, we will
first look at the impact of the interface on comment quality, after which
we will present the results of the user experience questionnaire.

**5.1 Comment Quality**

We used IBM SPSS software to perform statistical analyses. Since the
presumption of normal distribution of observed variables was not justi-
fied, we used non-parametric tests. See Table 2 for a complete over-
view of means and standard deviations by condition for the comment
quality variables and Table 3 for the results of the post-hoc tests.

Results of a Kruskal-Wallis test (with a Bonferroni-adjusted signifi-
cance level of .01 at a 95% confidence level) showed a significant dif-
ference between the four conditions for degree of civility (H(3) =
26.453, p < .001) and degree of relevance (H(3) = 14.048, p < .01).
When it comes to degree of argumentation, strength of argumentation
and sentence quality, no significant differences emerged.

Subsequent post-hoc pairwise comparisons (with a Bonferroni-ad-
justed significance level of .017 at a 95% confidence level) revealed
that the mean levels of degree of civility and degree of relevance were
significantly higher in each of the three experimental conditions com-
pared to the control group. Again, these findings indicate that present-
ing a discussion statement, whether or not combined with one of the
two opinion elements, had a positive impact on civility and topic rele-
vance when commenting in an online news environment. When com-
paring each of the two interfaces including an opinion element (STAT-
buttons and STATscale) with the statement-only condition (STATo-
pen), no significant differences emerged. Both opinion elements do not
have a significant impact on top of the discussion statement regarding
the single comment quality criteria. We discuss the implications of
these results in the next section.

<table><caption>**Table 2.** Means and Standard Deviations (in Parentheses) for Comment Quality Variables</caption>
   <thead>
    <tr>
     <td>
      Condition
     </td>
     <td>
      Argumentation
     </td>
     <td>
      Civility
     </td>
     <td>
      Relevance
     </td>
     <td>
      Orthography
     </td>
     <td>
      Length
     </td>
    </tr>
   </thead>
   <tbody>
    <tr>
     <td>
      Control
     </td>
     <td>
      .42 (.731)
     </td>
     <td>
      1.05 (.789)
     </td>
     <td>
      .63 (.487)
     </td>
     <td>
      .72 (.453)
     </td>
     <td>
      .46 (.503)
     </td>
    </tr>
    <tr>
     <td>
      STATopen
     </td>
     <td>
      .63 (.698)
     </td>
     <td>
      1.46 (.683)
     </td>
     <td>
      .84 (.371)
     </td>
     <td>
      .79 (.411)
     </td>
     <td>
      .53 (.504)
     </td>
    </tr>
    <tr>
     <td>
      STATbuttons
     </td>
     <td>
      .67 (.746)
     </td>
     <td>
      1.63 (.540)
     </td>
     <td>
      .88 (.331)
     </td>
     <td>
      .77 (.426)
     </td>
     <td>
      .47 (.502)
     </td>
    </tr>
    <tr>
     <td>
      STATscale
     </td>
     <td>
      .65 (.664)
     </td>
     <td>
      1.65 (.593)
     </td>
     <td>
      .84 (.371)
     </td>
     <td>
      .78 (.418)
     </td>
     <td>
      .50 (.504)
     </td>
    </tr>
   </tbody>
  </table>
<table><caption>**Table 3.** Post-Hoc Pairwise Comparisons</caption><thead><tr><th>Condition</th><th>Civility</th><th>Relevance</th></tr></thead><tbody><tr><td>Control vs. STATopen</td><td>U= 1168, z= -2.795, p < .01</td><td>U= 1264.5, z= -2.489, p < .05</td></tr><tr><td>Control vs. STATbuttons</td><td>U= 1244, z= -4.337, p < .001</td><td>U= 1570.5, z= -3.279, p < .01</td></tr><tr><td>Control vs. STATscale</td><td>U= 1132, z= -4.429, p < .001</td><td>U= 1537, z= -2.626, p < .01</td></tr></tbody></table>

## 5.2 User Experience

Across all conditions, the mean score for pragmatic quality or task-oriented aspects ranged between .77 and 1.32, which is a slightly positive evaluation. On the other hand, the mean score for hedonic quality or non-task-oriented aspects ranged between -.28 and -.19, which represents a rather neutral evaluation [60].

A Kruskal-Wallis test compared users' reported experience and revealed a significant difference between the four groups with regard to pragmatic quality (H(3) = 7.902, p < .05). When it comes to hedonic quality, we observed no significant differences (H(3) = .069, p > .05). Post-hoc pairwise comparisons with a Bonferroni-correction tested which groups differed from the control group in terms of evaluation of pragmatic qualities and showed a significant difference between the control condition and the condition STATscale (U = 1016, z = -2.442, p < .05). More specifically, participants evaluated the task-oriented aspects significantly more positive in the control condition (M = 1.3207, SD = 1.1434) compared to participants in the STATscale condition (M = .8156, SD = 1.1105). This suggests that, despite the positive evaluation of pragmatic aspects across all conditions, the novel interface with the discussion statement and opinion scale was seen as less clear, efficient, easy and supportive compared to the traditional design. With respect to the conditions STATopen (M = 1, SD = 1.1134) and STATbuttons (M = .7667, SD = 1.3050), pragmatic evaluation did not significantly differ from the control group.

## 6 Discussion and Conclusion

The aim of this study was to explore the effectiveness of debiasing in a news environment by presenting commenters a discussion statement related to the news article's topic, alone or combined with an opinion element. We expected that narrowing down the focus of the discussion using a conversational prompt, i.e. a statement, encourages users to overcome negativity bias and, hence, comment more qualitatively. With the
goal of providing users an anchor that guides them through the interac-
tion, the statement was expanded with an opinion element.

First, the results revealed that the three novel interfaces – a discus-
sion statement, a discussion statement with an ‘agree’ and ‘disagree’
button and a discussion statement with an opinion scale – produced sig-
nificant positive effects on civility and relevance of participants’ com-
ments compared to a traditional interface. This suggests that when read-
ers have a specific statement to comment on, in particular in the context
of controversial news, they are more likely to behave civil and stay on-
topic than when the focus of the comment section is more open. This
finding is in line with prior research arguing that using conversational
prompts may increase the quality of the commentary [16, 69, 49].

Second, although our results clearly illustrate that the statement has a positive impact on civility and relevance, we found no influence on argumentation, comment length and orthography. As we prompted participants to clarify whether they agreed or not with the statement, it was reasonable to expect that participants provided more and stronger arguments compared to the more open instruction of ‘Leave your comment below’ in the control condition. While research suggests that civility and using evidence go hand in hand [50], we noticed only an increased civility and relevance. In that respect, scholars found mixed evidence about the association between civility and the use of evidence. Coe and colleagues [13] for example concluded that uncivil commenters are slightly more likely to support their claims with evidence, especially with numbers. Chen [10] on the other hand argued that deliberation and mild incivility can overlap, stating that “communication that is too polite or too nasty has no potential for deliberation”.

Third, we found no added value of combining the discussion state-
ment with an opinion element. When the discussion statement was
combined with one of the two opinion elements, comment quality was
still higher compared to the control group, but not higher compared to
the statement-only condition. Our experimental design does not allow
us to make inferences about the opinion element itself.

Fourth, across all conditions, participants rated the pragmatic and he-
donic qualities of the writing task as positive and neutral respectively.
Compared to the control group, the mere addition of a statement in the
statement-only condition did not significantly influence users’ experi-
ence. This suggests that they did not felt restricted by the statement to
exert an opinion. However, the interface with the discussion statement
and the opinion scale received a significantly lower evaluation in terms
of pragmatic qualities compared to the traditional interface. In other
words, participants perceived the novel design as less clear, efficient,
easy and supportive. This could be the result of the higher degree of in-
teraction required from the participant to move the slider, compared to
writing a traditional comment.

Finally, we discuss some potential limitations of our study and for-
mulate directions for future research. A first limitation of our experi-
mental design is that we combined two interface elements in one condi-
tion, that is the statement and two opinion elements. Although we com-
pared the impact of those interfaces with both the impact of the control
and statement-only groups, we were not able to provide insights about
the impact of an opinion element in itself. Further research could use a
2 (no statement – statement) x 3 (open commenting – opinion buttons -
opinion scale) design to get a clearer view on how an opinion element
affects comment quality.

A second limitation is that it is not yet clear to which extent the re-
sults can be generalized to other news topics as well as to a real com-
ment section. In our experiment we pursued a high ecological validity,
for example by including actual comments reflecting various views and
by letting participants believe that others would see and respond to their
comments. However, despite the fact that its validity was confirmed in
a pilot study and that prior experimental research [30] used the same
platform to gather participants’ comments, the survey environment was
not a real comment section which could have affected the quality of the
comments. For instance, it was not possible to react on other com-
ments, and all participants were required to leave a comment. In a real-
life setting, readers have their own motivations for engaging in a de-
bate, or they can simply choose not to react. Moreover, enabling con-
structive conversations implies that only users that have something to
say should be encouraged to comment, in order to increase the topical
focus of the discussion and to avoid irrelevant posts.

To conclude, this paper demonstrated that particular interface ele-
ments can have an impact on the quality of online comments and there-
fore can bring about a more constructive debate on online news plat-
forms and social media. This finding is a single contribution to the
growing field of research, especially in the context of polarizing news,
and is a possible approach to pro-actively encourage respectful dia-
logue in the comment section itself, which in turn can also decrease
moderation efforts. Future research opportunities lie in conducting fol-
low-up experiments with other stimuli and on existing platforms within
real comment sections.

References

1. Acquisti, A. et al.: Nudges for Privacy and Security: Understanding and Assisting Users' Choices Online. ACM Comput. Surv. 50(3), 1–41 (2017). https://doi.org/10.1145/3054926

2. Anderson, A.A. et al.: The “Nasty Effect:” Online Incivility and Risk Perceptions of Emerging Technologies: Crude comments and concern. J Comput-Mediat Comm. 19(3), 373–387 (2014). https://doi.org/10.1111/jcc4.12009

3. Arkes, H.R.: Costs and Benefits of Judgment Errors. Psychological Bulletin 110(3), 486-98 (1991).

4. Bachleda, S. et al.: Individual-level differences in negativity biases in news selection. Personality and Individual Differences 155, 109675 (2020). https://doi.org/10.1016/j.paid.2019.109675.

5. Baumeister, R.F. et al.: Bad is Stronger than Good. Review of General Psychology 5(4), 323–370 (2001). https://doi.org/10.1037/1089-2680.5.4.323.

6. Beckert, J., Ziegele, M.: The Effects of Personality Traits and Situational Factors on the Deliberativeness and Civility of User Comments on News Websites. International Journal of Communication (Online), 3924 (2020).

7. Berg, J.: The impact of anonymity and issue controversiality on the quality of online discussion. Journal of Information Technology & Politics 13(1), 37–51 (2016). https://doi.org/10.1080/19331681.2015.1131654.

8. Blank, G., Reisdorf, B.C.: The Participatory Web: A user perspective on Web 2.0. Information, Communication & Society 15(4), 537–54 (2012).

9. Brückner, L., Schweiger, W.: Facebook discussions of journalistic news: Investigating article objectivity, topic, and media brand as influencing factors. SC|M 6(4), 365–394 (2017). https://doi.org/10.5771/2192-4007-2017-4-365.

10. Chen, G.M.: Online Incivility and Public Debate. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-56273-5.

11. Chen, M. et. al.: We Should Not Get Rid of Incivility Online. Social Media Society 5(3), 1-5 (2019).

12. Cheng, F.-F., Wu, C.-S.: Debiasing the framing effect: The effect of warning and involvement. Decision Support Systems 49(3), 328–334 (2010).

13. Coe, K. et al.: Online and Uncivil? Patterns and Determinants of Incivility in Newspaper Website Comments. J Commun. 64(4), 658–679 (2014). https://doi.org/10.1111/jcom.12104.

14. Correia, V.: Contextual Debiasing and Critical Thinking: Reasons for Optimism. Topoi. 37(1), 103–111 (2018). https://doi.org/10.1007/s11245-016-9388-x.

15. Croskerry, P. et al.: Cognitive debiasing 2: impediments to and strategies for change. BMJ Qual Saf. 22(2), ii65–ii72 (2013). https://doi.org/10.1136/bmjqs-2012-001713.

16. Curran, J. et al.: Media System, Public Knowledge and Democracy: A Comparative Study. European Journal of Communication 24(1), 5–26 (2009). https://doi.org/10.1177 /0267323108098943.

17. Darics, E.: Politeness in computer-mediated discourse of a virtual team. Journal of Politeness Research. Language, Behaviour, Culture 6(1) (2010).

18. Diakopoulos, N., Naaman, M.: Towards Quality Discourse in Online News Comments. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, pp. 133-142. Association for Computing Machinery, New York (2011).
19. Diakopoulos, N.A.: The Editor's Eye: Curation and Comment Relevance on the New York Times. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 1153–1157. CSCW '15. ACM Press, Vancouver (2015). https://doi.org/10.1145/2675133.2675160.

20. Engelke, K.M.: Enriching the Conversation: Audience Perspectives on the Deliberative Nature and Potential of User Comments for News Media. Digital Journalism 8(4), 447-466 (2020).

21. Esau, K., Friess, D., Eilders, C.: Design Matters! An Empirical Analysis of Online Deliberation on Different News Platforms. Policy and Internet 9(3), 321-342 (2017).

22. Fogg, B.J.: Persuasive technology: Using computers to change what we think and do. Morgan Kaufmann Publishers, Amsterdam (2003).

23. Fredheim, R. et al.: Anonymity and Online Commenting: The Broken Windows Effect and the End of Drive-by Commenting. In: Proceedings of the ACM Web Science Conference on ZZZ, pp. 1–8. WebSci ’15. ACM Press, Oxford (2015). https://doi.org/10.1145/2786451.2786459

24. Friess, D., Eilders, C.: A Systematic Review of Online Deliberation Research. Policy and Internet 7(3), 319-39 (2015).

25. Galtung, J., Ruge, M.H.: The Structure of Foreign News: The Presentation of the Congo, Cuba and Cyprus Crises in Four Norwegian Newspapers. Journal of Peace Research 2(1), 64–90 (1965). https://doi.org/10.1177/002234336500200104.

26. Gervais, B.T.: Incivility Online: Affective and Behavioral Reactions to Uncivil Political Posts in a Web-based Experiment. Journal of Information Technology & Politics 12,(2), 167–185 (2015). https://doi.org/10.1080/19331681.2014.997416.

27. Godi, M.: Beyond Nudging: Debiasing Consumers Through Mixed Framing. The Yale Law Journal 128(7), 2034-086 (2019).

28. Graf, J. et al.: The Role of Civility and Anonymity on Perceptions of Online Comments. Mass Communication and Society 20(4), 526–549 (2017). https://doi.org/10.1080/15205436.2016.1274763.

29. Halpern, D., Gibbs, J.: Social media as a catalyst for online deliberation? Exploring the affordances of Facebook and YouTube for political expression. Computers in Human Behavior 29(3), 1159–1168 (2013). https://doi.org/10.1016/j.chb.2012.10.008.

30. Han, S.-H. et al.: Is Civility Contagious? Examining the Impact of Modeling in Online Political Discussions. Social Media & Society 4(3) (2018). https://doi.org/10.1177/2056305118793404.

31. Han, S.-H., Brazeal, L.M.: Playing Nice: Modeling Civility in Online Political Discussions. Communication Research Reports 32(1), 20–28 (2015). https://doi.org/10.1080/08824096.2014.989971.

32. Herbst, S.: Rude democracy: Civility and incivility in American politics. Temple University Press, Philadelpia (2010).

33. Hermida, A., Thurman, N.: A Clash of Cultures: The integration of user-generated content within professional journalistic frameworks at British newspaper websites. Journalism Practice 2(3), 343-56 (2008).

34. Hullman, J., Diakopoulos, N.: Visualization Rhetoric: Framing Effects in Narrative Visualization. IEEE Trans. Visual. Comput. Graphics 17(12), 2231–2240 (2011). https://doi.org/10.1109/TVCG.2011.255.

35. Hung, H.T., Yeh H.C., Chou C.H.: An Investigation into English Language Learners' Argumentative Writing Performance and Perceptions. In: Wu T.T., Gennari R., Huang Y.M., Xie H., Cao Y. (eds) Emerging Technologies for Education - SETE 2016. Lecture Notes in Computer Science, vol 10108, pp. 712-720. Springer, Cham (2016).
36. Jang, S.M., Oh, Y.W.: Getting attention online in election coverage: Audience selectivity in the 2012 US presidential election. New Media & Society 18(10), 2271–2286 (2016).

37. Jensen, J.L.: Public Spheres on the Internet: Anarchic or Government-Sponsored - A Comparison. Scand Pol Stud. 26(4), 349–374 (2003). https://doi.org/10.1111/j.1467-9477.2003.00093.x.

38. Kirman, B., Lineham, C., Lawson, S.: Exploring Mischief and Mayhem in Social Computing or how we learned to stop worrying and love the trolls. In: CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2012), pp. 121-30. ACM, New York (2012).

39. Knobloch-Westerwick, S. et al.: Confirmation Bias, Ingroup Bias, and Negativity Bias in Selective Exposure to Political Information. Communication Research 47(1), 104–124 (2020). https://doi.org/10.1177/0093650217719596.

40. Kramp, L., Loosen, W.: The Transformation of Journalism: From Changing Newsroom Cultures to a New Communicative Orientation? In: Hepp, A. et al. (eds.) Communicative Figurations, pp. 205–239. Springer International Publishing, Cham (2018).

41. Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. 2nd ed. Thousands Oaks: Sage (2004).

42. Ksiazek, T.B., Springer, N.: User Comments and Moderation in Digital Journalism: Disruptive Engagement. Routledge, London (2020).

43. Laugwitz, B, Held, T., Schrepp, M.: Construction and Evaluation of a User Experience Questionnaire. Lecture Notes in Computer Science 5298, 63-76 (2008).

44. Lilienfeld, S.O. et al.: Giving Debiasing Away: Can Psychological Research on Correcting Cognitive Errors Promote Human Welfare? Perspect Psychol Sci. 4(4), 390–398 (2009). https://doi.org/10.1111/j.1745-6924.2009.01144.x.

45. Masullo Chen, G. et al.: We Should Not Get Rid of Incivility Online. Social Media & Society. 5(3), (2019). https://doi.org/10.1177/2056305119862641.

46. Meany, J., Shuster, K.: On that Point!: An Introduction to Parliamentary Debate. International Debate Education Association, New York (2003).

47. Meffert, M.F. et al.: The Effects of Negativity and Motivated Information Processing During a Political Campaign. Journal of Communication 56(1), 27–51 (2006). https://doi.org/10.1111/j.1460-2466.2006.00003.x.

48. Menon, S. et al.: Nudge for Deliberativeness: How Interface Features Influence Online Discourse. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. pp. 1–13 ACM, Honolulu (2020). https://doi.org/10.1145/3313831.3376646.

49. Noveck, B.S.: Wiki Government: How Technology Can Make Government Better, Democracy Stronger, and Citizens More Powerful. Brookings Institution Press, Washington, D.C. (2009).

50. Papacharissi, Z.: Democracy online: civility, politeness, and the democratic potential of online political discussion groups. New Media & Society 6(2), 259–283 (2004).

51. Park, D. et al.: Supporting Comment Moderators in Identifying High Quality Online News Comments. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 1114–1125 ACM, San Jose California USA (2016). https://doi.org/10.1145/2858036.2858389.

52. Peacock, C. et al.: The deliberative influence of comment section structure. Journalism. 20(6), 752–771 (2019). https://doi.org/10.1177/1464884917711791.

53. Pingree, R.J. et al.: Effects of Postdebate Coverage on Spontaneous Policy Reasoning. Journal of Communication. 62(4), 643–658 (2012). https://doi.org/10.1111/j.1460-2466.2012.01656.x.

54. Riffe, D. et al.: Analyzing Media Messages. Routledge, New York (2006).
55. Risch, J., Krestel, R.: Deep Learning-Based Approaches for Sentiment Analysis. Springer: Singapore (2020).

56. Rozin, P., Royzman, E.B.: Negativity Bias, Negativity Dominance, and Contagion. Pers Soc Psychol Rev. 5(4), 296–320 (2001). https://doi.org/10.1207/S15327957PSPR0504_2.

57. Santana, A.D.: Virtuous or Vitriolic: The effect of anonymity on civility in online newspaper reader comment boards. Journalism Practice 8(1), 18-33 (2014).

58. Santana, A.D.: Incivility Dominates Online Comments on Immigration. Newspaper Research Journal. 36(1), 92–107 (2015). https://doi.org/10.1177/0739532915580317.

59. Saris, W.E., Sniderman, P.M.: Studies in public opinion: Attitudes, nonattitudes, measurement error, and change. Princeton University Press, Princeton (N.J.) (2004).

60. Schrepp, M. et al.: Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S). IJIMAI. 4(6), 103 (2017). https://doi.org/10.9781/ijimai.2017.09.001.

61. Schrepp, M.: User Experience Questionnaire Handbook. (2015). https://doi.org/10.13140/RG.2.1.2815.0245.

62. Seering, J. et. al.: Designing User Interface Elements to Improve the Quality and Civility of Discourse in Online Commenting Behaviors. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1-14 (2019).

63. Sobieraj, S., Berry, J.M.: From Incivility to Outrage: Political Discourse in Blogs, Talk Radio, and Cable News. Political Communication. 28(1), 19–41 (2011). https://doi.org/10.1080/10584609.2010.542360.

64. Stoll, A., Ziegele, M., Quiring, O.: Detecting Impoliteness and Incivility in Online Discussions: Classification Approaches for German User Comments. Computational Communication Research 2(1), pp. 109-134 (2020).

65. Stromer-Galley, J.: Measuring Deliberation's Content: A Coding Scheme. Journal of Deliberative Democracy 3(1), 12 (2007). https://doi.org/10.16997/jdd.50.

66. Stroud, N.J. et al.: Changing Deliberative Norms on News Organizations' Facebook Sites. J Comput-Mediat Comm. 20(2), 188-203 (2015). https://doi.org/10.1111/jcc4.12104.

67. Sui, M., Pingree, R.J.: In Search of Reason-centered Discussion on China's Twitter: The Effects of Initiating Post and Discussion Format on Reasoning. International Journal of Communication 10(1) (2016): 416-31.

68. Sunstein, C.R., Thaler, R.H.: Nudge: improving decisions about health, wealth and happiness. Yale University Press, New Haven (2008).

69. Towne, W.B., Herbsleb, J.D.: Design Considerations for Online Deliberation Systems. Journal of Information Technology & Politics 9(1), 97–115 (2012). https://doi.org/10.1080/19331681.2011.637711.

70. Trénel, M.: Measuring the Deliberativeness of Online Discussions. Coding Scheme 2.4. report, Berlin: Social Science Research Centrex, (2004).

71. Tversky, A., Kahneman, D.: Judgment under uncertainty: Heuristics and biases. In: Kahneman, D. et al. (eds.) Judgment under Uncertainty. pp. 3–20 Cambridge University Press (1982). https://doi.org/10.1017/CBO9780511809477.002.

72. Wales, C. et al.: The structural features and the deliberative quality of online discussions. Presented at the Political Studies Association Conference, Edinburgh (2010).

73. Wessler, H.: Investigating Deliberativeness Comparatively. Political Communication 25(1), pp. 1-22 (2008).

74. Wilhelm, A.G.: Virtual Sounding Boards: How Deliberative Is On-line Political Discussion? Information, Communication & Society 1(3), 313-38 (1998).

75. Zhang, W. et al.: The structural features and the deliberative quality of online discussions. Telematics and Informatics 30(2), 74–86 (2013).

76. Ziegele, M., Quiring, O.: Conceptualizing Online Discussion Value: A Multidimensional
Framework for Analyzing User Comments on Mass-Media Websites. Annals of the Inter-
national Communication Association 37(1), 125–153 (2013).