interest or place of residence. Consequently, we cannot identify a clear equivalence
between the creation of these group and a specific degree of Arnstein's ladder. How-
ever, the acknowledgment of the existence of these groups allows citizens to produce
common contributions to online consultations in a collaborative workflow. This
specificity is important since the different platforms of our panel implement scarcely
any means for online collaboration between contributors. The creation of groups
inside participatory platforms can therefore be seen as a first step towards online
collaboration in citizen participation, even though this collaboration still takes place
outside of the platforms (e.g., in online CSCW platforms or in-person meetings).
Moreover, the acknowledgment of the existence of groups working on a given topic
allows institutions to include these groups in potential policy-making related with
their topics of interest. For example, in Interviewee #1's platform, even though the
city had created a citizen participation council to "review the tools for interacting
with citizens [...] and validate the guidelines of the platform", an independent group
was created by citizens to "define citizen participation" at the scale of their city. This
group organized local events and eventually did produce a list of recommendations
addressed to the municipality.

4.3.4 Co-construction of projects

The last objective we identify in participatory platforms is the possibility for citizens
and institutions to elaborate common projects. The implication of both sides can
greatly vary among platforms implementing this objective, from the possibility given
to citizens to propose modifications to a project already conceived by an institution,
to the collective elaboration of a project by representatives of the institution and
citizens. Theoretically, this objective would therefore correspond to the degrees of
participation between “consultation” and “partnership” inclusive [15]. However, in
practice, we could not identify in the platforms of our panel an example of online
co-construction where citizens were given actual decision power on the final project,
which would consequently categorize the implementations of this objective in the
“consultation” level of the ladder. This vertical organization, as well as the objec-
tive of allowing citizens to have an input on institutional projects, is described by
Interviewee #4 as follows:

The common objective [of our platforms] is to allow the participation of citizen in public decisions making. [...] They do not only aim at being a space for expression but, most importantly, a place where we give visibility to a full cycle of a project in which participation is involved and also where we can organize feedbacks to participants by showing precisely which of their proposals were included or not to the final project.

This objective is the one where interaction between contributors is the most fre-
quent. Indeed, platforms which aim at allowing co-construction of projects generally
implement a horizontal workflow where citizens – and in certain cases, institutions
representatives – are allowed not only to submit proposals, but also to interact with
one another’s proposals through comments or revised versions. However, the plat-
forms of our panel hardly implement any means for online collaboration outside
comments, vote-based validation and basic content modification. The possibility to
create and advertise groups of citizens working on a specific topic or living in a given
neighborhood can be seen as a first step towards the acknowledgment of this col-
laboration by maintainers of participatory platforms. However, as stated in § 4.3.3,
the means for collaboration are found outside of online participatory platforms, in
online CSCW platforms or in-person meetings [5].

Finally, the objective of co-construction of projects intrinsically creates a risk of
information overload, both for contributors and for administrators of participatory
platforms [13, 75]. Indeed, the existence of numerous contributions and replies to
contributions presents an issue concerning the scalability of participatory processes
themselves. This issue is accentuated by the interactive nature of the co-construction
of projects, even though it can be also faced when the objectives are different. While
Interviewee #2 and Interviewee #3, who work for the smallest platforms of our
panel concerning the number of contributions and do not propose co-construction
of projects, state that they simply read all the contributions published in their
platforms, other interviewees need to use more complex methods to summarize all
contributions. Interviewee #1 states that they “do a lot of data and natural language
processing using Modalisa<fnref n="3" /> to help the involved services of the municipality”. Facing
the same problem when they work for consultations with high volumes of contri-
butions which have to be processed quickly, Interviewee #4’s company “is working
on a R&D project to use contemporary Natural Language Processing techniques
on citizen consultations”. Interviewee #5 states that they follow the same goal to
reduce this scalability issue using different techniques assessed in [13]:

What we want now is to better take advantage of all this information.
Indeed, there is a project [...] for using machine learning techniques
to analyze the comments and proposals and other information in the
database to extract patterns, trends, sentiment analysis, visualization
and present this information to the different units that manage the topics
so they can use them easily, instead of ready and manually processing
thousands of contributions so that they can define their politics. [...] So
far, we have accumulated a lot of data, but is really difficult to manage
it and do something with it.

4.4 Different types of platforms for contemporary online citizen participation

During our interviews, we asked interviewees about the functionalities of the plat-
forms they work for. These platforms can be divided in two categories: “suggestion
box”-like and interactive platforms. Categorization of the platforms for which our
different interviewees work are displayed in Table 4.4.

4.4.1 “Suggestion box”-like platforms

In “suggestion box”-like platforms, the functionalities are designed to facilitate the
sending of proposals from the citizens to the organizing entity, in a vertical work-
flow. These platforms allow little to no interaction between contributors beyond
the public sharing of proposals. The content produced by contributors is generally

<footnote n="3"><sup>3</sup>Software tool for the analysis of interviews and survey results.</footnote>
<table><caption>Table 4.4: Categorization of platforms maintained by interviewees according to their features</caption><thead><tr><td>INTERVIEWEE #</td><td>PLATFORM FEATURES</td></tr></thead><tbody><tr><td>#1</td><td>Multiple use cases in the same platform. Participation is implemented both with and without interaction between citizens.</td></tr><tr><td>#2</td><td>Participation without interaction between citizens inside the platform.</td></tr><tr><td>#3</td><td>Participation without interaction between citizens.</td></tr><tr><td>#4</td><td>Multiple platforms both with and without interaction between citizens.</td></tr><tr><td>#5</td><td>Multiple use cases in the same platform. Participation is implemented both with and without interaction between citizens.</td></tr></tbody></table>

an answer to an open-ended question or a proposal with few details, matching the “idea collection” phase of the generic interaction workflow supporting participatory budgeting presented in [128] (see Figure 2.2). In these platforms, a large part of the synthesis work is done manually by agents of the institution. Interviewee #3 describes the process for their platform as follows:

A first examination is done by the service focused on relations with users and neighborhood councils. Then, the proposal is sent to the most relevant service [of the institution] which will look in detail at potential elements provided and make a complementary examination, i.e. look for additional data to make an estimate of the needed budget and analyze the related problems. [...] Finally, the investigation file is sent to neighborhood councils, municipal officials, etc.

The interactive dimension of these platforms is very limited since contributors cannot interact with one another's contributions using comments or other text-based contributions (e.g., submitting a revised version of a proposal). The means of interaction tend to be limited to the ability to signify one's interest in a proposal by following it or voting in favor of it. To justify this choice of a restrictive workflow, Interviewee #3 states that “the objective is mainly to facilitate the use [of the platform] and to let contributors freely present projects”. This choice of workflow is also the most commonly made when the objective of the platform is to report simple issues to a service –Interviewee #1 states that they use this workflow for their “potholes and dog mess signaling tool”. Furthermore, the absence of means for contributors to interact with one another's contributions can be related with the concern for misconduct of certain contributors. Interviewee #2 supports this hypothesis by stating that their city “didn't want a platform working like a social network, i.e with the possibility for people to remain anonymous when they submit a proposal or a comment”. This intuition that anonymous comments leads to incivility tends to be confirmed, with some degree of mitigation, by Santana [145]. Moreover, Interviewee #4 indicates that their participatory platforms are explicitly designed to limit interactions between contributors and justify their decision as follows:

Participatory platforms also aim [...] at avoiding the pitfall of forums. When we created our company, the most common means of participation were forums organized by news websites at the end of their articles. Many of these websites removed this feature because their moderation rate was extremely high and as soon as there were interactions, three people were monopolizing the conversation, talking only to each other and, after a maximum of ten messages, insulting each other.
During our interviews, administrators of these platforms mentioned neighborhood councils or similar structures more frequently than other interviewees. Interviewee #2 states that the designers of the platform they maintain explicitly designed it to mimic the workflow of offline neighborhood councils. Interviewee #3 indicates that the proposals submitted by contributors are reviewed by a neighborhood council before their evaluation by the related institution. These connections with offline equivalents of participatory platforms can be one of the reasons why the platforms produced tend to implement little to no means for interaction between citizens, since aforementioned neighborhood councils are constituted of small groups of citizens without any interaction workflow implemented by the city.

### 4.4.2 Interactive platforms

The platforms we consider as interactive are the ones designed to allow contributors to interact with one another's contributions in a horizontal workflow. Even though it is not mandatory, in the case of the platforms we have studied, they do also implement a vertical workflow to allow the organizing institution to gather and process contributions and publish replies to some of them. Platforms of that kind generally implement – although not being limited to – the three following forms of contributions:

1. User A submits an initial contribution, detailed or vague, to the platform.

2. User B comments on a contribution formerly submitted by User A.

3. User B votes for, against, or neutrally, on A's contribution.

These platforms tend to implement the “idea collection”, “proposals development” and “deliberation and voting” phases of Parra et al.’s generic workflow [128]. Consequently, they allow users to have an impact on a project on further steps of its elaboration than “suggestion box”-like platforms. Interviewee #1 states that in their consultation platform, they mainly act to inform citizens of their impact on the project, giving them in certain cases a wide scope of action in the synthesis process:

> We are the guarantors of the proper organization of the consultation, i.e. we have to include the citizen in the project as strongly as possible, to make them understand what questions we are asking, to let them know their degree of implication in the final decision, and to ensure that they receive at least some feedback by the end of the consultation.

As stated in § 4.3.4, interactive platforms tend to be used when the objective of the participatory initiative is to co-construct a project. Interviewee #4, whose company produces participatory platforms using different functional modules, confirms this observation by saying that modules allowing more in-depth interaction between contributors (e.g., the possibility to submit a modified version of another user's contribution) are implemented in a platform only “if they want to produce a structured, problematized and contradictory thought process to identify points of consensus and dissensus concerning a given topic”. Interactive features also tend to be implemented in the cases where the expected number of contributors is high,
like national-scale consultations (e.g., the Universal Activity Income consultation<fnref n="4" /> in France) or participatory platforms for large cities (see Table 4.4). This tendency may be explained by the need for a wide enough audience to make use of such features.

Finally, even though interactive platforms tend to implement more means for collaboration than their “suggestion box”-like counterparts, these means are very limited with in comparison with state-of-the-art online collaborative systems. The only example of collaborative features found in the platforms of our panel is the possibility to create contributions derived from another contributor’s own contribution in certain platforms. However, most features for collaborative editing of contributions (e.g., version control, joint writing...) are not implemented, therefore requiring external tools to produce common contributions as mentioned in § 4.3.3.

## 4.5 How to enrich online participatory platforms

The analysis of interviews of stakeholders of online citizen participation proposed in § 4.3 and § 4.4 allow us to identify two axes to differentiate participatory platforms: the objectives pursued and the presence or absence of features for interaction between citizens. These interviews also allow us to highlight flaws in some of the categories of participatory platforms identified. Based on pre-existing works, we propose potential solutions to address the aforementioned flaws in future participatory platforms. For the sake of clarity, solutions proposed to address issues identified in this chapter are summarized in Table 4.5.

### 4.5.1 Collaborative systems

The lack of means for collaboration in online citizen participation has been highlighted several times in this chapter. This impossibility for groups to collaborate inside participatory platforms does not encourage contributors to produce common, well-developed contributions. Implementing features for collaborative editing of contributions could be a mean to reduce the raw number of contributions by merging similar ones – therefore reducing the information overload previously identified [38, 132]. It could also increase the degree of participation by giving a wider space to deliberation between contributors than on existing platforms where the means of interaction with contributions are simple (vote, comment and, in certain cases, propose modifications).

The first question towards the implementation of collaborative editing in participation is whether the system has to be *synchronous* or *asynchronous*. In the former case, the changes that a user applies to a document are shared with the rest of the contributors as soon as they are performed, generally as character-based updates. Platforms such as Google Docs and EtherPad implement this method. In the latter case, the users decide when they want to share their contributions with –and receive the ones from– the other users. This is the approach implemented by, e.g., Version Control Systems (VCSs) such as Git [164] or Subversion [114], and the massively used wiki engine Wikimedia. We identify the two following means to enrich online citizen participation using synchronous and asynchronous collaborative editing:

<footnote n="4">⁴https://www.consultation-ria.gouv.fr/</footnote>
* Using asynchronous collaborative editing features, contributors could have the opportunity to alter one another's contributions to produce revised versions and to merge several proposals together. These features are partially implemented in certain platforms (e.g., République Numérique<fnref n="5" /> or Revenu Universel d'Activité<fnref n="6" /> consultations) but these platforms lack the tools implemented by contemporary asynchronous editing systems (e.g., VCSs, wikis) like the merging or the labeling of different versions of a document. The choice of asynchronous over synchronous editing in that case is motivated by the possibility to easily keep track of revisions and to prevent issues such as vandalism [3, 151] and edit wars [159, 180]. We acknowledge the fact that, in contemporary use cases, asynchronous editing tends to be implemented when contributors have a high level of computer literacy (e.g., collaborative software development, wiki-based information sharing) and remains little used by the general public. To reduce this issue, certain activist communities<fnref n="7" /> and academic courses [142] organize group training sessions to teach the use of platforms of this type, but the use of synchronous editing systems can also be seen as a solution to this issue.

* Participatory platforms could also leverage the specificities of synchronous collaborative editing (e.g., its ease of use or the possibility to do joint writing) to allow groups of users to collaboratively create a common proposal from the ideation phase to its sharing with the rest of contributors. As shown in [123], synchronous editing applications give users new means of collaboration using recognizable and specific patterns of work. However, the direct sharing of contributions to a document can also be the source of inconveniences for users, from the complexity induced by collaboration rules of a group to the difficulty to share contributions when they are yet not finished and do not fully represent the ideas their authors want to share [170]. We therefore recommend to implement synchronous collaboration only in contexts where contributors share the same objective, like an opinion group. To the best of our knowledge, such features are not implemented in contemporary citizen participation platforms.

### 4.5.2 Natural Language Processing for large-scale processing of contributions

In this chapter, we highlighted the risks induced by large volumes of contributions in citizen participation platforms, especially concerning the difficulty for platform administrators to produce an accurate summary of a high number of contributions (see § 4.3.4). Previous works [38, 132] have also identified the risk of information overload for citizens when the number of contributions published is too high for them to easily identify the most relevant contributions depending on their topic of interests or areas of expertise. These issues should be given particularly high importance in cases where the objective is the co-construction of a project between citizens and an institution and where means for text-based interaction between contributors are implemented.

<footnote n="5">⁵https://www.republique-numerique.fr/</footnote>

<footnote n="6">⁶https://www.consultation-rua.gouv.fr/</footnote>

<footnote n="7">⁷E.g., https://www.artandfeminism.org/</footnote>
<table><caption>Table 4.5: Summary of the issues identified and means proposed in Section 4.5 to solve them.</caption><thead><tr><th>ISSUE</th><th>PROPOSED SOLUTION</th><th>SOURCES</th></tr></thead><tbody><tr><td rowspan="2">Lack of means for collaboration</td><td>Asynchronous collaborative editing in cases where approval of new changes to a proposal is needed and where vandalism or edit wars are major concerns.</td><td>[3], [114], [151], [159], [164], [180]</td></tr><tr><td>Synchronous collaborative editing in cases where contributors share the same objective, i.e., vandalism and edit wars are minor concerns.</td><td>[8], [123], [170], [172]</td></tr><tr><td rowspan="2">Information overload in large-scale participation</td><td>Automated tag generation using NLP to help filter contributions and recommend the most relevant contributions for a given user.</td><td>[13], [26], [156]</td></tr><tr><td>Automated summarization of large sets of contributions by identifying the most relevant contributions and arguments for a given topic.</td><td>[13], [92], [125], [149], [150]</td></tr></tbody></table>

The automation of certain tasks currently performed by administrators of participatory platforms or by contributors themselves can be a mean to address the aforementioned risks. We consider that such automation can be done using different Natural Language Processing (NLP) tools. We identify the two following use cases where machine learning can be leveraged to improve large-scale online citizen participation:

* Participatory platforms could use NLP tools to generate tags for the different contributions published. Implementation of such feature would allow contributors to more easily identify the most relevant contributions based on their topics of interest. Platforms administrators would also be able to filter contributions focused on a given topic and therefore summarize them more efficiently. Contribution tagging can also be used to recommend relevant contributions to a user based on topical similarity with a given first contribution (e.g., the one they are currently reading). Dynamic identification of the most relevant tag words can be done using TF-IDF [156] or similar algebraic methods. However, topical clustering of contributions and contributors can also be done using machine learning-based algorithms such as LDA [26] to probabilistically assess the relevance of each contribution for a given topic. The two aforementioned methods have been experimentally implemented by Arana-Catania et al. [13] for the Decide Madrid platform. However, while Interviewee #5 states that they use similar NLP features to help platform administrators summarize participatory processes, none of the contributor-oriented features are implemented in the current version of the platform.

* Another use case for NLP in citizen participation platforms would be the summarization of contributions on a given topic. Indeed, given a large set of contributions on a given topic, both contributors could want to know the most frequent arguments or the most relevant contributions. Using cosine similarity of sentence vectors<fnref n="8" />, Arana-Catania et al. [13] produced a network of sentences from all the comments of an initial proposal and applied the PageRank [125] algorithm to produce a summary of the most relevant remarks to this proposal, with mixed results. However, other techniques such as argument mining can be leveraged to produce a collection of arguments about a given contribution or

<footnote n="8">⁸Sentence vector is a version of a sentence encoded by a sentence embedding technique to allow their use by a language understanding model (e.g., SBERT [138]).</footnote>
topic [92]. Segura-Tinoco et al. [149, 150] propose an encouraging step towards the implementation of such technique in citizen participation contexts.

## 4.6 Threats to validity

One potential threat to the validity of our study is the presence of online participatory initiatives in other countries or using different platforms than those taken into account by our interviews. Indeed, adding interviews from other citizen participation platforms administrators or from active contributors could improve our findings and add new keys to understanding collaboration in citizen participation platforms. However, the large diversity of the platforms managed by our interviewees –in terms of objectives, functionalities and size of the involved group of potential contributors– allows us to describe a wide spectrum of practices. This diversity of profiles is described in Section 4.2 of the chapter. We consider our set of interviews to be diverse enough to describe contemporary online citizen participation platforms.

A second issue is related to the fact that all our interviewees work for French and Spanish citizen participation platforms. Indeed, the addition of platforms used in countries outside of Europe could add new perspectives based on these countries' legal framework or culture concerning participation. However, the examples of participatory platforms that we found outside of Europe were either out of service (e.g., Argentinian *Partido de la Red*'s implementation of the DemocracyOS framework) or too close to other platforms already in our panel (e.g., New York City's participatory budgeting platform<fnref n="9" /> based on Decidim).

## 4.7 Conclusion

This chapter presents an analysis of online citizen participation platforms based on a series of semi-directed interviews with different stakeholders involved in the creation and maintenance of the said platforms. We propose a classification of contemporary participatory platforms based on two axes: the output they aim at producing and their types of features for interaction between contributors. We identify specific issues for the platforms aiming at co-constructing projects with citizens at a large scale and for those acknowledging the possibility for contributors to produce collective contributions. Finally, we propose to leverage different state-of-the-art technologies – namely collaborative editing and natural language processing – to solve these issues and help produce produce platforms more suitable for both large numbers of contributors and high degrees of participation [15]. Both the highlighting of issues on the different types of participatory platforms and the identification of state-of-the-art methods to solve the said issues are crucial means to help platforms administrators and designers improve online citizen participation. Therefore, in the next chapter, we will assess the relevance of an NLP-based method to improve large-scale citizen participation.

We consider that identifying the objectives of institutions organizing participatory initiatives as well as their motivations for implementing specific features is essential to understand the design – and potential flaws – of contemporary participatory platforms since they have the most direct influence on their creation. However,

<footnote n="9">⁹https://wwwparticipate.nyc.gov/</footnote>
we also acknowledge the importance of understanding the issues faced by contribu-
tors in online participation contexts to improve their involvement in public decision
making. Therefore, we argue that both the inclusion of citizens in the governance of
participatory platforms and quantitative analysis of the feedback of contributors [63]
are promising leads towards the creation of more viable participatory platforms.
# Chapter 5

# Improving large scale participation using NLI

## Contents

<table><tr><td>5.1</td><td>Introduction</td><td>85</td></tr><tr><td>5.2</td><td>Methodology</td><td>87</td></tr><tr><td>5.2.1</td><td>Problem statement</td><td>87</td></tr><tr><td>5.2.2</td><td>Weakly-labeled data for model training</td><td>89</td></tr><tr><td>5.2.3</td><td>Creating a gold dataset for evaluation</td><td>89</td></tr><tr><td>5.2.4</td><td>Automatically labeling long texts</td><td>90</td></tr><tr><td>5.3</td><td>Datasets</td><td>91</td></tr><tr><td>5.4</td><td>Evaluation</td><td>95</td></tr><tr><td>5.4.1</td><td>Fine-tuned models</td><td>95</td></tr><tr><td>5.4.2</td><td>Insights and error analysis</td><td>98</td></tr><tr><td>5.5</td><td>Assessing the integration of an NLI model into a citizen participation platform</td><td>100</td></tr><tr><td>5.5.1</td><td>Identifying idea communities in a consultation</td><td>101</td></tr><tr><td>5.5.2</td><td>Recommending contributions to citizens during their navigation</td><td>103</td></tr><tr><td>5.6</td><td>Interview with stakeholders</td><td>105</td></tr><tr><td>5.7</td><td>Conclusion</td><td>107</td></tr></table>

This chapter of my thesis is based on our paper “Finding Conflicts of Opinion on Citizen Participation Platforms”, written with Oana Balalau, Rafael Angarita and Valérie Issarny. At the time of submission of my Ph.D. thesis, this paper has not been published yet.

## 5.1 Introduction

In certain instances, citizen participation platforms allow citizens to see and interact with one another's contributions, generally in an asynchronous collaborative work-
flow (see Chapter 4). This chapter focuses on these latter cases where collaboration-
oriented interactions among citizens are made possible. However, even though citi-
zen participation platforms are valuable means to co-construct elaborated projects
on a large scale, several studies have reported that users and decision-makers need
help at different stages in the participation effort. In [75], the authors identify several
challenges, such as the lack of communication between authorities and participants
and unclear criteria used by the authorities for selecting among the proposed solu-
tions, or the unwillingness of collaborators to reach a consensus. Another critical
challenge for such projects is information overload, affecting decision-makers and
users [38, 132].

This work introduces a tool to detect equivalence and contradiction in users’
proposals. Our tool allows users to find proposals of interest - whether equiva-
lent or contradictory - and engage with their contributors, while decision-makers
can leverage it to create projects composed of semantically consistent sets of pro-
posals. Semantic consistency, as opposed to syntactic consistency implemented by
consistency management algorithms for collaborative editing (e.g., [8]), is defined
as the absence of contradiction between different texts. The contradictions of in-
terest can be of different natures, ranging from ideological contradiction to numeric
disagreements [50]. We focus on projects composed of proposals contributed by dif-
ferent users. Then, we consider a project to be semantically consistent if it does
not contain any pair of contradictory proposals. Automatically ensuring semantic
consistency will help citizens build joint well-developed, complex projects by high-
lighting the potential contradictions in their constituent parts. Moreover, detecting
equivalent or contradictory proposals will help platform maintainers identify idea
communities and summarize their claims.

We introduce an approach based on Natural Language Processing (NLP) to
identify pairs of conflicting proposals via text classification. More precisely, this ob-
jective can be identified as a Natural Language Inference (NLI) task, also referred to
in the literature as textual entailment. Parikh et al. [126] define NLI as “the problem
of determining entailment and contradiction relationships between a premise and a
hypothesis”. In our case, we try to predict whether two proposals are entailed, con-
tradictory or neutral. We then define our methodology to identify models enabling
the detection of contradictory proposals. We apply our method to French-speaking
citizen consultations, i.e., “République Numérique”<fnref n="1" /> (Digital Republic, RepNum for
short in the following) and “Revenu Universel d’Activité”<fnref n="2" /> (Universal Activity In-
come, RUA in the following). We chose these consultations because they received a
large number of proposals (respectively 692 and 1760) and allowed contributors to
submit comments for or against any given proposal, which are exploited for model
training. Moreover, the contributions that were submitted in these consultations are
typically larger than pairs of sentences found in most NLI corpora (e.g., SNLI [30],
XNLI [45]). Therefore, we consider our work as an important step toward applying
NLI tasks to large texts. To validate our approach, we perform a user study compar-
ing textual entailment with textual similarity, a technique proposed for organizing
citizen contributions [13, 19]. Finally, we present an interview with Cap Collectif,
the company which has created the participatory platform used in the consultations
we are analyzing.

<footnote n="1">¹https://www.republique-numerique.fr/</footnote>

<footnote n="2">²https://www.consultation-rua.gouv.fr/</footnote>
This chapter is organized as follows. Section 5.2 defines our methodology. Section 5.3 describes how we build the datasets we use to train our models according to the specifics of citizen participation data. Section 5.4 describes the results obtained using our methodology with the datasets previously presented. Section 5.5 assesses the benefits of using such a tool on two online citizen consultations that took place in France, while in Section 5.6 we present an interview with Cap Collectif, the company that created the collaborative platform hosting the two citizen consultations we chose to implement our methodology. Finally, Section 5.7 concludes with perspectives for future work.

The code and datasets used in the production of this chapter is available at
https://github.com/WilliamAboucaya/rua_repnum_contradiction_analysis.

# 5.2 Methodology

This section presents our methodology to improve online citizen participation platforms using NLI on the contributions produced.

## 5.2.1 Problem statement

In this work, we focus on a particular type of citizen participation platform whose purpose is to gather opinions and potentially reach a consensus with citizens on a given topic. The discussion topic is usually structured in several subtopics, referred to as sections. We suppose that the platform implements the following workflow: i) A discussion starts in a section with a proposal. A proposal has a title (i.e., a short description) and a body (i.e., a detailed description of the proposal). ii) Each proposal can receive comments; iii) Each comment has a stance towards the proposal, in support or contradiction. From now on, we will refer to platforms fulfilling such requirements for citizen participation platforms. This format is also typical of a debate platform, where proposals are claims for which users present pro and against arguments.

Based on previous works [38, 75, 132], we identify the following needs (N1, N2, N3), expressing issues we aim at mitigating, and requirements (R1, R2), expressing general criteria to meet, for a citizen participation platform:

(N1) Reducing information overload:

* User information overload: a citizen cannot find proposals relevant to their points of view.

* Admin information overload: an admin cannot understand all the points brought forward by the citizens.

(N2) Obtaining projects composed of semantically consistent sets of proposals. The absence of contradictions is obtained via conflict resolution between different proposals.

(N3) Identifying the main points brought forward by citizens. Identifying groups of proposals which are supporting the same ideas is important for offering a complete overview of a consultation.
(R1) Quality of results of algorithms integrated into the platform. When employed, algorithms should provide good-quality information to users.

(R2) Ease-of-use of algorithms integrated in the platform.

These needs and requirements are particularly crucial to maintain healthy citizen participation platforms at large scale. To address the needs and requirements of stakeholders of a citizen participation platform, we propose the use of a method that identifies if two proposals are entailed or contradictory. We validate this approach in Section 5.5.

We consider a pair of proposals to be **entailed** if they mutually support one another's claims while not having incompatible elements. On the other hand, **contra-dictory** pairs of proposals support incompatible or compatible claims but describe them with incompatible elements (e.g., different numeric values for the budgeting of the same idea). A pair of proposals is labeled **neutral** if it does not fit the criteria for entailed and contradictory labels, i.e., two proposals supporting unrelated claims without incompatible elements in their descriptions.

**Problem statement:** *Given a pair of proposals (p₁, p₂) from a citizen participation platform, predict its textual entailment classification, i.e. if the pair is entailed, contradictory or neutral.*

We address this problem by introducing a methodology to efficiently train NLI classifiers for citizen participation platforms based on data produced by their contributors. These classifiers must be able to handle contributions from participatory platforms, which are typically longer than the texts provided by NLI datasets.

We position the methodology presented in this section as a tool to facilitate large-scale online citizen participation in contexts where citizens have the possibility to express their different views on specific debates related to a broad topic (e.g., digital transformation of a country, universal income) defined by the organizer of the participatory initiative. Online citizen consultations chosen as examples in the following section of this thesis, as well as other similar cases<fnref n="3" />, can be used to implement our methodology since citizens are not bound to answer specific questions or vote on pre-existing items. In these types of platforms, contributions often contain very few elements of structured characterization (e.g., tags) to lower the level of computer literacy required to participate. This leads us to focus on the textual content of the different proposals to better identify their relation with one another – entailed, contradictory or neutral. However, our work is less relevant for other types of online citizen participation initiatives such as citizen surveys or the deliberation phase of participatory budgeting [128]. Other works have been proposed to tackle the specific issues faced in these citizen participation contexts. For example, Goel et al. [64] propose a voting method for deliberation in participatory budgeting called “knapsack voting”, presented as an alternative to the most commonly implemented restricted voting methods. Their method is particularly relevant for participatory budgeting in which the objective is to choose how to distribute a finite budget between multiple projects, and allows to better represent the distributions expected by each citizen. However, in citizen consultation contexts, applying complex voting rules is much less likely to fix problems already identified such as information

<footnote n="3">³e.g., https://discriminations.cap-collectif.com, https://decide.madrid.es</footnote>
overload since the objective is to help citizens directly express their different views on a topic through text-based contributions – mainly proposals and comments. On the opposite, our work aims at ensuring that the publication of a large number of text-based proposals is not an obstacle to the proper understanding of the content of a participatory platform by citizens and platforms administrators. This problem is much more relevant for online consultation than for deliberation in participatory budgeting, since in the latter case the number of proposals is typically lower than in large-scale citizen consultation – to the best of our knowledge, participatory budgeting is currently only implemented at the local scale – and certain proposals are generally filtered out or grouped together prior to this step.

### 5.2.2 Weakly-labeled data for model training

To train classifiers for NLI in citizen participation contexts, we need large datasets of NLI-labeled pairs of texts from similar contexts. However, to our knowledge, such datasets do not exist. Existing datasets for NLI tasks (e.g., XNLI [45], SNLI [30]) are generally composed of pairs of one-sentence statements. Therefore, using the content produced by contributors of citizen participation platforms, we create weakly-labeled datasets for the NLI task. The term **weakly-labeled dataset** is used to denote a dataset created based on rules or human feedback, but for which we do not have a strong guarantee of quality as it was not obtained via an annotation procedure, hence some labels might be incorrect. The objective is to train our models on pairs of texts concerning topics similar to those treated in the real case data – here, the proposals in a citizen participation platform – using a similar vocabulary and writing style and containing a comparable number of tokens.

As stated in § 5.2.1, proposals receive comments explicitly identified as for or against them. To create a relevant NLI dataset, we use pairs constituted of a proposal and one of its comments. We use comments supporting the proposal to produce entailed pairs and comments against the proposal to create contradictory pairs. We select a proposal and a comment associated with another proposal for neutral pairs. Hence we associate proposals with comments about unrelated topics. This weakly-labeled dataset can then be used to fine-tune either a language understanding model (e.g., BERT [52]) or a model already trained for the NLI task.

### 5.2.3 Creating a gold dataset for evaluation

The weakly-labeled dataset is close to the problem stated in § 5.2.1, which is relevant for training NLI models. However, assessing the different classifiers on pairs of proposals is important for a complete evaluation. Therefore, we create a gold dataset composed of pairs of proposals from the consultation manually annotated using the labels entailed, neutral and contradictory. Pairs of proposals were generated using the following method:

1. First, we select a random proposal from the consultation studied.

2. Then, using SBERT's multilingual models [137, 139], we compute the selected proposal's cosine similarity with all the other proposals in the consultation. SBERT is trained for semantic textual similarity and allows us to identify proposals with similar topics.
3. Finally, we retrieve five pairs composed of the initial proposal and its five closest neighbors according to cosine similarity.

We note that choosing random pairs of proposals to annotate would give us a highly imbalanced dataset with a large percentage of neutral pairs. We will show in the following that the top-$k$ closest neighbors of a proposal contain entailed, neutral and contradictory proposals, hence highlighting the relevance of models trained for NLI.

### 5.2.4 Automatically labeling long texts

Humans are able to understand and remember a lot of contextual information and to make informed decisions based on this information. Proposals can be quite long, and contradictions between two proposals can appear only in a handful of sentences. In this work, we experiment with the amount of context that is necessary for a classifier to make a good classification decision. The entire proposal is the maximum amount of context, but the classifier could be overwhelmed. Hence, we also experiment with smaller units of context, such as a sentence or two consecutive sentences. Therefore, we propose three different **labeling strategies** to automatically label a pair of proposals $p_1$ and $p_2$:

**Proposal-wise:** The model takes as input a pair of proposals and returns a label we define as the prediction. If the proposals are longer than the tokens limit imposed by the model, then the last tokens are not considered. This strategy is identical to the one used during the training of our models on the weakly-labeled datasets.

**Sentence-wise:** First, we split $p_1$ and $p_2$ into sentences. Then, we feed the model with every possible pair containing a sentence from $p_1$ and $p_2$, respectively. Finally, we compute $S_e$ and $S_c$, respectively, the share of pairs labeled as entailed or contradictory among all the pairs of sentences from $p_1$ and $p_2$. If $S_e$ (resp. $S_c$) is higher than a threshold $t_e$ (resp. $t_c$), then the proposals are labeled as entailed (resp. contradictory). Otherwise, neither $S_e$ nor $S_c$ are higher than their respective thresholds, and the proposals are labeled as neutral. Finally, if both $S_e$ and $S_c$ are higher than their respective thresholds, the label is chosen according to the highest value between $S_e - t_e$ and $S_c - t_c$.

**Sentence couple-wise:** This strategy is similar to the sentence-wise strategy with the following difference: the pairs used to feed the model are constituted of two consecutive sentences from $p_1$ and $p_2$. The objective is to give more context to the classifier than in the sentence-wise strategy. The label for the pair of proposals is then computed exactly the same way as for the sentence-wise strategy.

For sentence-wise and sentence couple-wise strategies, we compute $t_e$ and $t_c$ for each <model, strategy> pair using the weakly-labeled dataset. First, we compute $S_e$ and $S_c$ for each pair in the test split of the weakly-labeled dataset using either the sentence-wise or sentence couple-wise strategy. Then, we use the SciPy [166] implementation of the dual annealing optimization algorithm to find the $(t_e, t_c)$ pair for which the macro average of F1-scores is maximized for our NLI task. The pair of
<table><caption>Table 5.1: Distribution of NLI labels for pairs of texts used for models training</caption><thead><tr><th>DATASET</th><th>NB. OF ENTAILED PAIRS</th><th>NB. OF NEUTRAL PAIRS</th><th>NB. OF CONTRADICTORY PAIRS</th></tr></thead><tbody><tr><td>RepNum<sub>WL</sub></td><td>2,953</td><td>2,995</td><td>3,037</td></tr><tr><td>RUA<sub>WL</sub></td><td>4,954</td><td>4,489</td><td>4,024</td></tr><tr><td>XNLI<sub>FR</sub></td><td>133,399</td><td>133,400</td><td>133,403</td></tr></tbody></table>

thresholds computed on the weakly-labeled dataset of a given consultation is used for pairs of proposals from the same consultation.

## 5.3 Datasets

We apply the methodology described in the previous section to two major online citizen consultations in France: RepNum and RUA.

The **RepNum consultation** has already been presented in Section 3.2, therefore we will only present its most relevant characteristics for this chapter. This consultation aimed at allowing contributors to propose amendments to a bill on how the French state should be organized in a digital era<fnref n="4" />. The main themes of the consultation were the digital transformation of the French administration, open data, online privacy, and access to the internet. Proposals published in this consultation were regrouped in three different sections depending on their topic. The platform allows contributors to interact with one another's proposals using, among other things, comments labeled as for or against the proposal.

The **RUA consultation** was held between 10/9/2019 and 11/20/2019 on a dedicated participatory platform. The objective of the consultation was to help the French government gather citizens' opinions on a new solidarity system based on three main axes: (i) the objectives the system should pursue, (ii) its administrative and financial functioning, and (iii) the criteria to benefit from this new income. At the beginning of the consultation, 18 proposals had been submitted by the French Ministry of Health and Solidarity, followed by 1,742 additional proposals submitted by citizens and other stakeholders. Similarly to *RepNum*, this consultation was held on a platform based on the proprietary solution for citizen participation developed by the Cap Collectif company. Proposals published have a title and a body text explaining their content. The three axes of the consultation were split into eight sections regrouping proposals about a specific topic. The consultation took place on a platform based on the same proprietary solution as the RepNum consultation. To our knowledge, no prior academic research work has been published on this consultation. Contributions to the *RUA* consultation and related metadata are available online as open data.<fnref n="5" />

**Datasets for model training.** To train our models, we used three different datasets, namely the French-speaking section of XNLI (named XNLI<sub>FR</sub>) and two weakly-labeled datasets we produced from the RepNum and RUA consultations (respectively named RepNum<sub>WL</sub> and RUA<sub>WL</sub>).

<footnote n="4">⁴https://en.wikipedia.org/wiki/Loi_pour_une_République_numérique</footnote>

<footnote n="5">⁵https://www.data.gouv.fr/fr/datasets/consultation-vers-un-revenu-universel-dactivite-1</footnote>
<figure><img src="image_15.png" /><figcaption>Figure 5.1: A proposal (top) with the arguments pro (bottom-left) and against (bottom-right) it in the RepNum consultation platform (anonymized).</figcaption></figure>

**Translation of the proposal:** Our country is becoming less and less democratic. The citizen can only express themself by his vote. It is time to allow them to give their opinion on the texts which govern their life. This must be done without any intermediary, directly through a web platform. 1° Using a platform open to all, the government must allow the citizen to propose modifications on official texts. 2° These texts can be laws, the constitution or codes. 3° The actors of the State are morally committed to read and take into account the proposals. 4° The above-mentioned actors have to use the platform to participate in the discussion. 5° However, they must not have administrator rights, their voice must be equal to that of a citizen. 6° Every month a committee composed of citizens picked at random and members of the government must meet to discuss the proposals. 7° If a text undergoes significant modifications, a referendum may be conducted.

**Translation of the first comment pro:** 7° In case of important modifications, a referendum MUST be held.

**Translation of the second comment pro:** This consultation allows to enrich the parliamentary debate and not to compete with it. It allows a citizen debate and undoubtedly ideas that representatives could not have will appear. Moreover, with parachuted people, the availability of the elected representatives is quite limited.

**Translation of the third comment pro:** Yes but only if identification of a French citizen is ensured (today anyone can register even people who are not from this country)

**Translation of the first comment against:** It is not the citizen's job to do this, but the legislator's.

**Translation of the second comment against:** A platform of this type, and it is already the case on that one, generates irrelevant proposals that are difficult to review.

**Translation of the third comment against:** "citizens picked at random and members of the government"? No no: it is the job of the legislative power, elected by all the citizens on the principle of "1 citizen = 1 vote". And the voters are not picked at random
**XNLI<sub>FR</sub>:** The XNLI dataset [45] is a cross-lingual dataset for NLI tasks. It consists of pairs of sentences originally written in English and translated into 15 languages, including French. Each pair of sentences is annotated as entailed, contradictory or neutral. The French-speaking section of the XNLI dataset contains 400K datapoints. Each datapoint contains a pair of sentences, called the premise and the hypothesis, and a label. The distribution of labels in the XNLI<sub>FR</sub> is available in Table 5.1. This dataset is by far the largest we have used, but the natural-language texts are drastically smaller than standard texts written in online citizen consultations, with the median number of tokens being 23 for premises and 13 for hypotheses. The XNLI<sub>FR</sub> dataset is split as follows: a training set (≈ 98.1 % of the datapoints), a validation set (≈ 0.6 %), and a test set (≈ 1.3 %).

**RepNum<sub>WL</sub> and RUA<sub>WL</sub>:** The RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets are weakly-labeled datasets created from the textual content of the associated consultations. As said earlier, these consultations allow contributors to add comments to a given proposal and to label them as “for” or “against” the respective proposal (see Figure 5.1). The distribution of labels in these datasets is also available in Table 5.1. These datasets are smaller than XNLI<sub>FR</sub>. RepNum<sub>WL</sub> contains 8,985 datapoints and RUA<sub>WL</sub> contains 13,467 datapoints. However, the texts are longer: in RUA<sub>WL</sub>, the proposals have a median number of tokens of 58 and the comments of 42, while in RepNum<sub>WL</sub> the proposals have a median number of tokens of 234 and the comments of 40. Both datasets are split as follows: a training set (80 %), a validation set (10 %), and a test set (10 %).

**Datasets for model evaluation.** Even though the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets are large enough to be used for the training of our models, they are not representative of our real case problem where we want to identify entailed, contradictory and neutral pairs of proposals. To assess our models on data more closely related to our problem, we manually annotated 239 pairs of proposals from the RUA consultation and 107 pairs from the RepNum consultation, using the three labels, entailed, neutral and contradictory. Table 5.2 displays examples of pairs of proposals for the three different labels. These annotations represent our gold dataset, see Table 5.3 for the distribution of labels.

N.B.: The pairs in Table 5.3 have been produced using the method described in § 5.2.3. The high number of pairs of sentences annotated as neutral allows us to infer that cosine similarity by itself is not a meaningful indicator of the entailment or the contradiction of a pair of proposals.

Both datasets were annotated independently by two authors of the original paper, and agreement among annotators was measured using Cohen's kappa (κ) coefficient [41]. According to Landis and Koch's classification of κ score [88], our agreement is substantial (κ > 0.6) for both datasets and therefore, we consider that comparison with our labels is a valid means to evaluate our models. Furthermore, all the datasets are available in two versions: with and without the title of the proposals before their body text, but the annotators have worked on the version with the titles included. We note that the initial pool of sentence pairs contained a small number of contradictory labels. Therefore, a keyword-matching method was used to
<table><caption>Table 5.2: Examples of pairs of proposals from the RUA consultation with their gold label. Proposals translated by this work's authors. Opinions expressed do not reflect those of the authors of this work.</caption><thead><tr><td>FIRST PROPOSAL</td><td>SECOND PROPOSAL</td><td>LABEL</td></tr></thead><tbody><tr><td>Information and accessibility. Consumer information videos must be accessible to everyone, including deaf people. This means ensuring the presence of subtitles and French sign language interpretation. All consumer information posted by the relevant authorities and related to any government decision must be fully accessible.</td><td>Practices - Accessibility. Implement subtitles or French sign language translation for deaf and hearing-impaired consumers to better inform them concerning company policies and consumers' rights.</td><td>Entailed</td></tr><tr><td>An evaluation by a social worker (based half on an evaluation grid, half on human investigation) and multi-disciplinary support. A person must be assessed by a well-trained social worker to give them right to the Universal Activity Income and multi-disciplinary support must be offered according to their needs: support towards employment, help with budget management, support towards care or adaptation of housing (if unsafe or inaccessible). Working in partnership with the different stakeholders would make it possible to follow the evolution of the beneficiaries without putting the burden of monitoring on a single worker. It would be much more beneficial and appropriate.</td><td>Automated payment. The payment of the Universal Activity Income should not be subject to a recurrent resources/status reporting. Interconnections are possible (or even already exist between tax collection services, family allowance funds, social security, employment agencies, etc.) so that the calculation of the amount and the payment are made automatically. This would also avoid overpayments due to errors or delays in payment that could place the poorest people in even more difficult situations (eviction from housing, bankruptcies, etc.).</td><td>Contradictory</td></tr><tr><td>Taking into account people who live in couples without being married for reversionary pension! Hello! it seems normal to me that people who live in couple without being married for years and who have 1 or more children can benefit from the reversionary pension of the spouse! Nowadays marriage does not mean anything anymore, it is just an administrative act!</td><td>Take into account the income of each person rather than of the couple. Today, for certain aids, the income taken into account is that of the household, therefore reducing the aids for couples. If we use individual income, each person will be entitled to what is due to them according to their situation. The "adult with a disability" allowance for a couple is not the addition of two allowances for single people. Moreover, at age 60 (automatically retired), their income decreases but the disability is still there...</td><td>Neutral</td></tr></tbody></table>
<table><caption>Table 5.3: Distribution of NLI labels in the gold dataset</caption><thead><tr><th>CONSULTATION</th><th>NB. OF ENTAILED PAIRS</th><th>NB. OF NEUTRAL PAIRS</th><th>NB. OF CONTRADICTORY PAIRS</th><th>κ AGREEMENT</th></tr></thead><tbody><tr><td>RepNum</td><td>22</td><td>65</td><td>20</td><td>0.614</td></tr><tr><td>RUA</td><td>94</td><td>108</td><td>37</td><td>0.621</td></tr></tbody></table>

find additional contradictory examples. As a result, twenty additional contradictory pairs have been added to RUA, while twelve contradictory pairs have been added to RepNum. The proposals (body and title) in the manually annotated dataset for the RepNum (resp. RUA) consultation have a median length of 179 tokens (resp. 87 tokens).

# 5.4 Evaluation

## 5.4.1 Fine-tuned models

CamemBERT [105] is a state-of-the-art language model for French. Moreover, unlike certain other models such as GPT-3 [31], CamemBERT is available under an open source license, which is considered as a crucial feature for trustworthiness of participatory platforms [25]. CamemBERT has been pretrained on masked language prediction and is fine-tuned on the required downstream application. In this work, we propose several fine-tuning scenarios, and we investigate which has the best performance. We use CamemBERT<sub>LARGE</sub>, a neural network with an architecture of 24 layers, 1024 hidden dimensions, 12 attention heads, and 335M parameters. We fine-tune the CamemBERT model using the following seven combinations:

**RepNum:** CamemBERT model fine-tuned on the RepNum<sub>WL</sub> dataset.

**RUA:** CamemBERT model fine-tuned on the RUA<sub>WL</sub> dataset.

**RepNum + RUA:** CamemBERT model fine-tuned on both the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets.

**XNLI:** CamemBERT model fine-tuned on the XNLI<sub>FR</sub> dataset.

**XNLI → RepNum:** CamemBERT model fine-tuned on the XNLI<sub>FR</sub> dataset, then fine-tuned again on the RepNum<sub>WL</sub> dataset.

**XNLI → RUA:** CamemBERT model fine-tuned on the XNLI<sub>FR</sub> dataset, then fine-tuned again on the RUA<sub>WL</sub> dataset.

**XNLI → RepNum + RUA:** CamemBERT model fine-tuned on the XNLI<sub>FR</sub> dataset, then fine-tuned again on both the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets.

We train our models for 5 epochs, with a learning rate of 1e-5 as suggested in [105]<sup>6</sup>, a weight decay of 0.01 and a batch size of 8. We implemented our models using PyTorch [129] and the Transformers library [177].

<footnote n="6"><sup>6</sup>Default parameter proposed for the fine-tuning of RoBERTa on the MNLI task (see https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.glue.md).</footnote>
<table><caption>Table 5.4: Results on the RUA gold dataset: macro F1-score and precision on entailed label ($P_E$) and contradictory label ($P_C$) for the manually annotated pairs of proposals of the RUA consultation. Results obtained using only the content of the proposals, except starred (*) results where the title of the proposals and name of their sections were included. The macro F1-score of a random classifier is 31.01.</caption><thead><tr><th rowspan="3">MODEL TRAINED ON...</th><th colspan="12">LABELING STRATEGY</th></tr><tr><th colspan="6">WITH PAST TENSE SENTENCES</th><th colspan="6">WITHOUT PAST TENSE SENTENCES</th></tr><tr><th colspan="3">PROPOSAL</th><th colspan="3">SENTENCE</th><th colspan="3">SENTENCE COUPLE</th><th colspan="3">SENTENCE COUPLE</th></tr><tr><th></th><th>F1</th><th>P<sub>E</sub></th><th>P<sub>C</sub></th><th>F1</th><th>P<sub>E</sub></th><th>P<sub>C</sub></th><th>F1</th><th>P<sub>E</sub></th><th>P<sub>C</sub></th><th>F1</th><th>P<sub>E</sub></th><th>P<sub>C</sub></th><th>F1</th><th>P<sub>E</sub></th><th>P<sub>C</sub></th></tr></thead><tbody><tr><td>RepNum</td><td>30.8</td><td>43.1</td><td>29.0</td><td>31.8</td><td>44.8</td><td>20.4</td><td>37.3</td><td>49.3</td><td>17.6</td><td>30.8</td><td>43.9</td><td>27.8</td><td>35.8</td><td>41.8</td><td>26.9</td><td>37.9</td><td>48.8</td><td>20.0</td></tr><tr><td>RUA</td><td>45.7</td><td>55.0</td><td>29.6</td><td>43.5</td><td>53.9</td><td><u>55.6*</u></td><td>35.6</td><td>50.0</td><td>22.2</td><td>44.2</td><td>52.8</td><td>30.8</td><td>43.1</td><td>52.9</td><td>55.0*</td><td>35.7</td><td>51.7</td><td>28.6</td></tr><tr><td>RepNum + RUA</td><td>44.8</td><td>50.9</td><td>28.9</td><td>50.0</td><td>59.5</td><td>26.2</td><td>36.6</td><td>55.6</td><td>20.0</td><td>45.2</td><td>51.0</td><td>32.5</td><td>49.5</td><td>58.7</td><td>28.2</td><td>36.8</td><td>52.1</td><td>18.2</td></tr><tr><td>XNLI</td><td>45.7</td><td><u>76.5*</u></td><td>51.9</td><td>45.1</td><td>58.3</td><td>32.9</td><td>37.9</td><td>76.2</td><td>27.0</td><td>45.1</td><td>74.2*</td><td>50.0</td><td>42.7</td><td>54.8</td><td>32.9</td><td>37.9</td><td>76.2</td><td>27.8</td></tr><tr><td>XNLI → RepNum</td><td>30.6</td><td>44.4</td><td>20.5</td><td>39.1</td><td>46.9</td><td>23.7</td><td>40.6</td><td>48.4</td><td>31.8</td><td>30.3</td><td>45.2</td><td>20.5</td><td>39.5</td><td>45.2</td><td>25.6</td><td>39.9</td><td>47.8</td><td>31.8</td></tr><tr><td>XNLI → RUA</td><td>52.4</td><td>58.9</td><td>35.7</td><td><u>54.7</u></td><td>59.3</td><td>39.4</td><td>38.8</td><td>54.3</td><td>26.7</td><td>53.7</td><td>57.8</td><td>43.2</td><td><u>54.7</u></td><td>59.3</td><td>42.4</td><td>39.4</td><td>57.1</td><td>27.6</td></tr><tr><td>XNLI → RepNum + RUA</td><td>48.2</td><td>55.9</td><td>32.5</td><td>51.1</td><td>61.3</td><td>29.8</td><td>40.6</td><td>55.0</td><td>35.7</td><td>48.8</td><td>55.0</td><td>40.0</td><td>48.9</td><td>59.1</td><td>28.9</td><td>41.7</td><td>55.2</td><td>38.9</td></tr></tbody></table>

Once the models have been trained, we apply the strategies described in Section 5.2.4 to two different versions of the datasets: one containing the complete proposals without any pre-processing and another where past tense sentences were removed. This pre-processing is motivated by the fact that in French, sentences expressing instructions – i.e., the components and objectives of the law, that is, the elements of potential disagreement with other proposals – are written in present or future tense. Consequently, past tense sentences in our datasets can be seen as elements of context and, potentially, as superfluous and unnecessarily increasing the number of tokens. To perform this pre-processing, we trained a token classifier for Part-Of-Speech (POS) tagging using the PERCEO corpus [17], where verbs are labeled using the “VER” tag and a sub-tag identifying their tense. Then, we apply our POS-tagger to each sentence of our dataset, and we remove the sentences containing *passé simple*, *imparfait*, and *subjonctif imparfait* verbs since we identify them as past tense sentences.

The results obtained using the 42 different combinations of our fine-tuned models (7 models) and strategies on our two manually annotated datasets (6 strategies) are available in Tables 5.4 and 5.5. We compute the macro F1-scores for each combination and the precision for the labels entailed ($P_E$) and contradictory ($P_C$). The first metric informs us of the overall performances of a <model, strategy> pair. The second and third metrics allow us to assess the model’s suitability for the large-scale evaluation presented in § 5.5. We analyze the performance of the two datasets separately.

Given the definition of a contradictory pair we use (see § 5.2.1), one could argue that the sentence-wise and sentence couple-wise should label a pair of proposals ($p_1, p_2$) as contradictory if at least one of the sentences of $p_1$ is contradictory with any sentence from $p_2$, i.e., if $S_c > 0$. However, given the consistently low results for ($P_C$) obtained using this method, we have chosen not to include it in our experimental evaluation. For example, using the RUA model and sentence-wise strategy on the
<figure><img src="image_16.png" /><figcaption>Figure 5.2: Confusion matrix for the RUA manually annotated dataset without past tense sentences, using the “XNLI → RUA” model and sentence-wise strategy. The labels are as follows: entailed (0), contradiction (1), neutral (2).</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 5.3: Confusion matrix for the RepNum manually annotated dataset without past tense sentences, using the “RepNum + RUA” model and proposal-wise strategy. The labels are as follows: entailed (0), contradiction (1), neutral (2).</figcaption></figure>

RUA gold dataset, $P_C$ = 15.7, which is significantly lower than the results obtained with other strategies.

**Results for RUA** The best-performing model on RUA, according to the F1 score on our gold dataset, is the model fine-tuned first on XNLI and then on $RUA_{WL}$. For each labeling strategy, the results obtained using this model are consistently among the best, if not the best. We observe that fine-tuning on $RUA_{WL}$ generally improves the classifier’s performance, which is expected as this dataset is the most similar to the gold dataset. Using this model and the sentence-wise strategy, independently of the inclusion of past-tense sentences, we obtain a macro F1-score of 54.7. Furthermore, the confusion matrix (see Figure 5.2), which displays the distribution of datapoints depending on their true and predicted labels, shows that this combination of model and strategy struggles with identifying contradictory (1) pairs and on the opposite, is very effective in identifying entailed (0) pairs. However, we cannot identify a labeling strategy that consistently produces the best results for every model. Therefore, for every model, all the strategies (see § 5.2.4) should be tested to find the one giving the best results.

**Results for RepNum** The best-performing model on RepNum according to the F1 score on our gold dataset is the model fine-tuned using both the $RepNum_{WL}$ and $RUA_{WL}$ datasets. The best labeling strategy for this dataset is proposal-wise with past tense sentences removed, with the macro F1-score of 48.8.

The confusion matrix (see Figure 5.3) shows that neutral (2) and contradictory (1) pairs tend to be falsely identified as entailed pairs, while entailed (0) pairs tend to be correctly identified. Overall, the best results are mostly obtained using the proposal-wise strategy. This is an interesting finding, as the proposals from the RepNum consultation are on average longer than those from the RUA consultation (see Section 5.3). We hypothesize that this happens because of how the thresholds
<table><caption>Table 5.5: Results on the RepNum gold dataset: macro F1-score and precision on entailed label (P<sub>E</sub>) and contradictory label (P<sub>C</sub>). The results are obtained using the content of the proposals prepended with their titles. The macro F1-score of a random classifier is 26.3.</caption><thead><tr><th rowspan="3">MODEL<br>TRAINED<br>ON...</th><th colspan="9">LABELING STRATEGY</th></tr><tr><th colspan="3">WITH PAST TENSE SENTENCES</th><th colspan="3">WITHOUT PAST TENSE SENTENCES</th></tr><tr><th>PROPOSAL</th><th>S SENTENCE</th><th>S SENTENCE<br>C couple</th><th>PROPOSAL</th><th>S SENTENCE</th><th>S SENTENCE<br>C couple</th><th>PROPOSAL</th><th>S SENTENCE</th><th>S SENTENCE<br>C couple</th></tr></thead><tbody><tr><td>RepNum</td><td>35.6 | 27.8 | 20.0</td><td>26.1 | 23.8 | 25.0</td><td>23.1 | 22.3 | 33.3</td><td>34.3 | 26.7 | 20.0</td><td>30.6 | 26.3 | 30.8</td><td>27.6 | 22.8 | 50.0</td><td></td><td></td><td></td></tr><tr><td>RUA</td><td>42.6 | 25.4 | <b>83.3</b></td><td>30.2 | 24.4 | 25.0</td><td>33.4 | 27.0 | 23.1</td><td>40.8 | 25.0 | 80.0</td><td>30.8 | 23.7 | 25.0</td><td>33.2 | 24.4 | 37.5</td><td></td><td></td><td></td></tr><tr><td>RepNum +<br>RUA</td><td>47.2 | 29.0 | 54.5</td><td>37.0 | 26.0 | 40.0</td><td>45.1 | 30.8 | 42.9</td><td><b>48.8</b> | 30.0 | 53.8</td><td>33.9 | 24.4 | 44.4</td><td>43.3 | 29.0 | 44.4</td><td></td><td></td><td></td></tr><tr><td>XNLI</td><td>35.8 | 22.6 | 33.3</td><td>27.9 | 20.4 | 18.2</td><td>38.0 | 32.1 | 23.8</td><td>36.4 | 25.0 | 33.3</td><td>27.9 | 19.2 | 21.7</td><td>38.2 | <b>34.6</b> | 23.3</td><td></td><td></td><td></td></tr><tr><td>XNLI →<br>RepNum</td><td>45.7 | 28.6 | 50.0</td><td>31.5 | 22.0 | 60.0</td><td>31.5 | 24.4 | 28.6</td><td>45.1 | 26.9 | 62.5</td><td>31.3 | 21.2 | 75.0</td><td>32.9 | 24.1 | 40.0</td><td></td><td></td><td></td></tr><tr><td>XNLI →<br>RUA</td><td>39.5 | 24.1 | 61.5</td><td>32.6 | 22.5 | 41.7</td><td>35.7 | 25.8 | 26.9</td><td>42.0 | 25.0 | 64.3</td><td>25.3 | 20.2 | 38.5</td><td>37.0 | 26.7 | 28.6</td><td></td><td></td><td></td></tr><tr><td>XNLI →<br>RepNum +<br>RUA</td><td>45.7 | 30.4 | 41.2</td><td>29.8 | 21.7 | 66.7</td><td>38.5 | 23.5 | 75.0</td><td>44.3 | 29.8 | 42.9</td><td>32.1 | 21.7 | 50.0</td><td>43.1 | 25.3 | 75.0</td><td></td><td></td><td></td></tr></tbody></table>

on the percentage of entailed/contradictory pairs are computed. We recall that we chose the threshold using the weakly-labeled dataset to avoid biased results by computing it on the test set. However, in the weakly-labeled dataset, we have comments which are much shorter than proposals on RepNum. Hence, we might not be choosing the correct thresholds. This hypothesis is confirmed by the fact that, simply by changing the value of one of the two thresholds, we did reach significantly higher values for the macro F1-score – e.g., 58 for the XNLI → RepNum + RUA model with the sentence couple-wise strategy, while the value obtained with the pair of thresholds computed is 38.5.

## 5.4.2 Insights and error analysis

The results of our different <model, strategy> pairs allow us to draw two conclusions concerning NLI for citizen participation. First, as observed on the RepNum gold dataset, a model can learn using data from another consultation. Indeed, the best results for the metrics computed in this chapter – macro F1-score, precision on entailed, and contradictory labels – on the RepNum dataset are obtained using the RepNum + RUA, XNLI, and RUA models, respectively. Therefore, we consider the datasets produced for this chapter to be relevant as an initial corpus for future works on NLI for collaboration in the context of citizen participation (e.g., in the context of a platform where no weakly-labeled data can be extracted).

Second, confusion matrices highlight that classifiers tend to falsely label neutral pairs as entailed pairs, while the opposite is uncommon. This issue highlights a low recall on the neutral class for most classifiers. This could be because entailed pairs are selected in the weakly labeled data as pairs of proposals and comments “for”. However, a user might add a comment “for” to express an opinion that is on topic, but not addressing the main claims of the proposal, hence being according to our definition a neutral statement. These observations are validated by the fact that a model has a higher recall when trained on XNLI - a dataset where annotators
<table><caption>Table 5.6: Examples of pairs of proposals incorrectly labeled by the “XNLI → RUA” model with sentence-wise strategy. Proposals translated by the authors of this work.</caption><thead><tr><th>FIRST PROPOSAL</th><th>SECOND PROPOSAL</th><th>TRUE LABEL</th><th>PREDICTED LABEL</th></tr></thead><tbody><tr><td>Review the person's situation regularly.<br>A person's Universal Activity Income should be updated regularly to adapt to changes in their family or personal situation. It is also necessary to support people who go back to work without immediately taking away this income. When you get a new job, you have additional expenses and you need time to organize yourself, especially when you have children.</td><td>Allow the vulnerable self-employed entrepreneurs to have a safety net.<br>The Universal Activity Income should encourage people to have a salaried or non-salaried activity. From my experience, it is hard to have a non-salaried activity and to receive aid for the difficult months. It is not normal to discourage people who want to start their own business.</td><td>Neutral</td><td>Entailed</td></tr><tr><td>Everyone is equal so the financial aid should be the same.<br>800 € paid automatically to the persons living in France and older than 16 on the 1st of each month without any condition, no fraud possible, no paper to provide.</td><td>Universal income for all from birth.<br>Universal income from birth for all citizens legally residing in France; for a child, the payment will be blocked and made available upon reaching legal majority.</td><td>Contradictory</td><td>Entailed</td></tr></tbody></table>

gave entailed and neutral pairs. For example, for the RepNum dataset, the average recall on the neutral label for the <model, strategy> pairs using the XNLI model is 41.2, while it is 30.7 for the pairs using another model. Therefore, new methods to produce better quality weakly-labeled neutral pairs can be beneficial in the training of models.

False identification of neutral pairs as entailed is frequent in cases where both proposals have common objectives but different contexts. For example, the first row of Table 5.6 displays two proposals that aim to mitigate financial risks induced by changing life situations using a temporary income. However, these two proposals describe two different categories of persons to protect- persons who are in charge of a family and self-employed workers- which leads us to label the pair as neutral. This issue is common and is the first lead to improving the reliability of our classifiers to label neutral pairs. Another frequent issue in labeling concerns contradictory pairs of proposals falsely labeled as entailed. An example is shown in the second line of Table 5.6. Here, while the two proposals strongly agree on the principle of a universal income with as little restriction as possible, they differ on the minimum age required to receive the income – respectively starting at the age of 16 and at birth. This difference leads us to label the pair as contradictory while their agreement on the principle of an unconditional universal income probably leads the classifier to
label them as entailed. These examples can be considered as “contextual errors”, i.e. examples where both proposals advocate for similar measures but within different contexts or conditions. We, therefore, consider that higher quality of understanding of the context would improve the quality of models for NLI on texts larger than one sentence.

## 5.5 Assessing the integration of an NLI model into a citizen participation platform

In this section, we perform a large-scale investigation of the potential of our work. In particular, we will address how we can deal with the **requirements and needs** that we defined in Section 5.2.1. The proposed models can be used to identify groups of proposals defending similar ideas or proposals that are in contradiction, hence they can help solve N1 and N3. Moreover, finding entailed proposals can help citizens discover like-minded contributors and produce joint contributions, while platform maintainers can identify opinion groups to facilitate the synthesis of a consultation. In addition, retrieving contradictory proposals is the first step toward resolving the conflict and solving N2.

The thresholds for the percentage of entailed and contradictory pairs ($S_E$ and $S_C$) for the sentence-wise and sentence couple-wise strategies are computed in Section 5.4 such that we maximize the macro F1 score. However, one can optimize the thresholds for improved precision to show users only the pairs of proposals labeled with the highest certainty, that is, the pairs of proposals that are more likely to be contradictory or entailed, depending on the purpose. In addition, we can choose a model based on its precision on a specific class, not the overall F1 score.

For the RUA consultation, the best classifier for the class entailed is the XNLI model and the proposal-wise strategy. For the RepNum consultation, none of the classifiers produced have a high enough precision (we aim for a precision higher than 60%) for the entailed label to be reliable for this use case. Therefore, we use a modified version of the classifier composed of the “XNLI → RepNum + RUA” model and the sentence couple-wise strategy. We increase the $t_e$ threshold, which improves the precision ($P_E = 66.7$) while reducing the recall ($R_E = 28.6$) for the entailed label. We follow the same approach to select the best models for finding contradictory proposals. For the RepNum consultation, we use the RUA model and the proposal-wise strategy. For the RUA consultation, the classifiers need a higher precision for the contradictory label to be used for contradictory pairs detection. Consequently, we slightly increase the $t_c$ threshold, improving the precision ($P_C = 81.8$) and slightly reducing the recall ($R_C = 24.3$) for the contradictory label. We note that the contradictory model has a very high precision, meaning that a user who asks to see a proposal that contradicts a given one will more likely see a correct answer. For the entailed class, the precision is lower, however, we have shown in our error analysis (Section 5.4.2) that the proposals are still relevant to each other. Hence, we believe that this work is in line with requirement R1.

Once the classifiers have been chosen, we split the consultations using the different sections defined by the participatory platforms (see § 5.3). We then apply the classifiers to all the possible pairs of proposals inside the same section. Results for both the RepNum and RUA consultations are available in Table 5.7. For RepNum,
<table><caption>Table 5.7: Distribution of NLI labels for pairs of proposals in the RepNum and RUA consultations. Overlapping proposals are the ones labeled as entailed by the classifier chosen to maximize $P_E$ and labeled as contradictory by the classifier chosen to maximize $P_C$.</caption><thead><tr><th rowspan="2">CONSULTATION</th><th rowspan="2">TOTAL NB. OF PAIRS</th><th colspan="2">ENTAINED PAIRS</th><th colspan="2">CONTRADICTORY PAIRS</th><th colspan="2">OVERLAPPING PAIRS</th></tr><tr><th>NB.</th><th>%</th><th>NB.</th><th>%</th><th>NB.</th><th>%</th></tr></thead><tbody><tr><td>RepNum</td><td>86,398</td><td>210</td><td>0.2 %</td><td>1,548</td><td>1.7 %</td><td>5</td><td>0.006 %</td></tr><tr><td>RUA</td><td>295,796</td><td>28,582</td><td>9.6 %</td><td>9,659</td><td>3.2 %</td><td>736</td><td>0.2 %</td></tr></tbody></table>

we observe a relatively small percentage of entailed pairs compared to contradictory pairs. RepNum has a total of 692 proposals compared to RUA, which has 1,760 proposals. The small percentage of entailed pairs indicates that RepNum users have more easily identified proposals that correspond with their view on the platform. Consequently, they concentrate their effort on adding contradictory proposals when they disagree with the existing ones. On RUA, we observe a large percentage of entailed pairs, which could result from the difficulty in navigating the larger number of proposals. Finally, we note that many proposals are entailed or contradictory on both platforms, highlighting the important contribution that an NLI model would bring to a citizen participation platform.

### 5.5.1 Identifying idea communities in a consultation

A first use case for our models would be identifying communities of contributions defending similar ideas. This use case would be relevant to draw an initial summary of the ideas evoked in a given participatory platform when exhaustive reading of contributions is not an option (e.g., when the number of contributions produced is too high). Indeed, in this situation, the automated identification of communities would allow platform administrators to read only a few contributions in each community to identify the main ideas defended.

In the following, we focus on the largest sections regarding the number of proposals for RepNum and RUA. For RepNum, the section title is *La circulation des données et du savoir* (The flow of data and knowledge). For RUA, the section title is *À quels publics le revenu universel d'activité doit-il s'adresser ?* (Who should the universal activity income be aimed at?). We create a graph $G$ for each section where a vertex is a proposal, and two vertices $v_1, v_2$ are connected if the corresponding proposals were labeled as entailed. Finally, we find communities using scikit-network's [29] implementation of the Louvain algorithm. Proposals belonging to the same community are considered to defend similar ideas. For RepNum, we apply the Louvain algorithm with $\gamma = 1$, and for RUA with $\gamma = 1.5$. We then annotate the proposals in each community depending on their topic and claims, and we compute the purity [102] of the clusters. For RepNum, we also compute the purity of a baseline clustering produced for comparison using the subsections of the section of the consultation – this comparison cannot be performed on RUA as the section studied does not have subsections. Results are available in Table 5.8. Concerning the section from the RepNum consultation, the modularity obtained is high, with well-defined clusters (see Figure 5.4). The section from RUA consultation has a low modularity score, however, both the RepNum and RUA sections have a significant
<figure><img src="image_18.png" /><figcaption>(a) The nodes are colored and grouped according to the communities found by the Louvain algorithm.</figcaption></figure>

<caption>(b) The nodes are colored according to the subsections of the proposals.</caption>

Figure 5.4: RepNum proposals from the section “La circulation des données et du savoir”, on the flow of data and knowledge. A node represents a proposal and an edge connects two nodes when the corresponding proposals are entailed. Isolated vertices are not displayed. We observe that entailed proposals are likely to be placed by users in different subsections. A platform implementing our model would be able to guide users towards existing entailed proposals and help them better contribute to the consultation.

level of cluster purity. In particular, the purity of our clustering on RepNum has a very high purity when compared with the baseline clustering. This suggests that the communities in both RUA and RepNum have clearly defined topics and claims widely shared inside the clusters. In Figure 5.4, we observe the communities in RepNum. On the left, we have the distribution of proposals according to the community detection algorithm, while on the right, we see their distribution according to the subsection (the section we selected had 5 subsections). The entailed clusters contain proposals written in different subsections, showing that users might struggle to add contributions in the correct section/subsection, which is coherent with our observations on clustering purity. Hence a platform implementing an NLI model could guide a user toward the best section to make a contribution.

Overlapping pairs of proposals (see Table 5.7), similarly to those labeled as contradictory, are likely to contain two proposals placed in different clusters. Indeed, in the RepNum section, 2 of the 3 overlapping pairs are composed of two proposals placed in different clusters. For the RUA section, the community detection algorithm identifies the proposals of 156 of the 289 overlapping pairs (54.0 %) as belonging to two different clusters. Finding all the contradictory pairs of proposals between clusters of entailed proposals can significantly help decision-makers resolve
<table><caption>Table 5.8: Results of the clustering of the largest section of the RepNum and RUA consultations using the Louvain algorithm.</caption><thead><tr><th>CONSULTATION</th><th>NB. CONNECTED COMPONENTS</th><th>NB. OF CLUSTERS</th><th>MODULARITY (%)</th><th>PURITY (%)</th><th>PURITY OF SECTIONS (%)</th></tr></thead><tbody><tr><td>RepNum</td><td>6</td><td>13</td><td>67.0 %</td><td>68.3 %</td><td>31.7 %</td></tr><tr><td>RUA</td><td>1</td><td>19</td><td>8.5 %</td><td>56.5 %</td><td>N/A</td></tr></tbody></table>

all potential conflicts on a citizen collaboration platform.

### 5.5.2 Recommending contributions to citizens during their navigation

Another use case for our models is the recommendation of proposals to citizens to contradict or complement the views of the proposal they are currently reading. Indeed, in the case of large-scale citizen participation, information overload may lead to a higher difficulty for citizens to find different points of view on a given issue. We assess the suitability of our models using the following protocol:

1. We pick 100 random proposals in the RUA dataset. The choice of this specific dataset is motivated by the fact that most citizens have at least some degree of understanding of its topics (e.g., social welfare, taxes), while the RepNum consultation is focused on topics where a large share of the population may need an explanation to understand certain proposals (e.g., data protection, open science). Moreover, we filter our dataset to include only proposals between 30 and 250 tokens for ease of understanding.

2. For each proposal $p_i$, we pick both the proposal with the highest cosine similarity using SBERT $\tilde{p}_i$ and the contradictory proposal with the highest share of contradictory pairs $\bar{p}_i$. Here, we use $\tilde{p}_i$ as the baseline result, with which we compare the result obtained using our model. We choose the proposal with the highest cosine similarity rather than the lowest since, while high similarity does not seem to be correlated with entailment (as stated in § 5.3), the proposal with the lowest similarity is almost always off-topic. In cases where $\tilde{p}_i$ was a copy of $p_i$ (e.g., a user posting their contribution in two different consultation sections), we chose the proposal with the second highest similarity score to avoid easily fixed issues.

3. Using Amazon MTurk<sup>7</sup>, we present to each volunteer one of our 100 randomly picked proposals $p_i$ followed by $\bar{p}_i$. We ask them the following closed-ended question “Does the second proposal ($\bar{p}_i$) bring new points of view or elements of contradiction to the first proposal ($p_i$)?” and if their answer is yes, we ask them to list the said new elements brought. This question aims at identifying proposals that are focused on the same topic as $p_i$ while not being paraphrases.

4. We repeat step 3, replacing the second proposal – i.e., $\bar{p}_i$ – with $\tilde{p}_i$. Steps 3 and 4 are performed in random order to prevent risks of order-related biases. At the end of the experience, the annotator has been confronted to both pairs $\langle p_i, \bar{p}_i \rangle$ and $\langle p_i, \tilde{p}_i \rangle$.
<table><caption>Table 5.9: Results of the MTurk-based experiment assessing the suitability of our model and cosine similarity (baseline) to recommend new proposals to contributors.</caption><thead><tr><th>TYPE OF PAIRS</th><th>TOTAL NUMBER OF PAIRS</th><th>PAIRS WITH DIFFERENT VIEWS EXPRESSED</th><th>AVG. # OF NEW ELEMENTS IDENTIFIED (ALL PAIRS)</th><th>AVG. # OF NEW ELEMENTS IDENTIFIED (PAIRS WITH DIFFERENT VIEWS EXPRESSED)</th></tr></thead><tbody><tr><td>Contradictory pairs (from our model)</td><td>100</td><td>76</td><td>0.79</td><td>1.118</td></tr><tr><td>Similar pairs (baseline)</td><td>100</td><td>49</td><td>0.56</td><td>1.084</td></tr></tbody></table>

This protocol has been created to replicate the case of a citizen clicking on a proposal $p_i$ and who wants to read other proposals on the same topic – here, $\tilde{p}_i$ and $\bar{p}_i$ – to form their opinion on this topic. We do not include any question about the quality of $\tilde{p}_i$ and $\bar{p}_i$ since it would mainly measure the political stance of the annotators on the different topics presented. To evaluate the quality of $\tilde{p}_i$ and $\bar{p}_i$ independently, a distinct experimental protocol would be necessary. Hence, our evaluation primarily centers on assessing the ability of these two proposals to introduce new perspectives or elements of contradiction to readers of $p_i$. Each experiment is run with three different annotators to improve the reliability of results. We consider a pair of proposals to express different views if at least two annotators have identified differences of views between them. We compare results obtained using our model to those obtained using cosine similarity since this method has already been evaluated in previous papers for textual document recommendation [175]<fnref n="8" /> and other use cases such as documents clustering [13, 73]. A total of 24 different workers did all the assignments. Results of the experiment are available in Table 5.9. The first and most clear result is that proposals obtained using our model are 55.1 % more often labeled by workers as bringing new elements to the initial proposals than proposals obtained using cosine similarity.

This result leads volunteers to identify 41.1 % more differences of views in pairs of contradictory proposals than in similar proposals. This result implies that recommending proposals obtained using our model –rather than cosine similarity– to contributors will often lead them to read more diverse views on the topics of the proposal they were initially reading, therefore helping them build an informed opinion. We, therefore, assert that our model is more suitable than cosine similarity to produce proposal recommendations in citizen participation platforms. The second result is related to the number of differences of views identified by volunteers. Indeed, the average number of differences of views identified in pairs where different views have been detected is 3.1 % higher for pairs built using our model than for pairs built using cosine similarity. Nevertheless, in this case, the number of pairs for cosine similarity becomes too low to be representative. A potential explanation for this similar number of new views identified may be the fact that proposals used in our experiment have been filtered to contain up to 250 tokens, thus filtering out

<footnote n="7">⁷The reliability of MTurk for French-language tasks has been evaluated in [131].</footnote>

<footnote n="8">⁸Here, we prefer cosine similarity to other similarity measures such as BM25 since it tends to be more reliable on texts larger than one sentence.</footnote>
the most complex proposals presenting multiple ideas. However, as for now, while our results highlight a slight tendency for pairs produced using our model to contain more differences of views than pairs produced using cosine similarity, we do not draw any conclusion about the ability of our model to identify proposals bringing quantitatively more new views than the baseline model.

These results allow us to infer that our models can be reliably used in citizen participation platforms to recommend proposals to contributors based on the proposals they are currently reading. Indeed, even though the capability of our model to identify proposals with more new elements than cosine similarity is still uncertain, our results clearly show that proposals recommended using our model are more likely than those recommended using cosine similarity to add new elements of discussion to the initial proposal without being off-topic.

## 5.6 Interview with stakeholders

To assess the relevance of our work and of our use cases for online citizen participation, we conducted an interview with Cap Collectif, the company which has created the participatory platform used in the *RepNum* and *RUA* consultations. This company develops a modular platform to answer the needs of multiple citizen participation use cases (e.g., consultation, participatory budgeting, etc.). Their platform has been used by hundreds of organizations, companies and public institutions, primarily in France, but also across other countries like Canada. Cap Collectif is a stakeholder of online citizen participation, as they represent the administrators of the platform that need to analyse large amounts of textual input from users. Oana Balalau, Rafael Angarita and myself attended this in-person meeting on May 23rd, 2023, with the CEO and the counseling director of the company. The meeting began with a presentation of the works presented in this chapter, followed by a question-and-answer session. Quotes from this interview have been translated from French by myself.

The interviewees have presented the method they currently use to produce the synthesis of an online citizen consultation. First, for each initial proposal submitted by the organizing institution, workers read all the comments posted to produce a more comprehensive list of pro / con arguments and add extra categories to represent more mixed viewpoints, alternative solutions, implementation conditions or points of vigilance. This extensive synthesis of the contributions related to these initial proposals is due to the fact that, according to one of our interviewees, “[their] customers are mainly looking for assessments and arguments related to their proposals”. Then, proposals submitted by citizens in each category of the platform are read and grouped by theme. For each theme, proposals presenting similar ideas are replaced by a short summary of the said ideas. Comments on citizen proposals are dismissed. This tree representation of proposals aims at facilitating the understanding of the different ideas discussed by citizens on the platform.

We have asked our interviewees to quantify the amount of work needed to produce a synthesis of a consultation on a scale similar to the *RepNum* or *RUA* platforms. They gave us the example of the consultation they were currently working on and which had received 1,076 proposals, i.e., a number comparable to the two consultations studied in this paper. For this consultation, they expected the synthesis process to require “two persons working almost full-time for three weeks”. This high
workload induced by fully hand-made synthesis tends to validate our assumption concerning the need for a pre-processing to improve scalability of online citizen participation with regard to the number of contributions. Moreover, at another point in the interview, our interviewees state that they have started implementing NLP for certain tasks. Indeed, to help institutions explore the contributions produced in participatory platforms, they have started to implement sentiment analysis and vocabulary-based contributions clustering. They motivate this decision by stating that “[even though] in general, human work is qualitatively superior [to automated processing], there is a scalability issue so [they] are forced to go in this direction anyway”. We therefore consider that the use of our classifiers to identify idea communities – as proposed in Section 5.5.1 – is a solution to real issues faced by participatory platforms administrators and participatory processes organizers and can be integrated to the synthesis work.

Interviewees consider that the use of our tool for this use case is also relevant in other contexts than citizen consultation. They propose the example of “suggestion box”<fnref n="9" /> platforms in which, at the time of the interview, they were trying to implement proposals clustering with similar objectives to those presented in Section 5.5.1. Moreover, according to our interviewees, “once 50 % of the contributions have been read, sometimes even 20 %, new themes are no longer really created, contributions are simply added to the existing ones”. This leads us to propose the following example protocol leveraging idea communities identified by our classifiers in an online participatory platform to reduce the workload faced by people in charge of the synthesis work. For each cluster identified, we propose to only read an arbitrary percentage of the proposals chosen randomly. Our clustering allows us to reduce the risk to lose certain views because of the random sampling. A possibility to further reduce this risk would be to lower or increase the percentage of proposals read on a specific cluster based on the purity of the first proposals read in the said cluster. Then, the synthesis is written based on the views identified in the sample of proposals read. This protocol could be helpful to improve the scalability of participatory platforms with regard to the synthesis process.

However, we note that the interviewees also pointed out that our methodology to produce classifiers does not use extensively the categorization in sections of contributions proposed by their platforms. We acknowledge this remark since the categorizations proposed in both platforms studied contain valuable information concerning the content of proposals in each category. However, we consider that further use of the categorization in the training of our models would make it vulnerable to the risk that authors of proposals would classify them in an inappropriate category [91] such as the one placed at the top of the user interface [1], therefore reducing the reliability of our classifiers. This claim is supported by the purity metric computed for a section of RepNum based on its subcategories in Table 5.8. Nevertheless, categories defined in a platform can be used to narrow the scope of proposals processed for a specific use case, as proposed in the implementation of our first use case (Section 5.5.1).

Finally, we note that most of the discussion focused on the use case presented in

<footnote n="9">⁹A "suggestion box" participatory platform aims at gathering new ideas or vague projects from citizens. Certain ideas will then further developed and implemented by the organizing institution. These platforms typically implement very few features for citizens or institutions to give feedback to submitted ideas.</footnote>
Section 5.5.1. We suppose that this focus stems from the fact that our interviewees are actively seeking to address the problem tackled by this first implementation, unlike the implementation proposed in the second use case. Indeed, the platform built by our interviewees currently does not implement NLP-based features for contributors. They consider that “the role of the user interface is to make it easier for citizens to find the right category for their proposals, and not necessarily to [help them] interact with others’ proposals” since many citizens engage with the platform only once<fnref n="10" /> to submit their position on a given debate. Based on this remark, we consider that our tool could be used during the writing process of a proposal to highlight proposals entailed with the text written, for example to reduce the number of duplicate proposals. Additionally, interviewees highlight the risk of manipulating participation with recommendation systems. While we agree that this risk should not be underestimated, we consider that our approach successfully addresses it by highlighting contradictory proposals and leveraging open-source models fine-tuned on publicly accessible and transparently listed corpora. This ensures that the fairness of the participatory initiative is not compromised by inadvertently under-representing specific idea communities.

## 5.7 Conclusion

This work presents research focused on the textual content of contributions in citizen participation platforms to help improve the experience of contributors and administrators on such platforms. We propose a methodology for classifying proposals as entailed, contradictory or neutral. We highlight the potential of such tool in two national citizen consultations. Many proposals are entailed or contradictory, hence there is a lot of potentially duplicated content and differences in opinion that should be addressed in priority. Moreover, the tool presented is shown to be capable of identifying idea communities in a citizen consultation and recommending contributions on the same topic as a given proposal while at the same time bringing new elements of discussion. We consider our work to be a significant step towards the creation of more viable large scale citizen participation platforms. We also address the problem of performing NLI on large texts by proposing alternative strategies explicitly designed for texts containing more than one sentence. Finally, there are many exciting future directions, for example, testing the NLI functionality during an ongoing citizen consultation or providing a taxonomy of contradiction types. Implementing automated extraction of characteristics in text contributions is a key element to improve online citizen collaboration, especially at large scale, both to reduce the information overload experienced by contributors and to help platform maintainers summarize the contributions.

**Limitations.** A potential limitation of our work is that our analysis is done on French text. It would be interesting to test the validity of our method for other languages. Cross-lingual studies help to identify whether our conclusions could be generalized to other languages. Unfortunately, we could not find a comparable source of citizen participation data regarding the dataset size. Consequently, the only method we identify to produce cross-lingual analysis on citizen participation

<footnote n="10">¹⁰25 % for the *RepNum* consultation according to this satisfaction survey [44].</footnote>
contexts would be to translate the data sources we use in this study into other
languages.
# Chapter 6

## General conclusions

### Contents

<table><tbody><tr><td><strong>6.1 Contributions</strong></td><td><strong>109</strong></td></tr><tr><td>6.1.1 Analysis of a participatory platform's design</td><td>109</td></tr><tr><td>6.1.2 Building a knowledge graph for citizen consultations</td><td>110</td></tr><tr><td>6.1.3 Identification of the different objectives and features implemented in participatory platforms</td><td>110</td></tr><tr><td>6.1.4 NLI for large-scale participation</td><td>110</td></tr><tr><td><strong>6.2 Future research directions</strong></td><td><strong>111</strong></td></tr><tr><td>6.2.1 Knowledge graphs</td><td>111</td></tr><tr><td>6.2.2 Qualitative analysis of interviews with stakeholders of citizen participation</td><td>111</td></tr><tr><td>6.2.3 Interaction of citizen participation platforms with other participation channels</td><td>111</td></tr><tr><td>6.2.4 Fine-grained identification of contradictions in proposals</td><td>112</td></tr><tr><td>6.2.5 Formalized workflow for online citizen participation</td><td>112</td></tr></tbody></table>

Given the rise in popularity of online citizen participation in the recent years, solving issues impacting participatory platforms has become crucial to improve inclusion of citizens in public decision making. In this thesis, we identify multiple issues faced by contemporary online participatory platforms. Based on the issues we identified and on previous works, we propose different solutions to improve online participation. These solutions are focused on providing better support for citizens collaboration and on handling citizen participation at large scale. In the following, we summarize the contributions produced during this thesis and propose potential directions for future research.

### 6.1 Contributions

#### 6.1.1 Analysis of a participatory platform's design

In Chapter 3, we identified biases in the design of the *République Numérique* participatory platform based on the interactions between citizens and contributions. We
first identified a centralization of replies around certain proposals. We found that this centralization is unrelated with the authors of these proposals and is more certainly due to the content of proposals themselves. Then, we describe the behavior of the most active users, identifying both a low degree of involvement of active voters in other types of contributions and the coexistence of both versatile and specialized active contributors. Finally, we highlight the biases in participation induced by the user interface design of the platform, and especially the positioning of the different proposals. These different issues identified in the *République Numérique* online consultation allow us to formulate five recommendations for improving participatory platforms.

## 6.1.2 Building a knowledge graph for citizen consultations

In Section 3.6 of this thesis, we propose a methodology for building a knowledge graph for online citizen participation. We implement this methodology using the contributions produced in the *République Numérique* platform. This knowledge graph-based representation can be used to allow interoperability between different participatory platforms and to enrich the contributions published by citizens using external resources. In the case of the *République Numérique* platform, we link proposals with different topics and with resources from the DBpedia knowledge base to improve understanding and facilitate the retrieval of proposals mentioning specific entities.

## 6.1.3 Identification of the different objectives and features implemented in participatory platforms

In Chapter 4, we propose a method to describe citizen participation platforms based on the objectives pursued by platforms maintainers and the features for interaction implemented. This description framework is based on a series of semi-structured interviews with different stakeholders of citizen participation in France and Spain. Based on the descriptions proposed and issues identified for the different categories, we formulate recommendations to improve citizen collaboration in participatory platforms and to reduce information overload faced by large-scale participatory platforms users and administrators.

## 6.1.4 NLI for large-scale participation

In Chapter 5, we propose to facilitate online participation at large scale by identifying entailed and contradictory pairs of proposals in participatory platforms. We propose a methodology to produce classifiers for NLI tasks on citizen contributions. Then, we assess our methodology by implementing it on two different participatory platforms in French language. Finally, the classifiers produced are used to evaluate the relevance of our methodology for two different use cases: identification of clusters of entailed proposals and recommendation of proposals bringing new views or elements of contradiction to the readers of an initial proposal. The first use case is assessed using graph-related metrics. The second use case is assessed via an empirical evaluation conducted with independent testers.
# 6.2 Future research directions

## 6.2.1 Knowledge graphs

Online public consultations enable citizens to create proposals and participate in public debates and decision-making. However, given the large scale of these consultations – which cover various topics are presented in text format – managing and extracting meaningful insights from them can be a daunting task for both participatory platform administrators and institutions. Citizens may also find it challenging to locate information relevant to the consultation, identify proposals in a specific topic, and gain an overview of the relationship between existing proposals in a given consultation. To address these issues, we proposed a methodology in Section 3.6 for automatically constructing a consultation knowledge graph that can tackle these challenges. In the future, we plan to augment the consultation knowledge graph by incorporating additional data sources not stored as knowledge bases, such as the French National Assembly archive (https://data.assemblee-nationale.fr) to ease the creation of more informed and comprehensive proposals. Moreover, our topic modeling methodology could be improved by using automated labeling of topics [89], and empirically assessing the ability of DBpedia Spotlight to identify the most important words in proposals. We plan to perform this evaluation with a quantitative assessment of the relevance of the words annotated and of their associated annotation performed by independent testers.

## 6.2.2 Qualitative analysis of interviews with stakeholders of citizen participation

While we proposed a study of the different objectives pursued by platforms maintainers and the technical features they implement to achieve these objectives in Chapter 4, this analysis does not provide insights on the multiple objectives pursued and practices performed by citizens in participatory platforms. Previous studies have already tackled the topic of citizens’ motivators to get involved in specific participatory initiatives [74, 116, 146]. However, we did not identify any studies focused on multiple and different participatory platforms. Consequently, future research could focus on conducting a series of semi-directed interviews with citizens involved in different citizen participation platforms to understand their motivations as well as their practices (e.g., do they interact with other citizens’ contributions, do they produce proposals in collaboration with other citizens). These interviews will be the basis of a qualitative analysis focused on the differences and similarities identified between different citizen profiles.

## 6.2.3 Interaction of citizen participation platforms with other participation channels

Online citizen participation can take multiple shapes. In many cases, participatory platforms are not the only place where an institution aims at taking into account citizens’ opinions on certain issues. The most frequent example is probably social media services. However, there is little means for interaction between participatory platforms and external participation channels apart from link sharing. Therefore,
future research could focus on both comparative analysis of different online partici-
pation contexts (e.g., participatory platforms and social media) and implementation
of means for interaction between these different contexts using social media inter-
operability [9, 10].

6.2.4 Fine-grained identification of contradictions in propos-
als

In Chapter 5 of this thesis, pairs of proposals are classified using three different labels: entailed, neutral and contradictory. However, pairs of texts can be in contradiction for multiple reasons (e.g., numeric disagreement, support of opposing views, etc.) and therefore could be treated differently. Therefore, implementing a more nuanced identification of contradictions in NLI models for citizen participation could help stakeholders understand the content of participatory platforms more accurately. However, the datasets used in Chapter 5 would not be relevant as they do not allow citizens to characterize their comments further than the pro / con stances presented. Producing large NLI datasets with more fine-grained labeling of contradictory pairs (e.g., numeric or ideological contradiction) in large texts would be an important first step toward a better characterization of contradictions in NLI. In the specific case of online citizen participation, this would be feasible by giving the possibility to citizens to position their comments in a more nuanced scale than simply “for” or “against” the associated proposal in future consultations.

6.2.5 Formalized workflow for online citizen participation

The contributions presented in this thesis are focused on specific aspects of online
citizen participation and propose solutions to specific issues faced by their stake-
holders. However, we did not address the challenge of incorporating these different
contributions into a large-scale citizen participation platform. Consequently, a step
towards the design of more reliable participatory platforms would be to formalize a
new workflow for large-scale online participation based on our contributions, aiming
at improving means for citizens collaboration and reducing issues already identified
by us and other researchers. This workflow could be exemplified by the implemen-
tation of the associated technical features on an open-source participatory platform
such as Decidim [51] or Consul [47].
Bibliography

[1] William Aboucaya, Rafael Angarita, and Valerie Issarny. "Detecting Obstacles to Collaboration in an Online Participatory Democracy Platform: A Use-case Driven Analysis". In: *44th International Conference on Software Engineering (ICSE'22). Proceedings of the 2nd International Workshop on Equitable Data and Technology*. FairWare '22. Pittsburgh, Pennsylvania: Association for Computing Machinery, May 2022, pp. 25-33. ISBN: 9781450392921. DOI: 10.1145/3524491.3527307. URL: https://hal.inria.fr/hal-03654027.

[2] William Aboucaya, Sonia Guehis, and Rafael Angarita. "Building Online Public Consultation Knowledge Graphs". In: *Text2KG 2023: International Workshop on Knowledge Graph Generation from Text, Co-located with the ESWC 2023*. Ed. by CEUR-WS. Hersonissos (Crete), Greece, May 2023, p. 12. URL: https://inria.hal.science/hal-04076743.

[3] B. Thomas Adler, Luca de Alfaro, Santiago M. Mola-Velasco, Paolo Rosso, and Andrew G. West. "Wikipedia Vandalism Detection: Combining Natural Language, Metadata, and Reputation Features". In: *Computational Linguistics and Intelligent Text Processing*. Ed. by Alexander Gelbukh. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 277-288. ISBN: 978-3-642-19437-5.

[4] Leman Akoglu, Hanghang Tong, and Danai Koutra. "Graph based anomaly detection and description: a survey". In: *Data Mining and Knowledge Discovery* 29.3 (May 2015), pp. 626-688. ISSN: 1573-756X. DOI: 10.1007/s10618-014-0365-y. URL: https://doi.org/10.1007/s10618-014-0365-y.

[5] Lucie Alexis, Christine Chevret-Castellani, and Sarah Labelle. "Enjeux sociopolitiques d'un nouvel ordre de discours autour des dispositifs de participation en lien avec la loi pour une «République numérique»". In: *Cahiers du CRICIS* [Méta]morphoses numériques de la culture et des médias : quelques pistes de réflexion.Special Issue (2017), pp. 56-67.

[6] Davide Alocci, Julien Mariethoz, Oliver Horlacher, Jerven T. Bolleman, Matthew P. Campbell, and Frederique Lisacek. "Property Graph vs RDF Triple Store: A Comparison on Glycan Substructure Search". en. In: *PLOS ONE* 10.12 (2015). Publisher: Public Library of Science, e0144578. ISSN: 1932-6203. DOI: 10.1371/journal.pone.0144578. URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0144578 (visited on 01/30/2023).

[7] Isaac Alpizar-Chacon and Sergey Sosnovsky. "What's in an index: Extracting domain-specific knowledge graphs from textbooks". In: *Proceedings of the ACM Web Conference 2022*. 2022, pp. 966-976.
[8] Luc André, Stéphane Martin, Gérald Oster, and Claudia-Lavinia Ignat. "Supporting Adaptable Granularity of Changes for Massive-scale Collaborative Editing". In: *CollaborateCom - 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing - 2013*. Austin, United States, Oct. 2013. DOI: 10.4108/icst.collaboratecom.2013.254123. URL: https://hal.inria.fr/hal-00903813.

[9] Rafael Angarita, Nikolaos Georgantas, and Valérie Issarny. "Social Middleware for Civic Engagement". In: *2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)*. 2019, pp. 1777-1786. DOI: 10.1109/ICDCS.2019.00176.

[10] Rafael Angarita, Bruno Lefèvre, Shohreh Ahvar, Ehsan Ahvar, Nikolaos Georgantas, and Valérie Issarny. "Universal Social Network Bus: Toward the Federation of Heterogeneous Online Social Network Services". In: *ACM Trans. Internet Technol.* 19.3 (Oct. 2019). ISSN: 1533-5399. DOI: 10.1145/3323333. URL: https://doi.org/10.1145/3323333.

[11] Renzo Angles. "The Property Graph Database Model." In: *AMW*. 2018.

[12] Pablo Aragón, Andreas Kaltenbrunner, Antonio Calleja-López, Andrés Pereira, Arnau Monterde, Xabier E. Barandiaran, and Vicenç Gómez. "Deliberative Platform Design: The Case Study of the Online Discussions in Decidim Barcelona". In: *Social Informatics*. Ed. by Giovanni Luca Ciampaglia, Afra Mashhadi, and Taha Yasseri. Cham: Springer International Publishing, 2017, pp. 277-287. ISBN: 978-3-319-67256-4.

[13] Miguel Arana-Catania, Felix-Anselm Van Lier, Rob Procter, Nataliya Tkachenko, Yulan He, Arkaitz Zubiaga, and Maria Liakata. "Citizen Participation and Machine Learning for a Better Democracy". In: *Digit. Gov.: Res. Pract.* 2.3 (July 2021). ISSN: 2691-199X. DOI: 10.1145/3452118. URL: https://doi.org/10.1145/3452118.

[14] Miguel Arana-Catania, Rob Procter, Yulan He, and Maria Liakata. "Evaluation of Abstractive Summarisation Models with Machine Translation in Deliberative Processes". In: *Proceedings of the Third Workshop on New Frontiers in Summarization*. 2021, pp. 57-64.

[15] Sherry R. Arnstein. "A Ladder Of Citizen Participation". In: *Journal of the American Institute of Planners* 35.4 (1969), pp. 216-224. DOI: 10.1080/01944366908977225. eprint: https://doi.org/10.1080/01944366908977225. URL: https://doi.org/10.1080/01944366908977225.

[16] Dionysis Athanasopoulos. "Digital Ecclesia: Towards an Online Direct-Democracy Framework". In: *Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Society*. ICSE-SEIS '18. Gothenburg, Sweden: Association for Computing Machinery, 2018, pp. 91-94. ISBN: 9781450356619. DOI: 10.1145/3183428.3183432. URL: https://doi.org/10.1145/3183428.3183432.

[17] ATILF, INIST, and LIPN. *PERCEO : un Projet d'Etiqueteur Robuste pour l'Écrit et pour l'Oral*. ORTOLANG (Open Resources and TOols for LAN-Guage) -www.ortolang.fr. 2012. URL: https://hdl.handle.net/11403/perceo/v1.
[18] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. "DBpedia: A Nucleus for a Web of Open Data". In: *The Semantic Web*. Ed. by Karl Aberer, Key-Sun Choi, Natasha Noy, Dean Allemang, Kyung-Il Lee, Lyndon Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Mauroux. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 722–735. ISBN: 978-3-540-76298-0.

[19] Sergio Bachiller, Lara Quijano-Sánchez, and Iván Cantador. "A Flexible and Lightweight Interactive Data Mining Tool to Visualize and Analyze Digital Citizen Participation Content". In: *Proceedings of the 36th Annual ACM Symposium on Applied Computing*. SAC '21. Virtual Event, Republic of Korea: Association for Computing Machinery, 2021, pp. 413–416. ISBN: 9781450381048. DOI: 10.1145/3412841.3442081. URL: https://doi.org/10.1145/3412841.3442081.

[20] Romain Badouard. "Combining Inclusion with Impact on the Decision? The Commission's Online Consultation on the European Citizens' Initiative". In: *Is Europe listening to us?: Successes and failures of EU citizen consultations*. Ed. by Raphael Kies and Patrizia Nanz. Routledge, 2016, pp. 221–242. ISBN: 978-1-3155-8985-5. DOI: 10.4324/9781315589855.

[21] João Barroca, Abhishek Shivkumar, Beatriz Quintino Ferreira, Evgeny Sherkhonov, and João Faria. "Enriching a fashion knowledge graph from product textual descriptions". In: *arXiv preprint arXiv:2206.01087* (2022).

[22] H.S. Baum. "Citizen Participation". In: *International Encyclopedia of the Social & Behavioral Sciences*. Ed. by Neil J. Smelser and Paul B. Baltes. Oxford: Pergamon, 2001, pp. 1840–1846. ISBN: 978-0-08-043076-8. DOI: https://doi.org/10.1016/B0-08-043076-7/04439-9. URL: http://www.sciencedirect.com/science/article/pii/B0080430767044399.

[23] Ulrich Beck. *La Société du Risque: Sur La voie d'une autre modernité*. Aubier, 2001. ISBN: 2081218887.

[24] Lamjed Ben Jabeur, Lynda Tamine, and Mohand Boughanem. "Active Microbloggers: Identifying Influencers, Leaders and Discussers in Microblogging Networks". In: *String Processing and Information Retrieval*. Ed. by Liliana Calderón-Benavides, Cristina González-Caro, Edgar Chávez, and Nivio Ziviani. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 111–117. ISBN: 978-3-642-34109-0. DOI: 10.1007/978-3-642-34109-0_12. URL: https://doi.org/10.1007/978-3-642-34109-0_12.

[25] Martin Biéri, Régis Chatellier, Antoine Courmont, Déborah Zribi, Hajar El Aoufir, Estelle Hary, Émile Gabrié, Ahlam Ammi, Pauline Faget, and Marie Leroux. "Making technological black boxes transparent: open source and open data". In: *Civic tech, data and demos*. Vol. 7. Innovation and Foresight Reports. Paris, France: CNIL, Dec. 2019, pp. 23–24. URL: https://linc.cnil.fr/sites/default/files/atoms/files/cnil_ip_report_06_civic_tech_data_demos.pdf.

[26] David M. Blei, Andrew Y. Ng, Michael I. Jordan, and John Lafferty. "Latent dirichlet allocation". In: *Journal of Machine Learning Research* 3.Jan (2003), pp. 993–1022.
[27] Loïc Blondiaux. "La force d'une notion floue". In: *Le Nouvel esprit de la démocratie: Actualité de la Démocratie participative*. La République des Idées. Seuil, 2008, pp. 13-35. ISBN: 2020966751.

[28] Kirsten Boehner and Carl DiSalvo. "Data, Design and Civics: An Exploratory Study of Civic Tech". In: *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems*. CHI '16. San Jose, California, USA: Association for Computing Machinery, 2016, pp. 2970-2981. ISBN: 9781450333627.

[29] Thomas Bonald, Nathan de Lara, Quentin Lutz, and Bertrand Charpentier. "Scikit-network: Graph Analysis in Python". In: *Journal of Machine Learning Research* 21.185 (2020), pp. 1-6. URL: http://jmlr.org/papers/v21/20-412.html.

[30] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. "A large annotated corpus for learning natural language inference". In: *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*. Lisbon, Portugal: Association for Computational Linguistics, Sept. 2015, pp. 632-642. DOI: 10.18653/v1/D15-1075. URL: https://aclanthology.org/D15-1075.

[31] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared K plan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandl dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. "Language Models are Few-Shot Learners". In: (2020). arXiv: 2005.14165 [cs.CL].

[32] Axel Bruns and Jason Wilson. "Citizen Consultation from Above and Below : The Australian Perspective". In: *Politics, Democracy and E-Government: Participation and Service Delivery*. Hershey, PA, USA: IGI Global, 2010, pp. 332-346. ISBN: 9781615209330. DOI: 10.4018/978-1-61520-933-0.ch020. URL: http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-61520-933-0.ch020.

[33] *Cahier des charges de la plateforme citoyenne Participer Strasbourg*. 2019. URL: https://participer.strasbourg.eu/documents/2531722/3042639/Plan+PNU.pdf/740f2c9b-2fdf-df47-f252-f682af6f3397.

[34] Iván Cantador, María E. Cortés-Cediel, and Miriam Fernández. "Exploiting Open Data to analyze discussion and controversy in online citizen participation". In: *Information Processing & Management* 57.5 (2020), p. 102301. ISSN: 0306-4573. DOI: https://doi.org/10.1016/j.ipm.2020.102301. URL: https://www.sciencedirect.com/science/article/pii/S0306457320307962.

[35] Nico Carpentier. "Beyond the Ladder of Participation: An Analytical Toolkit for the Critical Analysis of Participatory Media Processes". In: *Javnost - The Public* 23.1 (2016), pp. 70-88. DOI: 10.1080/13183222.2016.1149760. eprint: https://doi.org/10.1080/13183222.2016.1149760. URL: https://doi.org/10.1080/13183222.2016.1149760.
[36] Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Spe-
cia. "SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and
Crosslingual Focused Evaluation". In: *Proceedings of the 11th International
Workshop on Semantic Evaluation (SemEval-2017)*. Vancouver, Canada: As-
sociation for Computational Linguistics, Aug. 2017, pp. 1–14. DOI: 10.18653/
v1/S17-2001. URL: https://aclanthology.org/S17-2001.

[37] Scott Chacon and Ben Straub. *Pro Git*. Apress, 2014. URL: https://github.com/progit/progit2/releases/download/2.1.194/progit.pdf.

[38] Kaiping Chen and Tanja Aitamurto. "Barriers for Crowd's Impact in Crowd-sourced Policymaking: Civic Data Overload and Filter Hierarchy". In: *International Public Management Journal* 22.1 (2019), pp. 99–126. DOI: 10.1080/10967494.2018.1488780. eprint: https://doi.org/10.1080/10967494.2018.1488780. URL: https://doi.org/10.1080/10967494.2018.1488780.

[39] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiri-donov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. *PaLM: Scaling Language Modeling with Pathways*. 2022. arXiv: 2204.02311 [cs.CL].

[40] Mabi Clément. "Les conceptions de la participation citoyenne inscrites au sein d'un dispositif de concertation". In: *Journées doctorales GIS Participation* (2011).

[41] Jacob Cohen. "A Coefficient of Agreement for Nominal Scales". In: *Educa-tional and Psychological Measurement* 20.1 (1960), pp. 37–46. DOI: 10.1177/001316446002000104. eprint: https://doi.org/10.1177/001316446002000104. URL: https://doi.org/10.1177/001316446002000104.

[42] Cap Collectif. *Bilan de la consultation nationale sur le projet de loi pour une république numérique*. Cap Collectif. Dec. 2015. URL: https://www.republique-numerique.fr/media/default/0001/02/2377a996950cadd5a093dc9e3a621c0e1f9a19f5.pdf.

[43] Cap Collectif. *Consultation sur le projet de loi pour une République Numérique*. 2015. URL: https://www.republique-numerique.fr/.

[44] Cap Collectif. *Résultats du questionnaire de satisfaction de la consultation nationale sur le projet de loi pour une république numérique*. Cap Collectif. Dec. 2015. URL: https://www.republique-numerique.fr/media/default/0001/02/52587d44cde47eacdba246c5625232454c06e33a.pdf.
[45] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. "XNLI: Evaluating Cross-lingual Sentence Representations". In: *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*. Brussels, Belgium: Association for Computational Linguistics, 2018, pp. 2475-2485. DOI: 10.18653/v1/D18-1269. URL: https://aclanthology.org/D18-1269.

[46] Michael Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Goncalves, Filippo Menczer, and Alessandro Flammini. "Political Polarization on Twitter". In: *Proceedings of the International AAAI Conference on Web and Social Media* 5.1 (July 2011), pp. 89-96. URL: https://ojs.aaai.org/index.php/ICWSM/article/view/14126.

[47] CONSUL. *The CONSUL Project*. URL: https://consulproject.org/.

[48] Robert A. Dahl. *On Democracy*. Yale University Press, 1998. ISBN: 9780300254051. URL: http://www.jstor.org/stable/j.ctv18zhcs4 (visited on 03/06/2023).

[49] Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. "Improving Efficiency and Accuracy in Multilingual Entity Extraction". In: *Proceedings of the 9th International Conference on Semantic Systems*. I-SEMANTICS '13. Graz, Austria: Association for Computing Machinery, 2013, pp. 121-124. ISBN: 9781450319720. DOI: 10.1145/2506182.2506198. URL: https://doi.org/10.1145/2506182.2506198.

[50] Marie-Catherine De Marneffe, Anna N. Rafferty, and Christopher D. Manning. "Finding contradictions in text". In: *Proceedings of ACL-08: HLT*. Columbus, Ohio, USA: Association for Computational Linguistics, 2008, pp. 1039-1047.

[51] Decidim. *Free open-source participatory democracy for cities and organizations*. URL: https://decidim.org/.

[52] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". In: *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*. Minneapolis, Minnesota: Association for Computational Linguistics, June 2019, pp. 4171-4186. DOI: 10.18653/v1/N19-1423. URL: https://aclanthology.org/N19-1423.

[53] Alex J.A. Donkers, Dujuan Yang, and Nico H.G. Baken. "Linked Data for Smart Homes: Comparing RDF and Labeled Property Graphs". English. In: *LDAC 2020 Linked Data in Architecture and Construction*. Ed. by María Poveda-Villalón, Ana Roxin, Kris McGlinn, and Pieter Pauwels. CEUR Workshop Proceedings. 8th Linked Data in Architecture and Construction Workshop, LDAC 2020, LDAC 2020 ; Conference date: 17-06-2020 Through 19-06-2020. CEUR-WS.org, 2020, pp. 23-36. URL: http://linkedbuildingdata.net/ldac2020/.

[54] C. A. Ellis and S. J. Gibbs. " Concurrency Control in Groupware Systems". In: *Proceedings of the 1989 ACM SIGMOD International Conference on Management of Data*. SIGMOD '89. Portland, Oregon, USA: Association for Computing Machinery, 1989, pp. 399-407. ISBN: 0897913175. DOI: 10.1145/67544.66963. URL: https://doi.org/10.1145/67544.66963.
[55] Douglas C Engelbart and William K English. "A research center for augmenting human intellect". In: *Proceedings of the December 9-11, 1968, fall joint computer conference, part I*. 1968, pp. 395-410.

[56] Roberto Falanga. "The national participatory budget in Portugal: Opportunities and challenges for scaling up citizen participation in policymaking". In: *Hope for democracy: 30 years of participatory budgeting worldwide* (2018), pp. 447-466.

[57] Cristina Flesher Fominaya. *Democracy Reloaded: Inside Spain's Political Laboratory from 15-M to Podemos*. Oxford University Press, June 2020. ISBN: 9780190099961. DOI: 10.1093/oso/9780190099961.001.0001. URL: https://doi.org/10.1093/oso/9780190099961.001.0001.

[58] Nancy Fraser. "Rethinking the Public Sphere: A Contribution to the Critique of Actually Existing Democracy". In: *Social Text* 25/26 (1990), pp. 56-80. ISSN: 01642472, 15271951. URL: http://www.jstor.org/stable/466240 (visited on 03/06/2023).

[59] Cornelia Fraune. "Gender matters: Women, renewable energy, and citizen participation in Germany". In: *Energy Research & Social Science* 7 (2015), pp. 55-65. ISSN: 2214-6296. DOI: https://doi.org/10.1016/j.erss.2015.02.005. URL: https://www.sciencedirect.com/science/article/pii/S2214629615000250.

[60] Bairan Fu, Wenming Zhang, Guangneng Hu, Xinyu Dai, Shujian Huang, and Jiajun Chen. "Dual Side Deep Context-Aware Modulation for Social Recommendation". In: *Proceedings of the Web Conference 2021*. WWW '21. Ljubljana, Slovenia: Association for Computing Machinery, 2021, pp. 2524-2534. ISBN: 9781450383127. DOI: 10.1145/3442381.3449940. URL: https://doi.org/10.1145/3442381.3449940.

[61] Li Gan, Bin Gu, Sirkka Jarvenpaa, and Geng Yang. "Habit formation in online communities". In: *ICIs 2009 Proceedings* (2009), p. 189.

[62] Marcel Gauchet. "Crise dans la démocratie". FR. In: *La revue lacanienne* 2.2 (2008), pp. 59-72. DOI: 10.3917/lrl.082.0059. URL: https://doi.org/10.3917/lrl.082.0059.

[63] Erik Glaas, Mattias Hjerpe, Martin Karlson, and Tina-Simone Neset. "Visualization for Citizen Participation: User Perceptions on a Mainstreamed Online Participatory Tool and Its Usefulness for Climate Change Planning". In: *Sustainability* 12.2 (2020). ISSN: 2071-1050. DOI: 10.3390/su12020705. URL: https://www.mdpi.com/2071-1050/12/2/705.

[64] Ashish Goel, Anilesh K. Krishnaswamy, Sukolsak Sakshuwong, and Tanja Aitamurto. "Knapsack Voting for Participatory Budgeting". In: *ACM Trans. Econ. Comput.* 7.2 (July 2019). ISSN: 2167-8375. DOI: 10.1145/3340230. URL: https://doi.org/10.1145/3340230.

[65] Max Goldman, Greg Little, and Robert C Miller. "Real-time collaborative coding in a web IDE". In: *Proceedings of the 24th annual ACM symposium on User interface software and technology*. ACM. 2011, pp. 155-164.
[66] Gérard Grunberg. "Les « gilets jaunes » et la crise de la démocratie représentative". FR. In: *Le Débat* 204.2 (2019), pp. 95-103. DOI: 10.3917/deba.204.0095. URL: https://doi.org/10.3917/deba.204.0095.

[67] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. "Annotation Artifacts in Natural Language Inference Data". In: *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)*. New Orleans, Louisiana: Association for Computational Linguistics, June 2018, pp. 107-112. DOI: 10.18653/v1/N18-2017. URL: https://aclanthology.org/N18-2017.

[68] Sara Hachem, Vivien Mallet, Raphaël Ventura, Animesh Pathak, Valérie Is-sarny, Pierre-Guillaume Raverdy, and Rajiv Bhatia. "Monitoring Noise Pollution Using the Urban Civics Middleware". In: *2015 IEEE First International Conference on Big Data Computing Service and Applications*. 2015, pp. 52-61. DOI: 10.1109/Big的服务. 2015.16.

[69] Lile Hattori and Michele Lanza. "Syde: a tool for collaborative software development". In: *Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 2*. 2010, pp. 235-238.

[70] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia D'amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann. "Knowledge Graphs". In: *ACM Comput. Surv*. 54.4 (July 2021). ISSN: 0360-0300. DOI: 10.1145/3447772. URL: https://doi.org/10.1145/3447772.

[71] J. Holston, V. Issarny, and C. Parra. "Engineering Software Assemblies for Participatory Democracy: The Participatory Budgeting Use Case". In: *2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C)*. Austin, TX, USA: Institute of Electrical and Electronics Engineers, 2016, pp. 573-582.

[72] Lester J Holtzblatt, Laurie E Damianos, and Daniel Weiss. "Factors impeding Wiki use in the enterprise: a case study". In: *CHI'10 Extended Abstracts on Human Factors in Computing Systems*. 2010, pp. 4661-4676.

[73] Anna Huang. "Similarity measures for text document clustering". In: *Proceedings of the sixth new zealand computer science research student conference (NZCSRSC2008), Christchurch, New Zealand*. Vol. 4. 2008, pp. 9-56.

[74] Shih-Wen Huang, Minhyang (Mia) Suh, Benjamin Mako Hill, and Gary Hsieh. "How Activists Are Both Born and Made: An Analysis of Users on Change.Org". In: *Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems*. CHI '15. Seoul, Republic of Korea: Association for Computing Machinery, 2015, pp. 211-220. ISBN: 9781450331456. DOI: 10.1145/2702123.2702559. URL: https://doi.org/10.1145/2702123.2702559.
[75] Mario Ianniello, Silvia Iacuzzi, Paolo Fedele, and Luca Brusati. "Obstacles and solutions on the ladder of citizen participation: a systematic review". In: *Public Management Review* 21.1 (2019), pp. 21–46. DOI: 10.1080/14719037.2018.1438499. eprint: https://doi.org/10.1080/14719037.2018.1438499. URL: https://doi.org/10.1080/14719037.2018.1438499.

[76] Abdessamad Imine, Michaël Rusinowitch, Gérald Oster, and Pascal Molli. "Formal design and verification of operational transformation algorithms for copies convergence". In: *Theoretical Computer Science* 351.2 (2006). Alge-braic Methodology and Software Technology, pp. 167–183. ISSN: 0304-3975. DOI: https://doi.org/10.1016/j.tcs.2005.09.066. URL: http://www.sciencedirect.com/science/article/pii/S030439750500616X.

[77] Insee. *Age pyramid in France (French)*. Insee. Jan. 2020. URL: https://www.insee.fr/fr/statistiques/2381472.

[78] Insee. *Socio-professional category by gender and age in France (French)*. In-see. Feb. 2020. URL: https://www.insee.fr/fr/statistiques/2489546.

[79] Hamed Jelodar, Yongli Wang, Chi Yuan, Xia Feng, Xiahui Jiang, Yanchao Li, and Liang Zhao. "Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey". In: *Multimedia Tools and Applications* 78 (2019), pp. 15169–15211.

[80] Robert Johansen. *Groupware: Computer support for business teams*. The Free Press, 1988.

[81] Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. "Ticket Tagger: Machine Learning Driven Issue Classification". In: *2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)*. 2019, pp. 406–409. DOI: 10.1109/ICSME.2019.00070.

[82] Maged N. Kamel Boulos, Bernd Resch, David N. Crowley, John G. Breslin, Gunho Sohn, Russ Burtner, William A. Pike, Eduardo Jezierski, and Kuo-Yu Slayer Chuang. "Crowdsourcing, citizen sensing and sensor web technologies for public and environmental health surveillance and crisis management: trends, OGC standards and application examples". In: *International Journal of Health Geographics* 10.1 (Dec. 2011), p. 67. ISSN: 1476-072X. DOI: 10.1186/1476-072X-10-67. URL: https://doi.org/10.1186/1476-072X-10-67.

[83] Martin Karlsson. "Understanding Divergent Patterns of Political Discussion in Online Forums—Evidence from the European Citizens’ Consultations". In: *Journal of Information Technology & Politics* 9.1 (2012), pp. 64–81. DOI: 10.1080/19331681.2012.635965. eprint: https://doi.org/10.1080/19331681.2012.635965. URL: https://doi.org/10.1080/19331681.2012.635965.

[84] Christopher Kelty, Aaron Panofsky, Morgan Currie, Roderic Crooks, Seth Erickson, Patricia Garcia, Michael Wartenbe, and Stacy Wood. "Seven dimensions of contemporary participation disentangled". In: *Journal of the Association for Information Science and Technology* 66.3 (2015), pp. 474–488. DOI: https://doi.org/10.1002/asi.23202. eprint: https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.23202. URL: https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/asi.23202.
[85] J. A. Khan, Y. Xie, L. Liu, and L. Wen. "Analysis of Requirements-Related Arguments in User Forums". In: *2019 IEEE 27th International Requirements Engineering Conference (RE)*. 2019, pp. 63–74. DOI: 10.1109/RE.2019.00018.

[86] Antti Knutas, Victoria Palacin, Giovanni Maccani, and Markus Helfert. "Software Engineering in Civic Tech: A Case Study about Code for Ireland". In: *Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Society*. ICSE-SEIS '19. Montreal, Quebec, Canada: IEEE Press, 2019, pp. 41–50. DOI: 10.1109/ICSE-SEIS.2019.14. URL: https://doi.org/10.1109/ICSE-SEIS.2019.14.

[87] Travis Kriplean, Jonathan Morgan, Deen Freelon, Alan Borning, and Lance Bennett. "Supporting Reflective Public Thought with Considerit". In: *Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work*. CSCW '12. Seattle, Washington, USA: Association for Computing Machinery, 2012, pp. 265–274. ISBN: 9781450310864. DOI: 10.1145/2145204.2145249. URL: https://doi.org/10.1145/2145204.2145249.

[88] J. Richard Landis and Gary G. Koch. "The Measurement of Observer Agreement for Categorical Data". In: *Biometrics* 33.1 (1977), pp. 159–174. ISSN: 0006341X, 15410420. URL: http://www.jstor.org/stable/2529310 (visited on 06/22/2022).

[89] Jey Han Lau, Karl Grieser, David Newman, and Timothy Baldwin. "Automatic Labelling of Topic Models". In: *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*. Portland, Oregon, USA: Association for Computational Linguistics, June 2011, pp. 1536–1545. URL: https://aclanthology.org/P11-1154.

[90] Victoria Laurent. "« Les citoyens auteurs de la loi » : la consultation en ligne République numérique, une étude des médiations à l’œuvre dans le processus de construction de la loi". 2021PA020076. Thèse de doctorat en sciences de l’information et de la communication. Paris 2, France, 2021. URL: http://www.theses.fr/2021PA020076.

[91] Victoria Laurent, Romain Badouard, Clément Mabi, Cécile Méadel, and Guillaume Sire. "Les outils numériques gouvernementaux au service de la loi : Le design de la plateforme République numérique". fr. In: *Terminal. Technologie de l'information, culture & société* 122 (June 2018). ISSN: 0997-5551. DOI: 10.4000/terminal.2071. URL: http://journals.openedition.org/terminal/2071.

[92] John Lawrence and Chris Reed. "Argument Mining: A Survey". In: *Computational Linguistics* 45.4 (Jan. 2020), pp. 765–818. ISSN: 0891-2017. DOI: 10.1162/coli_a_00364. eprint: https://direct.mit.edu/coli/article-pdf/45/4/765/1847520/coli/_a/_00364.pdf. URL: https://doi.org/10.1162/coli%5C_a%5C_00364.

[93] Rémi Lefebvre. "Les Gilets jaunes et les exigences de la représentation politique". In: *La Vie des idées* (Sept. 2019). ISSN: 2105-3030. URL: https://laviedesidees.fr/Les-Gilets-jaunes-et-les-exigences-de-la-representation-politique.html.
[94] Bruno Lefevre and Valerie Issarny. "Matching technological & societal innovations: the social design of a mobile collaborative app for urban noise monitoring". In: *2018 IEEE International Conference on Smart Computing (SMARTCOMP)*. IEEE. Taormina, Italy: Institute of Electrical and Electronics Engineers, 2018, pp. 33–40.

[95] Daniel Lévy-Bruhl, Jean-Claude Desenclos, Sylvie Quelet, and François Bourdillon. "Extension of French vaccination mandates: from the recommendation of the Steering Committee of the Citizen Consultation on Vaccination to the law". In: *Eurosurveillance* 23.17 (2018), pp. 18–00048.

[96] Matteo Lissandrini, Davide Mottin, Katja Hose, and Torben Bach Pedersen. "Knowledge graph exploration systems: are we lost". In: *Proc of the 12th Conference on Innovative Data Systems Research (CIDR)*. Vol. 22. 2022, pp. 10–13.

[97] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. "Roberta: A robustly optimized bert pretraining approach". In: *arXiv preprint arXiv:1907.11692* (2019).

[98] G. Longford and C. Hurrell. "Online Citizen Consultation and Engagement in Canada". In: *Encyclopedia of Digital Government*. Hershey, PA, USA: IGI Global, 2007, pp. 1261–1267. ISBN: 9781591407898. DOI: 10.4018/978-1-59140-789-8.ch191. URL: http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-59140-789-8.ch191.

[99] Martin Loughlin. "The Contemporary Crisis of Constitutional Democracy". In: *Oxford Journal of Legal Studies* 39.2 (Mar. 2019), pp. 435–454. ISSN: 0143-6503. DOI: 10.1093/ojls/gqz005. eprint: https://academic.oup.com/ojls/article-pdf/39/2/435/32374700/gqz005.pdf. URL: https://doi.org/10.1093/ojls/gqz005.

[100] Arthur Lupia. "Shortcuts Versus Encyclopedias: Information and Voting Behavior in California Insurance Reform Elections". In: *American Political Science Review* 88.1 (1994), pp. 63–76. DOI: 10.2307/2944882.

[101] Brendan Luyt. "Debating reliable sources: writing the history of the Vietnam War on Wikipedia". In: *Journal of Documentation* 71.3 (Jan. 2015), pp. 440–455. ISSN: 0022-0418. DOI: 10.1108/JD-11-2013-0147. URL: https://doi.org/10.1108/JD-11-2013-0147.

[102] Christopher D. Manning, Prabhakar Raghavan, and Hinric Schütze. "Evaluation of clustering". In: *Introduction to information retrieval*. Vol. 16. 1. Cambridge University Press, 2009, pp. 356–360.

[103] Frank Manola, Eric Miller, and Brian McBride. "RDF primer". In: *W3C recommendation* 10.1-107 (2004), p. 6.

[104] Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. "A SICK cure for the evaluation of compositional distributional semantic models". In: *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)*. Reykjavik, Iceland: European Language Resources Association (ELRA), May
2014, pp. 216–223. URL: http://www.lrec-conf.org/proceedings/lrec2014/pdf/363_Paper.pdf.

[105] Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. “CamemBERT: a Tasty French Language Model”. In: *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*. Online: Association for Computational Linguistics, July 2020, pp. 7203–7219. DOI: 10.18653/v1/2020.acl-main.645. URL: https://aclanthology.org/2020.acl-main.645.

[106] Tom McCoy, Ellie Pavlick, and Tal Linzen. “Right for the Wrong Reasons: Disagposing Syntactic Heuristics in Natural Language Inference”. In: *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*. Florence, Italy: Association for Computational Linguistics, July 2019, pp. 3428–3448. DOI: 10.18653/v1/P19-1334. URL: https://aclanthology.org/P19-1334.

[107] Heleen L.P. Mees, Caroline J. Uittenbroek, Dries L.T. Hegger, and Peter P.J. Driessen. “From citizen participation to government participation: An exploration of the roles of local governments in community initiatives for climate change adaptation in the Netherlands”. In: *Environmental Policy and Governance* 29.3 (2019), pp. 198–208. DOI: https://doi.org/10.1002/eet.1847. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/eet.1847. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/eet.1847.

[108] Pablo N. Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. “DBpedia Spotlight: Shedding Light on the Web of Documents”. In: *Proceedings of the 7th International Conference on Semantic Systems*. I-Semantics ’11. Graz, Austria: Association for Computing Machinery, 2011, pp. 1–8. ISBN: 9781450306218. DOI: 10.1145/2063518.2063519. URL: https://doi.org/10.1145/2063518.2063519.

[109] Pierre Mendès France. *La République Moderne*. Gallimard, 1962. ISBN: 2070350185.

[110] Laura Michel and Loïc Blondiaux. “L’expertise en débat : jeux d’acteurs et conflits de savoirs autour d’un débat public local dans le Lot”. In: *Les constructions de l’action publique*. Ed. by Fabrizio Cantelli, Steve Jacob, Jean-Louis Genard, and Christian de Visscher. L’Harmattan, Jan. 2006, pp. 181–202.

[111] Amita Misra, Brian Ecker, and Marilyn Walker. “Measuring the Similarity of Sentential Arguments in Dialogue”. In: *Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue*. Los Angeles: Association for Computational Linguistics, Sept. 2016, pp. 276–287. DOI: 10.18653/v1/W16-3636. URL: https://aclanthology.org/W16-3636.

[112] Chantal Mouffe. “Deliberative Democracy or Agonistic Pluralism?” In: *Social Research* 66.3 (1999), pp. 745–758. ISSN: 0037783X. URL: http://www.jstor.org/stable/40971349 (visited on 03/06/2023).

[113] David Nadeau and Satoshi Sekine. “A survey of named entity recognition and classification”. In: *Lingvisticae Investigationes* 30.1 (2007), pp. 3–26.
[114] William A Nagel. *Subversion Version Control: Using the Subversion Version Control System in Development Projects*. Vol. 14. Prentice Hall Professional Technical Reference, 2005.

[115] Laura Naismith, B-H Lee, and Rachel M Pilkington. "Collaborative learning with a wiki: Differences in perceived usefulness in two contexts of use". In: *Journal of Computer Assisted Learning* 27.3 (2011), pp. 228-242.

[116] Mijail Naranjo-Zolotov, Tiago Oliveira, Frederico Cruz-Jesus, José Martins, Ramiro Gonçalves, Frederico Branco, and Nuno Xavier. "Examining social capital and individual motivators to explain the adoption of online citizen participation". In: *Future Generation Computer Systems* 92 (2019), pp. 302-311. ISSN: 0167-739X. DOI: https://doi.org/10.1016/j.future.2018.09.044. URL: https://www.sciencedirect.com/science/article/pii/S0167739X18313396.

[117] Matti Nelimarkka. "A Review of Research on Participation in Democratic Decision-Making Presented at SIGCHI Conferences. Toward an Improved Trading Zone Between Political Science and HCI". In: *Proc. ACM Hum.-Comput. Interact*. 3.CSCW (Nov. 2019). DOI: 10.1145/3359241. URL: https://doi.org/10.1145/3359241.

[118] Matthieu Nicolas, Victorien Elvinger, Gérald Oster, Claudia-Lavinia Ignat, and François Charoy. "MUTE: A Peer-to-Peer Web-based Real-time Collaborative Editor". In: 2017.

[119] Michele Notari. "How to use a Wiki in education: 'Wiki based effective constructive learning'". In: *Proceedings of the 2006 international symposium on Wikis*. 2006, pp. 131-132.

[120] Natasha Noy, Yuqing Gao, Anshu Jain, Anant Narayanan, Alan Patterson, and Jamie Taylor. "Industry-scale Knowledge Graphs: Lessons and Challenges: Five diverse technology companies show how it's done". In: *Queue* 17.2 (2019), pp. 48-75.

[121] *Octoverse 2022: The global developer community*. 2022. URL: https://octoverse.github.com/2022/developer-community (visited on 01/19/2023).

[122] Claire Oger. "Fondements et formes de l'autorité en discours". Habilitation à diriger des recherches. Paris, France: CELSA, Université Paris-Sorbonne, Dec. 2013.

[123] Judith S. Olson, Dakuo Wang, Gary M. Olson, and Jingwen Zhang. "How People Write Together Now: Beginning the Investigation with Advanced Undergraduates in a Project Course". In: *ACM Trans. Comput.-Hum. Interact*. 24.1 (Mar. 2017). ISSN: 1073-0516. DOI: 10.1145/3038919. URL: https://doi.org/10.1145/3038919.

[124] Diler Öner. "Pre-service teachers' experiences with Wiki: challenges of asynchronous collaboration". In: *Proceedings of the 5th international Symposium on Wikis and Open Collaboration*. 2009, pp. 1-2.

[125] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. *The PageRank Citation Ranking: Bringing Order to the Web*. Technical Report 1999-66. Stanford InfoLab, Nov. 1999. URL: http://ilpubs.stanford.edu: 8090/422/.
[126] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. “A De-
composable Attention Model for Natural Language Inference”. In: *Proceedings
of the 2016 Conference on Empirical Methods in Natural Language Process-
ing*. Austin, Texas: Association for Computational Linguistics, Nov. 2016,
pp. 2249–2255. DOI: 10.18653/v1/D16-1244. URL: https://aclanthology.
org/D16-1244.

[127] Kevin Parker and Joseph Chao. “Wiki as a teaching tool”. In: *Interdisciplinary Journal of e-learning and Learning Objects* 3.1 (2007), pp. 57–72.

[128] Cristhian Parra, Christelle Rohaut, Marianne Maeckelbergh, Valerie Issarny, and James Holston. “Expanding the Design Space of ICT for Participatory Budgeting”. In: *Proceedings of the 8th International Conference on Communities and Technologies*. C&T ’17. Troyes, France: Association for Computing Machinery, 2017, pp. 213–221. ISBN: 9781450348546. DOI: 10.1145/3083671.3083702. URL: https://doi.org/10.1145/3083671.3083702.

[129] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. In: *Advances in Neural Information Processing Systems*. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett. Vol. 32. Curran Associates, Inc., 2019. URL: https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.

[130] Carole Pateman. *Participation and Democratic Theory*. Cambridge University Press, 1970. DOI: 10.1017/CBO9780511720444.

[131] Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. “The Language Demographics of Amazon Mechanical Turk”. In: *Transactions of the Association for Computational Linguistics* 2 (Feb. 2014), pp. 79–92. ISSN: 2307-387X. DOI: 10.1162/tacl_a_00167. eprint: https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl/_a/_00167/1566879/tacl/_a/_00167.pdf. URL: https://doi.org/10.1162/tacl%5C_a%5C_00167.

[132] Oren Perez. “Complexity, Information Overload and Online Deliberation”. In: *Journal of Law and Policy for the Information Society* (2009), pp. 10–09.

[133] Vicente Pina, Lourdes Torres, and Sonia Royo. “Comparing online with offline citizen engagement for climate change: Findings from Austria, Germany and Spain”. In: *Government Information Quarterly* 34.1 (2017). Open Innovation in the Public Sector, pp. 26–36. ISSN: 0740-624X. DOI: 10.1016/j.giq.2016.08.009. URL: https://www.sciencedirect.com/science/article/pii/S0740624X16301496.

[134] The Computational Democracy Project. *Polis*. URL: https://compdemocracy.org/Polis/.

[135] Robert D. Putnam. *Bowling Alone: The Collapse and Revival of American Community*. New York: Simon & Schuster, 2000. ISBN: 978-0-7432-0304-3. URL: http://bowlingalone.com/.
[136] Usha Nandini Raghavan, Réka Albert, and Soundar Kumara. "Near linear time algorithm to detect community structures in large-scale networks". In: *Phys. Rev. E* 76 (3 Sept. 2007), p. 036106. DOI: 10.1103/PhysRevE.76.036106. URL: https://link.aps.org/doi/10.1103/PhysRevE.76.036106.

[137] Nils Reimers and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks". In: *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3982-3992. DOI: 10.18653/v1/D19-1410. URL: https://aclanthology.org/D19-1410.

[138] Nils Reimers and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks". In: *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3982-3992. DOI: 10.18653/v1/D19-1410. URL: https://aclanthology.org/D19-1410.

[139] Nils Reimers and Iryna Gurevych. "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation". In: *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, Nov. 2020. URL: https://arxiv.org/abs/2004.09813.

[140] Ioannis Reklos and Albert Meroño-Peñuela. "MediCause: Causal Relation Modelling and Extraction from Medical Publications". In: *CEUR Workshop Proceedings*. Vol. 3184. CEUR-WS. 2022, pp. 1-18.

[141] Diego Rincon-Yanez and Sabrina Senatore. "FAIR Knowledge Graph construction from text, an approach applied to fictional novels". In: (2022).

[142] Ariella Rotramel, Rebecca Parmer, and Rose Oliveira. "Engaging Women's History through Collaborative Archival Wikipedia Projects". In: *The Journal of Interactive Technology and Pedagogy* 14 (Jan. 2019). ISSN: 2166-6245. URL: https://jitp.commons.gc.cuny.edu/engaging-womens-history-through-collaborative-archival-wikipedia-projects.

[143] Jorge Saldivar, Florian Daniel, Luca Cernuzzi, and Fabio Casati. "Online Idea Management for Civic Engagement: A Study on the Benefits of Integration with Social Networking". In: *Trans. Soc. Comput.* 2.1 (Jan. 2019). ISSN: 2469-7818. DOI: 10.1145/3284982. URL: https://doi.org/10.1145/3284982.

[144] Víctor Sampedro and Josep Lobera. "The Spanish 15-M Movement: a consensual dissent?" In: *Journal of Spanish Cultural Studies* 15.1-2 (2014), pp. 61-80. DOI: 10.1080/14636204.2014.938466. eprint: https://doi.org/10.1080/14636204.2014.938466. URL: https://doi.org/10.1080/14636204.2014.938466.
[145] Arthur D. Santana. "Virtuous or Vitriolic: The effect of anonymity on civility in online newspaper reader comment boards". In: *Journalism Practice* 8.1 (2014), pp. 18–33. DOI: 10.1080/17512786.2013.813194. eprint: https://doi.org/10.1080/17512786.2013.813194. URL: https://doi.org/10.1080/17512786.2013.813194.

[146] Lisa Schmidthuber, Dennis Hilgers, Thomas Gegenhuber, and Stefan Etzelstorfer. "The emergence of local open government: Determinants of citizen participation in online service reporting". In: *Government Information Quarterly* 34.3 (2017), pp. 457–469. ISSN: 0740-624X. DOI: https://doi.org/10.1016/j.giq.2017.07.001. URL: https://www.sciencedirect.com/science/article/pii/S0740624X16301502.

[147] Ronald Schroeter. "Engaging New Digital Locals with Interactive Urban Screens to Collaboratively Improve the City". In: *Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work*. CSCW '12. Seattle, Washington, USA: Association for Computing Machinery, 2012, pp. 227–236. ISBN: 9781450310864. DOI: 10.1145/2145204.2145239. URL: https://doi.org/10.1145/2145204.2145239.

[148] Daniel Schulz and Jens Newig. "Assessing Online Consultation in Participatory Governance: Conceptual framework and a case study of a national sustainability-related consultation platform in Germany". In: *Environmental Policy and Governance* 25.1 (2015), pp. 55–69. DOI: 10.1002/eet.1655. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/eet.1655. URL: https://doi.org/10.1002/eet.1655.

[149] Andrés Segura-Tinoco, Óscar G Borzdynski, and Iván Cantador. "An Argument-based Search Framework: Implementation on a Spanish Corpus in the E-Participation Domain". In: *CIRCLE 2022: Proceedings of the 2nd Joint Conference of the Information Retrieval Communities in Europe*. Ed. by CEUR-WS. Samatan, Gers, France, 2022, p. 18.

[150] Andrés Segura-Tinoco and Iván Cantador. "On the extraction and use of arguments in recommender systems: A case study in the e-participation domain". In: *Proceedings of the 3rd edition of knowledge-aware and conversational recommender systems and the 5th edition of recommendation in complex environments workshops*. Ed. by CEUR-WS. Amsterdam, Netherlands, 2021.

[151] Pnina Shachaf and Noriko Hara. "Beyond vandalism: Wikipedia trolls". In: *Journal of Information Science* 36.3 (2010), pp. 357–370. DOI: 10.1177/0165551510365390. eprint: https://doi.org/10.1177/0165551510365390. URL: https://doi.org/10.1177/0165551510365390.

[152] Sonali K Shah. "Motivation, governance, and the viability of hybrid forms in open source software development". In: *Management science* 52.7 (2006), pp. 1000–1014.

[153] Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. "Conflict-Free Replicated Data Types". In: *Stabilization, Safety, and Security of Distributed Systems*. Ed. by Xavier Défago, Franck Petit, and Vincent Villain. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 386–400. ISBN: 978-3-642-24550-3.
[154] Wei Shen, Jianyong Wang, and Jiawei Han. "Entity linking with a knowledge base: Issues, techniques, and solutions". In: *IEEE Transactions on Knowledge and Data Engineering* 27.2 (2014), pp. 443-460.

[155] Boaventura de Sousa Santos. "Participatory Budgeting in Porto Alegre: Toward a Redistributive Democracy". In: *Politics & Society* 26.4 (1998), pp. 461–510. DOI: 10.1177/0032329298026004003. eprint: https://doi.org/10.1177/0032329298026004003. URL: https://doi.org/10.1177/0032329298026004003.

[156] Karen Sparck Jones. "A Statistical interpretation of term specificity and its application in retrieval". In: *Journal of Documentation* 28.1 (Jan. 1972), pp. 11–21. ISSN: 0022-0418. DOI: 10.1108/eb026526. URL: https://doi.org/10.1108/eb026526.

[157] Manu Sporny, Dave Longley, Gregg Kellogg, Markus Lanthaler, and Niklas Lindström. "JSON-LD 1.1". In: *W3C Recommendation, Jul* (2020).

[158] Feng Su and Chris Beaumont. "Evaluating the use of a wiki for collaborative learning". In: *Innovations in Education and Teaching International* 47.4 (2010), pp. 417–431.

[159] R. Sumi, T. Yasseri, A. Rung, A. Kornai, and J. Kertesz. "Edit Wars in Wikipedia". In: *2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing*. 2011, pp. 724–727. DOI: 10.1109/PASSAT/SocialCom.2011.47.

[160] D. Sun and C. Sun. "Context-Based Operational Transformation in Distributed Collaborative Editing Systems". In: *IEEE Transactions on Parallel and Distributed Systems* 20.10 (2009), pp. 1454–1470.

[161] Minna Tamper, Rafael Leal, Laura Sinikallio, Petri Leskinen, Jouni Tuominen, and Eero Hyvönen. "Extracting knowledge from parliamentary debates for studying political culture and language". In: *Proceedings of the 1st International Workshop on Knowledge Graph Generation From Text and the 1st International Workshop on Modular Knowledge (TEXT2KG 2022 and MK2022)*. CEUR-WS. org. 2022.

[162] Nathan TeBlunthuis, Aaron Shaw, and Benjamin Mako Hill. "Density Dependence Without Resource Partitioning: Population Ecology on Change.Org". In: *Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing*. CSCW '17 Companion. Portland, Oregon, USA: Association for Computing Machinery, 2017, pp. 323–326. ISBN: 9781450346887. DOI: 10.1145/3022198.3026358. URL: https://doi.org/10.1145/3022198.3026358.

[163] Simon Tormey. "The Contemporary Crisis of Representative Democracy:" in: *Democratic Theory* 1.2 (2014), pp. 104–112. DOI: 10.3167/dt.2014.010211. URL: https://www.berghahnjournals.com/view/journals/democratic-theory/1/2/dt010211.xml.

[164] Linus Torvalds and Junio Hamano. "Git: Fast version control system". In: *URL* http://git-scm.com (2010).
[165] Bogdan Vasilescu, Vladimir Filkov, and Alexander Serebrenik. "StackOver-Flow and GitHub: Associations between Software Development and Crowd-sourced Knowledge". In: *2013 International Conference on Social Computing*. 2013, pp. 188-195. DOI: 10.1109/SocialCom.2013.35.

[166] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. "SciPy 1.0: fundamental algorithms for scientific computing in Python". In: *Nature Methods* 17.3 (Mar. 2020), pp. 261-272. ISSN: 1548-7105. DOI: 10.1038/s41592-019-0686-2. URL: https://doi.org/10.1038/s41592-019-0686-2.

[167] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems". In: *32 (2019). Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett.* URL: https://proceedings.neurips.cc/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf.

[168] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". In: *Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353-355. DOI: 10.18653/v1/W18-5446. URL: https://aclanthology.org/W18-5446.

[169] Chien-min Wang and David Turner. "Extending the wiki paradigm for use in the classroom". In: *International Conference on Information Technology: Coding and Computing, 2004. Proceedings. ITCC 2004*. Vol. 1. IEEE. 2004, pp. 255-259.

[170] Dakuo Wang, Haodan Tan, and Tun Lu. "Why Users Do Not Want to Write Together When They Are Writing Together: Users' Rationales for Today's Collaborative Writing Practices". In: *Proc. ACM Hum.-Comput. Interact.* 1.CSCW (Dec. 2017). DOI: 10.1145/3134742. URL: https://doi.org/10.1145/3134742.

[171] Jeremy K Ward, Florian Cafiero, Raphael Fretigny, James Colgrove, and Valérie Seror. "France's citizen consultation on vaccination and the challenges of participatory democracy in health". In: *Social Science & Medicine* 220 (2019), pp. 73-80.

[172] S. Weiss, P. Urso, and P. Molli. "Logoot: A Scalable Optimistic Replication Algorithm for Collaborative Editing on P2P Networks". In: *2009 29th IEEE International Conference on Distributed Computing Systems*. 2009, pp. 404-412.

[173] S. Weiss, P. Urso, and P. Molli. "Logoot-Undo: Distributed Collaborative Editing System on P2P Networks". In: *IEEE Transactions on Parallel and Distributed Systems* 21.8 (2010), pp. 1162-1174.
[174] Jianshu Weng, Ee-Peng Lim, Jing Jiang, and Qi He. "TwitterRank: Finding Topic-Sensitive Influential Twitterers". In: *Proceedings of the Third ACM International Conference on Web Search and Data Mining*. WSDM '10. New York, New York, USA: Association for Computing Machinery, 2010, pp. 261–270. ISBN: 9781605588896.

[175] Manjula Wijewickrema, Vivien Petras, and Naomal Dias. "Selecting a text similarity measure for a content-based recommender system". In: *The Electronic Library* 37.3 (Jan. 2019), pp. 506–527. ISSN: 0264-0473. DOI: 10.1108/EL-08-2018-0165. URL: https://doi.org/10.1108/EL-08-2018-0165.

[176] Alexander Wilson and Mark Tewdwr-Jones. "Let's draw and talk about urban change: Deploying digital technology to encourage citizen participation in urban planning". In: *Environment and Planning B: Urban Analytics and City Science* 47.9 (2020), pp. 1588–1604. DOI: 10.4000/terminal.2071.

[177] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. "Transformers: State-of-the-Art Natural Language Processing". In: *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*. Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45. URL: https://www.aclweb.org/anthology/2020.emnlp-demos.6.

[178] Scott Wright. "'Success' and online political participation: The case of Downing Street E-petitions". In: *Information, Communication & Society* 19.6 (2016), pp. 843–857. DOI: 10.1080/1369118X.2015.1080285. eprint: https://doi.org/10.1080/1369118X.2015.1080285. URL: https://doi.org/10.1080/1369118X.2015.1080285.

[179] W. Xing and A. Ghorbani. "Weighted PageRank algorithm". In: *Proceedings. Second Annual Conference on Communication Networks and Services Research, 2004*. Fredericton, NB, Canada: Institute of Electrical and Electronics Engineers, 2004, pp. 305–314.

[180] Taha Yasseri, Robert Sumi, András Rung, András Kornai, and János Kertész. "Dynamics of conflicts in Wikipedia". In: *PloS one* 7.6 (2012), e38869.

[181] Bo Yu and Guoray Cai. "Facilitating Participatory Decision-Making in Local Communities through Map-Based Online Discussion". In: *Proceedings of the Fourth International Conference on Communities and Technologies*. C&T '09. University Park, PA, USA: Association for Computing Machinery, 2009, pp. 215–224. ISBN: 9781605587134. DOI: 10.1145/1556460.1556492. URL: https://doi.org/10.1145/1556460.1556492.

[182] Mijail Naranjo Zolotov, Tiago Oliveira, and Sven Casteleyn. "Continued Intention to Use Online Participatory Budgeting: The Effect of Empowerment and Habit". In: *Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance*. ICEGOV '18. Galway, Ireland: Association for Computing Machinery, 2018, pp. 209–216. ISBN: 9781450354219. DOI: 10.1145/3209415.3209461. URL: https://doi.org/10.1145/3209415.3209461.