# Finding Conflicts of Opinion in Citizen Participation Platforms

William Aboucaya, Oana Balalau, Rafael Angarita, Valérie Issarny

► To cite this version:

William Aboucaya, Oana Balalau, Rafael Angarita, Valérie Issarny. Finding Conflicts of Opinion in Citizen Participation Platforms. 2024 IEEE International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Dec 2024, Bangkok, Thailand. pp.8. hal-04870985

HAL Id: hal-04870985

https://inria.hal.science/hal-04870985v1

Submitted on 7 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
to societal challenges. One success criterion for these platforms is the level of citizen participation. However, decision-makers, such as governments, often lack the resources to handle the high volume and complexity of citizen inputs [6]. In addition, citizens might struggle to assess whether their contributions bring new elements to the conversation and might be overwhelmed by the number of other contributions [18]. Several techniques have been proposed to alleviate this problem. In [2], the authors propose using topic modeling to generate tags for each proposal to enhance keyword-based search, extractive summarization for the comments associated with proposals, and user clustering based on contributions. In [20], the authors create a taxonomy of arguments and propose a rule-based approach using syntactic patterns to extract arguments. Our work focuses on a fine-grained understanding of user input by identifying entailed and contradictory proposals, aiming for their grouping or reconciliation.

**Natural Language Inference:** Natural Language Inference (NLI) is the task of deciding, given two texts - a premise and a hypothesis- if: *i*) the hypothesis can be inferred from the premise (entailment); *ii*) or, if the negation of the hypothesis can be inferred from the premise (contradiction); *iii*) or, if neither of the two situations is true (neutral). Several annotated datasets have been proposed for the task, such as the Cross-Lingual NLI Corpus (XNLI) [8]. Today, the best-performing systems on NLI benchmarks are Transformer-based models, such as BERT [10]; however, textual entailment remains challenging [16].

### III. METHODOLOGY

#### A. Problem statement

In this work, we focus on a particular type of citizen participation platform designed to collect opinions and potentially reach a consensus with citizens on a given topic. The discussion topic is usually structured in several subtopics called sections. We assume that the platform implements the following workflow: *i*) A discussion starts in a section with a proposal. A proposal has a title and a body. *ii*) Optionally, each proposal can receive comments. *iii*) Each comment is labeled with a stance towards the proposal, in support or contradiction.

We define a pair of proposals to be **entailed** if they mutually support each other's claims without having incompatible elements. On the other hand, **contradictory** pairs of proposals support incompatible claims or compatible claims containing incompatible elements (e.g., different numeric values for the budgeting of the same idea). A pair of proposals is labeled **neutral** if it does not fit the criteria for entailed and contradictory labels, i.e., two proposals supporting unrelated claims.

**Problem statement:** *Given a pair of proposals (p₁, p₂) from a citizen participation platform, predict its textual entailment classification, i.e., if the pair is entailed, contradictory, or neutral.*

In the following, we introduce a methodology to efficiently train NLI classifiers for participatory platforms based on data produced by their contributors. In these platforms, contributions typically exceed the length of texts provided in NLI datasets.

#### B. Weakly-labeled data

Training a classifier for NLI in citizen participation contexts raises challenges not supported by existing datasets. In particular, current approaches only compare two sentences(e.g., XNLI [8]), whereas we need to compare full texts. Therefore, using the content produced by contributors of citizen participation platforms, we create weakly-labeled datasets for the NLI task.

To create a relevant NLI dataset, we use pairs constituted of a proposal and one of its comments. We note that this approach has been used before, but we are the first one to study its relevance on long texts from citizen participation platforms. The comments supporting the proposal are considered as entailed pairs, and the comments against the proposal are the contradictory pairs. We select a proposal and a comment associated with another proposal for neutral pairs. This weakly-labeled dataset can then be used to fine-tune either a language understanding model (e.g., BERT [10]) or a model already trained for the NLI task.

#### C. Creating a gold dataset for evaluation

We create a gold dataset for the evaluation of the trained models. It comprises pairs of proposals from the consultation that have been manually annotated using the labels entailed, neutral, and contradictory. These pairs of proposals were generated by selecting random proposals and pairing them with their five closest neighbors based on semantic textual similarity. We compute this similarity using SBERT's multilingual models [19], which allows us to identify proposals with similar topics.

Choosing random pairs of proposals to annotate would give us a highly imbalanced dataset with a large percentage of neutral pairs. The actual composition of our gold datasets (summarized in Table III) shows that the top-k closest neighbors of a proposal contain entailed, neutral, and contradictory proposals, highlighting the relevance of models trained for NLI.

#### D. Automated labeling of long texts

Once we have trained an NLI model using the weakly-labeled data or other sources such as XNLI, we want to test it on our gold dataset of proposals, which are generally large texts. Comments are, on average, shorter than proposals; we will detail this in Section IV. Proposals can be quite long, and contradictions between two proposals can appear only in a few sentences. Our objective is to explore the impact of the amount of context necessary for a classifier to make effective classifications. Consequently, we propose three different **labeling strategies** to automatically label a pair of proposals p₁ and p₂:
<table><caption>Table I: Statistics on contributors and their activity in the RepNum and RUA consultations</caption><thead><tr><th>Consultation</th><th>RepNum</th><th>RUA</th></tr></thead><tbody><tr><td>Users</td><td>21,464</td><td>7,946</td></tr><tr><td>Proposals</td><td>692</td><td>1,760</td></tr><tr><td>Comments</td><td>5,990</td><td>8,984</td></tr><tr><td>Proposals with 1+ comment</td><td>631</td><td>1,072</td></tr></tbody></table>

**Proposal-wise:** The model receives a pair of proposals as input and returns a label as the prediction. If the proposals are longer than the tokens limit imposed by the model, then the last tokens are not considered. This strategy is identical to the one used during training our models on the weakly-labeled datasets.

**Sentence-wise:** We split $p_1$ and $p_2$ into sentences, and we feed the model with every possible pair containing a sentence from $p_1$ and $p_2$. Then, we compute $S_e$ and $S_c$, respectively, the share of pairs labeled as entailed or contradictory among all the pairs of sentences from $p_1$ and $p_2$. If $S_e$ (resp. $S_c$) is higher than a threshold $t_e$ (resp. $t_c$), then the proposals are labeled as entailed (resp. contradictory). Otherwise, neither $S_e$ nor $S_c$ exceed their respective thresholds, and the proposals are labeled as neutral. Finally, if both $S_e$ and $S_c$ are higher than their respective thresholds, the label is chosen according to the highest value between $S_e - t_e$ and $S_c - t_c$.

**Sentence couple-wise:** This strategy is similar to the sentence-wise strategy with the following difference: the pairs used to feed the model are constituted of two consecutive sentences from $p_1$ and $p_2$. The objective is to give the classifier more context than the sentence-wise strategy. The label for the pair of proposals is then computed the same way as for the previous strategy.

For sentence-wise and sentence couple-wise strategies, we compute $t_e$ and $t_c$ for each <model, strategy> pair using the weakly-labeled dataset. First, we compute $S_e$ and $S_c$ for each pair in the test split of the weakly-labeled dataset using either the sentence-wise or sentence couple-wise strategy. Then, we use a dual annealing optimization algorithm to find the ($t_e, t_c$) pair for which the macro average of F1-scores is maximized for our NLI task. The pair of thresholds computed on the weakly-labeled dataset of a given consultation is used for pairs of proposals from the same consultation.

## IV. DATASETS

We apply the methodology described in the previous section to two major online citizen consultations in France: RepNum and RUA. In Table I, we display certain statistics on participation in these consultations.

The RepNum consultation was held from 2015-09-26 to 2015-10-18. This consultation aimed at allowing citizens, institutions, non-profit and for-profit organizations to propose amendments to a bill on how the French state should be organized in the digital era. The main themes of the consultation were the digital transformation of the French administration, open data, online privacy, and

<table><caption>Table II: Distribution of NLI labels for models training</caption><thead><tr><th>Dataset</th><th>Entailed pairs</th><th>Neutral pairs</th><th>Contradictory pairs</th></tr></thead><tbody><tr><td>RepNum<sub>WL</sub></td><td>2,953</td><td>2,995</td><td>3,037</td></tr><tr><td>RUA<sub>WL</sub></td><td>4,954</td><td>4,489</td><td>4,024</td></tr><tr><td>XNLI<sub>FR</sub></td><td>133,399</td><td>133,400</td><td>133,403</td></tr></tbody></table>

access to the internet. The consultation took place before its examination by the French Parliament, and citizens proposed the creation of new articles or the modification of existing ones. The consultation contained 30 initial proposals, the 30 initial articles of the bill, and 662 new proposals were submitted during the consultation. Proposals were published in three different sections depending on their topic. The consultation was held on a platform based on the proprietary solution for citizen participation developed by Cap Collectif. The participatory platform used for this consultation has already been the subject of several analyses concerning the functionalities implemented [1, 13]. Contributions to this consultation and related metadata are available online as open data.<fnref n="4" />

The RUA consultation was held between 2019-09-10 and 2019-11-20. The objective of the consultation was to help the French government gather citizens' opinions on a new solidarity system based on three axes: (i) the objectives the system should pursue, (ii) its administrative and financial functioning, and (iii) the criteria to benefit from this new income. At the beginning of the consultation, 18 proposals had been submitted by the French Ministry of Health and Solidarity, followed by 1,742 additional proposals submitted by citizens and other stakeholders. The three axes of the consultation were split into eight sections regrouping proposals about a specific topic. The consultation took place on a platform based on the same proprietary solution as the RepNum consultation. Contributions to the RUA consultation and related metadata are available online as open data.<fnref n="5" />

### Datasets for model training:

**XNLI<sub>FR</sub>:** The XNLI dataset [8] is a cross-lingual dataset for NLI tasks. It consists of pairs of sentences originally written in English and translated into 15 languages, including French. Each pair is annotated as entailed, contradictory or neutral. The French-speaking section of XNLI contains 400K annotations. This dataset is the largest we have used, but the texts are drastically smaller than standard texts written in online citizen consultations, with the median number of tokens being 23 for premises and 13 for hypotheses. The XNLI<sub>FR</sub> dataset is split as follows: training (≈ 98.1 %), validation (≈ 0.6 %), and test set (≈ 1.3 %).

**RepNum<sub>WL</sub> and RUA<sub>WL</sub>:** The RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets are weakly-labeled datasets we

<footnote n="4">⁴https://www.data.gouv.fr/fr/datasets/consultation-sur-le-projet-de-loi-republique-numerique</footnote>

<footnote n="5">⁵https://www.data.gouv.fr/fr/datasets/consultation-vers-un-revenu-universel-dactivite-1/</footnote>
<table><caption>Table III: Distribution of NLI labels in the gold dataset</caption><thead><tr><td>Consultation</td><td>RepNum</td><td>RUA</td></tr></thead><tbody><tr><td>ENTAINED PAIRS</td><td>22</td><td>94</td></tr><tr><td>NEUTRAL PAIRS</td><td>65</td><td>108</td></tr><tr><td>CONTRADICTORY PAIRS</td><td>20</td><td>37</td></tr><tr><td>κ AGREEMENT</td><td>0.614</td><td>0.621</td></tr></tbody></table>

created from the consultation. These consultations allow contributors to add comments to a given proposal and to label them as “for” or “against” the respective proposal. RepNum<sub>WL</sub> contains 8,985 datapoints and RUA<sub>WL</sub> contains 13,467 datapoints. However, the texts are longer: in RUA<sub>WL</sub>, the proposals have a median number of tokens of 58 and the comments of 42, while in RepNum<sub>WL</sub> the proposals have a median number of tokens of 234 and the comments of 40. Both datasets are split as follows: a training set (80 %), a validation set (10 %), and a test set (10 %).

Table II contains the label distribution for the datasets.

**Datasets for model evaluation:** Even though the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets are large enough to be used for the training of our models, they are not representative of our real case problem where we want to identify entailed, contradictory and neutral pairs of proposals. To assess our models on data more closely related to our problem, we created a gold dataset by manually annotating 239 pairs of proposals from the RUA consultation and 107 pairs from the RepNum consultation, labeled as entailed, neutral or contradictory. See Table III for the distribution of labels.

*N.B.: The pairs in Table III have been produced using the method described in § III-C. The high number of pairs of sentences annotated as neutral allows us to infer that cosine similarity by itself is not a meaningful indicator of the entailment of a pair of proposals.*

Both datasets were annotated independently by two authors of this paper, and agreement among annotators was measured using Cohen's kappa (κ) coefficient [7]. According to Landis and Koch's classification of κ score [12], our agreement is substantial (κ > 0.6) for both datasets and therefore, we consider that comparison with our labels is a valid means to evaluate our models.

The proposals (body and title) in the manually annotated dataset for the RepNum (resp. RUA) consultation have a median length of 179 tokens (resp. 87 tokens).

# V. EVALUATION

## A. Fine-tuned models

CamemBERT [15] is a state-of-the-art language model for French. Moreover, unlike certain other models such as GPT-4 [17], CamemBERT is available under an open-source license, which is considered a crucial feature for the trustworthiness of participatory platforms [4]. In addition, CamemBERT is a small model that can be used with relatively few resources, while recent models, such as GPT-4 or LLAMA are at least one order of magnitude larger.

CamemBERT has been pretrained on masked language prediction and is fine-tuned on the required downstream application. In this work, we propose several fine-tuning scenarios, and we investigate which has the best performance. We use CamemBERT<sub>LARGE</sub>, a neural network with 335M parameters. The training is performed using an Nvidia v100 GPU with 12 GB RAM. We fine-tune the CamemBERT model using the following combinations:

**RepNum:** Fine-tuning on the RepNum<sub>WL</sub> dataset.

**RUA:** Fine-tuning on the RUA<sub>WL</sub> dataset.

**RepNum + RUA:** Fine-tuning on both the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets.

**XNLI:** Fine-tuning on the XNLI<sub>FR</sub> dataset.

**XNLI → RepNum:** Fine-tuning on the XNLI<sub>FR</sub> dataset, then on the RepNum<sub>WL</sub> dataset.

**XNLI → RUA:** Fine-tuning on the XNLI<sub>FR</sub> dataset, then on the RUA<sub>WL</sub> dataset.

**XNLI → RepNum + RUA:** Fine-tuning on the XNLI<sub>FR</sub> dataset, then on both the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets.

The intuition of these strategies is that XNLI can be seen as a first level of learning about entailment, as it is a large dataset with simple sentences. After achieving this first learning step, we aim to improve the performance by continuing the finetuning on specialized datasets, such as our weakly labeled data.

We train our models for 5 epochs, with a learning rate of 1e-5 as suggested in [15], a weight decay of 0.01, and a batch size of 8.

*N.B.: All the trainings have also been performed using a more recent multilingual Longformer [3] model, especially designed for longer inputs. However, since the results were worse than those obtained with CamemBERT, we will not report them to conserve space.*

Once the models have been trained, we apply the strategies described in § III-D to our gold datasets. The results obtained using the 21 different combinations of our fine-tuned models (7 models) and strategies (3 strategies) on our two manually annotated datasets are available in Table IV. We compute the macro F1-scores for each combination, and the precision for the labels entailed (P<sub>E</sub>) and contradictory (P<sub>C</sub>). The first metric informs us of the overall performances of a <model, strategy> pair. The second and third metrics allow us to assess the model's suitability for the large-scale evaluation presented in § VI. As a point of comparison, we have also performed the labeling using **Llama 3.1 8B-Instruct** (Llama for short in the following) as a Zero-Shot classifier. We report the results in § V-B and V-C.

## B. Results for RUA

The model fine-tuned first on XNLI and then on RUA<sub>WL</sub> is the best-performing model on RUA according to the F1 score on our gold dataset. For each labeling strategy, the results obtained using this model are consistently among the best, if not the best. We observe that
<table><caption>Table IV: Results of models evaluation on the RepNum and RUA gold datasets: macro F1-score and precision on entailed label ($P_E$) and contradictory label ($P_C$). For RepNum, results are obtained using the content of the proposals prepended with their titles. For RUA, results are obtained using only the content of the proposals, except starred (*) results where the title of the proposals and name of their sections were included. The macro F1-score of a random classifier is 26.3 on RepNum and 31.0 on RUA.</caption><thead><tr><th rowspan="3">Model trained on...</th><th colspan="3">Results on RepNum</th><th colspan="3">Results on RUA</th></tr><tr><th colspan="6">Labeling strategy</th></tr><tr><th>Proposal</th><th>Sentence</th><th>Sentence couple</th><th>Proposal</th><th>Sentence</th><th>Sentence couple</th></tr></thead><tbody><tr><td>RepNum</td><td>35.6 | 27.8 | 20.0</td><td>26.1 | 23.8 | 25.0</td><td>23.1 | 22.3 | 33.3</td><td>30.8 | 43.1 | 29.0</td><td>31.8 | 44.8 | 20.4</td><td>37.3 | 49.3 | 17.6</td></tr><tr><td>RUA</td><td>42.6 | 25.4 | <b>83.3</b></td><td>30.2 | 24.4 | 25.0</td><td>33.4 | 27.0 | 23.1</td><td>45.7 | 55.0 | 29.6</td><td>43.5 | 53.9 | <b>55.6*</b></td><td>35.6 | 50.0 | 22.2</td></tr><tr><td>RepNum + RUA</td><td><b>47.2</b> | 29.0 | 54.5</td><td>37.0 | 26.0 | 40.0</td><td>45.1 | 30.8 | 42.9</td><td>44.8 | 50.9 | 28.9</td><td>50.0 | 59.5 | 26.2</td><td>36.6 | 55.6 | 20.0</td></tr><tr><td>XNLI</td><td>35.8 | 22.6 | 33.3</td><td>27.9 | 20.4 | 18.2</td><td>38.0 | <b>32.1</b> | 23.8</td><td>45.7 | <b>76.5*</b> | 51.9</td><td>45.1 | 58.3 | 32.9</td><td>37.9 | 76.2 | 27.0</td></tr><tr><td>XNLI → RepNum</td><td>45.7 | 28.6 | 50.0</td><td>31.5 | 22.0 | 60.0</td><td>31.5 | 24.4 | 28.6</td><td>30.6 | 44.4 | 20.5</td><td>39.1 | 46.9 | 23.7</td><td>40.6 | 48.4 | 31.8</td></tr><tr><td>XNLI → RUA</td><td>39.5 | 24.1 | 61.5</td><td>32.6 | 22.5 | 41.7</td><td>35.7 | 25.8 | 26.9</td><td>52.4 | 58.9 | 35.7</td><td><b>54.7</b> | 59.3 | 39.4</td><td>38.8 | 54.3 | 26.7</td></tr><tr><td>XNLI → RepNum + RUA</td><td>45.7 | 30.4 | 41.2</td><td>29.8 | 21.7 | 66.7</td><td>38.5 | 23.5 | 75.0</td><td>48.2 | 55.9 | 32.5</td><td>51.1 | 61.3 | 29.8</td><td>40.6 | 55.0 | 35.7</td></tr></tbody></table>

fine-tuning on RUA<sub>WL</sub> generally improves the classifier's performance, which is expected as this dataset is the most similar to the gold dataset. Using this model and the sentence-wise strategy, we obtain a macro F1-score of 54.7, clearly outperforming Llama, which obtains a F1-score of 38.0, a $P_E$ of 44.8 and a $P_C$ of 35.0. Furthermore, a more precise analysis of misclassifications (Figure 1) shows that this <model, strategy> pair struggles to identify contradictory pairs and, on the opposite, is very effective to identify entailed pairs. However, we cannot identify a labeling strategy consistently producing the best results for every model. Therefore, for each model, all the strategies (see § III-D) should be tested to find the one giving the best results.

### C. Results for RepNum

The best-performing model on RepNum, according to the F1 score on our gold dataset, is fine-tuned using both the RepNum<sub>WL</sub> and RUA<sub>WL</sub> datasets. The best labeling strategy for this dataset is proposal-wise, with a macro F1-score of 47.2. In comparison, Llama obtains worse results with a F1-score of 36.1 and a $P_E$ of 23.2, but obtains results equivalent to our best model for $P_C$, with a score of 83.3. An analysis of misclassifications (Figure 1) shows that neutral and contradictory pairs tend to be falsely identified as entailed pairs, while entailed pairs tend to be correctly identified. Overall, the best results are mostly obtained using the proposal-wise strategy. This is an interesting finding, as the proposals from the RepNum consultation are, on average, longer than those from the RUA consultation (see § IV). We hypothesize that this happens because of how the thresholds on the percentage of entailed/contradictory pairs are computed. We recall that we chose the threshold using the weakly-labeled dataset to avoid biased results by computing it on the gold test set. However, in the weakly-labeled dataset, we have comments much shorter than proposals on RepNum. Hence, we might not be using the best values for the thresholds. This hypothesis is confirmed by the fact that,

<figure><img src="image_2.png" /><figcaption>Figure 1: Confusion matrix for the RUA (left) and RepNum (right): entailed (0), contradiction (1), neutral (2).</figcaption></figure>

simply by changing the value of one of the two thresholds, we did reach significantly higher values for the macro F1-score – e.g., 58 for the XNLI → RepNum + RUA model with the sentence couple-wise strategy, while the value obtained using the thresholds optimized for the weakly labeled test set is only 38.5.

### D. Insights and error analysis

The results of our different <model, strategy> pairs allow us to draw two conclusions concerning NLI for citizen participation. First, as observed on the RepNum gold dataset, a model can learn using data from another consultation: the best results for the metrics computed in this paper – macro F1-score, precision on entailed, and contradictory labels – on the RepNum dataset are obtained using the RepNum + RUA, XNLI, and RUA models, respectively. Therefore, we consider our datasets relevant as an initial corpus for future works on NLI for citizen participation.

Second, when examining the confusion matrices (see Figure 1), we observe that classifiers tend to falsely label neutral pairs as entailed pairs, while the opposite is less common. This issue highlights a low recall on the neutral class for most classifiers. This could be because entailed pairs are selected in the weakly labeled data as pairs of proposals and comments “for”. However, a user might add a comment “for” to express an opinion that is on topic, but not addressing the main claims of the proposal, hence being neutral according to our definition. These
observations are validated by the fact that a model has a higher recall when trained on XNLI - a dataset where annotators gave entailed and neutral pairs-. For example, for the RepNum dataset, the average recall on the neutral label for the <model, strategy> pairs using the XNLI model is 41.2, while it is 30.7 for the pairs using another model. Therefore, new methods to produce better quality weakly-labeled neutral pairs can be beneficial.

False identification of neutral pairs as entailed is frequent in cases where both proposals have common objectives but different contexts. For example, the first row of Table V displays two proposals that aim to mitigate financial risks induced by life-changing situations through temporary income. However, these proposals target different categories of individuals -those responsible for a family and self-employed workers- leading us to label the pair as neutral. This issue is common and is the first lead to improve the reliability of our classifiers in labeling neutral pairs. Another frequent concern involves contradictory pairs of proposals falsely labeled as entailed. The second row of Table V shows two proposals supporting universal income but differing on the minimum age to receive it – 16 years old and at birth, respectively. This difference leads us to label the pair contradictory, while the classifier may probably label them as entailed. These examples are “contextual errors” since both proposals advocate for similar measures within different contexts. Therefore, a better understanding of the context would improve the quality of NLI models on texts larger than one sentence.

## VI. ASSESSING THE INTEGRATION OF AN NLI MODEL INTO A PARTICIPATORY PLATFORM

In this section, we conduct a comprehensive investigation of the potential of our NLI models. First, we first describe how we can adapt our method such that it has a higher precision. The thresholds for the percentage of entailed and contradictory pairs ($S_E$ and $S_C$) for the sentence-wise and sentence couple-wise strategies are computed in § V such that we maximize the macro F1 score. However, one can optimize the thresholds for improved precision: to show users only the pairs of proposals labeled with the highest certainty, that is, the pairs of proposals that are more likely to be contradictory or entailed.

For the RUA consultation, the best classifier for the class entailed is the XNLI model and the proposal-wise strategy. For the RepNum consultation, none of the classifiers have a high enough precision (we aim for a precision higher than 60%) for the entailed label to be reliable for this use case. Therefore, we use a modified version of the classifier composed of the “XNLI → RepNum + RUA” model and the sentence couple-wise strategy. We increase the $t_e$ threshold, which improves the precision ($P_E = 66.7$) while reducing the recall ($R_E = 28.6$) for the entailed label. We follow the same approach to select the best models for finding contradictory proposals. For the RepNum consultation, we use the RUA model and the proposal-

wise strategy. For the RUA consultation, the classifiers need a higher precision for the contradictory label to be used to detect contradictory pairs. Consequently, we slightly increase the $t_c$ threshold, improving the precision ($P_C = 81.8$) and slightly reducing the recall ($R_C = 24.3$) for the contradictory label.

### A. Identifying idea communities in a consultation

A first use case for our models is identifying communities of contributions supporting similar ideas, serving as an initial summary of the ideas evoked in a given participatory platform when reading a high number of contributions is impractical. Finding entailed proposals can help citizens discover like-minded contributors and produce joint contributions. Platform maintainers can identify opinion groups to facilitate the synthesis of a consultation.

We take the RepNum and RUA sections with the largest number of proposals. For RepNum, the section is *La circulation des données et du savoir* (The flow of data and knowledge). For RUA, the section is *À quels publics le revenu universel d'activité doit-il s'adresser?* (Who should the universal activity income be aimed at?). We create a graph $G$ for each section where a vertex is a proposal, and two vertices $v_1, v_2$ are connected if the corresponding proposals are labeled as entailed.

Then, we find communities using the Louvain algorithm. Proposals belonging to the same community are considered to support similar ideas. For RepNum, we apply the Louvain algorithm with $\gamma = 1$, and for RUA with $\gamma = 1.5$. We then annotate the proposals in each community depending on their topic and claims, and we compute the purity [14] of the clusters. For RepNum, we also compute the purity of a baseline clustering produced for comparison using the subsections of the consultation section – this comparison cannot be performed on RUA as the section studied does not have subsections. Results are available in Table VI. Concerning the section from the RepNum consultation, the modularity obtained is high, with well-defined clusters. The section from the RUA consultation has a low modularity score. However, both the RepNum and RUA sections have a significant level of cluster purity. In particular, the purity of our clustering on RepNum is high when compared with the baseline clustering. This suggests that both RUA and RepNum communities have well-defined topics, and claims are widely shared inside the clusters. Moreover, a more detailed clustering analysis indicates that entailed clusters contain proposals from different subsections. This suggests that users might struggle to add contributions in the correct section/subsection, aligning with our observations on clustering purity. Therefore, a platform implementing an NLI model can assist contributors in placing a contribution in the appropriate section.

### B. Recommending contributions to citizens

Another use case for our models is the recommendation of proposals to citizens to contradict or complement the
<table><caption>Table V: Examples of pairs of proposals incorrectly labeled by the “XNLI → RUA” model with sentence-wise strategy. Proposals translated by the authors of this work.</caption><thead><tr><th>First proposal</th><th>Second proposal</th><th>True label</th><th>Predicted label</th></tr></thead><tbody><tr><td>Review the person's situation regularly. A person's Universal Activity Income should be updated regularly to adapt to changes in their family or personal situation. It is also necessary to support people who go back to work without immediately taking away this income. When you get a new job, you have additional expenses and you need time to organize yourself, especially when you have children.</td><td>Allow the vulnerable self-employed entrepreneurs to have a safety net. The Universal Activity Income should encourage people to have a salaried or non-salaried activity. From my experience, it is hard to have a non-salaried activity and to receive aid for the difficult months. It is not normal to discourage people who want to start their own business.</td><td>Neutral</td><td>Entailed</td></tr><tr><td>Everyone is equal so the financial aid should be the same. 800 € paid automatically to the persons living in France and older than 16 on the 1st of each month without any condition, no fraud possible, no paper to provide.</td><td>Universal income for all from birth. Universal income from birth for all citizens legally residing in France; for a child, the payment will be blocked and made available upon reaching legal majority.</td><td>Contradictory</td><td>Entailed</td></tr></tbody></table>

<table><caption>Table VI: Results of the Louvain clustering of the largest section of the RepNum and RUA consultations.</caption><thead><tr><th>Consultation</th><th># connected components</th><th># clusters</th><th>Modularity (%)</th><th>Purity (%)</th><th>Purity of sections (%)</th></tr></thead><tbody><tr><td>RepNum</td><td>6</td><td>13</td><td>67.0 %</td><td>68.3 %</td><td>31.7 %</td></tr><tr><td>RUA</td><td>1</td><td>19</td><td>8.5 %</td><td>56.5 %</td><td>N/A</td></tr></tbody></table>

views of the proposal they are currently reading. We created a protocol to replicate the case of a citizen clicking on a proposal $p_i$ and who wants to read other proposals on the same topic – here, $\tilde{p}_i$ and $\bar{p}_i$ – to form their opinion on this topic. The protocol is the following:

1) We pick 100 random proposals in the RUA dataset. We chose this dataset because most citizens have at least a partial understanding of its topics (social welfare, taxes, etc.), while some topics of the RepNum consultation (e.g., data privacy, open science) are completely unfamiliar to many citizens. We select proposals between 30 and 250 tokens to ensure clarity and readability.

2) For each proposal $p_i$, we pick both the proposal $\tilde{p}_i$ with the highest cosine similarity using SBERT (this method has been proposed for textual document recommendation [21]) and the contradictory proposal $\bar{p}_i$ with the highest share of contradictory pairs. We choose the proposal with the highest cosine similarity rather than the lowest since, while high similarity does not seem to be correlated with entailment (see § IV), the proposal with the lowest similarity is almost always off-topic.

3) Using Amazon MTurk, we present to each annotator one of our 100 randomly picked proposals $p_i$ followed by $\bar{p}_i$. We ask them the question “Does the second proposal ($\bar{p}_i$) bring new points of view or elements of contradiction to the first proposal ($p_i$)?” and if their answer is yes, we ask them to list the said new elements brought. This question aims to identify proposals focused on the same topic as $p_i$ while not being paraphrases.

4) We repeat step 3, replacing the second proposal – i.e., $\bar{p}_i$ – with $\tilde{p}_i$. Steps 3 and 4 are performed randomly to prevent risk of order-related biases.

Our evaluation centers on assessing the ability of these two proposals to introduce new ideas or elements of contradiction to readers of proposal $p_i$. Each pair of proposals

are labeled by three annotators, and we consider a pair of proposals to express different views if at least two annotators have identified differences in views between them. We had 24 annotators, and the results are available in Table VII. Annotators were paid on average 19 $ per hour, for a total of 480 $.

Proposals generated by our model are 55.1 % more frequently labeled by annotators as bringing new elements to the initial proposals than proposals obtained using cosine similarity. This result allows annotators to identify 41.1 % more differences of views in pairs of contradictory proposals than similar ones. This suggests that recommending proposals using our model –rather than cosine similarity– often provides more diverse views on the topics of a proposal, contributing to well-informed opinions. The second result concerns the number of differences of views identified by the annotators. The average number of differences of views (only possible to measure within pairs containing differences for both methods) is 3.1 % higher for pairs built using our model than for pairs built using cosine similarity. An explanation may be that proposals used in our experiment have been filtered to contain up to 250 tokens, thus filtering out the most complex proposals presenting multiple ideas. Our results show that proposals recommended using our model are more likely to add at least one new element of discussion to the initial proposal without being off-topic. Hence, our models can be used in participatory platforms to recommend proposals to contributors based on the proposals they are reading.

## VII. CONCLUSION

This research focuses on the textual contributions of citizen participation platforms. Citizen participation platforms represent powerful tools for democratic societies, hence we believe more research should be dedicated to the effort of improving them. We propose a methodology for
<table><caption>Table VII: Results of the MTurk-based experiment assessing the suitability of our model and of cosine similarity (baseline) to recommend new proposals to contributors.</caption><thead><tr><th>Type of pairs</th><th>Total number of pairs</th><th>Pairs with different views expressed</th><th>Avg. # of new elements identified (All pairs)</th><th>Avg. # of new elements identified (Pairs with different views expressed)</th></tr></thead><tbody><tr><td>Contradictory pairs (from our model)</td><td>100</td><td>76</td><td>0.79</td><td>1.118</td></tr><tr><td>Similar pairs (baseline)</td><td>100</td><td>49</td><td>0.56</td><td>1.084</td></tr></tbody></table>

classifying proposals as entailed, contradictory, or neutral. We highlight the potential of our approach in two national citizen consultations. Our tool can be used to identify idea communities in a citizen consultation, suggest relevant contributions to a proposal, and find controversial proposals. We also address the problem of performing NLI on long texts by proposing alternative strategies tailored for texts with more than one sentence. Moreover, automatically highlighting potential contradictions within a public consultation can help citizens build joint, well-developed, complex projects.

There are many exciting future directions, for example, testing NLI models during an ongoing citizen consultation or providing a taxonomy of contradiction types. Producing large NLI datasets with more fine-grained labeling [9] of contradictory pairs (e.g., numeric or ideological contradiction) in long texts would be an important first step toward better characterizing contradictions in NLI. Fine-grained characterization could also improve the quality of the weakly-labeled data used in this work in addition to the stances already identified by the platform.

REFERENCES

[1] William Aboucaya, Rafael Angarita, and Valerie Issarny. "Detecting Obstacles to Collaboration in an Online Participatory Democracy Platform: A Use-case Driven Analysis". In: FairWare '22. May 2022.

[2] Miguel Arana-Catania et al. "Citizen Participation and Machine Learning for a Better Democracy". In: *Digit. Gov.: Res. Pract.* (July 2021).

[3] Iz Beltagy, Matthew E. Peters, and Arman Cohan. "Longformer: The Long-Document Transformer". In: *arXiv:2004.05150* (2020).

[4] Martin Biéri et al. "Making technological black boxes transparent: open source and open data". In: *Civic tech, data and demos*. Vol. 7. Innovation and Foresight Reports. CNIL, Dec. 2019.

[5] Iván Cantador, María E. Cortés-Cediel, and Miriam Fernández. "Exploiting Open Data to analyze discussion and controversy in online citizen participation". In: *Information Processing & Management* (2020).

[6] Kaiping Chen and Tanja Aitamurto. "Barriers for Crowd's Impact in Crowdsourced Policymaking: Civic Data Overload and Filter Hierarchy". In: *International Public Management Journal* (2019).

[7] Jacob Cohen. "A Coefficient of Agreement for Nominal Scales". In: *Educational and Psychological Measurement* (1960).

[8] Alexis Conneau et al. "XNLI: Evaluating Cross-lingual Sentence Representations". In: *EMNLP 2018*.

[9] Marie-Catherine De Marneffe, Anna N. Rafferty, and Christopher D. Manning. "Finding contradictions in text". In: *Proceedings of ACL-08: HLT*. 2008.

[10] Jacob Devlin et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". In: *NAACL 2019*.

[11] Mario Ianniello et al. "Obstacles and solutions on the ladder of citizen participation: a systematic review". In: *Public Management Review* 21.1 (2019).

[12] J. Richard Landis and Gary G. Koch. "The Measurement of Observer Agreement for Categorical Data". In: *Biometrics* (1977).

[13] Victoria Laurent et al. "Les outils numériques gouvernementaux au service de la loi". In: *Terminal. Technologie de l'information, culture & société* (2018).

[14] Christopher D. Manning, Prabhakar Raghavan, and Hinric Schütze. "Evaluation of clustering". In: *Introduction to information retrieval*. Cambridge University Press, 2009.

[15] Louis Martin et al. "CamemBERT: a Tasty French Language Model". In: *ACL 2020*.

[16] Tom McCoy, Ellie Pavlick, and Tal Linzen. "Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference". In: *ACL 2019*.

[17] OpenAI et al. *GPT-4 Technical Report*. 2024. arXiv: 2303.08774 [cs.CL].

[18] Oren Perez. "Complexity, Information Overload and Online Deliberation". In: *Journal of Law and Policy for the Information Society* (2009), pp. 10–09.

[19] Nils Reimers and Iryna Gurevych. "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation". In: *EMNLP 2020*.

[20] Andrés Segura-Tinoco and Iván Cantador. "On the extraction and use of arguments in recommender systems: A case study in the e-participation domain". In: 2021.

[21] Manjula Wijewickrema, Vivien Petras, and Naomal Dias. "Selecting a text similarity measure for a content-based recommender system". In: *The Electronic Library* (2019).