# e-Consultation Platforms: Generating or Just Recycling Ideas?

Efthimios Tambouris, Anastasia Migotzidou, Konstantinos Tarabanis

## ► To cite this version:

Efthimios Tambouris, Anastasia Migotzidou, Konstantinos Tarabanis. e-Consultation Platforms: Generating or Just Recycling Ideas?. 7th International Conference on Electronic Participation (ePart), Aug 2015, Thessaloniki, Greece. pp.41-52, 10.1007/978-3-319-22500-5_4. hal-01587634

HAL Id: hal-01587634

https://inria.hal.science/hal-01587634

Submitted on 14 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
e-Consultation Platforms: Generating or just Recycling
Ideas?

Efthimios Tambouris, Anastasia Migotzidou and Konstantinos Tarabanis

University of Macedonia, Thessaloniki, Greece

tambouris@uom.gr

**Abstract:** A number of governments worldwide employ web-based e-consultation platforms to enable stakeholders commenting on draft legislation. Stakeholders' input includes arguing in favour or against the proposed legislation as well as proposing alternative ideas. In this paper, we empirically investigate the relationship between the volume of contributions in these platforms and the amount of new ideas that are generated. This enables us to determine whether participants in such platforms keep generating new ideas or just recycle a finite number of ideas. We capitalised on argumentation models to code and analyse a large number of draft law consultations published in *opengov.gr*, the official e-consultation platform for draft legislation in Greece. Our results suggest that as the number of posts grows, the number of new ideas continues to increase. The results of this study improve our understanding of the dynamics of these consultations and enable us to design better platforms.

**Keywords:** Online consultation, e-forum, e-rulemaking.

# 1 Introduction

During the last years, the Internet has been extensively used for engaging large numbers of citizens in policy making. Relevant research and practice is heavily influenced by deliberative democracy theory, which is based on the assumption that “public, plural discussions offer a superior form of collective decision making” [1]. Consequently, preferences should not just be aggregated (as in voting) but should be rather “revised in the light of a preceding debate” [2]. Despite its promises however, scholars soon concluded that putting the theory in practice is a challenging task [1][3].

A prominent area of relevant research and practice involves the use of Web-based platforms to support online deliberation on policy material such as draft laws and regulations, which is provided by relevant agencies. These platforms enable users to argue in favour or against the proposed draft legislation as well as to propose new ideas for the relevant policy topics. This area is sometimes termed *open consultation*, while in the USA it is closely associated with *e-rulemaking*, which refers to the use of digital technologies in the development and implementation of regulations [4]. For the
purposes of this study, we will restrict the use of the terms *open consultation platform* to refer to an *e-consultation platform dedicated to commenting on draft legislation*.

Research in this area has followed various areas and has provided some interesting findings. Early research revealed that various non-technological barriers, including deep motivational, cognitive, and knowledge-based chasms stand in the way of citizen participation in the regulatory process [5]. Other research suggests that the possibility of deliberation is affected by political choices made both about the format and operation of the online discussion [2]. More recent research revealed that online participation mechanisms require careful design considerations as well as a cultural change on behalf of public administrators [6]. The differences between those using online forums with those of online discussions in a forum designed according to deliberative principles have been also investigated [3]. A comprehensive meta-analysis of working with the pioneering “Regulation Room” provided significant insight in a number of areas ranging from identifying the different types of potential participants to presenting relevant barriers and how they can be overcome [7]. From a technological point of view, various platforms have been investigated. Regulations.gov, the official USA website for e-rulemaking, has been heavily investigated (e.g. [4]) however other platforms have been also studied (e.g. [5][8]).

In our work, we are interested in examining the characteristics of discussions that take place in such platforms. More specifically, we aim to investigate the relationship between the number of contributors and the new ideas that are contributed to the discussion. This can provide a better understanding of the social capital that is accumulated in such platforms as the level of participation increases. It can also facilitate designing better platforms to support the process.

Therefore, the study objective is to experimentally investigate the *relationship between the volume of contributions and the new ideas that are contributed in e-consultations on draft legislation*. For this purpose, data from the Hellenic official online consultation platform for draft legislation, namely *opengov.gr*, were obtained, coded and analysed.

The rest of this paper is organised as follows. In Section 2 we outline the motivation of this study along with background material. In Sections 3 and 4 we present the method of work and scope of our study respectively. In Section 5 we provide the employed data selection method. In Section 6 the main results of our study are presented while in Section 7 the main conclusions as well as study limitations and future work are discussed.

## 2 Background Material and Motivation

Online deliberation is associated with an emerging body of practices, research and software that purposely foster open, fair, and rational discussions over the Internet [9]. It has been argued that “the online environment erodes physical, psychological, and social barriers” [6]. On the other hand, online debates are often characterised by repetitive contributions, digressions, argumentative fallacies, rhetorical flourishes, manipulative framing and personal attacks that result in a high noise-to-signal ratio and con-
fusion rather than clarity [10]. Furthermore, the levels of engagements do not seem to increase while online deliberation results can be superficial and polarised particularly when using inappropriate tools [6]. A recent literature review has accumulated relevant challenges [11].

ICT tools used to support online deliberation range from general-purpose e-forums to Argument Visualisation (AV) tools and to tools developed exclusively for supporting online deliberations. We now briefly outline e-forums and AV tools.

An e-forum is structured around discussion topics (termed *threads*). Topics are pre-defined or can be created by users. In each topic, users can hold online conversations in the form of posted messages (aka *posts*). Replies to posts are often also possible. These tools however run the risk to drown under their own success. Indeed, as the number of posts increases, discussions within forums tend to be extremely large and complex resulting in poor deliberation quality (e.g. from repetitive contributions and argumentative fallacies).

AV tools represent discussions in terms of ideas, positions and arguments. Discussions are usually depicted as graphs (often called *argumentation maps*) containing nodes that can be joined by lines (or arrows) to display inferences [12]. Efforts to include argumentation concepts in e-forums have also emerged (e.g. [13]). Proponents of AV tools suggest they provide an unprecedented opportunity to facilitate rational and, therefore, concise online discussions.

<figure><img src="image_2.png" /><figcaption>**Figure 1:** Contributors vs. New Ideas Contributed in a deliberation (from [14])</figcaption></figure>

In a panel discussion of Politics Online Conference, figure 1 was presented to argue that using AV tools, online discussions will be much smaller since as the number of contributors in any discussion increases the number of new ideas contributed levels ([14]). This actually seems to be a logical claim. After all, how many new ideas can potentially exist in any single discussion? Interestingly enough, however, our literature review failed to reveal any experimental evidence to support this claim.

Figure 1 provided the main inspiration behind the work presented in this paper. Figure 1 suggests there is a logarithmic relationship between contributors and new ideas contributed in any discussion. In other words, our study objective is equivalent to
investigating the claim “*as the number of contributors in a discussion increases the number of new ideas levels*”.

However, we restrict our study to a specific category of online deliberations, i.e. open consultation. The difference is significant. In an online deliberation the participants are often asked to brainstorm, i.e. contribute new ideas to a specific topic or problem. It is often the case that there are no pre-existing solutions, thus ideas generation is a major objective. On the other hand, in open consultations, the draft law or regulation already provides a concrete set of ideas and solutions. Here, the participants are primarily asked to argue in favour or against these solutions. The participants can still contribute new ideas but this is not the main objective. Thus, ideas generation is not the primary objective of open consultations. As a result, one can assume that there are less ideas generated in open consultations when compared on online deliberations aiming to facilitate brainstorming.

## 3 Method of work

The method of work that we followed included four steps.

### Step 1: Define study scope

In this step we present important context information. Relevant research has revealed that the context is particularly important when studying or operating open consultations [1]. We therefore start by presenting the main design decisions behind Opengov, the e-forum consultation platform that will be used for analysis purposes.

### Step 2: Map posts to argumentation maps

In this step we present how posts from Opengov were mapped into new ideas. This is an important step as it outlines the method employed for coding the posts.

### Step 3: Select data for analysis

In this step we examine the content of Opengov and select a number of laws and articles for coding and analysis.

### Step 4: Conduct analysis and report results

In this step we present that analysis results. More specifically, we plot relevant scatter graphs and perform trend analysis using the standard functionality of MS Excel for a number of cases. Finally, we discuss the results.

## 4 Study scope

In Greece, online public consultations on draft legislation are conducted before the relevant laws are voted in the Hellenic Parliament. For this purpose, a specific technical infrastructure is employed, which is managed by the Greek Open Government Initiative (Opengov) of the Hellenic Parliament [8]. The corresponding technical infrastructure, also termed Opengov (www.opengov.gr), employs open source applications and tools and contains various applications. However, for simplicity we will use the term Opengov to refer to the specific application for online public consultations on draft legislation. Online public consultations have a life cycle comprising four successive phases.
* Preparation: The Opengov's team, in collaboration with the relevant ministry, sets up the platform and the material of consultation and ensures the overall approval of content by the prime minister's office.

* Public comment: Once approved, the consultation is published and opens to commentary from anyone interested (posts from both registered and anonymous users are allowed). Partners of the relevant ministry are moderating the comments before these are published.

* Edit Conclusions: After the end of the consultation period, the relevant ministry posts in Opengov a message containing initial consultation results. In addition, the ministry takes into consideration the comments and prepares a full report on the consultation, as suggested by relevant legislation.

* Completion: When the law is voted by the Parliament and the report on the consultation results is posted, the consultation is considered as completed.

In this study, *discussion* refers to online consultations on draft legislation. Furthermore, the term *contributions* refers to *posts*, since contribution to Opengov is usually anonymous and hence there is no way to determine who the actual contributor is.

The last point we need to clarify in our study objective is the concept of “*new ideas*”. In order to define new ideas, we examined the literature on argumentation models. This is a common approach followed when analysing online political content. More specifically, we examined the Issue-Based Information System (IBIS) [15], gIBIS [16], the Representation and Maintenance of Process Knowledge (REMAP) approach [17], the Procedural Hierarchy of Issues (PHI) approach [18], Questions, Options, and Criteria (QOC) [19] and Potts and Bruns approach [20] amongst others. Our review revealed there is no “*idea*” argumentation concept. However, in most models the concept *position* is proposed to suggest solutions to *issues*, which represent problems or topics for discussion. Thus, we will use *position* to denote ideas. We note here that for the purpose of our study, it is sufficient that our argumentation maps include two only concepts, namely *issue* and *position*. We acknowledge this is not sufficient for constructing a complete argumentation map as at least *argument* should be also included according to the IBIS model. Actually, in our research we have included various other concepts in the mapping. However, *issue* and *position* are still sufficient for the purposes of this paper.

Concluding, our study objective is now operationalised as follows: *to investigate the relationship between the number of posts and the number of citizens' new positions in an official e-consultation on draft legislation.*

# 5 Mapping framework

In this section, we develop a simple model and a process to enable us coding the content of Opengov platform using concepts from argumentation theory. We start by listing three important design characteristics of the Opengov platform:

* Citizens are allowed to provide posts only per article and not for the entire draft law.
* Citizens can only post on articles and not on other citizens' posts (i.e. replying to a post is not enabled).

* Both registered and anonymous posting is allowed.

In open consultations, issues (which are the actual topics of a debate) refer to the contents of draft laws and/or regulations. Draft laws are divided into articles with each article focusing on one or more topics. Thus, it seems reasonable to suggest that one article is represented by one issue. Our analysis however revealed that in certain cases, one article may contain more than one different issues. In addition, the main body of the draft law contains the proposed Government Positions (Gov Positions) on these issues. As a result, in our argumentation model we need to distinguish between Government Positions and Citizen Positions. Citizens' posts in Opengov platform contain, amongst others, citizen new positions, i.e. new ideas.

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Simple model for transferring Opengov content to an argumentation map</figcaption></figure>

The main entities in Opengov platform and their relationships are depicted in Figure 2 using UML notation. In UML, a diamond represents aggregation (e.g. *has* relationship) while an arrow represents generalization (e.g. *isA* relationship) between classes (i.e. concepts). A class in italics (e.g. *position*) denotes an abstract class, i.e. a class that cannot be instantiated. This actually means that in our coding we may have government positions or citizen positions; not just positions.

In order to investigate our study objective, we developed one simple argumentation map for each article consultation. We acknowledge existing research aiming to automatically derive arguments from text using natural language processing techniques (e.g. [21]). Nevertheless, we opted for a manual approach which is usually more accurate. Thus, the process that we followed in order to code Opengov contents (i.e. draft law text and posts) using our simple argumentation map is:

*Step 1: Analysis of draft law text*

We read each article to determine (a) the relevant problem statement, which was coded as *Issue*, and (b) the positions of the government in the proposed law to address that problem, which were coded as *Government Positions*.
### Step 2: Analysis of posts

We read all posts related to each Article. The aim was to identify new ideas to address the Issue. These were coded as *Citizen (New) Positions*.

## 6 Data selection

Since the start of its operation, a large number of consultations have been conducted in Opengov. Table 1 presents the consultations per ministry and the number of posts per consultation (as of June 2011 when data were collected).

<table><caption>**Table 1:** Draft laws published per ministry (from http://www.opengov.gr/, as in 2011)</caption><thead><tr><td>Ministry of</td><td>Consultations</td><td>Posts</td></tr></thead><tbody><tr><td>Interior, Decentralization & eGovernment</td><td>10</td><td>14.069</td></tr><tr><td>Finance</td><td>9</td><td>18.713</td></tr><tr><td>Foreign Affairs</td><td>0</td><td>0</td></tr><tr><td>National Defence</td><td>0</td><td>0</td></tr><tr><td>Regional Development and Competitiveness</td><td>24</td><td>4.138</td></tr><tr><td>Maritime Affairs and Fisheries Islands</td><td>0</td><td>0</td></tr><tr><td>Environment, Energy and Climate Change</td><td>29</td><td>7.760</td></tr><tr><td>Education, Lifelong Learning and Religion</td><td>19</td><td>9.577</td></tr><tr><td>Infrastructure, Transport and Networks</td><td>14</td><td>2.154</td></tr><tr><td>Labour and Social Insurance</td><td>7</td><td>1.790</td></tr><tr><td>Health and Social Solidarity</td><td>1</td><td>1.146</td></tr><tr><td>Rural Development and Food</td><td>8</td><td>3.053</td></tr><tr><td>Justice, Transparency and Human Rights</td><td>17</td><td>1.956</td></tr><tr><td>Citizen Protection</td><td>5</td><td>1.910</td></tr><tr><td>Culture and Tourism</td><td>2</td><td>580</td></tr><tr><td>OpenGov</td><td>5</td><td>366</td></tr><tr><td>Total</td><td>150</td><td>67.212</td></tr></tbody></table>

The first step was to calculate the number of posts per article per consultation. For this purpose, the following have been considered:

* Opengov platform accepted posts at the article only level. Thus, general posts on the whole draft law are not possible.

* The first article is not considered in the analysis. In most consultations, the first article contained a significantly larger number of posts than the rest of the articles. The reason is that citizens tend to post general comments referring to the whole draft law in the first article. Thus, in general, posts in the first article do not necessarily relate to the content of the first article only. For this reason, the analysis of these posts is excluded from our study.

* Consultations of very brief draft laws, i.e. having up to two articles, have not been considered.

* Obviously, articles with zero (0) number of posts have been excluded from the study. This includes “Locked for comment” articles. These are articles
that the Opengov platform administrators have selected to exclude from commenting. Thus, no posts exist for these articles.

* Taking into consideration the above restrictions, the total number of articles presented in all ministries' consultations is 1,132 while the total number of posts is 42,263. The average number of posts published per article is 30. The maximum number of posts in one article was 2,546. Due to the extremely large number of posts that some articles had, we decided to restrict our research to articles having less than 70 posts. These represent 91,43% of the total number of posts in all articles.

Based on the above, we decided to study consultations with number of posts in the following ranges: 0-10, 11-20, 21-30, 31-40, 41-50, 51-60, 61-69 posts per article. From each group below the average number of posts (i.e. 30 posts) we selected and analysed two articles that were close to the high-end of the area, while for groups above the average number we selected one article per group, except for the last group where we also sampled two articles. This gave us a total of 17 articles belonging to 5 different law consultations. These were: (a) Public Consultation on the implementation of the electricity social tariff (termed here Public Consultation 1, or PC1), (b) Public consultation on the implementation of the biodiversity strategy in Greece (PC2), (c) Public consultation on reengineering the supporting structures of the Ministry of Education (PC3), (d) Public consultation on the technological high education (PC4) and (e) Public consultation on gambling (PC5).

# 7 Results

Following the method of work presented in the previous section, we started by reading and coding each of the 5 draft laws and 17 articles under investigation. As a result, we were able to identify the relevant issues along with the government positions. A first result here is the fact that in most cases each article included one only issue. In other words, each article was concentrated around one topic. In some articles however, this was not the case. For analysis purposes, we decided to discard all articles containing more than one issue leaving us with 4 draft laws and 14 articles to examine. We thereafter read and coded one by one each and every post for each remaining article. As a result, we were able to identify the new ideas suggested by citizens. These were coded as citizen positions (or new positions). For each article, a table was created presenting new positions as the number of posts grows. Table 2 presents the overall results.

<table><caption>**Table 2:** Analysis Overall Results</caption><thead><tr><th>PC id</th><th colspan="3">PC1</th><th colspan="5">PC2</th><th colspan="1">PC3</th><th colspan="4">PC4</th></tr></thead><tbody><tr><th>Article id</th><td>2</td><td>3</td><td>15</td><td>12</td><td>11</td><td>10</td><td>9</td><td>8</td><td>7</td><td>3</td><td>2</td><td>3</td><td>5</td><td>6</td></tr><tr><th># Posts</th><td>3</td><td>7</td><td>6</td><td>6</td><td>9</td><td>18</td><td>12</td><td>10</td><td>12</td><td>25</td><td>69</td><td>38</td><td>61</td><td>58</td></tr><tr><th># Citizen Positions</th><td>2</td><td>3</td><td>2</td><td>6</td><td>5</td><td>8</td><td>5</td><td>9</td><td>10</td><td>13</td><td>23</td><td>11</td><td>15</td><td>20</td></tr></tbody></table>
Based on the detailed tables for each article we created a graph depicting the number of citizen positions as the number of posts increases. As the graph accommodates all 14 data sets (one for each article), we have employed a line chart instead of the, more accurate, scatter plot.

<figure><img src="image_4.png" /><figcaption>Figure 3: Number of new positions per post (for all Articles)</figcaption></figure>

The figures suggest that most consultations follow a similar pattern. We thereafter examined in detail two cases that we consider as having particular interest. In the first case, we consider all 14 articles that we analysed. Here, we calculate the average of all citizen positions for each value of the number of posts. Clearly, as the number of posts increases, the average is based on a continuously smaller number of consultations (with the last 8 values calculated from only one consultation). The standard deviation is about 22% different from the average value (without considering the last 8 values). In the second case, we consider only the consultation with the largest number of posts, i.e. Article 2 of PC4 having 69 posts. The resulting scatter plots are depicted in Figure 4.

<figure><img src="image_5.png" /><figcaption>Figure 4: Number of new positions per post</figcaption></figure>

We thereafter use the standard “Trendline” functionality of MS Excel in order to examine which trend/regression type better fits the data. We present the analysis results
for the largest article only in Figure 5. By observing the different trendlines, we conclude that, within the limitations of our study, there is no evidence that the number of new positions tends to level. Instead, it seems there is an almost linear relationship between the number of posts and number of new positions.

<figure><img src="image_6.png" /><figcaption>**Figure 5:** Number of new positions per post including trendline</figcaption></figure>

This result suggests that increased participation in open consultations has an impact in terms of new ideas generated. We should therefore expect that by mobilising more citizens we can anticipate more different ideas. Clearly, in this study we did not evaluate the quality of those ideas. However, we only kept legitimate positions, i.e. we excluded spam, personal attacks etc., following the principles of argumentation models. This finding might be also interesting to the developers of tools for massive participation in e-rulemaking, online deliberation or idea generation in general. More specifically, the evidence that the number of new positions seems to constantly increase poses a significant challenge to those designing user interfaces for such applications. It seems it cannot be assumed that the number of ideas will eventually level, hence there is a challenge on how to fit ideas in the screen when having massive participation. As a result, new ways of accommodating the volume of positions are needed.

# 8 Conclusion and Future Work

In this study, we investigated the relationship between citizens' posts and number of new ideas. For this purpose, we used data from the Hellenic Open Government's platform (Opengov). Specifically, consultations on draft legislation using Opengov were manually obtained, coded and transferred into argument maps. For analysis purposes, we concentrated on e-consultations having less than 70 posts, which represented
91.43% of the total number of consultations at the time when data were obtained. The remaining 8.57% comprised of articles consultations having from 71 up to 2,546 posts. The analysis was performed at the article level, since posts in Opengov are allowed only at that level. We thereafter randomly selected and coded the consultations of 17 articles with varying numbers of posts. The maximum number of posts in one article was 69.

Our results suggest that, for the range of values that we examined (up to 69 posts), the relationship under investigation does not seem logarithmic, as suggested in Figure 1. We should note, however, that our approach has a number of limitations. First, we examined only one platform. Second, we measured the number of posts instead of contributors i.e. actual persons. Third, we equated new ideas with citizens *positions* (as defined in argumentation models). Fourth, the extraction of new positions from posts was performed manually by one only author. Fifth, we only examined e-consultations having up to 69 posts. Although these represent 91.43% of the total number of e-consultations, there were still consultations with significantly more posts (up to 2,546 posts). All these limitations indicate possible routes for future research.

## Acknowledgments

The authors would like to acknowledge the high-quality comments received by the four peer reviewers. We have tried to address as many comments as possible within the existing time and space limitations. Many comments provided us with new ideas and perspectives for extending this work in the future.

## References

1. Gonzalez-Bailon, S., Kaltenbrunner, A., and Banchs, R. E. (2010) ‘The structure of political discussion networks: a model for the analysis of online deliberation’. *Journal of Information Technology*, 25, pp. 230–243

2. Wright, S., and Street, J., (2007) ‘Democracy, deliberation and design: the case of online discussion forums’. *New media & society*, Vol. 9 (5), pp. 849–869

3. Strandberg, K., (2015) ‘Designing for democracy?: an experimental study comparing the outcomes of citizen discussions in online forums with those of online discussions in a forum designed according to deliberative principles’. *European Political Science Review*. May 2015, pp 1–24.

4. Coglianese, C., (2004) ‘E-Rulemaking: Information Technology and the Regulatory Process’. *Administrative Law Review*, Vol. 56, p. 353, 2004; KSG Working Paper No. RWP04-002.

5. Coglianese, C., (2006) ‘Citizen Participation in Rulemaking: Past, Present, and Future’, *55 Duke Law Journal*, pp. 943-968

6. Schulz, D., and Newig, J., (2015) ‘Assessing Online Consultation in Participatory Governance: Conceptual framework and a case study of a national sustainability-related consultation platform in Germany’. *Environmental Policy and Governance*, Volume 25, Issue 1, pp. 55–69.
7. Farina, C. R., Epstein, D., Heidt J., Newhart, M. J. (2014) ‘Designing an Online civic Engagement Platform: Balancing “More” vs. “Better” Participation in complex Public Policymaking’. *International Journal of E-Politics*, 5(1), pp. 16-40

8. Deligiaouria, A., (2013) ‘Open Governance and E-Rulemaking: Online Deliberation and Policy-Making in Contemporary Greek Politics’. *Journal of Information Technology & Politics*, Volume 10, Issue 1, pp. 104-124.

9. Zhang W., (2010), ‘Simulating the Ideal eDeliberation: The Roles of Inclusion, Equalization and Rationalization’, In De Cindio F., Macintosh A., Peraboni C. (eds.), In e-Participation to Online Deliberation, Proceedings of the Fourth International Conference on Online Deliberation, OD2010. Leeds, UK, 30 June - 2 July 2010. pp. 117-125.

10. Van Eemeren, F. H., Garssen, B. (2009), Pondering on Problems of Argumentation: Twenty Essays on Theoretical Issues, Springer Series: Argumentation Library, Vol. 14, XXII, 306 p. 9 illus.

11. Jonsson, M. E., Åström, J., (2014) ‘The Challenges for Online Deliberation Research: A Literature Review’. *International Journal of E-Politics*, 5(1), pp. 1-15

12. Braak, S.W. van den, Oostendorp, H. van, Prakken, H., Vreeswijk, G.A.W. (2006), ‘A critical review of argument visualization tools: Do users become better reasoners?’, in Workshop Notes of the ECAI-2006 Workshop on Computational Models of Natural Argument (CMNA VI), Riva del Garda, Italy, pp. 67-75.

13. Xenakis A. and Loukis E. (2010), ‘An investigation of the use of structured e-forum for enhancing e-participation in parliaments’, Int. J. of Electronic Governance, Vol.3, No.2, pp.134-147.

14. Price, D. (2009) ‘Debategraph - Politics Online Conference 2009’, Presentation at Politics Online 2009 by David Price, Obtained through the Internet: http://www.slideshare.net/davidprice/debategraph-politics-online-conference-2009, [accessed 18/3/2015]

15. Kunz, W., and Rittel, H. (1979) ‘Issues as Elements of Information Systems’, Working Paper No. 131, California: Berkley.

16. Conklin, E. J. and Begeman, M. L. (1988), ‘gIBIS: A hypertext tool for exploratory policy discussion’, Proceedings of CSCW 88, New York, NY: ACM, pp. 140-152.

17. Ramesh, B. and Dhar, V. (1991), ‘Representation and Maintenance of Process Knowledge for Large Scale Systems Development’, Knowledge-Based Software Engineering Conference, In Proceedings, 6th Annual KBSE, Syracuse, pp. 223-231.

18. McCall, R.J. (1991). ‘PHI: A Conceptual Foundation for Design Hypermedia’, Journal of Design Studies, Vol. 12, No. 1, MEDINA-MORA, pp. 30-41

19. MacLean, A., Young, R., Bellotti, V. and Moran T., (1996), ‘Questions, Options, and Criteria: Elements of Design Space Analysis’, In: Moran, T. and Carroll, J., Design Rationale Concepts, Techniques, and Use, Lawrence Erlbaum Associates, pp. 53-106.

20. Potts, C. and Bruns, G. (1988), ‘Recording the reasons for design decisions’, In Proceedings of the 10th International Conference on Software Engineering (Singapore, Apr. 11–15), T. N. Nam, Ed. IEEE Computer Society Press, Los Alamitos, CA, pp.418–427.

21. Winkels R., Douw J., and Veldhoen S. (2013), ‘Experiments in automated support for argument reconstruction’, In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law (ICAIL ’13), ACM, New York, NY, USA, pp. 232-236.