# New Ways of Deliberating Online: An Empirical Comparison of Network and Threaded Interfaces for Online Discussion

Anna De Liddo, Simon Buckingham Shum

► To cite this version:

Anna De Liddo, Simon Buckingham Shum. New Ways of Deliberating Online: An Empirical Comparison of Network and Threaded Interfaces for Online Discussion. 6th International Conference on Electronic Participation (ePart), Sep 2014, Dublin, Ireland. pp.90-101, 10.1007/978-3-662-44914-1_8. hal-01396957

HAL Id: hal-01396957

https://inria.hal.science/hal-01396957

Submitted on 15 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
New Ways of Deliberating Online:
An Empirical Comparison of Network and Threaded
Interfaces for Online Discussion

Anna De Liddo, Simon Buckingham Shum

Knowledge Media Institute, The Open University, Walton Hall
MK76AA, Milton Keynes, United Kingdom
{anna.deliddo, simon.buckingham.shum}@open.ac.uk

One of the Web's most phenomenal impacts has been its capacity to connect and harness the ideas of many people seeking to tackle a problem. Social media appear to have played specific and significant roles in helping communities form and mobilize, even to the level of political uprisings. Nevertheless the online dialogue spaces we see on the Web today are often re-purposed social networks that offer no insight into the logical structure of the ideas, such as the coherence or evidential basis of an argument. This hampers both quality of citizen participation and effective assessment of the public debate. We report on an exploratory study in which we observed users interaction with a new tool for online deliberation and compared network and threaded visualizations of arguments. Results of the study suggest that network visualization of arguments can effectively improve online debate by facilitating higher-level inferences and making the debate more engaging and fun.

**Keywords:** Argumentation, Computer Supported Argument
Visualisation (CSAV), Online Deliberation, Collective Intelligence

# 1 Introduction

Online debate through social media is increasingly used to promote citizens' engagement in policy and decision-making. While common social media websites reach increasing number of users all around the globe, and successfully respond to the demand of scaling up citizen access to public debate [1], the way in which those tools support discourse structuring, representation and analysis remains quite limited [2, 3]. This hampers the very quality of citizen participation to the public discourse and undermines the impact of citizen contributions to e-government and e-participation processes. We argue that one of the main barriers to quality e-participation to public debate is the lack of support for discourse reading and understanding. It is hardly possible for citizens to quickly grasp the main issues involved in an online debate, understand the articulation of the arguments presented, and discovering emerging synergies and conflicts.
In this paper we present results of an exploratory study, which suggests that network visualization of arguments can effectively improve the way in which users read and understand online discourse in at least two ways:

* by facilitating higher-level inferences of how the online discourse contents relate; and

* by supporting the identification of argumentation chains, synergies and contrasts within the discourse.

Moreover, we observed that the dynamic animation of networks adds an element of fun and excitement to arguments exploration. This may play an important role in promoting users' engagements with the online debate, especially crucial for motivating younger citizens.

In the following we detail the results of the study, starting from describing background knowledge and motivating the research in the CSAV (Computer Supported Argument Visualization) and Online Deliberation literature (section 2). We describe the research question and method (section 3), report on the user study (design, and experimentation setting), and describe the tool used to investigate the research question (section 4). Finally we discuss the data analysis and interpretation and present main research findings (Section 5). We conclude by outlining potential applications and future research (section 6).

# 2 Background and Motivation

## 2.1 Background Knowledge: Computer Supported Argument Visualization for Online Deliberation

The rationale and motivation for this work have roots at the intersection of Computer Supported Argument Visualization (CSAV) and Online Deliberation (OD) research, which studies how computer mediated communication can be combined with argumentation theory to enhance public deliberation. We aim to investigate how multidimensional communication flows (between people, environments, time, topics and points of views) can be translated into coherent discourse and eventually lead groups to better understanding “wicked problems” [4]. Wicked problems are undetermined and complex by nature, and they require deliberative discussions and argumentation to be better understood and tackled. Supporting better understanding and deliberating is the first steps toward the development of informed participation to policy and decision-making, and it therefore sits at the art of e-participation research.
Computer Supported Cooperative Work (CSCW) and Computer supported Collaborative Argumentation have been recognized as two major areas of research that provide support for e-consultation (pp 53, [5]). Many projects prove the extent of interest and research in this area and its relevance to develop new technologies for e-participation (for a review of argumentation tools for e-participations see [6, 7]).
CSAV tools have proved to encourage debate and deliberation by citizens on public issues and have been applied f.i for policy formulation to provide a visual medium by which citizens can follow and join in public debates on policy issues [8]. Tambouris
et al. even identify CSAV between the key enabling characteristics to consider when
assessing eParticipation projects and tools [9].

Computer Supported Argument Visualization (CSAV) aims to augment the personal and collective ability of users to explore complex problems, make sense of difference in viewpoints, discuss options, and reflect on the implication of those by analyzing and constructing arguments.

CSAV is a research field which has its roots that span several disciplines, such as
Philosophy, HCI, hypertext, CMC, and several domain applications, such as design,
education, law and public policy [10]. The idea that people can augment their human
intellect and their capability to “comprehend and find solution to complex-problem
solving situations” by manipulating explicit and externalized “concept structures”
dates to Douglas Engelbart [11]. Concept Mapping [12] and Argument Mapping [13]
have been applied to support education and critical thinking and several authors
identified “visualization” as an important but unexploited dimension for refining and
communicating one’s thoughts [14]. Argumentation and the associated technological
support for argument analysis and construction, have been widely investigated in the
two decades from the 70ies and 90ies, when first class researchers in the hypertext
community developed prototype tools (between others NoteCards [15], gIBIS [13]
and AAA [16]) and carried out pilot studies to explore advantages and shortcoming of
this approach [17]

In design rationale studies several limitations have been identified which highlights
the difficulty to use computer supported argumentation for design practice,
particularly related to the conceptual and time overheads in fragmenting and
structuring thoughts before communication. Fisher and al. [18] suggest that there are
many limitations that need to be overcome to make argumentation (and argumentation
tools) serve design; and these concerns have been echoed in [19].

Nonetheless, in another domains such as Technology Enhanced Learning, Public
engagement and e-Participation the interest in argumentation and CSAV has
persistently been followed, also encouraged by the recognized advantages of this
approach in term of support to reflection, knowledge construction and learning [20].
Example tools for large-scale deliberation are currently classified as e-participation, e-
democracy or e-government systems [21, 22] some of which uses CSAV as a way to
structure and facilitate citizens consultation [8] and large scale participation to public
debates [23].

An established research literature documents the advantages of making the structure
and status of a dialogue or debate more visible [10]. Scheuer and al. [24] review the
state of the art of computer-supported argumentation, which proves the extensive
interest and production of research and technology in this field. Scheuer and al.
present a rich overview of both the types of argument representations, as well as the
variety of interaction designs and ontologies to support argumentation. Several
empirical studies are also presented that have been carried out to assess various
argumentation systems in different domains. While many of these systems are aimed
to support argumentation in a learning context [25](e.g Belvedere [26] and
ARGUNAUTUT [27]); many others have been designed to support argumentation in
other fields such as law (Carneades [28] and Rationale [29]; science (SenseMaker [30], WISE [31]) and decision making (QuestMap, Compendium [32, 33]).
We are interested in this last category of argumentation systems and their Web
successors which have been designed to support e-democracy and e-participation by
promoting citizen engagements in decision-making throughout online discourse
processes. These are specifically designed collective intelligence infrastructures and
large-scale argumentation systems to structure informal online discussion as
argumentation processes and use CSAV to make visually explicit users’ lines of
reasoning and (dis)agreements [34]. Naturally, the use of semantic networks to
provide computational intelligence has now converged with Web Science, resulting in
Web 2.0 Argumentation [35] and a semantic web standards-based Argument Web
[36].

## 2.2 Research Gap and Motivation

Many of these CSAV tools are now available, exhibiting a wide mix of network visualizations and more conventional threaded renderings. Whereas many systems use a common ontology to represent discourse element (such as IBIS [4] or its more design oriented version QOC [37], when it gets to argument visualization there seems to be no agreement on what is the best interface for argument visualization. Deliberatorium uses a linear-threaded visualization [3], while Cohere [35] supports a graph visualization of the online discourse. Debategraph¹ and CoPe_it! [38] enable linear, threaded and graph views. However, to date, we have not yet found a systematic evaluation of the merits or otherwise of CSAV vs. threaded interfaces.
In addition to this, the suitability of graphs as medium to support argumentation has been questioned by some, for three main reasons: complexity of the ontology, number of participants and domain of application. Some researchers argue that the bigger the number of participants to the discussion, and the higher the complexity of the discourse ontology, more clumsy and less usable graph visualizations become (pp.53 [24]). In addition, there seems to be resistance to the use of graph visualization of arguments in certain domains of application. Hair [39] reports on a study in the legal field in which lawyers have expressed a strong preference in favour of linear-threaded text representations of arguments compared to network visualization.

It therefore remains unclear what are the advantages and affordances of different graphical representation of arguments to support online discussion and large-scale deliberation. This motivates the research presented in this paper.

An exploratory user study was carried out to compare network visualizations and threaded discussions as the two most commonly used interfaces for argument visualization. We aimed to assess which one of these two visualizations better support online discourse in two main tasks: argumentation reading and understanding. Argumentation reading and understanding are at the cornerstone of effective participation to online discourse, and they also consist of the most prominent users

¹ DebateGraph 2013: Web tool for the visualization of idea and debates
http://debategraph.org
interaction modality. In fact, the internet usage pattern of “participation inequality” [40] confirms that typically, only a very small percentage of users contributes to a website, compared to vast majority who consume. So, while browsing, searching, reading and writing clearly cannot be divorced from each other, the focus of the study reported here is not on the authoring process, but on the experience and performance of the majority of users who will be reading and searching the online deliberation platforms.

# 3 RESEARCH QUESTION AND METHODOLOGY

Does an interactive, self-organizing network visualization of arguments provide advantages over a more conventional threaded interface for reading and search? The following research questions was formulated:

*RQ: What are the recognized advantages of threaded and network visualization of arguments for supporting online discourse reading and understanding?*

A grounded theory analysis has been conducted to study the use of different graphical interfaces for the representation of arguments, and to assess to what extent these affect the way in which users read and understand the online discourse. Grounded Theory is a qualitative research method which has been widely used in HCI to provide insights on people's views, behaviors, understanding and experience with technology [41]. Very recently the discussion has been revamped on what are the nuances and innovation to the method that may help to respond to the increasing request and diffusion of qualitative research studies in HCI [42].

We took a ‘constructivist’ stance to grounded theory which aims at creating a description of the context rather than discovering a description of it (objectivist approach). We started by keeping a Glaserian approach to data analysis and applied open coding of the data trying not to be influenced by prior ideas or extant theories. The initial open coding phase was performed on transcripts of the screencast video of users’ interaction in the experimentation. Video clips and transcripts were analyzed and coded in an unfocused manner. While reiterating the coding from one user experimentation to the following, we moved progressively to a more focused coding in which specific concepts and themes started emerging. From this analysis several categories emerged which provided the building blocks of our findings, and the main pillars of our narrative of the analyzed phenomenon.

The coding process generated over 350 codes, which have been edited and merged in several revising cycles, and then have been grouped into main sub-categories (23 in total) and higher-level categories of analysis (9 in total). The main higher-level categories emerged provided a taxonomy of the main components/variables which affect users capabilities to read and comprehend online argumentative discussion. The 9 categories have then been ordered on the base of what were more significant in terms of three metrics: number of quotations per code (N. of Videoclips/Quotation), number of participants who experienced the event captures by the code (N. of People affected), and time of interaction per code. As a result three main higher-level code categories emerged as key factors affecting users interaction and these have been used
as metrics to access users performance with the two interfaces (see section 5). We
then developed a narrative around these 3 emerging themes, but also around events
and findings, rooted in the in depth observation and analysis of the interaction. This
allowed us to draw some conclusions on what are the recognized advantages of
threaded and network visualization of arguments to support online discourse reading
and understanding.

# 4 User Study

## 4.1 Goal, Design and Participants

We studied an heterogeneous group of researchers and practitioners in Higher
Education engaged in the use of a collective intelligence tool for collaborative argu-
mentative discourse and knowledge construction. An exploratory user study was run
to observe users’ performance under three information-seeking tasks, and compare
their performances using two different user interfaces for arguments visualization
(threaded vs network visualization of arguments). Participants were divided in two
groups of 5 (Group1 and Group2). The 10 subjects were drawn from the members of
different Open University departments, so with widely mixed IT expertise. They were
randomly allocated to the two different groups, but we verified post-hoc that IT
expert/non ex-pert ratio in each group was approximately the same. The median age
was 40 (with range from 32 to 48) with the majority of users being either native or
near-native English speakers.

## 4.2 Online Environment: The Evidence Hub

The online discussion tool used throughout the study was the Evidence Hub
system[43]. The Evidence Hub is a collective intelligence and online deliberation tool
to support argumentative knowledge construction by crowdsourcing contributions of
issues, potential solutions, research claims and the related evidence in favor or against
those. The Evidence Hub is particularly oriented to support community of
practitioners to build evidence-based knowledge about specific key challenges that
are set up to the community. This key challenges can then be addressed by tackling
specific sub issues, that the community contribute to add to the system. Each sub
issue can then be tackled by proposing solutions to it, or by sharing specific research
claims that can help tackling the issue. Finally potential solutions and research claims
can be debated by the community by advancing evidence in favor or against those and
by providing relative resources backing them up. By scaffolding users contributions
in this way, the Evidence Hub aim to effectively crowdsource and support large-scale
deliberation in e-democracy and decision-making processes. To allow comparison
between users interaction with two graphical representations of arguments, two
different versions of the Evidence Hub have been set up which used different user
interfaces for arguments visualization. The two versions of the Evidence Hub pointed
at the same database to ensure that participants in the two groups would receive the
same quantity and type of information.

4.3 The Linear-threaded Interface for Arguments Visualization

The linear-threaded interface was used by Group1 (Figure 1). This interface for arguments visualization is similar to the most common threaded online discussion interfaces to support argumentative discourse activities. It consists of a classical threaded visualization in which issues set the focus of discussion (title of the page in Figure 1), and then potential solutions are listed below (light bulb icons are placed before each solution). Each solution can then be expanded on demand. The progressive indentation of text shows the supporting and challenging evidence for a solution, and the re-sources supporting evidence (Figure 1).

<figure><img src="image_2.png" /><figcaption>Figure 1. Linear-threaded Interface for Arguments Visualization</figcaption></figure>

4.4 The Network Interface for Arguments Visualization

The network interface was used by Group 2: This consists of an argument map built
by following the modified IBIS, PHI model of arguments [44]. Each statement added
to the discussion is here represented as a node in a semantic graph structure (Figure
2). In this visualization, node icons and colors represent the rhetorical role of the
statement in the discussion: issues are characterized by a red question mark icon in a
dark turquoise node (yellow in Figure 2 because the node is selected); solutions are
distinguished by a light bulb icon (see light green nodes), while pro and con are
characterized by purple nodes and are connected with a green link or red link to
respectively indicate “supporting” or “challenging” relations.
The network visualization of the Evidence Hub is built with a Java Applet, which lays out the nodes dynamically, and following a gravitational algorithm. This adds a specific animation component to most common argument network visualization tools.

<figure><img src="image_3.png" /><figcaption>Figure 2. Screenshot of the Network Interface for Arguments Visualization</figcaption></figure>

# 5 Data Analysis and Results

The experimentation design aimed to provide an answer to the RQ by focusing on three specific information-seeking tasks (for issues of space we skip detailed description of the tasks):

1. Identifying solutions to an issue (Task1)

2. Identifying synergies between solutions (Task 2)

3. Identifying contrasts in the wider debate (Task 3)

The tasks lasted about 2 to 3 minute and required users to scan 1-3 Webpages (about 1500 - 4000 words) depending on the task.

The three higher-level class of codes emerged from the grounded theory analysis of the user-interaction videos were used as metrics to compare the user's performance across the three tasks. The metrics are: Task Accomplishment; Data Model Interpretation and Emotional Reactions to the two interfaces. From the grounded theory analysis, these metrics have been found to represent the most relevant factors affecting interaction and therefore have been used to assess users' performance with the two interfaces. In the following we analyze and compare users' performance by focusing on each of these metrics and providing users code to motivate our understanding.

## 5.1 Task Accomplishment

Four main sub-categories of accomplishment emerged from the coding: Accomplished easily, Accomplish but incorrect, Not accomplished, and Give Up the
task. From the analysis of the quotations and of the screen capture video of the interaction we found that the main reason of failure was task's understanding. Most interaction events coded as "Accomplished but incorrect" were reported in Group 1. The linear structure of the interface is not designed to help interconnecting and comparing content and this has proved to be a burden to the user. As a consequence of this limitation, users who interacted with the linear interface focused more on issues of content's style and validity, rather than on the argumentation process, which lead to digressions and incorrect responses. On the contrary the interaction events coded as "Accomplished easily" were mainly experienced in Group 2. This is mainly due to the visual hints provided by the network representation. Specifically, links labels and colours seemed particularly predominant in determining success. We noticed that, the users who accomplished the task easily usually relayed on links colours and label, and paid less attention to the iconography of the nodes. The number of links (connections density) also proved to be a very effective way to provide answers to the task.

## 5.2 Comparing Data Model Interpretation

In order to compare linear and network visualization of arguments in term of how they supported Data Model Interpretation (DMI) we first identified the emerging codes related to DMI and merged them under the following 5 main classes (1. *get the model right*, 2. *get the categories right*, 3. *get the model but is unsure*, 4. *misinterpret categories*, 5 *does not get the model at all*). We then analysed the interaction events and quotations for each main class.

Results show that DMI was better supported by network visualization of arguments. Category misinterpretation and uncertainty in data model interpretation tend to occur more frequently in Group1 than in Group2, while network visualization of arguments seems to support a complete and correct understanding of both categories and data model. Findings suggest that one possible motivation for this is that argument maps provides examples of how the content of discussion is interpreted in the data model supported by the Evidence hub. In this way users can learn by example ("let's see what other people have put under this category" - P8) and at each exploration of a new argument map they reinforce their understanding of the data model ("this is my solution and this evidence support the solution, and these two nodes challenges it...the clue is the red "challenges" link -P7").

## 5.3 Comparing Emotional Reactions

Emotional reaction to the graphical representation of arguments is another emerging category that we used to measure users satisfaction and emotional attitude toward the two proposed interfaces. Results show that a general sense of surprise and positivity toward the network visualization was recorded, which easily sparks into linkeness and even exhitment ("ahhh so fabulous!"-P7; "wow this is amazing!"-P6). This positive emotional reaction also seems to be associated to an increase of user's confidence with the tool ("I feel confident, I'm pretty sure this is the answer"-P7). The main object of positivity toward the network visualizations was the self-arranging
graph applet. The Applet presents argument maps as floating nodes and edges, slowly
moving and arranging on the screen ("it is like a jelly, it is so fantastic!"-P7; "it is all
shifting! It is interesting... I quite like that!"-P8). Movements provoked surprise and
excitement and it was also recognized as a useful information feedback. By looking at
the floating network, users understood that the map was not static and they could
move nodes around and play with it. Users reported that they mostly enjoy this
feature. In summary, findings suggest that: moving arranging, zooming and pinning
are the network features which most augment users confidence and satisfaction with
the tool. On the contrary emotional reactions to the linear interface for arguments
visualization were general *skepticism* and *dissatisfaction*, and sometime also decayed
to *confusion* and *feeling lost* ("I think I am lost"-P1; "I am now really annoyed..
because I haven't worked out how to do it" "that is the all page I am looking
at?...ohhh ok I give up!"-P5). Some users explicitly identified the cause of their
frustration and said that they needed "too many clicks" to seek information and
frequent "change of context" which often provoked disorientation.

6 Contributions and Future Research

The grounded theory analysis of the experimentation's video showed that different
graphical interfaces for the representation of arguments strongly affect the way in
which users read and understand the online discourse. Network-like representations
and visual hints such as network structure, iconography, links' labels and colors seem
to facilitate the identification of argumentation chains, thus supporting indirect
connection and higher-level inferences of how the content connects. The results show
that data model interpretation is also improved by argument visualization. Notably,
exposing the data model in form of argument maps appears to enable learning-by-
example mechanisms, whereby users reinforce their understanding of the data as they
navigate through the user interface. Finally there is an element of fun and excitement
associated to dynamic network visualization of arguments. This may suggest that this
type of arguments visualization could be useful for e-participation processes in which
the element of "play" and positive emotional reaction are key factors to success. This
paper shows promising results on the capability of network visualization of arguments
to support reading and sensemaking of online discourse activity. Of course the
challenge for e-participation is: how can we move from lab experiments to real world
setting and scale the use of CSAV from small group to collectives. Future research
will be devoted to explore the use of CSAV and other form of visual analytics to
improve users' engagement and sensemaking of large-scale online deliberation
processes.

7 Acknowledgements

The authors thank the support of Catalyst (FP7 program - grant agreement
#6111188) under which this work has been finalised.
8 Reference

1. Shirky, C.: Here Comes Everybody: How change happens when people come together. (2009).

2. Sunstein, C.R.: Infotopia: How Many Minds Produce Knowledge. Infotopia: How Many Minds Produce Knowledge. (2006).

3. Klein, M., Iandoli, L.: Supporting Collaborative Deliberation Using a Large-Scale Argumentation System: The MIT Collaboratorium. (2008).

4. Kunz, W., Rittel, H.W.J.: Issues as elements of information systems. (1970).

5. OECD: Promise and Problems of E-Democracy: Challenges of Online Citizen Engagement. 1–162 (2003).

6. Benn, N., Macintosh, A.: Argument Visualization for eParticipation: Towards a Research Agenda and Prototype Tool. Electronic Participation. pp. 60–73. Springer Berlin Heidelberg, Berlin, Heidelberg (2011).

7. Gordon, T., Macintosh, A., Renton, A.: DEMO-net: D5. 2: Argumentation Support Systems. itc.napier.ac.uk.

8. Renton, A., Macintosh, A.: Computer-Supported Argument Maps as a Policy Memory. The Information Society. 23, 125–133 (2007).

9. Efthimios Tambouris, N.L.A.K.T.: A Framework for Assessing eParticipation Projects and Tools. 1–10 (2006).

10. Buckingham Shum, S.B.: The roots of computer supported argument visualization. Visualizing argumentation. (2003).

11. Engelbart, D.C.: Conceptual Framework for the Augmentation of Man's Intellect. (1963).

12. Novak, J.D.: Learning, creating, and using knowledge: Concept maps as facilitative tools in schools and corporations. (2010).

13. Conklin, J., Begeman, M.L.: gIBIS: a hypertext tool for exploratory policy discussion. ACM Trans. Inf. Syst. 6, 303–331 (1988).

14. Horn, R.E.: Visual language and converging technologies in the next 10-15 years (and beyond). Proceedings of the 10th WSEAS International Conference on Applied Informatics and Communications (2002).

15. Halasz, F.G., Moran, T.P., Trigg, R.H.: Notecards in a nutshell. Presented at the CHI '87: Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface April (1987).

16. Schuler, W., Smith, J.B.: Author's Argumentation Assistant (AAA): a hypertext-based authoring tool for argumentative texts. Hypertext: concepts, systems and applications. (1992).

17. Shipman, F.M., III, Marshall, C.C.: Formality Considered Harmful: Experiences, Emerging Themes, and Directions on the Use of Formal Representations in Interactive Systems. Computer Supported Cooperative Work. 8, (1999).

18. Fischer, G., Lemke, A.C., McCall, R., Morch, A.I.: Making Argumentation Serve Design. Human-Computer Interaction. 6, 393–419 (1991).

19. Buckingham Shum, S., Hammond, N.: Argumentation-based design rationale: what use at what cost? International Journal of Human-Computer Studies. 40, (1994).

20. Simon, S., Erduran, S., Osborne, J.: Enhancing the quality of argumentation in school science. Journal of research in science teaching 41.10 (2004): 994-1020.

21. Macintosh, A.: The emergence of digital governance. Significance. (2008).

22. Macintosh, A.: E-democracy and e-participation research in Europe. Digital Government. (2008).

23. Klein, M.: The MIT Collaboratorium: Enabling Effective Large-Scale Deliberation for Complex Problems, (2007).

24. Scheuer, O., Loll, F., Pinkwart, N., McLaren, B.M.: Computer-supported
argumentation: A review of the state of the art. International Journal of Computer-Supported Collaborative Learning. 5, 43–102 (2010).

25. Suthers, D.D.: Collaborative representations: Supporting face to face and online knowledge-building discourse. System Sciences. (2001).

26. Suthers, D., Weiner, A., Connelly, J.: Belvedere: Engaging students in critical discussion of science and public policy issues. Proceedings of the 7th World Conference on Artificial Intelligence in Education (1995).

27. De Groot, R., Drachman, R., Hever, R.: CSCL'07: Proceedings of the 8th international conference on Computer supported collaborative learning (2007).

28. Gordon, T.F., Prakken, H., Walton, D.: The Carneades model of argument and burden of proof. Artificial Intelligence. 171, (2007).

29. Van Gelder, T.: The rationale for Rationale™. Law. (2007).

30. Bell, P.: Using argument representations to make thinking visible for individuals and groups. Presented at the Proceedings of the 2nd international conference on Computer support for collaborative learning, Toronto, Ontario, Canada (1997).

31. Linn, M.C., Clark, D., Slotta, J.D.: WISE design for knowledge integration. Sci. Ed. 87, 517–538 (2003).

32. Conklin, J.E.: Conklin: "Designing organizational memory: preserving intellectual assets in a knowledge economy." Group Decision Support Systems 1 (1996): 362.

33. Buckingham Shum, S., Selvin, A.M., Sierhuis, M., Conklin, J., Haley, C.B., Nuseibeh, B.: Hypermedia Support for Argumentation-Based Rationale. Presented at the, Berlin, Heidelberg (2006).

34. De Liddo, A., Buckingham Shum, S., Convertino, G., Sándor, Á., Klein, M.: Collective intelligence as community discourse and action. Presented at the CSCW '12: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work Companion February (2012).

35. Buckingham Shum, S.: Cohere: Towards Web 2.0 Argumentation. Presented at the Proceeding of the 2008 conference on Computational Models of Argument: Proceedings of COMMA 2008 June (2008).

36. Rahwan, I., Zablith, F., Reed, C.: Laying the foundations for a world wide argument web. Artificial Intelligence. (2007).

37. MacLean, A., Young, R.M., Bellotti, V.M.E., Moran, T.P.: Questions, options, and criteria: elements of design space analysis. Human-Computer Interaction. 6, (1991).

38. Karacapilidis, N., Tzagarakis, M., Karousos, N.: Tackling cognitively-complex collaboration with CoPe_it! International Journal of Web-Based Learning and Teaching Technologies (IJWLTT), 4(3), 22-38. (2009).

39. Hair, D.C.: LEGALESE: a legal argumentation tool. SIGCHI Bulletin. 23, (1991).

40. Hill, W.C., Hollan, J.D., Wroblewski, D., McCandless, T.: Edit wear and read wear. Presented at the CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems June (1992).

41. Glaser, B.G., Strauss, A.L., Strutzel, E.: The discovery of grounded theory; strategies for qualitative research. Nursing Research. (1968).

42. Brown, E., Cairns, P.: A grounded investigation of game immersion. CHI'04 extended abstracts on Human factors in computing systems ACM (2004).

43. De Liddo, A., Buckingham Shum, S.: The Evidence Hub: harnessing the collective intelligence of communities to build evidence-based knowledge. (2013).

44. McCall, R.J.: PHI: A conceptual foundation for design hypermedia. Design Studies. (1991).