# IdeaTracker: An Interactive Visualization Supporting Collaboration and Consensus Building in Online Interface Design Discussions

Roshanak Zilouchian Moghaddam, Brian P. Bailey, Christina Poon

► To cite this version:

Roshanak Zilouchian Moghaddam, Brian P. Bailey, Christina Poon. IdeaTracker: An Interactive Visualization Supporting Collaboration and Consensus Building in Online Interface Design Discussions. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.259-276, 10.1007/978-3-642-23774-4_23 . hal-01590573

HAL Id: hal-01590573

https://inria.hal.science/hal-01590573

Submitted on 19 Sep 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# IdeaTracker: An Interactive Visualization Supporting Collaboration and Consensus Building in Online Interface Design Discussions

Roshanak Zilouchian Moghaddam, Brian P. Bailey, and Christina Poon

University of Illinois at Urbana
Department of Computer Science
{rzilouc2, bpbailey, cpoon2}@illinois.edu

**Abstract.** With the rapid growth of open source and other geographically distributed software projects, more interface design discussions are occurring online. Participation in such discussions typically occurs via issue management systems or similar interactive discussion forums. While such systems have a low learning curve, they do not support key elements of design discussion such as comparing alternatives, maintaining awareness of the arguments for and against the alternatives, or building consensus. To better understand these and other challenges, we conducted a study of online interface design discussion. The study consisted of analyzing a large corpus of online discussion content and conducting interviews with designer and developer participants. We discuss the findings of our study and use them to motivate the implementation of an interactive visualization tool - IdeaTracker. The tool offers explicit support for tracking and comparing ideas and gaining an abstract summary of the overall discussion as well as specific alternatives. It also provides a voting system to support consensus building. The tool extracts and visualizes useful information from the discussions that would otherwise be hidden but without interfering with the current method of participation. Our tool is compatible with the issue management system of one open source project but can be extended for others. Initial user feedback is positive and confirms the need for an alternative visual representation of interface design discussions online.

**Keywords:** Design, open source software, user interface, visualization

## 1 Introduction

The User Interface accounts for a large part of the design and development effort in the production process of any interactive software system [18]. With the number of geographically distributed software projects and, in particular, open source software projects growing [10], more interface design and development discussions are unfolding online. For example, in open source projects, designers typically engage in interface design discussions through the project's issue management system. These systems offer interactive Web-based forums for discussing design issues. These forums are similar to other types of Web-based newsgroups and social media where participants discuss a variety of topics from food and books to religion and politics.

However, interface design discussions have many unique characteristics that are ignored by the current metaphor of interactive forums. For example, common practices that are paramount to producing effective interface designs such as
generating, discussing, and comparing multiple ideas [9, 26] are not specifically supported. This problem is exacerbated due to the diverse levels of expertise and viewpoints of the discussion participants [17].

A few studies have analyzed design discussions in open source software. For example, Barcellini et al. studied software discussions occurring in mailing lists with the goal of extracting design relevant information [7]. Unlike their study, our work focuses on analyzing and enhancing design discussions relating to the *interface* of a software project. Twidale et al. analyzed usability reports from Bugzilla [25]. They recommended several improvements such as enhancing the classification of usability bugs and adding explicit representations of design arguments to the reports. However, these enhancements were never implemented and their study considered only usability bug reports rather than, e.g., proposals for new interface features.

Various tools have also been developed to support distributed interface design. For instance, Designers' Outpost [11], Synergy [15] and the Envisionment and Discovery Collaboratory [4] are examples of collaborative design tools that can be used for co-located or distributed team situations. In addition, a myriad of tools have been developed to support the creation of design artifacts in distributed settings [1-3]. Though these types of tools can facilitate the early stages of interface design, our work is original in that we are targeting improving online *discussions* of interface design issues. Though our current focus is on open source projects due to the public availability of discussion data, we believe our results are applicable to any situation where Web-based interactive forums are being used as the primary means for discussing interface design issues.

To better understand designers’ challenges participating in online design discussions, we studied a large corpus of interface design discussions from two popular open source projects: Ubuntu and Drupal. In our study we analyzed the content of 1560 messages collected from thirty discussion threads from the Ubuntu and Drupal issue management systems. Figure 1 shows an example of a design discussion extracted from the Drupal issue management system. Our analysis included counting the posts, alternatives, and participants per discussion as well as the fluidity of the topic flow. Our data analysis was complemented by interviews with UI designers (N=6) and developers (N=6) participating in the design discussions. The interviews helped interpret the results from the content analysis and uncover the current nature of participation, idea generation, and the consensus building process.

From data analysis and interviews, we found that designers are struggling to track the design alternatives and the arguments for and against each alternative, want to maintain better awareness of others’ preferences, and prefer that the interface design and development activities remain tightly integrated. We have incorporated these and other findings into the design of a new interactive visualization tool called IdeaTracker for managing online interface design discussions. The visualization highlights the proposed alternatives in a visual timeline, allows participants to filter and review messages that refer to each alternative, and provides an abstract visual summary of each message to indicate length and affective tone of each message. IdeaTracker also helps with decision making by enabling designers to compare selected subsets of alternatives and supports consensus building by implementing a voting system. We believe that using IdeaTracker can result in more effective design discussions online, which can lead to higher quality interfaces. Initial user feedback is
positive and confirms the need for an alternative visual representation of interface
design discussions online.

Our work makes two contributions. First, we identify key challenges participants
face when engaging in interface design discussion online, which include tracking and
comparing alternatives, maintaining awareness of community opinion, and building
consensus around specific alternatives. Second, we demonstrate how these challenges
can be addressed through the implementation of a novel interactive visualization tool.
The tool extracts and visualizes useful information from the discussions that would
otherwise be hidden but without interfering with the current method of participation.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** A sample interface design discussion thread in Drupal. The interface design problem discussed in this thread is the lack of visual connection between a selected tab and its content. An alternative has been proposed along with the problem description. In the first few comments, participants are discussing the scope of this problem and whether it is a valid issue.</figcaption></figure>

## 2 Related Work

We describe prior work for representing online content in different domains and how
our study builds upon and extends that work for online interface design discussion.
Also, we describe open source and the usability enhancement movement within open
source and how our work can aid this movement.

### 2.1 Distributed Collaborative User Interface Design

A myriad of tools have been developed to support creation of design artifacts in
distributed settings. For instance Axure [1], Pencil [3], and Balsamiq [2] are part of a
large class of tools that have been developed to manage online collaborative creation of interface mockups. These tools assume that collaborators utilize existing tools such as interactive discussion forums or synchronous messaging systems for discussing the designs. Other tools have been developed to support remote collaboration in UI design. For instance, Distributed Designers' Outpost is a collaborative web site design tool that captures the history of an evolving information architecture [11]. Synergy is another collaborative design environment that supports remote collaboration during the early stages of the design process [15]. However, there has been little research into interfaces that specifically support the unique nature of *online design discussions*. We have taken a first step in this direction by first understanding specific challenges of such discussions and showing how they can be addressed in a novel visualization.

## 2.2 Visual Representation of Online Content

A visual representation of online content expressed in narrative form can foster sense making of the information. Such visual representations have been created for visualizing content in myriad online communities. For instance, Opinion Space is an online interface designed for collecting and visualizing users' opinions from comments in online news articles, blogs, videos, and product reviews [12]. Shared Space is another tool that analyzes students' chat messages to visualize discussion and agreement during online discussions [14]. The Conversation Map system creates a graphical interface by analyzing the text of archived newsgroup messages. The interface can be utilized to read and search all of the messages in the archive [22].

All of these tools were developed to represent a specific type of discussion content in an online community: Opinion Space represents opinions, Shared Space represents chat messages, and Conversation Map shows newsgroup posts. Similarly, interface design discussions have certain elements that, if incorporated into the design of a visual representation, can foster participation and improve the quality of discussions. Inspired by this previous work, this paper explores the design of an interactive visualization tool that supports interface design discussions, capturing the unique characteristics of this content.

## 2.3 User Interface Design in Open Source

A usability movement has been ongoing in the Open Source community [5, 8, 13, 16, 19, 23, 24]. Corporations were one of the first contributors to this movement by providing interface guidelines and recruiting usability experts to work on open source projects [5, 8, 19]. The open source community itself helped this movement by promoting new techniques such as “Design-by-blog” where personal blogs are used for design discussions [20]. In many projects, screenshots can now be posted in mailing lists and issue management systems which facilitate design discussion [25]. Also, usability experts have started joining interface design discussions in open source through blogs, issue management systems, and wikis [5, 27]. As more designers, usability experts, and developers engage in discussions of interface design
issues for open source, there is a growing need for studies to characterize their
participation challenges and identify feasible ways to address them.

As a starting point, several studies have been conducted to identify the challenges of improving the user interfaces of open source projects [5, 8, 13, 16, 19, 23, 24]. Some solutions propose to resolve the challenges by funding usability experts to work on open source projects, adapting automatic approaches for evaluating interfaces, encouraging HCI students to join open source, and providing a social and technological infrastructure for usability experts [19]. Bach et al. expanded this last solution by arguing for a separate design space for usability experts at the same level as developers in the project's information (and social) architecture [5].

Our work shares the long-term goal of improving the user interface of open source projects, but emphasizes enhancing the discussions of interface design issues, thereby leading to more effective participation. In contrast to Bach et al., our approach is to provide a more effective visual representation of the existing design discussions, which involve both designers and developers, rather than advocate for a separate discussion space targeted only for designers.

Finally, in relation to a prior study of online interface design discussion for open source [25], our work is original in that it analyzes a broader sample of design discussions, complements this analysis with results from stakeholder interviews, and implements the findings within an interactive visualization tool.

## 3 A Study of Online Interface Design Discussion

We studied the evolution of interface design discussions in two well-known open source projects: Ubuntu and Drupal. Ubuntu is a popular Linux distribution and Drupal is a widely used content management system. We chose these two projects because they have an active interface design team, the design discussions are publicly accessible, and they contain a considerable amount of lively design discussion.

Within both of these projects participants post interface design problems, solution proposals, and related comments to the respective issue management system. These issue management systems are Web-based interactive forums dedicated to discussing software related issues including interface design issues. These forums are similar to the interactive forums used for discussing topics of interest in many other online communities or social media sites.

As shown in Figure 1, each interface design discussion (a new thread) starts when a participant posts the description of a design problem. Other participants can then contribute to the discussion by proposing design alternatives, arguing for or against the proposed alternatives, attaching an implementation of an alternative (called a *patch*), reporting the results of a patch review, or raising other concerns such as clarifying the scope of the problem or how it relates to other ongoing design efforts.

Generating multiple alternatives and having an engaged discussion are critical for producing an effective solution to the design problem and having confidence in it. Designers also need mechanisms to share opinions and increase awareness of others' perspectives to facilitate consensus. This is difficult even during face-to-face design meetings, yet the issue management systems used for interface design discussions
only support text postings along with the option of attaching an image. They do not have explicit support for key elements of the discussion itself such as tracking which ideas are favored, arguments for or against the ideas, or building consensus around specific alternatives. Our study therefore centered on answering these questions:

1. How many ideas are typically shared in online design discussions? How extensive is the debate? How do designers maintain awareness of the ideas?

2. What techniques are used to promote consensus around specific alternatives and how effective are these techniques?

3. What are the strengths and weaknesses of the issue management systems used for discussing and managing ideas?

4. What are other challenges that designers encounter participating in online design discussions and what strategies are used for addressing them?

To answer these questions, we analyzed content from online interface design discussions and conducted interviews with participants. For the quantitative analysis, we examined 1560 messages that spanned thirty discussion threads. Fifteen of the threads came from a pool of 300 in Ubuntu (average number of messages in each thread=35.4 sd=12.9). The other fifteen come from a pool of 500 threads in Drupal (average number of messages in each thread=68.6 sd=23.0). In both cases, we selected the fifteen threads from the most active threads. From inspecting a sample of the threads; we defined two characteristics for ‘activity;’ (i) the number of images, where we considered an image to be a proxy for a proposed design alternative and (ii) the number of messages in that discussion. We rank ordered the pool of discussion threads based on these criteria and selected fifteen threads from the top fifty threads in each project.

We then analyzed the content of the messages. First, we divided the message into a smaller set of topical chunks. A new chunk was created when there was a transition from one topic to another. Then, we manually coded the chunks into eight categories: Issue, Alternative, Criterion, Clarification, Project Management, Implementation, Digression, and Other. We adapted these categories from the coding scheme developed by Olsen et al. [21] for capturing and analyzing the core elements of design discussions in collocated settings. Though the original coding schema had eleven categories; our adaptation only used seven. When testing the schema and resolving inconsistencies (based on five discussion threads from each project), the evaluators agreed that the other four categories were not applicable. An implementation category was added to capture the technical messages in the discussions.

The data analysis was complemented by a set of semi-structured interviews with twelve participants from both Ubuntu and Drupal projects. We interviewed six active designers, three from Drupal with an average of 7 years of experience (sd = 1) and three from Ubuntu with an average of 7 years of experience (sd = 3.26). We will refer to Drupal designers and Ubuntu designers as DD# and DU# accordingly. We also interviewed six developers participating in resolving UI design and usability issues: three from Drupal with an average of 6.5 years of experience (sd = 3.51) and three from Ubuntu with an average of 7.5 years of experience (sd = 4.5).

Each interview lasted about an hour and was conducted via phone and instant messaging. The latter was used to share Web links, images, and other data artifacts during the interview. The subjects were compensated with a $30 Amazon gift card.
The interview questions reflected the main research questions of our study and were informed by prior work (e.g. [5]). For example, the questions included: what are the main challenges in discussing design ideas online? How do you maintain awareness of the ideas? What techniques are used to promote consensus? What are the strengths and weaknesses of the issue management systems for discussing ideas?

# 4 Challenges and Implications

We discuss key challenges that designers encounter while participating in interface design discussions online and the methods used for addressing those challenges. Although our study focused on open source projects, the challenges identified were mostly due to the intersection of the interaction design of the forums and the topic of the discussion. As a result, we believe these observations apply to any interactive forum which serves as the primary means for discussing interface design issues.

## 4.1 Designers Struggle to Track Design Alternatives

A total of 299 alternatives (ideas) were submitted in the 1560 messages analyzed. On average, nine alternatives were proposed in each thread of discussion (sd=5.88, max=27, min=0), indicating that multiple alternatives were welcomed and considered for each design problem. Of all the alternatives proposed, 63% were described solely in narrative form, 13% included only a patch (a file that updates the implementation to provide a working preview of the idea), and 18% included only a screenshot of a proposed solution. Figure 2 shows a distribution of alternatives for each combination of modality. Although we chose part of our data set from the discussions with the most images, more than half of the alternatives were presented in narrative form. This is partly due to the lower cost of expressing alternatives in narrative form. Other reasons for having fewer visual representations may be the text-based structure of the discussion threads and lack of emphasis on the importance of visually demonstrating alternatives by the discussion interface. However, all of the designers were aware of the importance of screenshots and other visual representations in explaining a design alternative. They mentioned that alternatives with visual representations have a better chance of receiving comments from community members. For instance DU2 said: "...wireframes or other visual stuff legitimize ideas. Because they're very memorable. [...] People comment on images more, or codes more than paragraphs."

One way to improve the discussion interface is to better emphasize the importance of alternatives and the significance of providing visual representations for them by adopting a more visual structure.

Confirming the importance of alternatives, our quantitative analysis revealed that 48% of the conversation is spent discussing alternatives (Figure 3). Designers also mentioned alternatives as a vital piece of information when contributing to a design discussion. As DD1 said: "[When participating in a new discussion] I would want to know, in its current state, what is the exact problem the issue is trying to deal with, what are the proposed solutions so far, and what direction have people been taking on each of the proposed solutions."
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Distribution of alternatives over each combination of modality, aggregated across all messages (N = 1560).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Distribution of each coding category. Summing the categories for Alternative, Criterion, Implementation, and Clarification indicates that 48% of all discussion was devoted to alternatives.</figcaption></figure>

However the current systems do not support tracking the proposed alternatives. As DU3 said: "*They (bug reports and mailing lists) aren't good for keeping track of all of the ideas. [...] There usually isn't anybody keeping track of these are all the possible options and these are some [discussions] about each option.*" Today, the only way a participant can maintain awareness of the proposed alternatives is to rely on their memory, maintain notes in a separate tool, or scan the entire discussion thread to review the alternatives and rationale for and against each of them. The first method is unreliable as human memory is fragile and has a limited capacity [6]. The latter two methods are cumbersome and do not promote shared awareness among participants.

One of the consequences of using an interactive forum for managing a design discussion is that many of the proposed alternatives get buried in the midst of the discussion. As DU4 said: "*Some ideas that people have are actually really good, but then they kind of get lost in the thread...*" This means that not all alternatives will be evaluated thoroughly or may be forgotten during a lengthy discussion.

<figure><img src="image_6.png" /><figcaption>**Figure 4.** An event timeline for five complete threads of interface design discussion from the Drupal project. The timeline shows the fluidity of topics along with visual indicators of when new alternatives were posted and when new community members joined the discussion thread.</figcaption></figure>

Figure 4 demonstrates some of these challenges by showing an event timeline for five complete threads of interface design discussion from the Drupal project. Though we only show five, the other threads analyzed exhibited similar patterns. The timeline
shows the fluidity of topics (i.e. whether each message continues the topic of the previous message or changes it) along with visual indicators of when new alternatives were posted and when new participants joined the discussion. This figure offers interesting insights about the flow of conversation in the interface design discussions.

For example, within these threads, the proposal of design alternatives is distributed throughout the discussion, rather than batched at the very beginning reminiscent of commonly applied face-to-face brainstorming techniques. This may be a consequence of the distributed nature of the discussion in time and space as well as its integration with an issue management system. It may also reflect the fact that participants are able to generate and submit new alternatives as a function of the ongoing discussion. However, an important consequence of having the alternatives distributed throughout the discussion is that some may be overlooked or even fade from community memory. As DU3 explained: "Even if 20 or 30 ideas get generated during the mailing list discussion, a few days on people will be discussing [only] one or two which might be the worst ones because they might be the most controversial."

A related pattern is that the topic of the design discussion changes frequently. For example, in the first thread, there is a topic change after A3 (Alternative 3), A6, A8, and A11. Because of these topic changes there may be a reduced chance for these alternatives to be evaluated. In this case, A8 and A11 didn't receive any comments, and A3 only received one. Indeed, the lack of structure in the discussion sometimes prompts a participant to write a summary of the discussion to date, including the alternatives and opinions of those alternatives. One such summary is called out in the first thread in Figure 4. In this case, the participant wrote the summary mainly to compensate for the lack of awareness in the system. Twelve ideas were proposed and people were struggling with deciding which one works best. The summary reminded them of the goals and the description of each alternative.

<figure><img src="image_7.png" /><figcaption>**Figure 5.** The timeline from Figure 4 showing only the introduction of alternatives and patches.</figcaption></figure>

Another interesting pattern is that the majority of the participants joined during the first half of the discussions, but continued to join throughout. As the discussion grows, those who join later or otherwise do not keep up must review the messages to track the alternatives, the arguments for and against them, and the current consensus of the other participants. The common method for acquiring this information is to (re-)read the discussion to date. But, since this is time consuming, some participants will
post irrelevant comments that hinder the flow of the discussion. As DD1 said: "one thing that gets very frustrating in this, it gets very frustrating when I'm involved in a long discussion and have been for the whole time, someone will often come in and just kind of jump in to the discussion and either drill it and say "Oh, this is such a great discussion I also noticed this other problem with this other issue or this other thing" and people will go off on a tangent for two weeks talking about this other thing and we've gotten away from what the core issue is which is can be frustrating."

To avoid losing bright ideas and to have a more organized discussion, the current systems should support better tracking of ideas. Highlighting the alternatives and connecting the messages that reference them can greatly aid participants. It could also reduce the time required to identify, compare, and consider the alternatives without having to sift through all of the textual comments in the discussion thread.

## 4.2 Integration of UI Design and Development Activities is Essential

Designers and developers currently participate in the discussions through a centralized issue management system. This centralized venue helps designers in building trust and gaining merit by enabling them to interact with developers and exhibit their skills. Once they gain respect as a designer, they can more easily convince developers to implement their suggested improvements [5, 23, 27].

Integrating design and development activities also helps designers collaborate with developers. This collaboration is necessary for designers to receive feedback on the feasibility of their desired improvements [17] and for developers to be advised on the interaction design of their implementation. This iterative process of interface design and development is visible in current discussion threads. As shown in Figure 5 the proposed alternatives and submitted patches are distributed throughout the discussion threads, where each alternative is usually followed by a number of patches.

Any new interface for supporting interface design activities online should be fully integrated into the respective issue management systems. This will allow designers and developers to build mutual trust and collaborate more effectively.

## 4.3 Participants Need to Be Aware of Others' Opinions Regarding Alternatives

Discussion participants typically demonstrate agreement by writing “+1” for a favored alternative or they simply state that they like the idea. In order to determine the current direction and the favored ideas, participants must read through the messages. Another option would be to ask others to clarify the current direction.

However, participants may have inconsistent perceptions about the direction of the issue. As DD1 said: "... It's often hard to figure out what is the current direction. That's definitely hard to do. Often it takes getting someone to clarify it. And not everyone would clarify it the same way. If there are two people, and they're each kind of pushing their own ideas, within a Drupal issue...and you were to go on IRC and ask each of them individually, "So what's the current direction?", you'll get two very different answers."
The current issue management systems lack a formal way of expressing one’s preferred idea and visualizing others’ preferences. The absence of a mechanism to share opinions can hinder the consensus building process. Today, the consensus building process can be lengthy and it can be difficult to determine whether consensus has been reached at all. As DU3 said: “*People can keep on arguing the point, long after the decision was made [...] The nature of the way that many online discussions work is that they let the discussion continue [indefinitely]. That’s the main difficulty.*”

Implementing a mechanism to share preferences and formalize the consensus building technique (e.g. a voting system) may help facilitate the decision-making process. Also, it will be effective to highlight the alternative that has the consensus so far. Bringing the consensus to light can help developers determine which alternatives need to be implemented to improve the project or further inform the discussion.

<figure><img src="image_8.png" /><figcaption>**Figure 6.** The main screen consists of an interactive visual timeline, highlighting alternatives and offering an abstract summary of the comments. (a) The timeline shows the chronological order of comments and alternatives. (b) The alternatives are shown in callouts so designers can easily track them. (c) The comments are represented by a rectangle whose width corresponds to the length of the comment. The rectangles are colored based upon their affective tone.</figcaption></figure>

## 5 IdeaTracker

We describe how we translated our implications into the implementation of an interactive visualization tool for reviewing online interface design discussions - IdeaTracker. Our tool can be used by designers and developers participating in a design discussion as well as facilitators who may join a discussion to facilitate the decision-making process. Our system was developed through an iterative design process, starting with four different prototypes that addressed the challenges identified in our study. An informal user study was conducted on these prototypes. For the study, each prototype was seeded with data from an actual design discussion. Four users representative of our target audience were recruited and asked to perform similar tasks (e.g., identify the idea that reflects community consensus) with each
prototype and the existing interactive forum interface. The users were then asked to explain the strengths and weaknesses of each prototype. From the results, we implemented our prototype of IdeaTracker. We first discuss the main interface components of our system and then illustrate its value through a user scenario. All of the figures illustrating the use of our system are based on data imported from an actual design discussion in Drupal where participants are proposing and debating alternatives for a revised password checker. To facilitate use and learning of the interface, all interactive controls in IdeaTracker have a tooltip which explains their functionality. For the visual elements, the user can access a short description of each element via a context menu.

## 5.1 Tracking Alternatives

IdeaTracker's main screen consists of an interactive visual timeline, highlighting the alternatives proposed in the current thread of discussion and offering an abstract summary of the posted comments (Figure 6). The timeline illustrates when a specific comment or alternative has been posted (Figure 6a). This timeline allows participants to gauge the amount of activity that has occurred within a specific timeframe as well as the overall progress and pace of the discussion. The alternatives are shown in separate callouts so designers can easily identify and track them (Figure 6b). If an alternative has an attached screenshot, the screenshot is shown in the callout; otherwise, the first few sentences describing the alternative are shown. All other comments are represented by a thin rectangle, with the width of the rectangle corresponding to the length of the comment (Figure 6c). The comments are colored based upon their affective tone. If a comment has a negative tone, the rectangle is colored red. If it has a positive tone, the rectangle is colored green. If the comment has both positive and negative words, then it is colored yellow. The number of negative and positive words is computed by looking up each word in a commonly used dictionary. This color coding allows designers to quickly assess the community opinion of a certain alternative. Designers can easily skim through the comments related to a specific alternative without having to read the text of each message. To aid in exploring alternatives and the comments regarding each alternative, three interaction mechanisms have been implemented:

* **Expand Alternatives and Comments:** The user can select the expand/collapse button next to each alternative and read the entire post explaining the proposed alternative (Figure 7). Also, hovering over each comment representation will open a window containing the first few sentences of that comment (Figure 8).

* **Filter Unrelated Comments:** To examine the comments related to a particular alternative, designers can press the filter button next to the alternative. This dims all of the representations that do not reference this alternative (Figure 9). This interaction isolates the pros and cons of an alternative pointed out by other designers. It also aids in detecting the alternatives that have received insufficient or controversial discussion.

* **Link to the Original Post:** The user may want to read the original post corresponding to a comment or an alternative. To make this interaction possible, a link is provided in both expanded versions which redirect the user to the original post
corresponding to that particular alternative or comment. Also, the title of the issue at the top of the main screen links to the original discussion thread.

<figure><img src="image_9.png" /><figcaption>**Figure 7.** The user can press the expand button to reveal the entire post explaining the proposed alternative.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 8.** Placing the cursor over a comment's representation will open a window containing its first few sentences.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Figure 9.** Users select 'filter' to dim comments unrelated to the alternative. Here selecting filter for the alternative in the top left shows one message referencing it in context of the discussion.</figcaption></figure>

## 5.2 Comparing Alternatives

To compare different ideas, IdeaTracker offers a comparison view. Users select the ideas they would like to compare by clicking on the check-box at the bottom of each idea. Then, selecting the compare link will redirect them to the compare view (Figure 10). This view shows a timeline for each idea and the representations of comments referencing those ideas are available on the timelines for comparison. To provide users with a reference point for comparison, all of the comments are shown under each idea. But, the comments that are not related to a particular idea are dimmed.

## 5.3 Voting System

A voting system has been implemented in IdeaTracker to aid designers in promoting and reaching consensus. The number of votes for each alternative is shown on the
vote button next to the alternative. Hovering over the vote button will show the list of people who voted for the idea. The user can vote for an alternative by clicking on the vote button. If the user clicks the vote button, the number of votes for that idea will increase by one and the vote button will be highlighted to indicate which idea the user has voted for. To synchronize IdeaTracker with the original issue, a comment will be automatically posted to the original issue on behalf of the user stating that the user favors that particular alternative. Conversely, if a user posts a comment using the common notion of "+1" for an alternative in the original discussion thread, the number of votes for that alternative will be updated in the IdeaTracker.

Through IdeaTracker each user can only vote for one idea. If a user votes for a different idea, her initial vote will be re-assigned to the new idea. This feature enables users to retract their votes if a better idea is proposed or an existing idea is refined. This way IdeaTracker reflects the participants' recent views about the proposed alternatives. To promote awareness of the current consensus, the idea with the highest number of votes is highlighted in the system (Figure 11).

<figure><img src="image_12.png" /><figcaption>**Figure 10.** The compare view shows a timeline for each selected alternative and the comments referencing them. All of the comments are shown under each alternative, but the comments unrelated to the particular alternative are dimmed.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Figure 11.** To promote awareness, the alternative currently with the most votes is highlighted.</figcaption></figure>

## 5.5 User Scenario

Bob is a UI designer who contributes to open source projects in his spare time. Looking through the usability issues in Drupal, he finds an issue about improving the usability of Drupal's password checker. He launches IdeaTracker and enters the URL corresponding to the discussion thread for that design issue.

He first wants gain awareness of the proposed alternatives and their pros and cons as pointed out by other participants. He quickly scans the list of alternatives highlighted on the main screen and notices the idea posted by Lisa that suggests borrowing the design of Google's password checker. Bob then expands the idea to read it in detail and selects the filter button to identify the comments that reference
that idea. He immediately notices a negative comment colored in red and hovers the
cursor over the comment to read it. The comment has been posted by Mark who
thinks that copying and pasting a design from Google will spoil Drupal's trade mark.
Bob agrees, so he continues to scan the alternatives to determine if someone else has
proposed a better solution. At the end of the discussion, he finds Anne's idea. Anne's
idea is a tweak of Lisa's idea. She suggests that instead of using red, orange, and
green to indicate a weak, medium, and strong password, they can use different shades
of green. Bob decides to compare Anne's and Lisa's ideas. He selects the compare
checkbox next to these ideas, selects the compare link, and is redirected to the
compare view. He reads the comments posted regarding each idea and compares
them. He decides that he likes Lisa's idea more than Anne's. He returns to the main
screen and votes for Lisa's idea. A comment is generated on Bob's behalf and posted
to the original discussion thread indicating that Bob is in favor of Lisa's idea.

In contrast, using only the current interface, Bob needs to review each comment to identify the alternatives and their pros and cons. He then either needs to use another tool to create a summary of the discussion or rely on his memory. Comparing the two alternatives would also be challenging because it is difficult to isolate the comments that specifically address only the desired alternatives. Finally, since there is no running tally of "+1" votes, it is difficult to identify the currently favored idea.

6 Implementation

IdeaTracker is fully implemented and its interface was written in ActionScript using Adobe Flex 3 interface framework. The software consists of two layers: the data and visualization layers. The data layer parses the collected data and translates it into an internal format understood by the visualization layer. The data layer receives the data in XML format. The XML data consists of a set of `<comment>` tags, and each `<comment>` tag should have `<author>`, `<content>`, `<date>`, and `<image>` tags.

When the user launches IdeaTracker to view a particular discussion thread, an
adaptor component parses the html source of the thread and converts it to the XML
format readable by the data layer. The data layer is independent of the html format
and only depends on the XML format. In order to apply IdeaTracker to design
discussions on other interactive forums, an adaptor component needs to be written
that translates the html source of that forum to the XML format readable by our tool.

The data layer processes the XML file to find all the posts. In order to find the
alternatives we use two heuristics: (i) we consider posts with image attachments as
alternatives and (ii) we consider posts that have been referred to by other posts as
alternatives. Our testing indicates that these heuristics accurately detect most
alternatives in a thread of discussion. In future work, discussion participants could be
allowed to insert a simple tag in their comment stating they are posting an alternative.

After detecting the alternatives, we use natural language processing techniques to
infer the comments related to each alternative and the tone of each comment. To find
the comments related to an alternative, we look for certain key phrases participants
commonly use to reference a comment, for instance: "#34" to refer to comment
number 34 or "@Lisa" to refer to the latest comment posted by Lisa.
In order to determine the affective tone of a comment, we find the number of negative and positive words in the comment using standard online dictionaries. Based on the percent of negative and positive words, we assign respective values to that comment, which is used by the visualization layer for color coding.

# 7 Preliminary Evaluation

We conducted a qualitative evaluation to assess design choices and gauge initial user reactions to IdeaTracker. The evaluation was performed using the implementation of our tool as described in the previous section. It involved eight designers and six developers who actively contribute to interface discussions in Drupal and Ubuntu. The evaluation started with an introduction to IdeaTracker and a demonstration of its main features. Afterward we asked participants about their perceptions of the overall direction and different features of the tool (e.g. what do you think about showing the ideas in separate callouts, providing an abstract visualization of the comments, and using color codes for the affective tone of the comments?) and encouraged them to respond openly. Each session lasted about thirty minutes.

Overall, the participants reacted positively to IdeaTracker. All of the participants appreciated the visual separation of ideas from the other comments and being able to filter comments related to a particular idea. For example, one Drupal designer said: "The most useful feature to me is the callout of the major ideas that cuts through all lot of the crufty comments" while another said "I like this sort of compressing it... here is some big comments, here is a bunch of small comments, and if they are generally in favor or not, sort of at a glance as an overview is very cool." Most of the participants appreciated having access to an abstract visualization of the comments and felt we were using reasonable decision rules for identifying ideas and filtering comments. Participants also appreciated the fact that IdeaTracker was seeking to complement the existing issue management systems rather than trying to replace them.

The evaluation also highlighted several opportunities for improvement. For example, some of the participants were unsure of the utility of the idea-centric comparison view. Instead they preferred the ability to filter comments based on user id, thereby allowing them to see the comments that one user made across all of the ideas. Participants also expressed that the content of the negative and positive arguments for an idea was more important than the number of votes. For example, as one Drupal designer said: "Often there are issues though where an idea has lots and lots of "likes" until one person discovers why it shouldn't be done..." It may therefore be useful to extract the arguments for and against an idea and represent them within the main visualization. Most of the designers were concerned about accuracy of coloring comments based on affective tone and suggested to color comments based on their type (e.g. code review or patch). Participants also asked for more information to be included in the visualization of each comment (e.g. who posted the comment).
# 8 Conclusions and Future Work

With the emergence of open source software and the geographic distribution of many design teams, more interface design discussions are occurring online. The discussions are carried out using typical interactive Web-based forums, which lack support for the unique nature of design discussions. This paper makes two contributions. First, we conducted a study examining the challenges faced by participants when using the current interactive forums. The study included analyzing the discussion content from two popular open source projects and conducting interviews with active participants in these projects. From the results, we identified key challenges of using these types of systems and implications for how they could be addressed. Second, we built a new interactive visualization tool called IdeaTracker to demonstrate how to manage online design discussion more effectively. The interface offers specific support for tracking alternatives, comparing alternatives, and building consensus, all of which were identified as significant challenges in our study but which are not supported by existing interactive forum models. The interface was also implemented such that it can collect actual data from existing discussion threads for several open source projects and can be adapted for others.

We see at least three promising directions for future work. First, we want to conduct a longitudinal study to compare the impact of our system on the process and outcomes of design discussions relative to the use of existing interfaces. Second, we would like to test different techniques for building consensus, for example, voting for versus ranking the ideas. Finally, we would like to integrate more sophisticated machine learning techniques to better identify key elements of the design discussions such as identifying the alternatives and the affective tone of a discussion.

# References

1. *Axure*. Available from: http://www.axure.com.

2. *Balsamiq*. Available from: http://www.balsamiq.com.

3. *Pencil*. Available from: http://www.evolus.vn/pencil.

4. Arias, E., et al., Transcending the individual human mind - creating shared understanding through collaborative design. *ACM TOCHI, 7*(1): 84-113.

5. Bach, P.M., R. DeLine, and J.M. Carroll, Designers wanted: participation and the user experience in open source software development. *Proc. ACM Conference on Human Factors in Computing Systems*, 2009, 985-994.

6. Baddeley, A.D., Working memory. *Science*, 1992. 255: p. 556–559.

7. Barcellini, F., et al., *A Study of Online Discussions in an Open-Source Software Community*, in *Communities and Technologies 2005*, P. Besselaar, et al., Editors. 2005, Springer Netherlands. p. 301-320.

8. Benson, C., M. Muller-Prove, and J. Mzourek, Professional usability in open source projects: GNOME, OpenOffice.org, NetBeans, in *CHI '04 extended abstracts on Human Factors in Computing Systems*, 2004, 1083-1084.

9. Brown, D. and B. Chandrasekaran, *Design problemsolving: Knowledge structures and control strategies*. Morgan Kaufmann, 1989.
10. Deshpande, A. and D. Riehle, *The Total Growth of Open Source*, in *Open Source Development, Communities and Quality*, B. Russo, et al., Editors. 2008, Springer Boston. p. 197-209.

11. Everitt, K.M., et al., Two worlds apart: bridging the gap between physical and virtual media for distributed design collaboration. *Proceedings of the ACM conference on Human Factors in Computing Systems*, 2003, 553-560.

12. Faridani, S., et al., Opinion space: a scalable tool for browsing online comments, *Proceedings of the ACM Conference on Human Factors in Computing Systems*, 2010, 1175-1184.

13. Frishberg, N., et al., Getting to know you: open source development meets usability, *CHI '02 extended abstracts on Human Factors in Computing Systems*, 2002, 932-933.

14. Janssen, J., et al., Online visualization of agreement and discussion during computer-supported collaborative learning, *Proc. Conference on Computer Supported Collaborative Learning*. 2007, 314-316.

15. Liapis, A., Synergy: a prototype collaborative environment to support the conceptual stages of the design process, *Proc. International Conference on Digital Interactive Media in Entertainment and Arts*. 2008, 149-156.

16. Muller-Prove, M., Community experience at OpenOffice.org. *Interactions*, 2007. 14(6): p. 47-48.

17. Muller, M.J., Retrospective on a year of participatory design using the PICTIVE technique. *Proceedings of the ACM Conference on Human Factors in Computing Systems*, 1992, 455-462.

18. Myers, B.A. and M.B. Rosson, Survey on user interface programming, *Proc. ACM Conference on Human Factors in Computing Systems*, 1992, 195-202.

19. Nicholas, D.M. and M.B. Twidale, The usability of open source software. *First Monday*, 2003. 8(1).

20. Nicholas, D.M. and M.B. Twidale, Usability processes in open source projects. *Software Process: Improvement and Practice*, 2006. 11(2).

21. Olson, G.M., et al., Small group design meetings: an analysis of collaboration. *Hum.-Comput. Interact.*, 1992. 7(4): p. 347-374.

22. Sack, W., Conversation map: a content-based Usenet newsgroup browser, *Proc. ACM Conference on Intelligent User Interfaces*, 2000, 233-240.

23. Terry, M., M. Kay, and B. Lafreniere. Perceptions and practices of usability in the free/open source software (FoSS) community. *Proc. ACM Conference on Human Factors in Computing Systems*, 2010, 999-1008.

24. Thomas, M.P. *Why Free Software has poor usability, and how to improve it.* 2008; http://mpt.net.nz/archive/2008/08/01/free-software-usability.

25. Twidale, M.B. and D.M. Nichols, Exploring Usability Discussions in Open Source Development, *Proceedings Hawaii International Conference on System Sciences - Volume 07*. 2005, IEEE Computer Society. 198.3.

26. Vora, P. and M. Helander, *A review of design models and a proposal for a taxonomy of design*. M.Helander & M.Nagamachi (Eds.), Design for manufacturability: A systems approach to concurrent engineering and ergonomics, 1992: p. 78-90.

27. Zilouchian, R., M. Twidale, and K. Bongen, *Open Source Interface Politics: Identity, Acceptance, Trust, and Lobbying*, U.o. Illinois, Editor. 2010.