# StoreAnt: A System to Support Finding Collaborative Systems Evaluation Methods

Marcella Souza, Lidia Silva Ferreira, Raquel Oiliveira Prates, Marília Lyra Bergamo

## ► To cite this version:

Marcella Souza, Lidia Silva Ferreira, Raquel Oiliveira Prates, Marília Lyra Bergamo. StoreAnt: A System to Support Finding Collaborative Systems Evaluation Methods. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.482-485, 10.1007/978-3-319-22723-8_43. hal-01610795

HAL Id: hal-01610795

https://inria.hal.science/hal-01610795

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# StoreAnt: A System to Support Finding Collaborative Systems Evaluation Methods

Marcella L. C. de Souza, Lidia S. Ferreira, Raquel O. Prates, Marília L. Bergamo

Federal University of Minas Gerais, Belo Horizonte, MG, Brazil
{marcellasouza, lidiaferreira, rprates}@dcc.ufmg.br,
marilialb@eba.ufmg.br

**Abstract** This paper presents StoreAnt, a virtual repository tool containing information about collaborative systems evaluation methods. It supports researchers and practitioners in finding and comparing information about methods, and identifying methods that comply to specific criteria (e.g. how the data is collected). The system is functional but has not yet been deployed publicly. Hopefully it will provide the HCI and CSCW communities with a valuable support regarding collaborative systems evaluation methods.

**Keywords** Collaborative systems evaluation methods; Repository, Groupware.

## 1 Introduction

Although collaborative systems are now integrated to people's everyday lives, designers of such systems still face challenges when developing them. One of the challenges involved is in evaluating these systems [2]. There have been many evaluation methods proposed specifically to or adapted to collaborative systems, however the great majority cannot be considered consolidated [5].

As a result, from time to time, researchers perform and publish a survey of existing methods (e.g. [4][5][6]). In these surveys, authors propose (usually based on the literature) criteria to classify and discuss existing collaborative methods. Antunes et al. [1] have gone a step further and proposed a framework to characterize collaborative evaluation methods that will support evaluators in comparing or choosing among them.

In spite of such efforts, identifying among the proposed methods which one would be (the most) appropriate in a given context may be a challenge. The surveys would allow evaluators to choose among the methods they have analyzed, but any updates – new methods or new empirical data about existing methods – would be left to the evaluators to identify and contrast to the work. In the case of the framework, although it supports evaluators in comparing methods of interest, the comparison would require the evaluator to identify the potential methods to be compared, as well as classify them according to the framework's proposed dimensions.

The goal of this work is to present StoreAnt, a tool developed to create a central repository in which information about collaborative evaluation methods can be stored, supporting researchers and practitioners in finding and comparing existing methods.
## 2 Goals and Requirements

StoreAnt was motivated by the challenge to identify and compare collaborative evaluation methods. To address this challenge, our solution was to create a collaborative information system that could be a repository for information for collaborative evaluation methods, as well as a tool that could support researchers or practitioners learn and compare about existing methods]. To achieve this goal, the requirements for StoreAnt were:

* Store information about each method: name, description, publications and tools;

* Provide a search tool according to different aspects of the methods;

* Allow the comparison of two methods;

* Allow researchers and practitioners to share their experience in applying a method;

* Maintain the method's base updated;

In order to search and compare methods it was necessary to classify methods in different dimensions. We chose to classify the methods using the dimensions used in dos Santos et al. [4] which adopted general terms usually known in the HCI field:

* **Source:** refers to the origin of the method, and each method can be classified as: *new* (new collaborative evaluation method proposed), *adapted* (the method exists for single-user applications but has been adapted to collaborative systems, or combines existing collaborative systems' methods); *existing* (the method has been proposed for single-user applications but can be directly applied to collaborative systems). Besides indicating the origin of the method, this criterion can also be an indicator of how consolidated it is.

* **Focus:** refers to what type of collaborative system the method is aimed at. Each method can be classified to *specific* (methods created for a specific domain or technology – e.g. collaborative learning) or *general* (apply to all collaborative systems independently of domain or technology).

* **Data collection method:** describes how the data is collected in the method: *inspection, observation in controlled environment, observation in natural setting, users' opinion* or *experiment measurements*. This information is useful for evaluators in considering the resources available, as well as application costs.

* **Moment of application:** refers to the moment of the systems development in which the method should be applied – *formative* (during the development), or *summative* (once the system is ready).

* **Type of analysis:** refers to what type of analysis the method yields – *quantitative* or *qualitative*. This dimension can be relevant for evaluators to identify methods that allow for generalization and comparison of data (quantitative) and those that explore motivations, experiences or interpretations (qualitative).

To support users in sharing their experiences and keeping the methods repositories updated, StoreAnt should be accessible from different locations, using different devices and operational systems. Therefore, the decision was to implement a web system.
# 3 StoreAnt

StoreAnt allows users to search for the methods by name or by choosing one or more of the classification dimensions. The resulting methods will be ranked and displayed according to how well rated it is by users (5 stars being the most well rated). For each method its name and short description will be displayed. The user can choose one of the methods to see its full description containing its name, a descriptive abstract, authors, resources (publications and links) and existing tools to support the method's application, as well as to have access to other user's comments. Fig. 1 depicts StoreAnt's screen for a full method description.

<figure><img src="image_2.png" /><figcaption>Fig. 1. - StoreAnt Method Description Screen</figcaption></figure>

The interface was designed to support users in identifying an appropriate method for the situation at hand. Therefore, the search tool and visual elements were chosen to convey an overall view of each evaluation method. The icons and graphics of visual interface elements were developed specifically for StoreAnt. The sketches were made using an open source application named Inkscape. As for the interfaces' implementation, front-end frameworks like Bootstrap and Jquery were used. The database structural organization and management was achieved using MySQL. Finally, the PHP language was used to develop the search engine and the system's data processing.

The system is currently functional, but not yet available for public use¹. StoreAnt already allows the registration of users and methods. The user can evaluate the registered methods by rating stars. The possibility to post comments has not been imple-

¹ Link: http://www.storeant.dcc.ufmg.br/.
mented yet. Preliminary informal evaluations of the interface have been conducted.
The next steps involve populating the database with the information about the meth-
ods described in the research surveys [4][5] and evaluating the systems.

4 Final Remarks

StoreAnt has been developed as a curation system [2], that is a system that will al-
low members to identify, organize and assess collaborative systems evaluation meth-
ods As soon as StoreAnt is deployed to the scientific community, we hope that re-
searchers will be interested in using it, as well as including the information about
methods they have proposed or used in the system. The success of the system depends
on it being adopted by the community (critical mass) which is crucial for it to be al-
ways updated. One of the factors that can positively influence its use is that members
can benefit from StoreAnt for different reasons – evaluators can be supported in find-
ing and comparing methods; researchers who have proposed methods will be able to
have a unique repository to share information and collect feedback about their meth-
ods; students, practitioners, or those new to the field can easily learn about different
methods. As people use StoreAnt more broadly, its interface and functionality may
also evolve based on feedback and comments by users. Finally, StoreAnt has initially
been developed for collaborative evaluation methods, but it could easily be used for
other methods or models.

**Acknowledgments.** Authors thank CNPq and Fapemig for partially funding their research.

References

1. Antunes, P., Herskovic, V., Ochoa, S. F., Pino, J. A. (2012). Structuring dimensions for collaborative systems evaluation. ACM Comp. Surveys (CSUR), 44(2), 8.

2. Duh, K., T. Hirao, A. Kimura, K. Ishiguro, T. Iwata, and C.-M.A. Yeung. Creating Stories: Social Curation of Twitter Messages. In *Proc. ICWSM '12*.

3. Grudin, J., Poltrock, S.: Computer Supported Cooperative Work. In: Soegaard, M. and Dam, R.F. (eds.) The Encyclopedia of Human-Computer Interaction, 2nd Ed. The Interaction Design Foundation, Aarhus, Denmark (2013)

4. Pinelle, D. Gutwin, C.: A review of groupware evaluations. In Proc. of 9th IEEE WETICE 2000. (2000)

5. Santos, N. S., Ferreira, L. S., Prates, R. O.: An Overview of Evaluation Methods for Collaborative Systems. In Proc. IX Brazilian Symposium in Collaborative Systems, pp. 127-135 (2012) (In Portuguese).

6. Wainer, J., Barsottini C. Empirical research in CSCW — a review of the ACM/CSCW conferences from 1998 to 2004. In Journal Of The Brazilian Computer Society. Volume 13, Number 3, 27-35 (2007)