# Development of Usability-Criteria for the Selection Process of Document Management Systems

Antje Heinicke, Christina Bröhl, Ioannis Dokas, Katrin Walbaum, Jennifer

Bützler, Christopher Schlick

<figure><img src="image_1.png" /></figure>

## To cite this version:

Antje Heinicke, Christina Bröhl, Ioannis Dokas, Katrin Walbaum, Jennifer Bützler, et al. Develop- ment of Usability-Criteria for the Selection Process of Document Management Systems. 15th Human- Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.514-517, 10.1007/978-3-319- 22723-8_51 . hal-01610809

HAL Id: hal-01610809

https://inria.hal.science/hal-01610809

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Development of Usability-Criteria for the Selection Process of Document Management Systems

Antje Heinicke, Christina Bröhl, Ioannis Dokas, Katrin Walbaum, Jennifer Bützler, Christopher Schlick

Chair and Institute of Industrial Engineering and Ergonomics of RWTH Aachen University
Bergdriesch 27, 52062 Aachen Germany

**Abstract.** As the overload of digital information in the SMEs requires an adequate management, document management systems (DMS) increasingly gain significance since they enhance the automation of processes within documents' registration, classification, processing, archiving and forwarding. However, in current selection processes of DMS the usability aspect seems to be unconsidered. For this purpose criteria are developed that measure the usability of existing DMS and allow the selection of a DMS according to usability aspects.

**Keywords:** document management, DMS, usability, software ergonomics, selection criteria

## 1 Introduction

The current scientific methods for software selection are not capable of representing the differences in the usability of the DMSs. The methods largely consider functional system requirements [1]. Objective criteria concerning the usability of DMS do not exist. Consequently, usability problems usually become visible only after the implementation. This study introduces an approach to determine objectively measurable usability criteria which can be applied by non-usability-experts for the evaluation and selection of DMS with regard to usability.

The evaluation of whether a system has a good or bad usability depends largely on the product itself and as well on the user and the context of use [2]. Accordingly, the identification of usability weaknesses of the existing DMSs, as well as the characterization of the users and their requirements was carried out in a first step [3]. Based on the analysis of requirements, a user test was performed to detect the relevant usability criteria.

## 2 Method

The study included 22 subjects aged 27 to 60 years (M=43.41, SD=10.9). All Participants had experience in document management, yet little or no experience in dealing with document management systems. Only novices where included in the analysis in order to find as many aspects as possible for the formulation of the criteria which
may not be important for experienced DMS users and DMS experts. To make valid statements, two commercial DMS were exemplarily used for the tasks: A system with a graphic layout very similar to Microsoft Office and a system with an individual user interface design were used.

The requirement-analysis indicated that the invoice receipt, a frequently used pro-
cess in many companies, is often supported by a DMS. Consequently, specific test
tasks, which are associated with this process, have been developed and tested.

## 2.1 Procedure

A training video was presented at the beginning of the test, providing an introduc-
tion to the field of document-management and a description of DMS' aim. This en-
sured that also novice users of DMS gain a certain degree of knowledge about the
features as well as the spectrum of uses and that all subjects had a similar knowledge
about DMS. The training video was followed by the processing of the three tasks:
Task type 1 included the searching and displaying of a filed document; task type 2
included the filing of a document in the DMS, while in task type 3 the participants
were asked to share and comment on a document. To minimize sequence effects, both
the sequence of task performance, as well as the sequence of the systems was permut-
ed. While working on the tasks, the participants were encouraged to think aloud
(Concurrent Thinking Aloud). After completing each task participants were asked to
reflect their experience with the DMS in form of a structured interview.

## 2.2 Formulation of Criteria

With the aid of the comments made during the task execution and during the inter-
views, the formulation of the usability criteria took place. For this purpose the com-
ments regarding the participants’ subjective experience, were systematically analyzed,
indexed and categorized with an expert evaluation. The first step hereto was the tran-
scription of the comments into a list, sorted by subject, system and task type. In every
comment, the key messages were highlighted and the comments were multiplied ac-
cording to the number of key messages. Additionally, the key messages were com-
pressed and assigned to the categories function, layout, wording, user guidance, sys-
tem feedback, to the characteristics positive or negative system feature, and to the
keywords depending on the type of task. Using the compressed key messages, the
formulation of criteria was carried out starting with the first key message. If the fur-
ther key messages could not be assigned to already existing criteria, a new criterion
was formulated. Regarding the formulation of criteria it was ensured that the criteria
can be evaluated with yes/no or available/not available.

## 3 Results

More than seventy usability criteria were formulated based on which the usability
of a DMS can be evaluated. The usability criteria are to be considered as DMS-
specific operating requirements that are related to the design of the entire user inter-
face or to individual elements of the user interface and to the interaction with the
system. The criteria show different characteristics, enabling various methods of eval-
uation and application.

**3.1 Criteria Classification**

For the specific use of the usability criteria in the DMS selection process, the crite-
ria were separated in two categories. Criteria related to characteristics of the entire
DMS user interface are of cross-functional nature (suitable for the specification
sheet). Criteria, concerning specific controls or properties of the main functions
"search", "import", " workflow" were correspondingly assigned as function related
characteristics. In the following four examples of the most frequently mentioned crite-
ria are listed (for the number of mentions, subjects were counted, whose comments a
criterion could be assigned to at least once):

- Cross-functional: Is a graphical representation of the filing structure available (e.g. in form of a tree structure)?

- Functional reference to "Search": Do document details (nature/ type of the docu-
ment) are displayed in a search results?

- Functional reference to "Import": Does the user receive a visual feedback on the successful filing of a document into the DMS?

- Functional reference to "Workflow": Are high priority tasks graphically highlight-
ed in the task list (e.g. by a special symbol or colored highlighting)?

In case of the comparison between an already reduced number of possible DMS
solutions, the usability of the main functions can be considered separately based on
the functional classifications. This allows, for example, the usability evaluation of the
search function separately from the usability of the workflow function. Thus, at the
DMS-selection it can be focused on the usability of the particularly relevant functions
for the user enterprise. Furthermore, the criteria could be characterized regarding their
affiliation to norm-based dialogue principles [4]. This allows a standardized evalua-
tion of a system’s operating requirements.

**3.2 Criteria list**

The developed criteria were listed in a catalog. A weighting of the criteria based on
the number of mentions was not made. The relevance of a criterion cannot be inferred
from the number of mentions. Moreover, frequent mentions could be caused by the
obviousness of a deficiency or a positive feature. The identified criteria primary con-
cern general system properties as well as the three main features search, import and
workflow. This fact is due to the type of tasks that were carried out. Therefore, the
catalog of criteria does not claim to be complete. With the help of project partners,
both DMS users and DMS provider companies, the set of criteria could be validated
in terms of relevance, practical application and comprehensibility. For this purpose,
the categorized catalogue of criteria was given to the users, expanded by an evalua-
tion option according to the specification sheet. Users should evaluate each criterion
as to whether it is critical for them, required, optional, or not required. A note below
each category intended to provide information on whether the review decision was
easy to make or not. If not, the users could name the criteria which were difficult to
evaluate. Furthermore, optional comments could be included. Providers also received
an expanded catalogue including the evaluation of whether a criterion is met, not met
or partially met by their own system. Furthermore, they were able to insert comments
when understanding problems occurred. On the basis of the comments made by both
the users and the providers, criteria were reformulated and redundancies as well as
irrelevant criteria were removed from the catalogue.

4 Summary and Outlook

In the DMS selection process, established procedures are applied that often neglect usability aspects. With the aid of user tests, over seventy objectively measurable usability criteria were formulated, which can add the factor of usability to the future selection process. Due to the classification in cross-functional and function-related criteria, these can be used in different phases of the DMS selection process.

5 References

1. Naß E, Scheibmayer M, 2011 3-Phasen-Modell zur DMS-Auswahl, Unternehmen der Zukunft, FIR-Zeitschrift für Organisation und Arbeit in Produktion und Dienstleistung, Schwerpunkt "Informationsmanagement", 12 Jg. Ausgabe 2/2011

2. Herczeg M. (2005) Software-Ergonomie. Grundlagen der Mensch-Computer-Kommunikation, 2. Auflage, München: Oldenbourg 2005

3. Heinicke, A.; Bröhl, C.; Bützler, J.; Schlick, C.: Usability of Document Management Systems Considering Users' Level of Experience: A Survey, In: Proceedings of the 6th International Conference on Applied Human Factors and Ergonomics 2014 (AHFE), Hrsg.: Ahram, T.; Karwowski, W.; Marek, T., The Printing House, Inc., Stoughton, FL, USA 2014, ISBN 978-1-4951-1572-1, S. 359-367

4. ISO 9241 1999 – 2011, Ergonomics of human-system interaction, Part 8, Part 110, Part 129, Parts 11-17, Part 171, Part 210, Beuth Verlag, Berlin 1999 – 2011

**Acknowledgements:** The joint research project uSelect DMS (01MU12018A) is supported by the Federal Ministry of Economics and Technology within the framework of the research program "SME-Digital" initiative "Simply intuitive - usability for SMEs" program.