# Towards a Semantic Modelling Framework in Support of Multimodal User Interface Design

Elena Tsiporkova, Tom Tourwé, Nicolás González-Deleito

► To cite this version:

Elena Tsiporkova, Tom Tourwé, Nicolás González-Deleito. Towards a Semantic Modelling Framework in Support of Multimodal User Interface Design. 13th International Conference on Human-Computer Interaction (INTERACT), Sep 2011, Lisbon, Portugal. pp.636-639, 10.1007/978-3-642-23768-3_101 . hal-01596925

HAL Id: hal-01596925

https://inria.hal.science/hal-01596925

Submitted on 28 Sep 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Towards a Semantic Modelling Framework in Support of Multimodal User Interface Design

Elena Tsiporkova, Tom Tourwé and Nicolás González-Deleito

Sirris – Software Engineering & ICT Group
A. Reyerslaan 80 – 1030 Brussels – Belgium
{elena.tsiporkova,tom.tourwe,nicolas.gonzalez}@sirris.be

**Abstract.** In this paper, we propose a semantic modelling framework to capture the available domain knowledge in the field of multimodal interface design and to support designers in their daily design tasks.

## 1 Introduction

Parallel with the evolution towards multimodal applications, the research toward the establishment of formal principles and guidelines for multimodal interaction design is gaining increasing interest and importance in recent years (e.g. [1,3,4]). However, as observed by Sarter [5], the existing guidelines mostly focus on high-level design objectives and do not provide support on how to map them to the needs of an actual application. They do not capture and reflect on the considerable practical experience and valuable expert knowledge that interface designers rely on during their daily activities. Moreover, a considerable gap exists between the theory (formal guidelines) and the practice of multimodal human interface design, as different experts might approach the same interface design tasks in different ways based on personal expertise, background and intuition.

Our aim in this article is to work toward bridging this gap via the application of semantic technologies (e.g. ontologies) for capturing the available domain knowledge in the field of multimodal interface design. There are several advantages associated with such an approach: it guarantees a uniform approach across different designers within the same organisation, allows for semantic inter-usability of the formal guidelines across different applications and domains, and is open to allow for knowledge evolution and growth.

## 2 Multimodal User Interface Design: Current Practices and Challenges

A recent survey [1] on the current state-of-the-art of multimodal interfaces covers exhaustively the foundations and features of multimodal interaction, current developments in modelling languages and programming frameworks, and existing principles and guidelines for multimodal interface design. Several authors worked on
establishing formal principles for multimodal user interface design. Sarter [5] reviewed the existing design guidelines for multimodal information presentation, approaching the problem from the point of view of main decisions and considerations involved in multimodal interface design. She identified four themes of guidelines: 1) the selection of modalities; 2) mapping modalities to tasks and types of information; 3) the combination, synchronisation and integration of modalities; 4) adaptation of multimodal information input and presentation to accommodate changing task contexts and circumstances.

Sarter argued that most shortcomings in the aforementioned guidelines are due to
the fact that there remain a substantial number of open research problems in the area
of multimodal interaction design. However, we consider that guidelines resulting
from research would always remain of a rather conceptual and less empirical nature
and thus of little practical use. In our opinion, an extensive intuitive and empirical
knowledge base exists already within the designer community, and the challenge is to
develop methods, formal languages and frameworks that allow for capturing and
exploiting this knowledge. The semantic modelling framework proposed in this work
is an initial attempt in this direction. The selected approach is based on Sarter's insight
of considering the interface design process from the perspective of the decision
dilemmas designers face daily when executing their design tasks.

3 Semantic Modelling Framework

Our semantic modelling framework allows us to capture general *domain
knowledge* and *expert knowledge*, the latter is referring to Human-Computer
Interaction (HCI) knowledge related to multimodal application design. Both domain
and expert knowledge are described via an ontology, a formal representation of
knowledge by a set of key domain concepts and the relationships between those
concepts.

## 3.1 Competency Questions and Domain Concepts

To determine the scope of an ontology and identify the questions that an ontology-
based knowledge repository should be able to answer, we need to state so-called
*competency questions* [2]. Subsequently, from these questions the information that
needs to be contained within the ontology, i.e. the concepts and the relationships
between concepts, can be derived.

We consider that at this stage of the research, our ontology should be able to
answer the following competency questions, which are inspired by the major themes
identified by Sarter [5] (see above): 1) Which input and output modalities are
available to the user of an application? 2) What are the different factors that affect the
use of particular input and output modalities? 3) What are the appropriate
(combinations of) modalities to support users in a particular task?
environment with limited noise, since the accuracy of voice technology is heavily
dependent on environmental noise conditions (e.g. background noise); 3) the
application is used in a private environment, since controlling an application by voice
might disturb surrounding colleagues; 4) the user only uses the application as a
support for his primary activities which require the use of both hands, i.e. the
application itself is not the primary focus of the user.

We model this by introducing a subclass VoiceInputApplication of Application,
representing applications that could use vocal commands, and by defining a necessary
and sufficient condition representing the above conditions as follows:

(has_access_to some VoiceInputComponent) and (used_in some PrivateWorkingSpace) and (usedwhile_performing some (PrimaryActivity that requires some BothHands))

4 Conclusions

This paper presents an initial attempt to formally model and exploit relevant
domain knowledge and expertise in support of selecting appropriate modalities during
the human machine interface design process. The work presented here is inspired and
results from our interactions with HCI practitioners. Besides further refinement of the
presented semantic modelling framework, future research includes considering other
competency questions that our ontology could support, expanding toward broad range
of working contexts and types of users, and supporting synchronisation of modalities
in time. All this research will be performed in close collaboration with HCI
practitioners.

**Acknowledgements.** This research is supported by the ARTEMIS JU and IWT (www.iwt.be) through the SMARCOS project (www.smarcos-project.eu), and by Innoviris (www.innoviris.be) through the VariBru project (www.varibru.be).

References

1. Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: A survey of principles, models and frameworks. Human Machine Interaction, pp. 3--26 (2009)

2. Gruninger, M., Fox, M., et al.: Methodology for the Design and Evaluation of Ontologies. In: Proceedings of the Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI. vol. 95 (1995)

3. Oviatt, S.: Multimodal interfaces. The human-computer interaction handbook: Fundamentals, evolving technologies and emerging applications, pp. 286--304 (2003)

4. Reeves, L., Lai, J., Larson, J., Oviatt, S., Balaji, T., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J., et al.: Guidelines for multimodal user interface design. Communications of the ACM 47(1), pp. 57--59 (2004)

5. Sarter, N.: Multimodal information presentation: Design guidance and research challenges. International journal of industrial ergonomics 36(5), pp. 439--445 (2006)