<figure><img src="image_1.png" /></figure>

HAL
open science

# Interacting with signifier-less designs - the case of swhidgets

Nicole Ke Cheng Pong

<figure><img src="image_2.png" /></figure>

## To cite this version:

Nicole Ke Cheng Pong. Interacting with signifier-less designs - the case of swhidgets. Human-Computer Interaction [cs.HC]. Université de Lille, 2020. English. NNT: tel-03133122

HAL Id: tel-03133122

https://inria.hal.science/tel-03133122

Submitted on 10 Feb 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

Université
de Lille

École Doctorale Sciences Pour l'Ingénieur
Laboratoire CRISTAL UMR 9189

# INTERACTING WITH SIGNIFIER-LESS DESIGNS

– the case of swhidgets –

Thèse présentée par

NICOLE KE CHEN PONG

en vue de l'obtention du grade de docteur en informatique de l'Université de Lille

<table><tr><td>Sous la direction de :</td><td>Stéphane Huot</td><td>directeur</td></tr><tr><td></td><td>Nicolas Roussel</td><td>co-directeur</td></tr><tr><td></td><td>Sylvain Malacria</td><td>encadrant</td></tr></table>

Thèse soutenue le 21 octobre 2020
devant un jury composé de :

<table><tr><td rowspan="2">Rapporteurs</td><td>Nadine Couture</td><td>Professeure des universités, École Supérieure des Technologies Industrielles Avancées (ESTIA)</td></tr><tr><td>Kris Luyten</td><td>Full Professor, Hasselt University</td></tr><tr><td rowspan="2">Examinateurs</td><td>Caroline Appert</td><td>Directrice de recherche, CNRS</td></tr><tr><td>Laurent Grisoni</td><td>Professeur des universités, Université de Lille, président du jury</td></tr><tr><td></td><td>Sylvain Malacria</td><td>Chargé de recherche, Inria, encadrant de thèse</td></tr><tr><td>Directeur de thèse</td><td>Stéphane Huot</td><td>Directeur de recherche, Inria</td></tr><tr><td>Invité</td><td>Nicolas Roussel</td><td>Directeur de recherche, Inria</td></tr></table>
<figure><img src="image_5.png" /></figure>
Le cas d'un utilisateur confronté à une interface qui ne l'informe pas d'une possibilité d'interaction au moment où il en a besoin est un problème fondamental d'IHM. La présence de telles possibilités d'interaction *non-signalées* est fréquente dans les interfaces homme-machine modernes et potentiellement problématique, rendant nécessaire l'étude des interfaces et techniques d'interaction dites *sans signifiants*.

Un exemple de conception "sans signifiants" moderne est ce que j'appelle un *swhidget* pour “SWIpe-revealed HIDden WIDGET”: un composant d'interface normalement caché sous les bords de l'écran ou sous un autre objet, pouvant être révélé en le tirant à l'aide d'un geste de balayage selon une métaphore de manipulation physique. Les *swhidgets* sont des composants importants des interfaces de téléphones et tablettes à écran tactile, et sont la principale conception sans signifiant étudiée dans cette thèse.

En présence d'une conception sans signifiant, les utilisateurs peuvent être confus quant à ce qu'ils doivent faire pour atteindre leur but, ou être réduits à utiliser des méthodes sous-optimales parce qu'ils ne sont pas conscients de l'existence de meilleures alternatives. Il est donc généralement recommandé d'éviter de concevoir des interfaces sans signifiants. De telles interfaces sont pourtant courantes bien que les concepteurs soient conscient des problèmes qu'elles causent. Elles méritent donc une analyse plus approfondie, au delà du simple conseil de les éviter. En effet, il pourrait y avoir de bonnes raisons de concevoir des interfaces sans signifiants, qu'elles aient des qualités difficiles à mettre en évidence en l'état actuel de notre compréhension ou qu'il soit simplement impossible de les éviter.

Dans cette thèse, j'analyse les raisons pouvant inciter à la conception d'interfaces qui n'exposent pas clairement les possibilités d'interaction qu'elles offrent, en prenant les *swhidgets* comme objet d'étude principal. Pour cette étude initiale des *swhidgets*, je me concentre sur les points suivants : que sont les conceptions sans signifiants et quels aspects des *swhidgets* leur sont propres ? Les utilisateurs connaissent-ils les *swhidgets* de leurs systèmes ? Comment les ont-ils connus malgré l'absence de signifiants ? Quels avantages y a-t-il à ne pas avoir de signifiants ?

Les contributions de cette thèse sur ces points sont :

* Une definition des conceptions sans signifiants basée sur des observations de telles conceptions dans des interfaces.
* Une analyse des notions fondamentales requises pour la définition des conceptions sans signifiants : affordances, signifiants et sémiotique.

* Un modèle de la découverte et adoption par les utilisateurs de techniques d'interaction en général, reposant sur trois dimensions et leurs relations : les compétences et connaissances actuelles des utilisateurs, leur motivations, et les moyens par lesquels une interface peut informer ses utilisateurs.

* Les notions de degré de connaissance et de source de connaissance, dérivées de ce modèle, qui permettent d'évaluer expérimentalement à quel point les utilisateurs connaissent une technique d'interaction et comment ils l'ont découverte.

* La conception et les résultats de deux expériences sur les *swhidgets* d'iOS pour évaluer la connaissance qu'en ont les utilisateurs, comment ils les ont découvert, leurs éventuelles raisons de ne pas les utiliser, comment ils les perçoivent globalement et les intègrent dans leur façon de penser l'interaction. Ces études montrent que les *swhidgets* sont globalement appréciés et relativement bien connus, tout en laissant de la place pour des améliorations, surtout pour certains *swhidgets*.

Cette thèse ouvre des perspectives concernant le transfert de connaissances entre applications, la pertinence du concept d'expérience utilisateur pour la compréhension des *swhidgets*, et la possibilité de favoriser leur découverte lors de transitions animées entre vues.
<figure><img src="image_6.png" /></figure>
<figure><img src="image_7.png" /></figure>
At the heart of this thesis is a common but problematic situation that users of digital systems often face in their daily interactions: to interact with the system, they need some knowledge of an interaction possibility, some piece of information about the interface, but this information is not provided in the context in which they need it. I call such interaction possibilities *non-signified*, and *signifier-less designs* the interfaces and interaction techniques that rely on non-signified interaction possibilities.

An example of modern signifier-less design is what I call "swhidgets" for "SWIpe-revealed HIDden WIDGETS": widgets that are hidden under the screen bezels or other interface elements, out of view and not advertised by any graphical mark, but that can be revealed by dragging them into view with a swipe gesture relying on a physical manipulation metaphor. Swhidgets are an important component of touch-based smartphone and tablets interfaces, and will be the principal signifier-less design studied in this thesis.

When facing a signifier-less design, users may be confused about what they should do and how to achieve their goals; or they might have to use suboptimal ways of achieving their goals because they are unaware of the existence of more efficient options. It is thus usually advised to avoid signifier-less designs. Yet, despite designers' awareness of the problems they may cause, signifier-less designs are common in user interfaces. They thus deserve a deeper analysis than simply advising to avoid them in interface design. Indeed, there might be good reasons to apply this design: maybe they provide some benefits that are hard to see with our current understanding of these designs, or maybe there is no way to avoid them.

In this thesis, I study the question of why designers would create interfaces that do not clearly expose some of their interaction possibilities, taking the case of *swhidgets* as an example and focus of inquiry. As a preliminary work on *swhidgets*, I focus on the following questions: What are signifier-less designs and what aspects of *swhidgets* design make them unique? Do users know the *swhidgets* provided by their system? How did they get to know them despite their lack of signifiers? What are the benefits of not having signifiers in the design of *swhidgets*?

My contributions to these questions are:

* I define signifier-less designs and provide observations of this type of design in user interfaces.

* I provide an analysis of the fundamental notions required to define signifier-less designs: affordances, signifiers and semiotics.
* I propose a model of user discovery and adoption of interaction techniques in general, relying on three dimensions and their relationships: users' current knowledge and skills, users' motivations, and the design means of informing users provided by the interfaces.

* I propose the notions of Degree of Knowledge and Source of Knowledge derived from this model, that can be used in experiments to evaluate how well the participants know an interaction technique and how they discovered it.

* I present the design and results of two studies on iOS *swhidgets* that investigate how well users known them, how they discover them, their reasons for not using them, how they generally feel about them, and how they integrate them in the way they think about their interactions with the system. These studies revealed that *swhidgets* were globally appreciated and relatively well known by users, although there is still room for improvement, notably for some specific *swhidgets*.

I conclude with perspective for future works regarding the transfer of knowledge about *swhidgets* from one application to another, the persistence of considering all aspects of user experience to understand the design of *swhidgets*, and the possibility to increase the discoverability of *swhidgets* by using animated transitions between interface views.
<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /></figure>
Pong, Nicole Ke Chen (Aug. 2017). "Understanding and Increasing Users' Interaction Vocabulary." In: 29ème conférence francophone sur l'Interaction Homme-Machine. Ed. by AFIHM. Rencontres doctorales. AFIHM. Poitiers, France, 4 p. URL: https://hal.archives-ouvertes.fr/hal-01577901.

Pong, Nicole Ke Chen and Sylvain Malacria (2019). "Awareness, Usage and Discovery of Swipe-Revealed Hidden Widgets in IOS." In: *Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces*. ISS '19. Daejeon, Republic of Korea: Association for Computing Machinery, pp. 193-204. ISBN: 9781450368919. DOI: 10.1145/3343055.3359713.
<figure><img src="image_10.png" /></figure>
This dissertation would not have been possible without the support I
received from an extended community of colleagues, friends, families,
and people working in various places. I express here my sincere
appreciation to these people and organizations.

First of all, I would like to thank my advisors and jury members for
their sincere care, interest, and thoughtful feedback to the manuscript.
To my director Professor Nicolas Roussel - thank you for giving me
the opportunity to work with you as a student who came from more
of a design domain, I appreciate your guidance and your always-
heartwarming introduction of me in public events including my final
defense, I am very grateful of your thoughtful attention all along my
thesis. To my advisor Sylvain Malacria and my director Professor
Stéphane Huot, thank you both for your vast reserve of patience,
knowledge and long-term support in guiding me through this re-
search.

To Professors Nadine Couture and Kris Luyten, I am deeply grateful
to both of you for providing helpful and precious suggestions on
reviewing my dissertation. Your various pieces of advice are precious
for me and to this research, I shall carry them on. To Professors
Caroline Appert and Laurent Grisoni, thank you both for participating
in this journey right from the beginning of this thesis, for giving
me good reflections enlarging my vision from the first year of my
PhD (Caroline) and for participating in the experiment (Laurent).
My deepest gratitude for all your thought provoking questions and
suggestions during the defense.

I would also like to thank my colleagues from Team Loki and Team
Mjolnir, where I get to work with the best ambiance and the best
discussions: Marcelo, Ed, Géry, Fanny, Thomas, Mathieu, Karine,
Julie, Jonathan, Alix, Justin, Amira, Thibault, Christian, Sébastien,
Hakim, Axel, Mark, Philippe, Grégoire, Raiza, and all the remarkable
persons I have met in Inria Lille, Gwenaël Desfontaines, Anne Rejl,
Charlotte Hespel, Marion Blasquez, Alice De Raphaël, and especially
Marie-Agnès Énard and Caroline Van Hoy (who made possible the
interactive paper poster at IHM 2017). All your thoughtful help,
interesting discussions and hanging-outs have made working with
you these four years a great pleasure.

I would also like to extend my deepest gratitude to my beloved
family and friends around the world. To my family in Taiwan, US, and
France: my mum Yi-An and my brother Shuo-Hsiu Hsu for making
themselves very good examples for me to carry on in the field of
research and their countless support and encouragement alongside
all my studies in France; my sister Chia-Jen and my brother in law
Ted Ku, my dear cousins Christine and Lucia Hsu along with my aunt
in the States. Also to the Pillias family: Anne-Françoise, Camille and
my love Clément, thank you for your endless support. To my friends
around the world: To Can Liu, Jessalyn Alvina and Isabelle Sage, thank
you for your encouragements and warm discussions during the most
daunting moments of my PhD. To Shin-Yi Yu and Estelle Garenaux,
James Eagan and Intissar Abdelhak, thank you for all the support that
you gave me. To James JW Lin and Yung-wen Liu, thank you both
for your warm encouragement and your advice that accompanied me
through this thesis.

Much grateful thoughts to the persons who introduced me to the
fields of computer science and game design, and led my research
reflection in good direction: Professor Stéphane Natkin, Professor
Pierre Cubaud, and the very regretted Alexandre Topol. To Wendy
Mackay and Gilles Bailly, thank you for giving me a chance to present
my ideas and encouraging me pursuing research in HCI. Also my
appreciation to my colleagues and students in Asia University, Tatong
University and National Taiwan University of Science and Technology
in Taiwan: you made me realize how much I love teaching and this
was an important motivation for getting a PhD.

I am also deeply indebted to my friends Miowowo Kuo (萬磁木),
Chia-Chia Yang (楊家家), Chiafen Tsai, Olga Chang, Jenny Chang,
Natalia Yu and all the friends in the 六喵軒 group, who were an
endless source of what a PhD student needs the most: fun, cat, food
and drinks distractions. They have to be thanked for this as well as
for their dedicated and much more serious participation in the online
survey they made possible and valuable, altogether with the Reddit
members and administrators of /r/iphone and /r/ipad.

Finally, I want to thank my little « roi du monde », my cat Gucci,
who has accompanied me from my master study in France all the way
to my PhD, the best cat in the world.

<figure><img src="image_11.png" /></figure>
<figure><img src="image_12.png" /></figure>
<figure><img src="image_13.png" /></figure>
# CONTENTS

1 INTRODUCTION 3

1.1 Observations 3

1.2 A context favoring signifier-less designs 5

1.2.1 Diversification of interactive device types 5

1.2.2 Design philosophies 5

1.2.3 Lack of knowledge about how users adopt signifier-less designs 7

1.3 Approach and contributions 9

1.3.1 Approach 9

1.3.2 Contributions 9

1.4 Plan of the dissertation 10

2 SWHIDGETS: DESCRIPTION, ANALYSIS, AND RESEARCH QUESTIONS 15

2.1 Types of Swhidgets 15

2.2 Roles of Swhidgets 17

2.3 Do users interact with Swhidgets? 19

2.4 Voluntarily signifier-less: why? 20

2.5 Discovery of Swhidgets 21

2.6 General research questions identified 22

2.6.1 Integration of swmidgets in relation with other interface features 22

2.6.2 Interaction with specific swmidgets 23

2.6.3 Discovery and adoption of Swhidgets in the long-term 24

2.6.4 Methodological aspects of the study of Swhidgets 26

2.7 How I Address These Research Questions 27

2.7.1 What is a signifier-less design and how *swhidgets* are different from other signifier-less designs? 27

2.7.2 Do users know *swhidgets*? 28

2.7.3 How can users know *swhidgets* despite their lack of signifiers? 28

2.7.4 What are the benefits of not providing signifiers? 29

# I DISCOVERING INTERACTION POSSIBILITIES

3 AFFORDANCES 35

3.1 Gibsonian affordances 36

3.1.1 Origin of the concept in perception theory 37

3.1.2 Affordances and ecology 38
3.1.3 Tools and learning: Affordances change for equipped actors 40

3.1.4 Perception of affordances is learned 40

3.2 The role of affordance structures in HCI 41

3.2.1 Means-end relations 41

3.2.2 The problem of unclear relations between affordances 42

3.2.3 Instrumental affordances 43

3.3 Debated Aspects of Affordances in HCI 45

3.4 Debate 1: Meaningfulness of Affordances 46

3.4.1 Norman's view: affordances as action possibilities only 46

3.4.2 Meaningfulness as potential 47

3.4.3 Norman's view: Meaning is conveyed by signifiers 47

3.4.4 Meaningfulness from complementarity 48

3.5 Debate 2: Social Aspects of HCI Affordances 50

3.5.1 Norman's view: Affordances, conventions and symbols 50

3.5.2 Gaver's view: Culture only highlights some affordances 52

3.5.3 Gibson's notion of Ecology 53

3.6 Debate 3: Symbols and Affordances in Human Displays 54

3.6.1 Symbols involved in affordances 54

3.6.2 From Gibson's human displays to Norman's signifiers: affording knowledge 55

3.7 Conclusion and recap 57

4 THE SEVEN STAGES OF ACTION 61

4.1 The Seven Stages 61

4.2 Limitations and ambiguities of the model 65

4.2.1 Status of the model and nature of the stages 65

4.2.2 Distinction between intentions and sequences of actions 66

4.2.3 Focus on problem-solving rather than discovery 67

4.2.4 Focus on information used in short-term interaction 69

4.3 Conclusion 70

5 SIGNIFIERS & DESIGN MEANS 73

5.1 Perception of design means 73

5.1.1 Affordances as output of perception 73

5.1.2 Sensory affordances 74

5.1.3 Gestalt Laws 76

5.2 Definition of Signifiers in Peirce's semiotics 78
<table><tbody><tr><td>5.2.1</td><td>Components of a Sign: Signifier, Object, and Interpretant</td><td>79</td></tr><tr><td>5.2.2</td><td>Combination of Signs</td><td>80</td></tr><tr><td>5.2.3</td><td>Nature of the signifier-object relation</td><td>80</td></tr><tr><td>5.3</td><td>Interpretation of design means</td><td>82</td></tr><tr><td>5.3.1</td><td>Perceived affordances</td><td>82</td></tr><tr><td>5.3.2</td><td>Labels</td><td>82</td></tr><tr><td>5.3.3</td><td>Interface Metaphors</td><td>83</td></tr><tr><td>5.3.4</td><td>Natural Mappings</td><td>83</td></tr><tr><td>5.3.5</td><td>Interaction Frogger</td><td>84</td></tr><tr><td>5.3.6</td><td>Natural Signals</td><td>86</td></tr><tr><td>5.3.7</td><td>Feedback and feedforward</td><td>86</td></tr><tr><td>5.3.8</td><td>Constraints</td><td>88</td></tr><tr><td>5.4</td><td>Evaluation of design means</td><td>89</td></tr></tbody></table>

## II SIGNIFIER-LESS DESIGNS

<table><tbody><tr><td>6</td><td>SIGNIFIER-LESS DESIGNS</td><td>97</td></tr><tr><td>6.1</td><td>Defining Signifier-less Designs</td><td>97</td></tr><tr><td>6.1.1</td><td>Edge cases and limitations of the definition</td><td>97</td></tr><tr><td>6.1.2</td><td>Directly signified affordances</td><td>99</td></tr><tr><td>6.2</td><td>Historical Signifier-less designs</td><td>101</td></tr><tr><td>6.2.1</td><td>Command-line interfaces</td><td>101</td></tr><tr><td>6.2.2</td><td>Signifier-less designs in Direct Manipulation</td><td>102</td></tr><tr><td>6.2.3</td><td>WIMP interfaces</td><td>103</td></tr><tr><td>6.2.4</td><td>Post-WIMP interfaces</td><td>106</td></tr><tr><td>6.3</td><td>Types of Signifier-less Design</td><td>108</td></tr></tbody></table>

## III PROPOSITIONS FOR A MODEL OF DISCOVERY

<table><tbody><tr><td>7</td><td>DISCOVERING AND ADOPTING AFFORDANCES</td><td>117</td></tr><tr><td>7.1</td><td>About this chapter</td><td>117</td></tr><tr><td>7.2</td><td>current knowledge and skills</td><td>118</td></tr><tr><td>7.2.1</td><td>Domain of the knowledge and skills</td><td>119</td></tr><tr><td>7.2.2</td><td>Improving knowledge and skills in time</td><td>122</td></tr><tr><td>7.2.3</td><td>Dependency across levels</td><td>123</td></tr><tr><td>7.3</td><td>Interaction between current knowledge and design means</td><td>125</td></tr><tr><td>7.3.1</td><td>Current understanding affects the usefulness of means</td><td>125</td></tr><tr><td>7.3.2</td><td>Learnability of the interface</td><td>128</td></tr><tr><td>7.4</td><td>Motivations</td><td>130</td></tr><tr><td>7.4.1</td><td>Extrinsic vs. intrinsic motivation</td><td>130</td></tr><tr><td>7.4.2</td><td>Self-determination theory</td><td>131</td></tr><tr><td>7.4.3</td><td>Hassenzahl's model of UX</td><td>132</td></tr><tr><td>7.4.4</td><td>Non-motivational causes</td><td>134</td></tr><tr><td>7.5</td><td>Interaction between current knowledge and motivation</td><td>134</td></tr><tr><td>7.5.1</td><td>Knowledge motivated by performance</td><td>134</td></tr><tr><td>7.5.2</td><td>The Knowledge-motivation loop</td><td>137</td></tr></tbody></table>
7.6 First contribution: Degree of Knowledge 138

7.7 Second contribution: Sources of Knowledge 140

7.7.1 Distance of the source of knowledge 140

7.7.2 Intentionality of the discovery 142

7.7.3 Sources of Knowledge as an extension of Design Means 143

7.8 Conclusion 143

IV STUDIES

8 TWO STUDIES ON USERS' KNOWLEDGE AND RECEPTION OF IOS SWHIDGETS 151

8.1 The laboratory and online surveys 151

8.1.1 Rationale and objectives 151

8.1.2 Scope limitation 153

8.1.3 Applications and operations studied 154

8.2 Laboratory Study 157

8.2.1 Factorial design of the experimental tasks 157

8.2.2 Experimental tasks 158

8.2.3 Participants 158

8.2.4 Material and Apparatus 158

8.2.5 Protocol 159

8.2.6 Data Collection 160

8.2.7 Data processing 161

8.3 Results of The Laboratory Study 162

8.3.1 Participants 162

8.3.2 Completion of Operations 162

8.3.3 PRIMARY usage of input methods 164

8.3.4 Degree of knowledge of Swhidgets vs. Navigation 164

8.3.5 PRODUCTIVE knowledge 166

8.3.6 Awareness and RECEPTIVE knowledge 167

8.3.7 Preference for swhidgets 169

8.3.8 How participants discovered swhidgets 169

8.3.9 Participants' reaction about swhidgets 170

8.4 Discussion of the Results of the Laboratory Study 171

8.4.1 Reflection on the study 171

8.4.2 When participants did not use swhidgets 171

8.4.3 Task-based vs. System-based Interaction Models 172

8.5 Study 2: Online Survey on Awareness, Usage and Discovery of swhidgets 173

8.5.1 Design and Procedure 173

8.5.2 Apparatus and Participants 174

8.6 Results of Online Study 174

8.6.1 Participants Background and Experience 174

8.6.2 General Usage 174

8.6.3 Awareness and Usage of Swhidgets 175
8.6.4 Discovery of Swhidgets 178

8.6.5 General perception of Swhidgets 182

8.6.6 Satisfaction of basic needs 183

8.7 Discussion of the results of the two Studies 185

8.7.1 Awareness of Swhidgets 185

8.7.2 Usage of (And Reasons Not to Use) Swhidgets 185

8.7.3 Discovery of Swhidgets 187

8.7.4 Future work 188

8.7.5 Dataset 189

V PERSPECTIVE AND CONCLUSION

9 PERSPECTIVES AND CONCLUSION 195

9.1 From results back to models 195

9.1.1 Transfer of Knowledge 196

9.1.2 Hedonic aspects of Swhidgets 198

9.2 Improving swhidgets 202

9.2.1 Revealing Swhidgets with Animated Transitions 202

9.2.2 Considerations for the Design of Animated Transitions 203

9.2.3 Proposed Animated Transitions 204

9.2.4 Testing the animations 207

9.3 Conclusion 207

VI APPENDIX

J FIRST QUESTIONNAIRE OF THE LABORATORY STUDY 218

K SECOND QUESTIONNAIRE OF THE LABORATORY STUDY 224

BIBLIOGRAPHY 226
<figure><img src="image_14.png" /></figure>
The principle of visibility is violated over and over again in
everyday things. In numerous designs crucial parts are carefully
hidden away. Handles on cabinets distract from some design
aesthetics, and so they are deliberately made invisible or left out.
The cracks that signify the existence of a door can also distract
from the pure lines of the design, so these significant cues are also
minimized or eliminated. The result can be a smooth expanse of
gleaming material, with no sign of doors or drawers, let alone of
how those doors and drawers might be operated. Electric switches
are often hidden: many electric typewriters have the on/off switch
in the rear, difficult to find and awkward to use; and the switches
that control kitchen garbage disposal units are often hidden away,
sometimes nearly impossible to find.

Many systems are vastly improved by the act of making visible
what was invisible before.

Donald A. Norman,
The Psychology of Everyday Things, 1988.

1.1 OBSERVATIONS

At the heart of this thesis is a common but problematic situation that users of digital systems often face in their daily interactions. This situation can be characterized simply like this: to interact with the system, users need some knowledge of an interaction possibility, some piece of information about the interface, but this information is not provided in the context in which the user needs it. Or, to paraphrase Norman: the information about a possible interaction is not "in the world" when it is needed and has therefore to be "in the head" of the user [Nor88]. In such situations, users may be confused about what they should do and how to achieve their goals; or they might have to use sub-optimal ways of achieving their goals because they are unaware of the existence of more efficient options.

The quote introducing this chapter comes from The Psychology of
Everyday Things by Donald Norman [Nor88], a book that certainly
had a great influence on the evolution of user interface design practice.
In particular, Norman advised to expose interaction possibilities by
making things visible. To do so, he discussed design concepts such as
*affordances* (design elements that directly convey interaction possibili-
ties to users), *mappings* (that create a similarity between an action and
its result), and *constraints* (that limit possible actions to the meaningful
ones). These concepts have been widely discussed by HCI researchers, who also proposed related notions such as *feedforward* (that informs about the purpose or consequence of an action) [Ver+13]. Norman finally proposed to simply call “signifiers” any design element of a system that conveys information about possible interactions with the system [Noro8]. Signifiers have become popular concepts for UI and product designers, taught in many popular introductory books to the field [Shn97; Wei11; LHB10; Coo+14; Spoo8]. Ignoring these concepts, with the consequence of hiding some interaction possibilities, would have been perceived as a design flaw by most designers for decades after the first publication of Norman’s book. I call *signifier-less* the designs that provide such non-signified interaction possibilities to users, which I will define more formally and study in this thesis.

Despite a general awareness by designers of the issues they can cause, non-signified interaction possibilities have been provided by the user interfaces of most systems in the past, and are still frequently used today. In desktop computer interfaces, signifier-less designs include: contextual menus revealed by clicking with the right mouse buttons on some objects (although nothing signifies which objects will provide specific entries in the menu); fast navigation keys such as using the tabulation key to navigate to the next UI element that can receive keyboard input, or using arrow keys with modifiers to move to the end of a word, line, or the end of the text; and hotkeys such as the famous Ctrl-C to copy, which are not signified when the user needs them, but only hinted in the application menu when users explicitly search for them or when they do not need the information anymore because they are actually using the menu to trigger the command. In the interfaces of handheld touch-based devices, short contextual menus may be triggered by a long press (or by pressing the screen harder), navigation in larger-than-screen lists and images is done with gestures such as pinch or flick but nothing in the interface informs users they can do so, and interface objects and screen bezels can be swiped to reveal controls that were hidden outside the screen. Even physical interfaces as simple and ubiquitous as a button may lack signifiers [Jan14]. For instance, we may understand that to get a coffee, we need to push the most prominent button on a coffee machine, and yet be unsure if the coffee will stop flowing when we release the button, when we press it again, or automatically after some predetermined time – of course, the same problem exists for virtual buttons.

As signifier-less designs are common in user interfaces despite designers' awareness of the problems they may cause, these designs deserve a deeper analysis than simply advising to avoid them. Indeed, there might be good reasons to use them: maybe they provide some benefits that are hard to see with our current understanding of these designs, or maybe there is no way to avoid them. In both cases, design-
ers lack guidelines to decide when to include non-signified designs
in the interfaces they create and how to minimize the problems it
may cause. This thesis aims at providing theoretical concepts and
user studies to better apprehend the issue of creating signifier-less de-
signs, focusing particularly on swipe gestures that reveal non-signified
widgets in touch-based handheld devices.

1.2 A CONTEXT FAVORING SIGNIFIER-LESS DESIGNS

The current industrial and scientific context favors the creation of
signifier-less designs in three ways that I will describe in the next
subsections: a diversification of interactive devices that put stronger
constraints on possible signifiers, the emergence of new design philoso-
phies in the last decade that puts usability concerns in new perspec-
tives, and a lack of publicly available data and knowledge about how
users adopt signifier-less designs.

1.2.1 *Diversification of interactive device types*

Today, with the broad availability and acceptance of devices like smart
watches, communicating furniture in "smart houses", wearable com-
puting, tangible computing toys, etc., designers face a diversification
of digital devices and have to exercise their design skills far from the
well-known model of the graphical interface on a screen which size
can be expressed in inches. Some devices have simply not enough pix-
els to display signifiers, others must be operated in environments with
a lot of disturbance or noise, or without even looking at the device.
While researches are being conducted to create new modalities for
signifiers, such as using device temperature or haptic clues, designers
may still have to rely on signifier-less designs.

With the popularity of handheld devices like smartphones and
tablets, designers are also incited to integrate in the applications
they create for these devices most of the features that are available
on the desktop computers versions of these applications. As these
devices have limited output resources to provide signifiers for all these
features, and since the techniques that provide hierarchies of features
– like menus – are not well suited to these devices, designers may
prefer to only signify the most basic interaction possibilities and use
signifier-less designs for the more advanced ones.

1.2.2 *Design philosophies*

Designers put a high value in the aesthetics of the products they create,
and while this is often misunderstood for a simple will to make things
look nice, designers' aesthetic sensibility actually encompasses many
aspects of the design, including usability concerns. Yet, what consti-
tutes a good design, including designers' perspective on usability, is often defined by higher-order design philosophies, which evolve with time. In the next paragraphs, I present three design philosophies that became popular in the last two decades and had a great impact on the emergence of some modern signifier-less designs.

**NATURAL USER INTERFACES (NUIS)** Touchscreen-based smartphones and tablets became mainstream after the release of the iPhone in 2007, and this popularity was built in part on advertising campaigns that praised the simplicity and “naturalness” of some gestures (mostly swipe and pinch), which in practice were not signified to the user.

These interactions were claimed to be “natural” in that they rely on skills that users already possess from interacting with the real world, and users should thus come up with such gestures by themselves without any additional “hint” [Jac+08; WW11]. In this sense, NUIs focus strongly on usability concerns, since they bring the premise that anybody can use an interface without training or even wondering how to do things. Some signifier-less designs may thus be the consequence of designers’ belief that the interactions involved in these designs are so obvious that they do not need to be signified to users, and that using them in the design will improve the usability of the interface.

Yet, despite many papers presenting new interaction techniques and interfaces claimed as “natural”, HCI researchers have also criticized the NUI design philosophy for the lack of evidence supporting its fundamental assumptions, from the lack of evidence that users effectively use spontaneously the interaction techniques claimed as natural and the lack of agreed methods to measure naturalness [AHN17], to the fact that some similarity between physical and digital interactions does not systematically imply that skills can be transferred from the physical to the digital world [BR15], and up to the fact that the skills in question may not be universal and innate but might instead be cultural constructs learned by people [Nor10; Hor13].

**FLAT DESIGN** The visual design of touch-based GUIs evolved in the mid-2010s towards the so-called “flat design”. Inspired by the Bauhaus design movement, and particularly the “international typographic style”, flat design follows a philosophy that seeks harmony between the function of some product and its design, rejecting the use of decorations or non-functional elements to create aesthetic value.<fnref n="1" />

In particular, it rejects skeuomorphic designs that create familiarity with a new product by mimicking aspects of other products although such features are non-functional and not strictly needed in the new product – which was a type of design used by the first generations of iPhones. Graphical elements mimicking real-world materials and depth/elevation of surfaces were rejected for this reason, hence the

<footnote n="1">¹ https://en.wikipedia.org/wiki/Flat design</footnote>
name “flat design”. However, these elements were also sometimes
affordances according to Norman, as a button that seems to protrude
signifies it can be pressed, and a hollow field signifies it can receive
input [Gav91]. Together with the idea of natural user interfaces, such
removals opened the way for the removal of other signifiers and
may have contributed to a lesser attention to signifying interaction
possibilities, going farther than what strictly belongs to the flat design
philosophy.

**UX DESIGN** Inspired by industrial design practice, and as a reaction to the main focus on usability in User Interface Design, the concept of User Experience (UX) design gained popularity in the 2000s. At its core, UX assumes that usability is only one facet of the experience that people have with products, and that products are also meaningful to people by the emotions they convey, the place they have in people's life, and how they support changes in the lives of their users (including self-improvement) [Noro4].

A popular framework thus distinguish between pragmatic and hedonic attributes of a product [Haso3], where the pragmatic qualities concern the achievement of behavioral goals and include usability concerns, but the hedonic qualities relate to users’ self –Why I use this product?– and can be subdivided into "stimulation" and "identification". Stimulation reflects the basic human need of self improvement and relates to the challenges and the novelty of using such products, while identification addresses the human need to express one’s self through products.

With a focus on taking all aspects of UX into consideration when
designing a product, trade-offs may appear between different qualities,
and sometimes, usability concerns may be perceived as less important
than hedonic ones. In the case of signifier-less designs, an interaction
possibility may thus be voluntarily hidden to users so that its discovery
–however it happens– is an event scoring high on the stimulation axis,
contributing to building a good user experience in the long term
[Kar+09].

1.2.3 Lack of knowledge about how users adopt signifier-less designs

Unfortunately, there are very few studies and very little public data
available to even start questioning or validating the design rationales
behind modern signifier-less designs. In the next paragraphs, I will
highlight three areas in which basic knowledge is missing: how users
adopt signifier-less designs, how they discover them without signifiers,
and how such designs affect users.

USER ADOPTION OF SIGNIFIER-LESS DESIGNS Studies on user adoption of signifier-less designs are very few but show that users have
different levels of knowledge and actual usage of these techniques – i.e. different levels of *proficiency* in their use, between completely ignoring them and perfectly mastering the art of how and when to use them.

A study of iPad multitasking gestures adoption by Avery and Lank showed that despite being non-signified, efficient gesture-based techniques can get high levels of user awareness and willingness to use [AL16]. On the other hand, it also showed that users “often find [the gestures] difficult to discover and challenging to learn”. Studies that compared hotkeys to other means of invoking commands in Microsoft Word [Lan+05; TWV13] revealed the following user behaviors: 1) belief of rarely using hotkeys, despite observed usage actually ranging from sporadic to almost systematic depending on the type of command the hotkey invokes; 2) users frequently missing opportunities to use the hotkeys they know; 3) perception of hotkey efficiency that is often wrong and depends on the command it invokes; 4) failure to use the technique they believe to be the fastest among the ones they know, even under time pressure.

Such studies call for further analyses and need to be conducted on more types of signifier-less designs. They also show that users have complex usage patterns of signifier-less designs –especially when considering generic designs like hotkeys that can have multiple distinct realizations in a same interface. This complexity is also an obstacle to the evaluation of signifier-less designs, since it makes the design of informative studies harder.

DISCOVERY OF SIGNIFIER-LESS DESIGNS In recent decades of HCI research, numerous studies addressed how users gain expertise as they use the interface, how they learn and what keeps them from becoming more skilled [Coc+14]. Yet there is almost no data explaining how users *discover* an interaction possibility before they “learn” to master it. Most HCI works simply assume that techniques are discovered, thanks to signifiers or exposure of the controls, preexisting skills or metaphors, or simply because they belong to a small core of basic knowledge that every users should be taught quickly before being able to use the system – including knowledge that has become a cultural evidence, like how to hold a mouse [Shn97; GFA09].

USER REACTION TO SIGNIFIER-LESS DESIGNS Very few research works investigate how signifier-less designs affect users beside the impossibility to use these interaction techniques, and users’ frustration of not being able to accomplish their tasks [Nor88]. There is a lack of observations about how users react to the discovery of a signifier-less design, if they appreciate this design and consider using it systematically as their preferred way to accomplish a task, or if they feel fooled for not having been told about it – or incompetent for not
discovering it by themselves. There is also a lack of studies assessing
if users can adapt and build on their new knowledge of this design to
discover other similar non-signified interaction possibilities, integrate
this knowledge in a logic of their own, envision new usage patterns,
and eventually, see opportunities to change the way they think about
their daily tasks for more efficient ones.

1.3 APPROACH AND CONTRIBUTIONS

1.3.1 *Approach*

Despite the well-known issues they cause, signifier-less designs have
maintained a continuous presence in successive generations of user
interfaces. There must be a reason for this persistence, possibly that
users appreciate these designs, or maybe designers think users will
use or appreciate them. My approach during this thesis was thus to
study such designs from a few different perspectives, not focusing
only on their known issues.

A first aspect of my work was to investigate the design space of
signifier-less designs. I reviewed and analyzed historical and mod-
ern signifier-less designs to identify a few different dimensions. I
particularly investigated a modern smartphone and tablet design in
which widgets are hidden on the side of the screen and revealed with
a non-signified swipe - I call this design *swhidgets* for *Swipe-revealed
hidden widgets*, and will provide a more complete definition of it in the
next chapter. This investigation revealed different types of *swhidgets*,
different roles for *swhidgets*, and a strong integration in the global de-
sign of the interface. Finally, I have also devised a categorization space
about how users may discover or learn about signifier-less designs in
the first place.

A second aspect of my work is the realization of two user studies to improve knowledge about how users discover, adopt and react to signifier-less designs – focusing more particularly on *swhidgets*. A first laboratory study allowed me to observe face to face how the participants react to –and generally think about– *swhidgets*. An online questionnaire with a larger and broader population of users allowed me to know more in detail how users discover, practice, think of, and feel about *swhidgets*. From these two studies, I draw some implications for new *swhidget*-like signifier-less designs and discuss some opportunities for improved design.

1.3.2 *Contributions*

During this thesis, I studied the question of why designers would
create interfaces that do not clearly expose some of their interaction
possibilities, taking the case of *swhidgets* as an example and focus of inquiry. My contributions to this research are:

* An analysis of the fundamental notions required to define precisely this research question, including affordances, signifiers and semiotics.

* The definition of *signifier-less designs* and the observation that this type of design is common in user interfaces and deserves more attention from HCI research.

* A model of user discovery and adoption of interaction techniques, relying on the categorizations of three dimensions and their interactions: user's current knowledge and skills, user's motivations, and the design means of informing users provided by the interface.

* The notions of *Degree of Knowledge* and *Sources of Knowledge* derived from this model, that can be instrumented in experiments to evaluate respectively how well participants know an interaction technique and how they discovered it.

* Two studies on *swhidgets* that investigated how well users known them, how they discovered them, how they feel about them, and how they integrate them in the way they think about the interaction. These studies revealed that *swhidgets* were globally appreciated and relatively well known by users, although there is still room for improvement, notably for some specific *swhidgets*.

## 1.4 PLAN OF THE DISSERTATION

This dissertation is divided in four parts:

**INTRODUCTION AND SWHIDGETS** To complete this introduction, Chapter 2 provides a more detailed example of signifier-less designs and exposes the diversity of problems and research questions raised by their study. It describes *swhidgets*, the modern signifier-less design that this thesis mainly focuses on. I categorize different types of *swhidgets*, analyze their roles in the interface, and discuss their discoverability and lack of signifiers in relation to the state of the art and other signifier-less designs. The chapter ends with a presentation of research questions that needs to be addressed to fully understand *swhidgets* and other signifier-less designs, defining a framework for this research and future researches, although not all of these research question can reasonably be addressed in this thesis.

**FUNDAMENTAL NOTIONS AND DISCOVERY OF INTERACTION POS-
SIBILITIES** The first part of this thesis aims at providing essential
definitions for the main concepts involved in the study of signifier-less
designs – notably *interaction possibilities* and *signifiers* – while also dis-
cussing how these concepts relate to the research questions identified
in Chapter 2. In particular, this part provides a state of the art of
how users may discover interaction possibilities. It starts by defining
precisely the notion of interaction possibility as Gibsonian affordances
in Chapter 3. Then, in Chapter 4, I present Norman's Seven Stages of
Action, a common HCI model of how users interact with the world
and interfaces, and discuss how it accounts for user discovery of new
interaction possibilities. Finally, Chapter 5 describes the means by
which design elements of the user interface – mostly signifiers – can
help users discovering new interaction possibilities, the different types
of such “design means”, and how they are used in the different stages
of Norman’s model.

**SIGNIFIER-LESS DESIGNS** The second part of this thesis introduces signifier-less designs in Chapter 6, proposes a definition based on the concepts of affordances and signifiers, discusses their history and how their understanding evolved over time, to finally distinguish three sub-types of signifier-less designs depending on what affordance is not signified.

**A MORE COMPLETE MODEL OF DISCOVERY** In the third part of this thesis, I propose models of how users may discover non-signified interaction possibilities and develop their interaction vocabulary to become expert users, to extend the state of the art presented in the first part. In Chapter 7, I describe the necessary conditions for the discovery and adoption of new interaction possibilities, considering a user's current knowledge and motivations, and how they affect or are affected by the design means already discussed in Chapter 5. I introduce *degrees of knowledge* as a measure of a user's current knowledge that is suitable for experimental studies. A characterization space of the possible sources of knowledge of an interaction possibility is also provided as a contribution, leading to a discussion of user exploration of the interface.

**STUDIES** The fourth part of this thesis presents in Chapter 8 the design and results of two studies investigating the knowledge of *swhidgets* by iOS users and their reactions to this design: a preliminary laboratory study and an online study that complements the first study with more precise investigations of what iOS users know about *swhidgets*, how they discovered them, how they feel about them, and how they integrate them in their way of thinking the interface. I discuss the results of these two studies and draw implications for designs that include or get inspiration from *swhidgets*.
**PERSPECTIVES** The dissertation ends with a discussion of perspectives and future work, in which I highlight some possible research and design directions to improve signifier-less design in general and *swhidgets* in particular.
<figure><img src="image_15.png" /></figure>
<figure><img src="image_16.png" /></figure>
It is now common for smartphone users to slide a finger from the edge of the screen toward the center of the screen to drag into view the hidden drawer that was secretly hiding under the bezel. I define *swhidgets* (for swipe-revealed *hidden widgets*) as widgets that are normally entirely hidden from the user, who can uncover them with a simple swipe gesture relying on the metaphor of sliding physical objects to drag the *swhidget* into view.

The use of a physical drag metaphor is a distinctive feature of *swhidgets* that may provide the type of user guidance discussed by Norman as lacking from other gestural interfaces [Nor10] and builds over the design of navigation gestures omnipresent in modern smartphone interfaces. As such, *swhidgets* differ from swipe-based interactions techniques that do not rely on a sliding metaphor such as Bezel Menus or Bezel Taps [JB12; SLG13].

In this chapter,<fnref n="1" /> I analyze *swhidgets* from popular commercial products, examine the apparently voluntary lack of elements signifying their presence, and discuss how users can discover them. While *swhidgets* can be found on various touch-based operating systems, I will focus on the iOS platform as this is a popular and consistent operating system without additional manufacturer overlay, relying on simple physical metaphors for interaction [App19]. However, the categories proposed in Section 2.1 are still likely to also apply to *swhidgets* on other platforms.

## 2.1 TYPES OF *swhidgets*

Surveying how *swhidgets* are used in iOS interfaces, I recognized three different types of *swhidgets*, each relying on different interface schemes that can be classified from the perspective of the physical metaphor they use and the type of functions they provide, as follows:

**SYSTEM SWHIDGETS** are “pulled over” the main view from an edge of the display, following the metaphor of a roll-up projection screen. Examples of *system swhidgets* are the Notification Center and the Control Center (Figure 2.1), which are revealed with sliding gestures from (and across) the edge of the display – respectively downward

<footnote n="1">¹ Sections 2.1 to 2.5 of this chapter have been partially published as parts of a paper for the ISS’19 conference [PM19].</footnote>
<figure><img src="image_17.png" /><figcaption>Figure 2.1: Examples of *system swhidgets* in iOS containing various controls that can be accessed from anywhere in the OS.
Left: a downward swipe from the top edge of the screen reveals the Notification Center.
Right: an upward swipe from the bottom edge of the screen reveals the Control Center.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure 2.2: Examples of *view* and *item swhidgets* in iOS containing various controls that are provided by some applications.
Left: A *view swhidget* in the Message application, where a leftward swipe from anywhere in the discussion view reveals the exact time each message was sent.
Right: An *item swhidget* in the Mail application, where a leftward swipe on an e-mail from the list reveals a set of buttons.</figcaption></figure>
from the top and upward from the bottom.<fnref n="2" /> *System swhidgets* can be
revealed regardless of the application/view<fnref n="3" /> and overlay the entire
main view to provide various system commands and information.

**VIEW SWHIDGETS** are concealed on the side of a view or container, outside its viewport, and revealed by pulling them into view, i.e., by sliding the finger in the opposite direction of the *view swhidget*. Examples are the exact arrival time of SMS messages, revealed with a leftward sliding gesture from anywhere in the discussion view (Figure 2.2-left), or the search text field that is located at the top of most element lists (typically e-mail list in the Mail application) but hidden by default, only to be revealed when scrolling over or back to the top.

**ITEM SWHIDGETS** are hidden next to or under an object of interest within the application and provide contextual functions to interact with this object. Examples are the commands operating on e-mails in the Mail app, that can be revealed by sliding horizontally an individual e-mail (Figure 2.2-right). *Item swhidgets* differ from *view swhidgets* in the sense that their revealing gesture must be performed on a specific element.

## 2.2 ROLES OF *swhidgets*

*Swhidgets* are used conjointly with other types of controls in the design of mobile device interfaces and cannot be understood fully if taken in isolation. One question is thus: do *swhidgets* replace, complement, or double other interaction techniques?

Most *swhidgets* provide a faster access to some functionalities that can also be accessed through traditional widgets.<fnref n="4" /> Yet, some *swhidgets* provide access to advanced features with no real equivalent, and where the conventional *tap and scroll* interaction paradigm would only provide access to limited versions of the feature. In iOS, this is typically the case of exact SMS arrival times that can only be accessed through a *view swhidget*, whereas conventional interaction would only provide discussion start and resume times. Similarly, the search feature in the Mail application is not accessible through another widget than the

<footnote n="2">2 On the iPhone X and iPad, both are revealed with a downward swipe, starting respectively from the left and right sides of the top bezel.</footnote>

<footnote n="3">3 In some full-screen applications like games and video players, a protection against an involuntary activation of the *swhidget* is provided, in which the swipe gesture first only displays a small un-intrusive handle, and a second swipe gesture needs to be performed to confirm and trigger the opening of the *system swhidget*.</footnote>

<footnote n="4">4 Often via a modal interface triggered by a button with a generic label like "modify" or an icon representing gears.</footnote>
dedicated *view swhidget*,<fnref n="5" /> but users can still search a mail by browsing the mail list.

In some cases, *swhidgets* are merely one of several ways to accomplish a task, which relative efficiency can depend on how users prefer to accomplish the task. For example, on an iPhone 6S running iOS 11, users have at least seven different ways to delete an e-mail in the Mail application:

1. From the list of mails/threads in a mailbox, by performing a leftward swipe on the mail/thread to delete (as shown in Figure 2.2-right):

a) A short swipe to reveal an *item swhidget*, then tap the rightmost icon it contains (a red trashcan labeled "Delete").

b) A short swipe to reveal the same *item swhidget*, then tap on the leftmost icon it contains (blue with three dots labeled "More" and suggesting "more options"), which opens a menu at the bottom of the screen, from which the user can choose "move to...," and then select the trash mailbox.

c) A long swipe leftward to directly delete the mail.<fnref n="6" />

2. From the list of mails/threads in a mailbox, by tapping the "Edit" button, then selecting the mail(s) to delete, then tap the "Delete" button.

3. From the view to read a specific e-mail or thread:

a) By taping the trashcan icon at the bottom of the screen.

b) By taping the folder icon at the bottom of the screen, which opens a list of the mailbox and folders, then choosing the trash folder.

c) If there are responses to the mail, it is a conversation view rather than a mail view, and users can then make a leftward swipe anywhere on the body of an e-mail to reveal three icons on its right, and use the rightmost one with a trashcan icon.

Here, users may prefer to read every incoming e-mail individually before eventually deciding to delete it with the trashcan icon at the bottom of the screen (option 3a), which deletes the mail and automatically opens the next one. Or they may prefer to first delete all irrelevant e-mails directly from the e-mails list, either using a swipe that directly provide feedback about the deletion (options 1a or 1c), or using the modal "modify" button to delete all mails in a batch (option

<footnote n="5">5 The search feature in mail is also accessible through Siri, which relies on a different interaction paradigm</footnote>

<footnote n="6">6 This interaction can be seen as an extension of the *swhidget* design, an "expert mode" to trigger the main command in the *swhidget*. Although it is an interesting design that deserves a study, in this thesis I focus first on the basic *swhidget* design.</footnote>
2). The preference, here, can depend on multiple factors: a user's expertise with the interface and e-mail management activities, the amount of unwanted e-mails that she receives, or external constraints affecting the ease of performing some gestures that can be imposed by the context of interaction (e.g., in a crowded subway train, or while walking).

2.3 DO USERS INTERACT WITH *swhidgets*?

It remains unclear how many *swhidgets* users actually use and how they are discovered. To the best of my knowledge, there is very little public data on current usage of *swhidgets*. The only available data set I am aware of can be found on a blog post reporting proportions of users (among 6 tested) who knew how to interact with a subset of *system swhidgets* in iOS 11 [Ram17], which suggests there are great differences in the knowledge of the different interactions, some being spontaneously used by all participants and others being understood by none of the participants. Unfortunately, the author does not accurately describe the methodology and experimental procedure employed, questioning the generalization of the results.

Academic research has mostly investigated the performance of novel edge-based interaction techniques, such as menus or keyboards [JB12; RT09; SLG13] that do not rely on a physical metaphor that may help users uncovering these features. Similarly, Schramm et al. [SGC16] investigated the transition to expertise in *hidden toolbars*, a name describing various types of user interfaces that make functionality available only when users explicitly expose them through a dedicated interaction, including menu bars (revealing commands when the menu bar is clicked), the “charms” in Windows 8 [Hop19], and some *swhidgets* (typically, *system swhidgets*). Their work focuses on the performance of such interfaces, proposing four hidden toolbar designs (all different from actual *swhidgets* designs) and comparing their performance in terms of selection time and learning of item locations. As a result, their work is informative regarding the performance of the hidden toolbar designs they propose, but is not adequate to fully understand how users currently interact with *swhidgets*.

Similarly, studies conducted on other related "hidden" or "expert" interaction techniques can help apprehending interaction with *swhidgets*. This is the case of the work conducted by Avery and Lank surveying the adoption of three multitasking multi-touch gestures on the iPad [AL16]. Their results show that users tend to have varying levels of knowledge and actual usage of these techniques, between completely ignoring them and perfectly mastering the art of how and when to use them. They also show that despite being "hidden", these gesture-based techniques can get high levels of user awareness and willingness to use. Regarding keyboard shortcuts on desktop, studies
conducted on Microsoft Word [Lan+05; TWV13] revealed that users do not systematically use the hotkeys they know. This behavior may be explained in part by the fact that users often underestimate the potential benefits of hotkeys. This is however not the whole explanation, since users also often fail to use hotkeys when they know it will be faster, even under time pressure.

## 2.4 VOLUNTARILY SIGNIFIER-LESS: WHY?

By definition, *swhidgets* are instances of *signifier-less designs*, since their design provides no *signifier*, i.e., no perceptible indicator of the presence of the *swhidget* or of the possibility to make a swipe to reveal it (the notion of signifiers will be introduced in a broader context in Chapters 3 and 5, and the notion of signifier-less design in Chapter 6).

Some signifier-less designs lack of signifiers because there seems to be no satisfying way of permanently displaying signifiers where the input method can be used [May+18]. This is however not the case of *swhidgets*, where a handle could easily be displayed on the movable interface elements to provide a hint of their presence [RT09]. Indeed, a small dash-shaped handle used to be displayed near the bottom edge of the screen when the control panel was introduced in iOS, but more recent releases show it on extraordinary occasions only (see Footnote 3), or use it as a replacement for the physical home button (typically on iPhone X, which has no physical home button).

*Swhidgets* are by default “hidden” under the screen bezel, yet their existence is not signified to users, possibly as an intention to minimize visual clutter [SGC16] and keep the UI clean in order to improve user experience and performance [Noro4]. The iOS *Human Interface Guidelines* echo these expected benefits in the themes of *clarity* and *deference*: “Content typically fills the entire screen, [...] minimal use of bezels, gradients, and drop shadows keep the interface light and airy, while ensuring that content is paramount” [App19].

This focus on “clean” interfaces benefits all users, but *swhidgets*-aware users might additionally benefit from other aspects of the design. For instance, it could be said that *swhidgets* provide sometimes a more uniform way to interact with the system globally than alternative designs, since they rely on similar gestures and metaphors than navigation gestures. However, none of these benefits seems to justify a total removal of signifiers. On the other hand, the discovery of a *swhidget* could also be a positive experience for users and such a positive experience may justify the lack of signifiers, although there currently seem to be very little evidence in the HCI field supporting such an hypothesis.
## 2.5 DISCOVERY OF *swhidgets*

As pointed by Mayer *et al.*, the swipe gestures of *swhidgets* are not visually communicated to the users: Instead, device manufacturers rely on dedicated tips and animations that are either shown to the user when setting up the device, or showcased on stage when the system is presented to the technology-oriented press [May+18].

A way to discover *swhidgets* that is better integrated in everyday interaction is to explore the interface by performing series of inputs and expecting the system to respond to them. Schramm *et al.* [SGC16] advocate that with edge-based interactions becoming more common, users might appropriate the physical metaphor of sliding objects and become more likely to expect *swhidgets*, thus exploring the operating system to discover them. The design guidelines and *Design language*<fnref n="7" /> (typically, *Material Design* on Android and the *Human Interface guidelines* on iOS) eventually declare a consistent interaction environment that promotes such metaphors and swipe-based interaction, which might lead users to discover these *swhidgets* through interactive exploration.

Moreover, *swhidgets* are by design exploration-friendly, since 1) they do not trigger commands directly but only reveal widgets, and it is thus safe to try revealing them; 2) testing the presence of a *swhidget* has a low interaction cost since it only requires to slide a little bit a finger on a candidate surface or edge; and 3) in case of success, this test provides immediate feedback with a visibly moving element.

However, Schramm *et al.* also acknowledge that one cannot expect novice users to guess the availability of *swhidgets* nor the actions used to access them regardless. Therefore, users still need to have discovered a first *swhidget* before being able to reproduce a similar input somewhere else in the interface. In addition, it still requires users to *explicitly* explore the interface in the first place, while they may be too engaged in their tasks to do it, even if it would eventually improve their performance as users [CR87].

Another possible way to discover *swhidgets* is through *accidental revelation*, that is, when the *swhidget* is revealed whereas it was not the first intention of the user. For instance, one of iOS *swhidgets* that is likely to be accidentally revealed is the search *view swhidget* located on top of some lists, that may be brought into view when the user overshoots while scrolling back to the top of the list. Other *swhidgets* may be less likely to be accidentally discovered. For instance, with the *item swhidgets* in the Mail application, there is no reason to expect users to perform an horizontal swipe on an item of a vertically scrollable list. Moreover, accidental activation can be confusing and users might be unable to reproduce the input operation that triggered that accidental activation.

<footnote n="7">7 https://en.wikipedia.org/wiki/Design_language</footnote>
Finally, users might discover the existence of *swhidgets* from their own social network, typically through friends, colleagues or family. Indeed, impact on input mechanism adoption of witnessing others performing them has already been observed in the context of keyboard shortcuts [Per+04].

## 2.6 GENERAL RESEARCH QUESTIONS IDENTIFIED

In this section, I will recall and formalize the questions that were quickly raised about the design of *swhidgets* in the previous sections. These questions point at individual phenomena that need to be understood before being able to get a full understanding of the design of *swhidgets*. However, the scope of these phenomena extends far beyond the single case of *swhidgets*, and understanding each of these phenomena is a general topic in the field of HCI. It is therefore not possible, in this thesis, to address all these phenomena, nor to contribute directly to the understanding of all of them. The list of phenomena and research questions that I give below is therefore defining a framework into which research on *swhidgets* has to be conducted, and highlights the benefits of studying *swhidgets* to improve general knowledge about human-computer interaction as a phenomenon [Bea04; HO17]. The next chapters will discuss basic notions implied by this framework.

### 2.6.1 Integration of swhidgets in relation with other interface features

We have identified three types of *swhidgets* (Section 2.1), in which specific interaction techniques (position and semantic of the swipe gesture) match with domains of application of commands: *system swhidgets* provide system-wide commands and are attached to the screen bezels, a feature of the device itself; *view swhidgets* provide commands about the set of data displayed in a view and are revealed by scroll-like interactions with this view; *item swhidgets* require to interact with the item targeted by the command. This matching raises two types of questions, depending on whether its role is considered in a specific instance of interaction or in multiple instances of interactions across different applications.

#### OVERARCHING INTERFACE LOGIC
This matching between interaction techniques and domains of application of commands is an overarching logic in the structure of the interface, a general principle ruling the design of the whole operating system and specific applications. It suggests that users can learn this overarching logic (Section 2.5), and later use it to deduce where they should search for a specific command, even if the widget providing this command is hidden. Do users really exploit such an overarching logic? How do they learn it, and how do they think about exploiting it? What
cognitive processes are activated to exploit this knowledge? These questions fit in the general HCI research about how users develop a conceptual understanding of the interface (as rules, strategies, mental models, etc.) and exploit them in their goal-oriented, problem solving interactive behaviors [Nor88].

**CONSISTENCY WITH OTHER INTERACTIONS** The overarching interface logic is only one aspect of a consistent use of *swhidgets* in the design of the system and applications, but other aspects of the design can also be consistent. For instance, the mailbox view in the Mail application (Figure 2.2-right) and the conversations view in the Message application both share a similar list layout with similar *item swhidgets*, although there are slight variations between the two designs. Can users notice this similarity, infer a pattern from it, and use their knowledge of the pattern to discover *swhidgets* in other applications? Are there aspects of the design, of the activity supported by an application, or user skills that can help or prevent noticing the pattern or similarities with other interfaces? Beyond the consistency of uses of *swhidgets* in design, what role plays the consistency of *swhidgets* with other types of interactions such as navigation gestures (Section 2.4), or the consistency of patterns that involve both *swhidgets* and other types of controls? How important is a consistent use of *swhidgets* in design for their adoption by users? These questions fit in the general topics of transfer of knowledge from other contexts of use, and other benefits of consistency in user interfaces [Gru89].

**BENEFITS OF A CLEAN INTERFACE** Since the design of *swhidgets* seems to purposefully avoid the use of signifiers (Section 2.4), and since a possible reason for this is to reduce interface clutter, improve readability, and focus on content, it seems important to understand the benefits of having a “clean” interface and how these benefits are obtained.

### 2.6.2 Interaction with specific swhidgets

**ACTION-FUNCTION COUPLING** Beside the existence of an overarching interface logic, is there something specific about how *swhidgets'* designs connect the semantics of the interactions with the semantics of the commands they trigger? *Swhidgets* use an interface element that connects the domain of commands and the interaction technique by being more or less loosely related to both: a bezel, a view, or an item – but does the strength of this relation affects users’ ability to understand it or to integrate this understanding in how they think about *swhidgets* and commands? Behind these questions, there is the whole domain of HCI research about the consequences of relating
the purpose or effect of an interaction with the way it is performed
[WDO04; Dja+04].

PHYSICAL METAPHOR *Swhidgets* rely on the physical metaphor of sliding interface objects to reveal what is under them. Unlike action-function coupling, such metaphors are concerned with the way the interaction if performed but not with the effect of the interaction or its purpose (beside “revealing some widgets”). As such, they fit in the general study of metaphors as a tool to help users make sense of the interface and learn how to use it by themselves.

However, in the case of *swhidgets*, there is a little more than that, since the metaphor also contributes to unify the design of *swhidgets* with other types of interactions promoted by the system (Section 2.4), and is thus related to consistency questions.

The specific metaphor used for *swhidgets* also allows to bridge the gap between uni-stroke gestures and direct manipulation, with consequences on the learnability of the gestures or users’ understanding of what they did or are doing (feedback). It therefore has to be analyzed both relatively to the problem of learning how to interact with a system, and to the problem of facilitating interaction itself.

2.6.3 *Discovery and adoption of Swhidgets in the long-term*

MULTIPLE WAYS TO DO A TASK *Swhidgets* can coexist with other methods for triggering the commands they provide, with sometime differences in performance or ease of use (Section 2.2). It raises the general questions of what methods users prefer to use, and of what aspects of the methods, task, and context affect these preferences. But it also raise questions about the evolution of such preferences in time as users get more familiar with them and better understand the advantages of each method. In particular, are some methods supposed to replace others as users get more experienced, and if so, what aspects of the design of each interaction method makes it better suited for different types of users?

REASONS NOT TO ADOPT The example of hotkeys (Section 2.3) reveals that measuring users' performance with an interaction technique and the proportions of users who know how to use it at a given level of performance is not enough to understand users' adoption of the technique: we also need ways to better explain the non-usage behaviors observed – including reasons for users not to use a technique despite performing well with it. A study of *swhidgets* should thus also evaluate users' perception of the technique's performance, ease of use, reliability, mental and physical costs of activation, etc., to determine if an inaccurate perception of these qualities justifies the non-usage of *swhidgets*.
Beyond the evaluation of such factors in experimental studies, there are also opportunities to analyze how the design of the interface helps users evaluating these qualities of an interaction technique. Such analyses may have to be performed in relation with other aspects of the design – e. g., for interface consistency: the metaphor of sliding physical objects and its consistent use in *swhidgets* and other navigation techniques may help users making more accurate evaluations of *swhidgets'* performance and ease of use, by leveraging their knowledge of other parts of the interface that rely on similar gestures and animations.

**USER EXPERIENCE OF DISCOVERY** Because they rely on interaction possibilities that are not signaled to users, a major obstacle to the use of *swhidgets* is their discovery. But conversely, what does the discovery of a *swhidget* entail for the user experience? In addition to the benefits of being able to use a new interaction technique, what are the consequences of the discovery on users, especially from the point of view of emotions, value judgments, and other reactions? The study of user experience (UX) has become an important theme in HCI research, although it is often understood from the point of view of a specific situation (notably, customers' first contact with a design) rather than as developing in the long term [Haso3].

**EXPLORATION OF THE UI** Exploration of the user interface is one way to discover *swhidgets* (Section 2.5), but despite a long term interest in HCI for the phenomenon of user exploration of the UI, there seems to be a lack of models that can explain how and when users explore the interface, and how they could do it to discover hidden features, or what properties of the hidden interaction possibilities can encourage users to explore the interface and search for them.

**EXTERNAL AND ACCIDENTAL SOURCES OF KNOWLEDGE** Users may learn about *swhidgets* from other sources than the interface itself or through accidental revelation (Section 2.5). While HCI research with a focus on usability usually considers that the knowledge about the interface should be conveyed through or by the interface itself, the reality of interaction is that it happens in a cultural and technological context that provides users with multiple opportunities to learn about the interface, to the point that it can be questioned if this knowledge really needs to be also conveyed by the interface itself. What opportunities have designers to leverage external sources of knowledge, and what are their chances of success in doing so?

Conversely, designers also have opportunities to create interaction techniques that have greater chances to be discovered through accidental activation, or by otherwise exploiting interactions that users may
perform without assuming they will be understood by the system.<fnref n="8" /> Game designers, for instance, frequently force players in situations to which they will naturally react with some action, to teach players that these actions are meaningful in the game<fnref n="9" /> – to what extent could UI designers rely on similar tricks, and what would be the guidelines to do it well?

### 2.6.4 Methodological aspects of the study of Swhidgets

#### UNDERSTANDING OF SIMILAR INTERACTION TECHNIQUES
Swhidgets are quite unique interaction techniques in that they have both aspects of gestures and direct manipulation of objects, in addition to being hidden. It makes it relatively difficult to understand this design by comparing it with other well-known interaction techniques. For this reason, I have chosen in this thesis to extend the analysis of *swhidgets* to a larger category of interactions that are not signified (Section 2.3) – which turn out to be more frequent than I initially believed. However, to be able to compare *swhidgets* with other *signifier-less designs*, it will be necessary to define as precisely as possible the notions of signifiers and interaction possibilities – notions that are complex and have been the subject of many debates in the HCI literature, as we will see in Chapter 3. Doing so helped me identify some of the previous research questions in this list, and I hope that in turn, my contributions about *swhidgets* may suggest new approaches to the study of other signifier-less designs.

#### MEASURING PROFICIENCY
Because *swhidgets* are a family of interaction techniques rather than a single one, it is difficult to answer the question “how well are they known by users”. Indeed, not only different users can have different knowledge of *swhidgets*, but one user’s knowledge can also vary for different instances of *swhidgets*. Knowledge itself is a vague term, as there can be various levels of awareness, understanding, ability to use or showing expertise in use. Knowledge can evolve in time and be closely connected to user’s preferences, motivations, and activities practiced. For these reasons, I prefer to consider the notion of proficiency in use to evaluate the design of *swhidgets* and the consequences of their reduced discoverability due to the lack of signifiers.

<footnote n="8">⁸ An example of the latter is provided by a macOS feature: when the user do not find the mouse pointer, they tend to shake the mouse to spot the pointer thanks to this motion, but the system actually recognizes this interaction and then displays temporally a much bigger pointer to make it even easier to locate.</footnote>

<footnote n="9">⁹ See for instance Egoraptor's popular and humorous YouTube video about the game Mega Man X: https://youtu.be/8FpigqfcvlM.</footnote>
## 2.7 HOW I ADDRESS THESE RESEARCH QUESTIONS

Although all the general research questions presented in the previous section deserve dedicated researches, I could only address some of them in this thesis. I therefore defined the following sequence of specific research questions that could reasonably be addressed during a thesis, and needed to be addressed in this order:

1. What is a signifier-less design and how *swhidgets* are different from other signifier-less designs?

2. Do users know *swhidgets*?

3. If yes, how is it possible that they know *swhidgets* despite their lack of signifiers?

4. Even if there are reasons to believe users can get to know *swhidgets* despite their lack of signifiers, what are the benefits of not providing signifiers?

I will now give a quick overview of how each of these questions relates to the general research questions from the previous section, and what contributions I provide in this thesis to address them.

### 2.7.1 *What is a signifier-less design and how swhidgets are different from other signifier-less designs?*

In the general theme "Understanding of similar interaction techniques" (Section 2.6.4), I will provide in Chapter 6 a definition of signifier-less designs and a review of historical signifier-less designs.

For obvious reasons, this definition relies on the notion of *signifier*, which will be discussed in Chapter 5. There are two additional reasons to discuss signifiers in the study of signifier-less designs: First, it is important to discuss signifiers to understand what information they usually convey but is missing in signifier-less designs. Second, users' ability to interact with signifier-less designs could rely on their ability to learn recognizing as signifiers some aspects of designs that would initially be considered as signifier-less – a phenomenon related to concerns about “Consistency with other interactions” (Section 2.6.1). I thus propose to consider signifiers in the framework of semiotics (from which they originate) so as to understand how users can learn to recognize signifiers and associate them with interaction possibilities.

Since we are interested in signifiers that inform more or less directly about “interaction possibilities”, this notion needs to be precised, and I claim it can be done through Gibson’s notion of *affordances*. The definition of signifier-less designs that I propose relies on affordances for this reason, but also to precise the relation between the missing signifiers and all the interaction possibilities (affordances) that are
connected to the one considered, either as intermediary steps or as the
purpose of the interaction (e. g., opening a menu to select a command).
Since the notion of affordance is complex and different definitions
have been used in the HCI literature, it will be discussed in Chapter 3
from the point of view of some concerns that are relevant for signifier-
less designs, including the meaningfulness of interaction possibilities
for users, their learning and acquisition by users, how they relate to
other affordances, and the possibility to interpret them.

2.7.2 *Do users know swhidgets?*

Answering this question is the main objective of the two studies that I will present in Chapter 8. To do so, and in the general theme "Measuring proficiency" (Section 2.6.4) – and in particular its implication "what does it mean to know a family of interaction techniques?" – I will propose in Chapter 7 the notion of *Degrees of Knowledge*, a simple characterization of how well a user knows an interaction technique that can be instrumented in experiments.

However, answering this question would be of little scientific value
without a consideration for the phenomena that can explain users'
lack of knowledge of *swhidgets*. I study two such phenomena in
this thesis: First, the consequences of having "multiple ways to do a
task" (Section 2.6.3) will be discussed theoretically in Chapter 7, and
investigated in the studies of Chapter 8 as the knowledge of *swhidgets*
is compared to that of other interaction techniques that can be used to
complete the same tasks and users' preferences are evaluated. Second,
"Reasons not to adopt" *swhidgets* (Section 2.6.3) will also be discussed
theoretically in Chapter 7 and investigated in the second study of
Chapter 8 with questions targeting specifically this concern.

2.7.3 *How can users know swhidgets despite their lack of signifiers?*

To answer this question, I first ask how the discovery of *swhidgets* can
happen. I discuss in Chapter 4 the general theme of "Exploration
of the UI" (Section 2.6.3) in relation with user's goals. In the theme
of "External and accidental sources of knowledge" (Section 2.6.3), I
also propose in Chapter 7 a characterization space of the *sources of
knowledge* based on the dimensions of *distance* and *intentionality*, which
I use to investigate the sources of knowledge of *swhidgets* in the two ex-
periments reported in Chapter 8, showing participants claimed to use
mainly the interface itself as the source of their knowledge although
other sources are important as well. I also identify as perspectives
(Chapter 9) some opportunities for new designs that may increase the
discoverability of *swhidgets* by providing sources of knowledge in a
rather unexplored area of the characterization space (small distance
and intentionality).
I also investigate some aspects of *swhidgets* and signifier-less de-
signs that help users remembering those they have discovered, taking
the habit to use them, and leveraging on their current knowledge to
discover other *swhidgets*. The online study presented in Chapter 8 con-
tains questions specifically included to get preliminary results on the
topics of “Overarching Interface Logic” (Section 2.6.1), “Consistency
with other interactions” and transfer of knowledge (Section 2.6.1),
“Action-Function Coupling” (Section 2.6.2), and “Physical metaphor”
(Section 2.6.2).

2.7.4 What are the benefits of not providing signifiers?

I have identified three general themes in the previous section that can
suggest possible benefits of not providing signifiers: "Benefits of a
clean interface" (Section 2.6.1), "Consistency with other interactions"
(Section 2.6.1), and "User experience of discovery" (Section 2.6.3). I
discuss in Chapter 7 how these concerns fit in the model of discovery
and adoption of interaction possibilities that I propose, respectively
as interactions between design means that affect their efficiency, as
interactions between design means and user's current knowledge, and
as interactions between user's current knowledge and motivations.
The online study presented in Chapter 8 contains questions specifically
included to get preliminary results on these topics, and perspectives
for future works on the role of user experience considerations on the
design of *swhidgets* are discussed in Chapter 9.
<figure><img src="image_19.png" /></figure>
# Part I

## DISCOVERING INTERACTION POSSIBILITIES

State of the arts about affordances, signifiers, and discovery models.
<figure><img src="image_20.png" /></figure>
<figure><img src="image_21.png" /></figure>
<figure><img src="image_22.png" /></figure>
An important aspect of this thesis is to understand the phenomenon of people managing (or not) to discover relevant interaction possibilities that are not signified to them in the environment/interface. The terms "relevant interaction possibilities" thus need some clarification and examples.

Instead of reinventing the wheel developing the concept behind these terms, I will claim that it is exactly the concept of affordances, as originally introduced by Gibson. This concept has many important but subtle aspects that Gibson covered in length in his books and papers, but that he never synthesized in a short definition of affordances. As a consequence, many works in HCI have been based on the following partial definition:

"The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill."

[Gib79, page 127]

Since the term *affordance* is frequently used in HCI works (see [Kap14] for a review), but often in a way that is not compatible with Gibson's view [KN12], there is a risk that readers of this dissertation already have an understanding of the term that does not match the one I want to present here. It is also possible that despite the warnings raised by some authors [Nor99; Toro3], the word has been used in so many ways that it has become meaningless or unclear for some readers.

Therefore, before providing more detailed explanations of Gibson's concept of affordances in the remainder of this chapter, I will temporarily provide the following personal interpretation and wording of the concept of affordance so that readers are not confused by the possible differences between their own understanding and the one I present in this chapter:<fnref n="1" />

**AFFORDANCE:** an interaction possibility offered by an environment to a specific individual in it, that has meaning and value in an

<footnote n="1">¹ Unlike the other parts of this dissertation, this chapter will rely heavily on quotes from other authors rather than simply citing their works. Indeed, the concept of affordance has many non-obvious implications and it is thus very easy to apply your own understanding on the concept to what an author wrote, and misinterpret it. I have myself many times read articles about affordances that attributed to other authors some interpretation of the concept of affordance that I could not see in the cited works. I hope that relying on actual quotes will highlight the parts of the cited works that convey the interpretation I discuss.</footnote>
ecological system, expresses the complementarity of the individ-
ual and the environment, and is independent of the individual’s
awareness of such a possibility, of her ability to perceive it, or of
her current need for it.

Understanding the concept of affordances is crucial when inves-
tigating signifier-less designs, since the notion of signifier has been
introduced as an alternative to the perception of affordances. Indeed,
there is a common belief that signifiers are necessary only when affor-
dances cannot be directly perceived by users. As Norman said when
he introduced the concept to the HCI community:

"When affordances are taken advantage of, the user knows what to do just by looking: no picture, label, or instruction is required. Complex things may require explanations, but simple things should not. When simple things need pictures, labels, or instructions, the design has failed."

[Nor88, page 9]

In other words, the potential of a signifier-less design to cause usability issues depends on users' ability to directly perceive its affordances. It thus seems that to investigate signifier-less designs, we cannot avoid a careful examination of how people perceive, and more generally, get aware of affordances.

As we will see later in this chapter, there is much debate in HCI
about what affordances are. The reason may be that there are many
subtleties behind the concept of affordances, which cannot be conveyed
easily by short presentations and definitions like the one I provided
above.

I will thus discuss in detail some aspects of affordances in the next
sections, starting with Gibson's work in Section 3.1 and continuing
with other works from authors in the field of human-computer interac-
tion. I will discuss aspects of affordances on which HCI authors agreed
and focused, notably their organization in hierarchical structures (Sec-
tion 3.2), as well as debated aspects (Section 3.3). By presenting the
different views that authors have expressed about affordances, a few
concerns will be highlighted that go beyond the pure theoretical anal-
ysis of affordances. The discussion will thus inform more generally on
how users can get to know about the interaction possibilities provided
by their devices and applications. It will notably highlight the possible
roles of the various ways interaction possibilities can make sense to
users (Section 3.4), of an usage culture (Section 3.5), and of explicit
communication from the designer (Section 3.6).

3.1 GIBSONIAN AFFORDANCES

The term and concept of *affordance* were created by the psychologist
James Jerome Gibson [Gib77; Gib79], and were later introduced (with
modifications) to the HCI community by Donald Norman [Nor88]. The HCI community has embraced the term but has also often extended the concept in ways that are arguably incompatible with Gibson's original motivations and approach [KN12; Kap14]. I will thus attempt to clarify this complex and often-misunderstood concept.

### 3.1.1 *Origin of the concept in perception theory*

When he introduced the concept of affordances, Gibson's focus was on understanding human (and animal) visual perception, which is a topic that far extends the concerns of this thesis. I will however have to present quickly some aspects of his theory of perception that will have implications later in this dissertation, concerning how affordances can be discovered.

Gibson introduced the concept of affordances as a way to challenge the prevalent theories of perception. Notably, he challenged the cognitive view, for which perception is a sequence of information processing stages, which start with raw sensations from the sensing organs, integrating this sensory information with memory to build symbolic representations of the environment, in a process influenced by mental states and needs of the perceiving individual. Instead, Gibson believed in the *direct perception of affordances* through a process of *information pickup* based on the recognition of *invariants*:

**DIRECT PERCEPTION AND INFORMATION PICKUP** These two concepts are defined in the following quote:

> "[...] when I assert that perception of the environment is direct, I mean that it is not mediated by retinal pictures, neural pictures, or mental pictures. Direct perception is the activity of getting information from the ambient array of light. I call this a process of information pickup that involves the exploratory activity of looking around, getting around, and looking at things. This is quite different from the supposed activity of getting information from the inputs of the optic nerves, whatever they may prove to be."

[Gib79, page 147]

**INvariants** How is information picked up? Influenced by his background in Gestalt perception, Gibson argues that we globally recognize *invariants*, i.e. structures of the environment that do not change as the perceiver evolves in the environment during the information pickup, causing changes in what she sees: the structures are *invariant* but what she sees *varies*.

For instance, the appearance of a static solid surface changes as the observer moves relatively to it, but in a way that is correlated to the observer's own movement: as she moves the head to her left,
the surface moves to the right of her field of view and is distorted
so that its parts closest to the observer move faster than the ones
further away – a phenomenon called *parallax*. During this movement,
the appearance of the surface varies in a way that correlates with
the perceiver’s movement and reveals the invariant structure of the
surface.

Of course, the combination of multiple invariants is itself an invari-
ant, and the information pickup aggregates information from all the
senses into complex invariants – from isolated surfaces to the layouts
of complete rooms and landscapes, up to more complex physical
events and animal behaviors.

AFFORDANCES In Gibson’s view, the recognition of invariants is neither the goal nor the result of perception, but only the mean by which it operates. The purpose of perception is to allow the perceiver to focus her attention on specific invariants, which are the *affordances*: what the environment offers to the individual. As Gibson says:

"[Objects] can be said to have properties or qualities: color, texture, composition, size, shape and features of shape, mass, elasticity, rigidity, and mobility. Orthodox psychology asserts that we perceive these objects insofar as we discriminate their properties or qualities. [...] The psychologists assume that objects are composed of their qualities. But I now suggest that what we perceive when we look at objects are their affordances, not their qualities. We can discriminate the dimensions of difference if required to do so in an experiment, but what the object affords us is what we normally pay attention to. The special combination of qualities into which an object can be analyzed is ordinarily not noticed."

[Gib79, page 134]

3.1.2 *Affordances and ecology*

The focus on affordances as the object of perception was the novelty
of Gibson’s theory. As special cases of invariants, affordances must
be defined as objective relationships between an individual and the
environment. But affordances must also express possible interactions
between the individual and the environment that bring something to
the individual, independently on her needs or interest in such things:

"I have described the environment as the surfaces that separate substances from the medium in which the animals live. But I have also described what the environment affords animals, mentioning the terrain, shelters, water, fire, objects, tools, other animals, and human displays. How do we go from surfaces to affordances? And if there is information in light for the
perception of surfaces, is there information for the perception of what they afford? Perhaps the composition and layout of surfaces constitute what they afford. If so, to perceive them is to perceive what they afford. This is a radical hypothesis, for it implies that the "values" and "meanings" of things in the environment can be directly perceived. Moreover, it would explain the sense in which values and meanings are external to the perceiver.

The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, but the noun affordance is not. I have made it up. I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment."

[Gib79, page 127]

Gibson's theory thus rely on the possibility to define the meaning of actions objectively by grounding the actions into a broader system that contains the actor and its environment: the ecological system [Gib79, chapter 3]. As Gibson says:

"The theory of affordances is a radical departure from existing theories of value and meaning. It begins with a new definition of what value and meaning are. The perceiving of an affordance is not a process of perceiving a value-free physical object to which meaning is somehow added in a way that no one has been able to agree upon; it is a process of perceiving a value-rich ecological object. Any substance, any surface, any layout has some affordance for benefit or injury to someone. Physics may be value-free, but ecology is not."

[Gib79, page 140]

This is an important point of the theory that seems to me often overlooked in HCI works, and departs from cognitivist psychology views in which the meaning of actions can only be understood as higher-level cognitive constructs built over the representations created by perception and the subject's psychological state and needs. Gibson is clear on this point:

"Note that all these benefits and injuries, these safeties and dangers, these positive and negative affordances are properties of things taken with reference to an observer but not properties of the experiences of the observer. They are not subjective values; they are not feelings of pleasure or pain added to neutral perceptions."

[Gib79, page 137]
### 3.1.3 Tools and learning: Affordances change for equipped actors

The field of Human-Computer Interaction is not really concerned with the general interactions between animals and their environment, but focuses on human beings interacting with technological artifacts. Gibson's theory of affordances can thus be of interest to HCI researchers and practitioners only if it provides an account of the phenomena that happen when people interact with artifacts. In particular, if an affordance like seating provided by a chair only exists for the animal that has a body compatible with this action (excluding, for instance, children too small to reach the seat), can we say that an affordance only exists for the person who has the right tool, who is in the right condition for doing the action, or knows how to do it?

Gibson embraces the idea that tools become extensions of their user's body. This idea is quite common in HCI, notably in the field of embodied interaction, following the views of philosophers like Heidegger and Merleau-Ponty [Sva13; Bon09]. Gibson indeed said:

> "When in use, a tool is a sort of extension of the hand, almost an attachment to it or a part of the user's own body, and thus is no longer a part of the environment of the user. But when not in use, a tool is simply a detached object of the environment, graspable and portable, to be sure, but nevertheless external to the observer. This capacity to attach something to the body suggests that the boundary between the animal and the environment is not fixed at the surface of the skin but can shift. More generally it suggests that the absolute duality of "objective" and "subjective" is false. When we consider the affordances of things, we escape this philosophical dichotomy."

[Gib79, page 41]

What we learn from this quote is that, for Gibson, the "animal" is not a static entity physically defined by its body. It can thus acquire new abilities, and the environment may hold affordances only for the actor that has become equipped for doing the action, including by learning to do it:

> "Of course, a horizontal, flat, extended surface that is nonrigid, a stream or lake, does not afford support for standing, or for walking and running. There is no footing, as we say. It may afford floating or swimming, but you have to be equipped for that, by nature or by learning."

[Gib79, p. 132]

### 3.1.4 Perception of affordances is learned

If affordances can exist only for the people with the right tools and knowledge, it means that the direct perception of such affordances
should itself be possible only in these conditions. Indeed, as discussed
by Bærensten et Trettvik [BT02], the perception of affordances is *always*
learned, and Gibson provides some accounts of how this learning can
happen in his perceptual framework (including when the perception
itself relies on tools, e.g., binoculars).

Kaptelinin, on the other hand, note that experimental validations of
Gibson’s theories “predominantly deal with processes that take place
in stable life conditions (e.g., perceptual learning during infancy)”
[Kap14, page 79]. As such, Gibson does not account for how people
learn to perceive “disruptive” affordances, which he claims is often
the case with new technologies. Kaptelinin thus concludes:

> "explicitly taking affordances into account means that supporting users' discovery of affordances and learning how to use them should be a key designer's concern. Currently there is a lack of evidence on how exactly people learn, unlearn, and re-learn new affordances."

[Kap14, page 80]

## 3.2 THE ROLE OF AFFORDANCE STRUCTURES IN HCI

Early HCI works on affordance by Vincente and Rasmussen, Gaver, and McGrenere and Ho [VR90; Gav91; MH00] simply accepted that computer systems have affordances without really questioning the possibility to apply the concept to such systems. They all highlighted, however, an aspect of affordances that Gibson did not comment much, which is the existence of structure and relations between affordances, and the role of such structures and relations in how people make sense of the world around them. They claim that the set of affordances provided by some (computerized) environment to a user is actually structured, and that understanding this structure may be beneficial to the design of user interfaces. The next subsections will describe some aspects of the structure in affordances.

### 3.2.1 Means-end relations

Vincente and Rasmussen remark that the affordances provided by a
system are not independent, as some may be involved in others [VR90].
For instance, a hammer affords striking, which allows to use it for
various purposes such as destroying things or forcing a sharp object
into another thing (typically, a nail into a piece of wood). The specific
affordance of striking provided by a hammer can thus be a *mean* to
provide the more general affordances of destroying something and
forcing something into another thing.

Conversely, the affordance of destroying something can be provided
in many other ways than by striking it, such as throwing it or throwing
something else on it. According to Vincente and Rasmussen [VR90],
the relation between the affordances of striking and destroying things provided by a hammer can be seen as *means-end relations*: a first affordance can be a mean to provide a second affordance, which conversely would be an *end* (goal/reason) for using the first affordance.

Obviously, the means-end relations form more complex structures: the affordance of destroying something can itself be a mean for other affordances like getting access to some place by destroying an obstacle, obtaining the materials of the destroyed thing, playing, preventing other people to use the object, etc. A first affordance can be a mean to provide a second affordance, which in turn can be a mean to provide other affordances, etc.

Gaver invoked a similar idea in his concepts of *sequential* and *nested* affordances [Gav91]. Although he was more interested in the way affordances are perceived (or not), his examples of first grabbing the handle of a door (or scrollbar) so that it can be rotated and pulled (or slid, for the scrollbar), contain implicitly the idea that one affordance (grabbing) can be a preliminary condition for – and thus, a mean for – another affordance (pulling/sliding). In addition, in his examples, the second affordance is itself a mean toward a third affordance: a door handle is rotated to unlock the door or pulled to open the door, a scrollbar handle is slid to scroll the view.

McGrenere and Ho also acknowledge that Gibson's theory of affordances implies they can be nested [MHoo], mentioning that "a word processor affords writing and editing at a high level, but it also affords clicking, scrolling, dragging and dropping". For them, "The functions that are invoke-able by the user are the affordances in software" and "affordances exist (or are nested) in a hierarchy" although "the levels of the hierarchy may or may not map to system functions".

According to Vincente and Rasmussen, creating such a means-end hierarchy for a specific computer system provides an account of how users of the system can learn its operation and find ways to achieve their goals by moving up or down the hierarchy [VR90].

### 3.2.2 The problem of unclear relations between affordances

In [Nor88], Norman introduced the concept of affordances in HCI to explain how people could successfully interact with objects that they had never seen before, yet have issues interacting with other everyday objects. Among the numerous examples he provides of taking advantage of affordances, we can cite the case of someone perceiving that a metal plate fixed on a door affords pushing the plate to open the door. Conversely, Norman provides examples of affordances that are not perceived, like a glass door that can be pushed to be opened, but the hinges of the door are hidden in such a way that people have no clue whether they should push on the left or right side of the door – both sides looking alike. He also provides examples
of cases where people do not know which of multiple perceived affordances will allow them to reach their goals: a doorknob affords pushing as well as pulling, so people might be confused about which one of these two actions they should perform to open the door.

However, when it comes to opening doors, Norman only considers the affordances for pushing plates and bars, pulling handles, grabbing knobs, etc. Such actions are only means toward a higher end: opening the door. This is also an affordance, and as Gaver acknowledges in his concept of nested affordances (as we have seen above), both types of affordances are connected by a means-end relation.

What Norman's examples make clear is the general problem of a broken perceived structure of affordances: when people need to use some mean to reach some end, they need to perceive the affordances of both the mean and the end, but they also need to perceive the relationship between the two. In Norman's examples, problems arise either because some affordance is not perceived, or because a mean-affordance is not perceived to be uniquely and unambiguously connected to an end-affordance. Although these two problems have different causes and may require different fixes design-wise,<fnref n="2" /> they have the same consequences on people confronted to them: confusion, distraction from their tasks, need to focus on the problem, and to find solutions by experimenting, searching clues, or asking people. Signifiers (and other ways to instruct users of a system how to use it) are only required when there is a risk that affordances and their means-end relations are not perceived clearly.

### 3.2.3 Instrumental affordances

The problem of unclear relations between affordances leads to the question of how people can understand the relationship between a mean-affordance and an end-affordance. Gaver's concept of nested and sequential affordances [Gav91] provide some elements of answer, by exposing the role of spatial and temporal relations between the objects that provide the affordances. This idea is more clearly established in the concept of *handling* and *effecter* affordances that Kaptelinin and Nardi introduced in their discussion of the affordances of instruments as mediators of an activity:

> "The distinction between handling affordances and effecter affordances is obvious in case of traditional instruments, discussed by Gibson, such as knives. Such instruments usually comprise two distinct parts, responsible, respectively, for handling (e.g., a handle) and affecting objects (e.g., blade). Apparently, the distinction also applies to digital technologies. For instance, common

<footnote n="2">² For instance, Norman proposes to use *natural mappings* when multiple devices are controlled by similar controllers and it is thus unclear which controller affords controlling the targeted device.</footnote>
user interface widgets, such as the scroll bar, enable two types of
actions [Beaoo]; operating on the widget (e.g., dragging the scroll
box) and operating on the object of interest (e.g., displaying in
the window a certain portion of the document). These two facets
correspond, respectively, to handling and effecter affordances of a
widget."

[KN₁₂]

In Kaptelinin and Nardi's view, the *instrument* embodies the relation between mean-affordances (its handling affordances, defined as possibilities for interacting with the artifact in question) and end-affordances (its effecter affordances, defined as possibilities for employing the artifact to make an effect on an object of interest), the combination of which they call *instrumental affordances*. But the tool can also provide affordances to ease its maintenance (for instance, a battery indicator), its aggregation with other instruments (e.g., a jack plug to connect a headset), or help learning about the instrument itself (e.g., a tooltip) [KN12]. Other authors have suggested similar notions to instrumental affordances, notably Hartson's notions of *physical* and *functional* affordances [Haro3] which are closely related to handling and effecter affordances respectively.

Obviously, the concept of instrument is useful to analyze human
interaction with physical tools and computers, as suggested by Michel
Beaudouin-Lafon [Beaoo]. Yet, if we are concerned with people un-
derstanding relations between affordances, the concept of instrument
is only useful if users recognize in the interface an instrument that
binds these affordances together: When Beaudouin-Lafon mention an
instrument for drawing rectangles with a mouse by making a drag ges-
ture that starts at one corner of the rectangle and ends at the opposite
corner, the instrument is not a visible object of the interface, and users
might not think about binding the handling affordance of making a
drag gesture with the effecter affordance of drawing a rectangle.

For all interaction techniques, and particularly for signifier-less de-
signs like this rectangle drawing instrument, user's ability to discover,
understand, and recognize or remember the binding between the han-
dling affordance and the effecter affordance is an important factor to
explain their (non-)use of the instrument and the usability issues it
can cause. This ability is an element of answer to the research ques-
tion "How can users know *swhidgets* despite their lack of signifiers?" (Section 2.7.3) and justifies considerations for aspects of the design of *swhidgets* that affect this ability, such as "Overarching Interface Logic" (Section 2.6.1), "Consistency with other interactions" (Section 2.6.1), "Action-Function Coupling" (Section 2.6.2), and "Physical metaphor" (Section 2.6.2).
3.3 DEBATED ASPECTS OF AFFORDANCES IN HCI

Does a button afford clicking? Does a scrollbar afford scrolling? Such questions are actually not easily answered. There has indeed been much debate in HCI about what should be considered as affordances in graphical user interfaces and digital devices [Nor99; BT02; Toro3; Kap14; KN12]. The debate might be technical, but it highlights some subtleties in the application of the concept of affordances to user interfaces and digital devices. To discuss these subtleties, I will start with the divergence of views about the affordances of buttons and scrollbars expressed by Gaver and Norman, coworkers and early promoters of affordances in HCI.

For Gaver [Gav91], “onscreen buttons seem to protrude from the screen; they afford pushing, but not moving or editing”. Indeed, for him:

"[the concept of affordances] implies that the physical attributes of the thing to be acted upon are compatible with those of the actor, that information about these attributes is available in a form compatible with a perceptual system, and (implicitly) that these attributes and the action they make possible are relevant to a culture and a perceiver."

[Gav91]

For Norman, on the other hand, “affordances reflect the possible relationships among actors and objects: they are properties of the world” [Nor99]. Hence, for him, “the computer, with its keyboard, display screen, pointing device, and selection buttons (e.g., mouse buttons) affords pointing, touching, looking, and clicking on every pixel of the screen” but a button does not provide an affordance for clicking:

"It is wrong to claim that the design of a graphical object on the screen 'affords clicking.' Sure, you can click on the object, but you can click anywhere. Yes, the object provides a target and it helps the user know where to click and maybe even what to expect in return, but those aren't affordances, those are conventions, and feedback, and the like."

[Nor99]

The opposition between Gaver's and Norman's views stem from different interpretations of Gibson's concept of affordances. A careful examination of these interpretations revealed that they were essentially built upon different assumptions about three specific concepts related to affordances:

1. The *meaningfulness* of what is afforded for the actor: to say that an environment affords something, should we simply require that this thing is possible, or should we additionally require that
it has some meaning for the actor? And if so, where does this
meaning come from?

2. *Social aspects* of the environment: does the distinction between physical and social aspects of the environment matter in the definition of affordances?

3. *Interpretation* of affordances: does the concept of affordances allow for interpretations of affordances or only their direct perception?

These elements will be addressed in the next three sections, respec-
tively in relation to Gibson's notions of *complementarity*, *ecology*, and *human display*.

3.4 DEBATE 1: MEANINGFULNESS OF AFFORDANCES

3.4.1 Norman's view: affordances as action possibilities only

As we have seen in his quotes above, Norman associates affordances with properties of the (physical) world, with no reference to their meaningfulness for the actor. The button only exists in software and as a virtual onscreen object, but not in the physical world, so for Norman it does not afford clicking more than would any other area of the screen: only the possibility of performing the physical action is considered in Norman's definition of affordances.<fnref n="3" /> This focus on the possibility of performing the action is made obvious in the following quote:

"It is possible to change the physical affordances of the screen so that the cursor appears only at spots that are defined to be "clickable." This would indeed allow a designer to add or subtract the affordance of clicking, much as many computer forms afford the addition of characters only in designated fields. This would be a real use of affordances.

In today's screen design sometimes the cursor shape changes to indicate the desired action (e.g., the change from an arrow to hand shape in a browser), but this is a convention, not an affordance. After all, the user can still click anywhere, whatever the shape of the cursor. Now, if we locked the mouse button when the wrong cursor appeared, that would be a real affordance."

[Norgg]

<footnote n="3">³ Note that for Norman, "physical" actions can however involve visually tracking the position of an onscreen virtual object like the mouse pointer.</footnote>
### 3.4.2 Meaningfulness as potential

Norman's examples of affordances in user interfaces totally ignore the question of the meaningfulness of actions: clicking anywhere in the screen is certainly a possible action, but it is also a meaningless one in the sense that it has been abstracted from its purpose, as Hartson remarked in his introduction of the concept of *functional affordances* [Haro3] (see Section 3.2.3).

Such an abstraction of purpose is common in Gibson's examples of affordances, too. For instance, a flat ground affords walking, and it does not matter if you have a reason for doing so or not. This kind of abstraction of purpose implies however that the meaningfulness of an action comes from its potential to be a mean toward higher-level ends, or to be a mean for higher-level affordances that can have a clear ecological value. For instance, you can walk to reach food or a safe place to rest.

Unfortunately, Norman does not mention the existence of such higher-level affordances, even if he acknowledges that clicking on a button is a meaningful action for the user: "This is what the interface designer should care about: Does the user perceive that clicking on that object is a meaningful, useful action, with a known outcome?" [Nor99]. Yet, this meaning is only defined in relation with the user's goals and needs in a concrete situation. The action of clicking on a button, abstracted from the context of use, does not seem to be meaningful in itself for Norman.

On one hand, Norman acknowledges the purpose-abstracted affordance of clicking anywhere, but on the other hand he seems to dismiss the higher-level purpose-abstracted affordance of clicking on a button. I can only see one reason for this apparent inconsistency in Norman's concept of affordance, which is that it ignores any notion of meaningfulness to only consider the possibility of a physical action.<fnref n="4" />

### 3.4.3 Norman's view: Meaning is conveyed by signifiers

Norman's view implies that after having perceived an affordance of clicking anywhere on screen, the user has to interpret this meaningless affordance to deduce the possibility of a meaningful action (clicking the button to achieve something). For Norman, the meaning of the action has to be added on the perceived affordance, which is exactly the view to which Gibson objected, as we have seen in section 3.1.2. Adding meaning to a meaningless "perceived affordance" makes it a *signifier*, a concept that Norman introduces like this:

<footnote n="4">⁴ It is also possible, however, that there is a little ambiguity in Norman's wording, and that he does not reject the idea of a real (gibsonian) affordance for clicking on buttons, but only rejects it for his own concept of *perceived* affordances. This possible interpretation will be addressed in the next section.</footnote>
"People need some way of understanding the product or service they wish to use, some sign of what it is for, what is happening, and what the alternative actions are. People search for clues, for any sign that might help them cope and understand. It is the sign that is important, anything that might signify meaningful information. Designers need to provide these clues. What people need, and what designers must provide, are signifiers. Good design requires, among other things, good communication of the purpose, structure, and operation of the device to the people who use it. That is the role of the signifier.

> The term signifier has had a long and illustrious career in the exotic field of semiotics, the study of signs and symbols. But just as I appropriated affordances to use in design in a manner somewhat different than its inventor had intended, I use signifier in a somewhat different way than it is used in semiotics. For me, the term signifier refers to any mark or sound, any perceivable indicator that communicates appropriate behavior to a person."

[Nor13, page 14]

As things that convey "appropriate behavior", signifiers can provide information about affordances but are not restricted to that. For instance, according to Norman, if you're running to catch a train and finally reach the station, an empty platform can be a signifier – in this case informing you that you have missed the train. Affordances are only concerned about whether you can get in the train or not, but do not depend on your will to do so. On the other hand, signifiers can provide information related to the actor's goals and expectations: as we will see in Chapter 5, they can provide feedback or feedforward, inform about the mapping between controls and the things controlled, etc.

### 3.4.4 Meaningfulness from complementarity

Unlike Norman, Gaver explicitly states that affordances should be relevant to a culture and a perceiver. For him, clicking on a button is a relevant action for the user, not only because it allows her to reach her goals, but also because the mouse and onscreen buttons are complementary elements of a system. He says for instance that "just as affordances of door handles imply the complementarity of handles and the motor system, so do the affordances of onscreen buttons imply the complementarity of buttons and mouse-driven cursors" [Gav91].

This notion of complementarity is another way to define the meaningfulness of affordances, in agreement with Gibson's statement that an affordance "refers to both the environment and the animal in a way that [...] implies the complementarity of the animal and the environment" [Gib79, page 127]. There are multiple definitions of the
term “complementarity” in dictionaries, so one can ask which of these
definitions is used by Gibson and Gaver. I will argue that they use the
term “complementarity” to describe a situation where two elements
(or more) form a system in which none of these elements can exist
without the others. Animals and their environment are complemen-
tary in that sense, since an animal cannot live all by itself without an
environment to support its life, and a physical world without living
animals in it would not deserve to be called an “environment” (what
would it be an environment of?). Moreover, because the environment
also includes other species and resources, its ecological stability de-
pends on the presence of the animal considered, or at least of a set of
animals that inhabit the same ecological niche.

In that sense, it is obvious that buttons and mouse-driven cursors<sup>5</sup> are complementary: A button would not be usable if there was no way to point at it somehow and then trigger an event (both possibilities being provided by moving the mouse and clicking on the mouse's physical button). Conversely, if there was no buttons and no other interface elements that make a good use of the mouse, there would be no point in having mouse-driven cursors in the system. Although there are other ways to use a mouse,<sup>6</sup> an onscreen button provides a reason to use the mouse and to exploit its affordance for clicking at positions on screen, and its design fits well this action. Indeed, the onscreen button has some area and a regular and convex shape, which both make it well adapted to the action of pointing at it and clicking on it,<sup>7</sup> in the same way than the shape and size of a doorknob is well adapted to the action of grabbing it.

As long as a user has integrated the operation of a mouse – i.e.,
how to manipulate it to move the cursor at the desired position,
and how to press the physical buttons of the mouse – then she has
integrated that the system affords clicking at a particular position. She
is equipped for clicking at positions in the screen, as would Gibson say
(see Section 3.1.3). But what positions should be clicked at? Onscreen
buttons (as well as other graphical user interface elements) allow users
to translate the action of clicking at positions into the action of clicking
on things. In this sense, the button’s design and the action of clicking
on it are complementary, and this complementarity is a property of
the system that can be understood by – and makes sense to – users
who are equipped to operate mice. There is a complementarity of the
button and the mouse-equipped user, which makes clicking on the

<footnote n="5">5 Or, more generally, pointing devices, including touch screens or keyboards with arrow keys and action keys like ‘Enter’.</footnote>

<footnote n="6">6 For instance, in some video games there is no pointer, and moving the mouse controls instead the camera orientation in a 3D world. Even when there is a pointer, using the mouse to draw a shape is a very distinct action than clicking on a button.</footnote>

<footnote n="7">7 On the contrary, clicking a specific pixel of the screen can be very hard: a mouse and display may afford clicking “anywhere in the screen” but the affordance should rather be defined as “clicking in any sub-region of the screen” with the pixel as the smallest division of the screen, because there is no such thing as absolute pointing.</footnote>
button a meaningful action, and thus allows to consider clicking on
buttons to be an affordance of the system.

For similar reasons, one can talk about the affordances of dragging
an onscreen handle, scrolling with a scrollbar, and other more elabo-
rate interaction techniques, as long as the user has integrated the basic
principles of these interactions.<fnref n="8" /> This however implies that affordances
may only exist for people with the required prior knowledge and skills.
Gaver and Gibson embrace this view, while Norman rejects it to only
consider the actions than an actor can physically perform with her
body.

For Norman, an onscreen button is a signifier conveying the mean-
ing that it can be clicked to trigger some event, and it conveys this
meaning through a mix of mechanical, logical, and cultural constraints,
as well as through a user mental model of how the system operates
[Nor99]. These are representations that exist in the head of the user,
while Gibson and Gaver see the complementarity of the user and the
design of some user interface element as an objective property of the
user+GUI system.

A consequence of Gibson's and Gaver's view is that affordances
cannot be understood in isolation but need to be studied as elements
of a system of affordances. Similarly, user's ability to exploit an
affordance may depend on her understanding of a whole subset of
the system of affordances, which is a concern related to the general
research questions of *overarching interface logic, consistency with other
interactions, physical metaphor* that convey a system of affordances,
and *multiple ways to do a task*, that I have identified for *swhidgets*
(Section 2.6).

3.5 DEBATE 2: SOCIAL ASPECTS OF HCI AFFORDANCES

3.5.1 Norman's view: *Affordances, conventions and symbols*

As we have seen, Norman acknowledges that clicking on a button
should be a meaningful action for the user in relation to her goals
and needs. However, the goal-related meaning of clicking on a button
and of similar interaction techniques is for him a cultural, learned
convention:

"Cultural constraints are conventions shared by a cultural group. The fact that the graphic on the right-hand side of a display is a "scroll bar" and that one should move the cursor to it, hold down a mouse button, and "drag" it downward in order to see objects located below the current visible set (thus causing the image itself to appear to move upwards) is a cultural, learned convention.

<footnote n="8">8 This is not restricted to human-computer interaction techniques: it is also what allows to say that a bike affords locomotion, which obviously only holds for someone who knows how to ride a bike.</footnote>
> The choice of action is arbitrary: there is nothing inherent in the devices or design that requires the system to act this way. The word "arbitrary" does not mean that any random depiction would do equally well: the current choice is an intelligent fit to human cognition, but there are alternative methods that work equally well."

[Nor99]

He insists that “a convention is a constraint in that it prohibits some activities and encourage others” and “symbols and constraints are not affordances” [Nor99].

Strangely, however, the fact that a mouse pointer moves in the same direction than the mouse itself, or even that a mouse has to be moved to control the pointer<fnref n="9" /> – which are also conventions – does not prevent him from saying that the mouse and display together afford pointing. And yet, someone who has never seen a mouse may not think about moving it to move the cursor (or for that matter, that it is useful to move the pointer). Indeed, the choice of action is also arbitrary. So what makes the basic operation of the mouse an affordance for Norman but not the operation of a scrollbar?

As I have discussed in sections 3.4.1 and 3.4.3, Norman distinguishes affordances and signifiers: the former are (for him) meaningless physical action possibilities, and the later are what can be interpreted so as to convey the meaning of actions. In that view, it does not matter that moving a mouse to move the pointer is a convention, because it is also a physically possible action which meaning is not considered, and thus Norman can say that it is an affordance for him. Yet, the mouse as an object is also a signifier that conveys the meaning that it can be used to move the pointer, which is a convention. The scrollbar, on the other hand, is only a signifier because dragging an onscreen handle and scrolling the view are not physical actions.

After having distinguished affordances and signifiers, Norman finally focused on signifiers and highlighted their social origin, minimizing the importance of the role of (perceived) affordances in the interaction with everyday objects. For instance, after having talked so much about the affordances of door handles in [Nor88], he later said the following:

> "A doorknob has the perceived affordance of graspability. But knowing that it is the doorknob that is used to open and close doors is learned: it is a cultural aspect of the design that knobs, handles, and bars, when placed on doors, are intended to enable the opening and shutting of those doors. The same devices on fixed walls would have a different interpretation: they might offer support, for example, but certainly not the possibility of

<footnote n="9">⁹ Which is not the only way to use a mouse, as we have seen in Footnote 6.</footnote>
> *opening the wall. The interpretation of a perceived affordance is a cultural convention.*

[Nor13, page 145]

If we follow Gaver's parallel between mouse-driven cursors and door handles, Norman's ideas about doorknobs would translate as follows: the affordance of clicking anywhere on screen can be interpreted differently by the user whether the click happens on a button or elsewhere, and this interpretation is a cultural convention. Hence, the onscreen button is a signifier that conveys the meaning of clickability through a cultural convention.

Unlike Gibson who acknowledges that users can understand the complementarity of mouse-driven cursor and button designs and use this complementarity to perceive the affordances of buttons, Norman only acknowledges interpretations of perceived affordances rooted in conventions. In the field of semiotics from which Norman borrowed the concept of signifiers, the signs that convey their meaning through a cultural convention are called *symbols*, and are only one of the three primary ways signs can convey meaning (with *icons* and *indexes*). Section 3.6 will discuss more in details the link between affordances, symbols, and other types of signs.

### 3.5.2 Gaver's view: Culture only highlights some affordances

Contrarily to Norman, Gaver explicitly states that affordances should be relevant to a culture and a perceiver. The fact that interactions are based on conventions or not does not seem to matter in his understanding of affordances. A button affords clicking on it because this action has a specific and relevant meaning for the user, even if the notion of onscreen buttons is a social construct. Buttons afford clicking even if this design is a convention.

Like Norman, Gaver acknowledges that culture has an impact on the perception of affordances. However, for him, it is more a question of learning to pay attention to some affordances that are then directly perceived, rather than learning to interpret affordances:

> *"The actual perception of affordances will of course be determined in part by the observer's culture, social setting, experience and intentions. Like Gibson I do not consider these factors integral to the notion, but instead consider culture, experience, and so forth as highlighting certain affordances. Distinguishing affordances and the available information about them from their actual perception allows us to consider affordances as properties that can be designed and analyzed in their own terms. Learning can be seen as a process of discriminating patterns in the world, as opposed to one of supplementing sensory information with past experience. From this perspective, my culture and experiences*
> may determine the choice of examples I use here, but not the existence of the examples themselves."

[Gav91]

This view is shared with Gibson and other authors, particularly those who approach HCI from the angle of Activity Theory – notably by Bærentsen and Trettvik [BT02] who also comment on how the design of artifacts emerge from activity in a way that necessarily account for people’s ability to learn perceiving their affordances, before becoming a cultural aspect of the activity.

### 3.5.3 Gibson's notion of Ecology

The origin of the divergences in Norman's and Gaver's views on affordances may be found in Gibson's notion of an *ecology*, which is the framework in which the meaningfulness of the interactions is defined. The word *ecology* can be understood as a system of animals living in their natural physical environment, as seems to do Norman – and Gibson provides many examples of affordances that fit in such an understanding. But Gibson actually uses the word *ecology* in a much broader way, which includes aspects of human life in a technological society that can seem very remote from life in natural environments. For instance, he insists that...

> "[... the man-built environment] is not a new environment – an artificial environment distinct from the natural environment – but the same environment modified by man. It is a mistake to separate the natural from the artificial as if they were two environments; artifacts have to be manufactured from natural substances. It is also a mistake to separate the cultural environment from the natural environment, as if there were a world of mental products distinct from the world of material products."

[Gib79, page 130]

Gibson also gives examples of affordances of man-made tools and even mentions that a "postbox affords letter-mailing to a letter-writing human in a community with a postal system" [Gib79, page 139]. It implies that his notion of ecology is broad enough to incorporate postal systems, and that in such an ecology, "letter-mailing" is a meaningful action beyond simply inserting a flat floppy object called letter in a slot of a box called postbox. Where Norman would only acknowledge the affordance of inserting a flat floppy object in a slot and consider that this affordance has to be interpreted as the possibility to post a letter, Gibson acknowledges an affordance for posting letters that can be directly perceived.

Since Gibson rejects the distinction between natural and cultural worlds, and accepts that affordances can exist for actions that only have a culturally-defined meaning, Norman's view on affordances
seems a little bit limited. It thus seems acceptable to talk about the affordances of digital devices and widgets in graphical user interfaces, as did Gaver, but also Vincente and Rasmussen, and McGrenere and Ho (see Section 3.2). However, Gibson only sets up a framework that is compatible with an HCI notion of affordance, but does not provide more practical insights for a proper use of affordances in HCI. For this reason, HCI authors like Gaver, Bærensten and Trettvik, or Kaptelinin and Nardi [Gav91; BT02; KN12] have discussed how Gibson's original concept could be precised or extended for the needs of HCI.

Although entering the details of these works is not necessary in this thesis, they provide important concepts for the study of the general research questions on *external and accidental sources of knowledge* (Section 2.6.3) and *understanding of similar interaction techniques* (Section 2.6.4). They also provide hints about the possibility to answer the question *how can users know swhidgets despite their lack of signifiers?* (Section 2.7.3) in terms of propagation of knowledge through social networks or by building an "interaction culture".

## 3.6 DEBATE 3: SYMBOLS AND AFFORDANCES IN HUMAN DISPLAYS

### 3.6.1 *Symbols involved in affordances*

We just saw with the example of the postbox that Gibson's notion of affordances includes interactions with man-made objects and designs that have a socially-constructed meaning. This view presents however some challenges when it comes to understanding what aspects of an object create a specific affordance, as it sometimes implies conventions and symbols (i.e., marks that have a conventional meaning). A naive understanding of the role of symbols and other interpreted signs – including Norman's signifiers – in the creation and perception of affordances might thus lead to nonsensical conclusions, such as an affordance being conditioned by the presence of a signifier. For instance, Bærentsen and Trettvik discuss the affordances of bills and coins for paying [BT02]: one could believe they afford paying because they display the symbols of their monetary value and proofs that they are legit.

Affordances often come from the fact that people follow some convention, but not from the fact that this convention is signaled by a specific design or symbol. For instance, as long as people accept your bills for payment, the bills afford paying, whatever they look like. A fake bill still affords paying if people do not realize it is fake. As long as people put letters in a box and the postman collects them, the box affords mail posting, even if it does not display the logo of the postal service. Conversely, a box that is in all points identical to the postboxes installed by the postal service may be a fake one designed
to steal your mails, or an old postbox that is not used anymore: in both cases it does not afford mail posting despite its design.

This is not even an aspect of affordances that is specific to objects with a conventional design: pears afford eating, and they have a specific shape that is not a conventional design but the result of a process of natural (or human) selection. That people use the shape of the pear and conventional designs to perceive affordances make no doubt, but the affordance itself does not depend on an individual's ability to perceive it. And for Gibson, the shapes, symbols and conventional designs are not perceived as such to be interpreted as signifying affordances: instead, the affordance is directly perceived in a process that integrates these conventional features without focusing on them (see Section 3.1.1).

### 3.6.2 *From Gibson's human displays to Norman's signifiers: affording knowledge*

In many aspects, a graphical user interface fits the definition of what Gibson calls *human displays*:

> "A display, to employ a useful generic term, is a surface that has been shaped or processed so as to exhibit information for more than just the surface itself. For example, a surface of clay is only clay, but it may be molded in the shape of a cow or scratched or painted with the profile of a cow or incised with the cuneiform characters that stand for a cow, and then it's more than just a surface of clay."

[Gib79, page 42]

Gibson insists that the perception of human displays is different and more complex than the direct perception of affordances in the environment [Gib79, chapters 15 and 16], and that the affordances of human displays are two-fold: the affordances of the object itself (e.g. grabbing it) and the affordance of communicating and holding knowledge: "It can be suggested in a preliminary way, however, that images, pictures, and written-on surfaces afford a special kind of knowledge that I call *mediated or indirect*, knowledge at second hand." [Gib79, page 42]. This is particularly clear in his analysis of the perception of pictures that represent scenes:

> "A picture always require two kinds of apprehension that go on at the same time, one direct and the other indirect. There is a direct perceiving of the picture surface along with an indirect awareness of virtual surface [in the scene depicted] – a perceiving, knowing, or imaging, as the case may be."

[Gib79, page 283]
The idea that human displays can afford mediated knowledge is quite close to Norman's concept of signifiers. In that spirit, other researchers have proposed to consider similar types of affordances:

Hartson proposes to distinguish four types of affordances: *cognitive affordances*, *physical affordances*, *sensory affordances*, and *functional affordances* [Har03]. While he associates *physical affordances* with Norman's definition of (perceived) affordances, he also recognizes *cognitive affordances* that match Norman's more recent concept of signifiers. Indeed, Hartson defines cognitive affordances as "*a design feature that helps, aids, supports, facilitates, or enables thinking and/or knowing about something*". He proposes "*a button label that helps users know what will happen if they click on it*" as a canonical example of cognitive affordance, and discusses how the design of doors and doorknobs includes both physical and cognitive affordances – in a way that is not dissimilar with the analysis of affordances and signifiers that I have done in Section 3.5.1.

Kaptelinin and Nardi propose a mediated action perspective on affordances [KN12], which also distinguishes between four types of affordances: *instrumental*, *maintenance*, *aggregation* and *learning* affordances. Although they do not provide an explicit definition of learning affordances, it seems clear from their examples that they imply something similar to Hartson's cognitive affordances – maybe only more limited in scope: learning affordances can teach how to use the artifact that provides the affordance while cognitive affordances can inform about anything. Indeed, Kaptelinin and Nardi give the examples of tooltips, help screens, and standardized icons (such as an USB icon inscribed next to an USB socket), and mention that shoelaces lack a learning affordance since most people would not figure out by themselves how to lace shoes.

Gibson, Hartson, Kaptelinin and Nardi thus seem to support the idea of what could generically be called "knowledge affordance". It can however been questioned whether this concept really fits in Gibson's framework or if it should be considered close to, but distinct from affordances. Indeed, Gibson only evokes the idea very quickly, taking precautions ("it can be suggested in a preliminary way"...). His wording does not allow to determine with absolute certitude if he means that, for instance, the word "pull" engraved on a door's metal handle affords the knowledge that the door should be pulled to be opened (which Norman would disagree with), or if he means in a general way that surfaces like metal afford being carved to communicate knowledge – the affordance would then not be to convey some knowledge, but to be turned into a signifier. The former would confuse the notions of affordances and signifiers, while the later would reduce the notion of knowledge affordance to a question of being able to produce signifiers.

In the definition of signifier-less designs in Section 6.1, I will use an intermediary interpretation, which I think is closer to Gibson's
thought: the idea that some object in the environment affords to hold information if it is interpreted by the individual, although this affordance exists and can be recognized by the individual even if she is unable to actually interpret the information (e. g., a poster that is too far to be read, or with text written in a foreign language).

## 3.7 CONCLUSION AND RECAP

### AFFORDANCES AS INTERACTION POSSIBILITIES
In the introduction of this chapter I claimed that the terms “relevant interaction possibilities” needed some clarification and examples and that what I meant by that was exactly the concept of affordances as originally introduced by Gibson. Affordances are interaction possibilities for an individual in an environment, but the terms “interaction” and “environment” have to be taken in a broad sense: “Interaction” includes high-level interactions like nutrition or safety as well as low-level interactions like breathing, walking, or grabbing an object (Section 3.2.1). “Environment” also includes socially-constructed artifacts like the user interfaces of computerized devices, where interactions can range from mouse clicks and keystrokes at the lowest level, to complex application functionalities such as editing text or chatting with friends at higher levels (Section 3.2.1).

### AFFORDANCES ARE MEANINGFUL
Affordances are not only interaction possibilities, but *meaningful* ones: They have a meaning that comes from the *ecological system* in which the individual is involved (Section 3.1.2), where “ecological” has to be understood in a way that includes socio-cultural systems and activities (Section 3.5.3). Their meaning also comes from their *potential to reach higher goals* (Section 3.4.2), a potential that can be expressed in the form of means-end hierarchies of affordances (Section 3.2.1). Their meaning also comes from the *complementarity* of the individual and the environment in the sense that the afforded interaction possibility can only exist in the specific system created by the environment, system that would not exist without this interaction possibility (Section 3.4.4).

### DYNAMICS OF AFFORDANCES
Affordances can change for an individual as she *equips tools* or *learns new skills* like swimming or clicking with a mouse (Section 3.1.3). For human beings, the ability to perceive a specific affordance is not innate but *always learned* (Section 3.1.4). Once learned, perception of affordances is *direct*: it does not rely on intermediary representations or interpretations (Section 3.1.1). But affordances exist independently of an individual's ability or need to perceive them (Section 3.1.2).
**AFFORDANCES HELP KNOWING WHAT TO DO** Affordances play an important role in knowing how to interact with the environment: Individuals can exploit *means-end relations* between affordances to determine what to do to reach their goals (Section 3.2.1). The discovery of means-end relations between affordances can exploit spatial or temporal relations between the objects providing the affordances, notably when they can be understood as handling and effecting affordances of an *instrument* (Section 3.2.3). However, when affordances and their means-end relations cannot be perceived by an individual, *usability issues* may arise (Section 3.2.2). Such issues can to some extent be avoided by designs that embed *signifiers* (Section 3.4.3), i.e., anything that can be interpreted by individuals to inform on what they can/should do – although there is some unresolved debate about the possibility to also interpret affordances to get such information (Section 3.6).

The notion of affordance will be useful for the derivation of a precise definition of signifier-less designs in Chapter 6, as an answer to the research question *what is a signifier-less design and how swhidgets are different from other signifier-less designs?* (Section 2.7.1). It is also important to provide possible answers to the research question *how can users know swhidgets despite their lack of signifiers?* (Section 2.7.3), from the idea that users could learn to perceive affordances even when they are not explicitly signified, to the idea that affordances are meaningful and belong to systems of affordances and means-end hierarchies that can help users foreseeing unknown affordances to discover them through an exploration of the interface. This last point is the reason why I will discuss in the next chapter a classical model of interaction with the world that integrates stages of perception and interpretation.
<figure><img src="image_23.png" /></figure>
<figure><img src="image_24.png" /></figure>
To answer the research question *how can users know swhidgets despite their lack of signifiers?* (Section 2.7.3), we need to understand why signifiers would be necessary to discover and adopt interaction techniques in the first place. I have presented in Section 3.2.2 how Norman understands the issues caused by unclear relations between affordances and how such issues could be solved with signifiers. But this only says that signifiers can convey useful information (which is basically how they are defined by Norman, as seen in Section 3.4.3), without specifying how people will actually use such information to interact with the world. The model of the Seven Stages of Actions, also proposed by Norman, is a popular model to investigate users' informational needs as they interact with the world (e. g., [Har03; Ver+13]), and it will be used to organize the discussion around the different types of signifiers in Chapter 5. The idea that affordances should be signified if not unambiguously perceptible is born from the way of thinking human-computer interaction that also gave birth to this model: the two are closely connected. Therefore, if we want to understand why interaction techniques like *swhidgets* are designed without signifiers, it might be useful to go beyond the HCI philosophy sustaining this model – which is why, in this chapter, I will also highlight its limits.

## 4.1 THE SEVEN STAGES

Hutchins, Hollan, and Norman proposed the model of the Seven Stages of Action as a way to discuss the feeling of "directness" in direct manipulation interfaces [HHN85]. Norman later popularized the model and promoted it as design aids [Nor88; Nor13]. This model of human interaction with the world, illustrated in Figure 4.1, describes the mental and physical activities that people perform while completing task goals, and has become a classical concept of HCI to explain the way people interact with digital devices.

Norman structures the action cycle into two aspects: *executions* and *evaluations* (shown respectively on the left and right sides of Figure 4.1). The idea is that we achieve what we want to happen by executing some actions and then evaluating how these actions affected the world. Both execution and evaluation can bring their share of difficulties as we interact with the world to satisfy goals, which is why Norman talks about the *Gulf of Execution* and the *Gulf of Evaluation* to express the difficulties that can happen between the formulation of a goal and the actual interaction with the world (or interface) that one does to
<figure><img src="image_25.png" /><figcaption>Figure 4.1: The Seven Stages of Action.</figcaption></figure>

reach this goal. These two aspects of execution and evaluation are then divided into three stages each, to give six stages – which makes seven stages by adding the formulation of a goal as a stage. I will now describe these stages in the order of the cycle starting from the formulation of a goal.

**GOALS** According to Norman, “A *goal* is something to be achieved, often vaguely stated”. Norman starts his presentation of the Seven Stages of Action in [Nor88] with the example of a precise goal: preparing a film projector for a presentation. But he quickly acknowledges that everyday behavior is usually concerned with much less precisely defined goals, and he gives examples such as “get something to eat”, “get to work”, “get dressed”, or “watch television”. The important point is that if goals specify something to be achieved, they do not specify what should be done to achieve it: goals define *what* should be achieved, not *how*. Specifying this *how* is the matter of the next two stages in the Gulf of Execution.

**INTENTION** For the Gulf of Execution, the first stage after the formulation of a *goal* is the stage of *Intention*, where people consider the various possible ways to reach their goals and select one as an intention to act. Norman defines an *intention* as “a specific action taken to get to the goal” and a “specific statement of what should be done” [Nor88, page 46]. He gives the example of a goal defined as
"having more light so that I can read" and different ways to satisfy this goal: turning the bed light on, turning the ceiling light on, opening the curtains to let daylight enter the room, moving to a place where there is more light, etc. In the stage of Intention, people thus need to translate their goals into intentions to act, i.e., things to achieve, which expected effect in the world will satisfy their goals. Yet, as Norman says, "even intentions are not specific enough to control actions" [Nor88, page 46].

**SEQUENCE OF ACTIONS** After the stage of Intention in the Gulf of Execution, comes the stage of *Sequence of Actions*. In this stage, sequences of actions are specified to realize the intention to act selected in the stage of Intention. An important difference between the two stages is that they consider actions either from an external or internal point of view: intentions to act are concerned with the results of actions in the world, and thus consider actions from a point of view that is external to the actor. But in the stage of Sequence of Actions, we consider what Norman calls "internal commands", which he defines as actions "that can control my muscles" [Nor88, pages 47 and 48]. In this stage, we thus consider actions from a point of view that is internal to the actor.

We can illustrate the difference by expending on Norman's example of getting more light to read, with the case where "turning on the bed light" is the selected intention to act. Then, multiple sequences of "internal" actions can be devised to reach this result: one could reach the lamp switch with the right hand, or with the left hand, or throw an object on it from a distance, or even ask another person to do it. Note that these actions can involve other persons and objects, but are nonetheless defined in terms of what the actor has to do. While the stage of Intention was characterized by a search of effects of actions, this stage is characterized by a search for a specific sequence of actions that one can do among those that will cause the chosen effect, and the result of this stage is a plan for action.

**EXECUTION STAGE** There is not much to say about the stage of *Execution*<fnref n="1" />: once a plan for action has been devised in the actor's head, she just has to physically perform the planned actions. This closes the Gulf of Execution.

**PERCEPTION** The Gulf of Evaluation starts with the *Perception* of the state of the world, after this state has been affected (or not) by the execution of actions. Here, perception should not be understood as in Gibson's theory (see Section 3.1.1), but rather as in the cognitivist view

<footnote n="1">¹ One subtle terminology issue with the model of the Seven Stages of Action is that the Gulfs of Execution and Evaluation are named after the last of their three stages. The reader should then take care of not mixing up the stage of *Execution* with the *Gulf of Execution*, or the stage of *Evaluation* with the *Gulf of Evaluation*.</footnote>
on perception that Gibson challenged – i.e., the creation of a mental representation of the world's state.

In the case of the lamp switch example, perception could be described as building a 3D mental representation of the lamp, switch, and environment involved in the interaction, as well as the tactile perception of touching the switch, the auditory perception of the sound made by the switch when it flips, and of course, the visual perception of light coming out of the lamp.

**INTERPRETATION** Norman provides much less definitions and examples for the Gulf of Evaluation than he did for the Gulf of Execution, and as a consequence, the definition of its stages can be relatively unclear. In the case of the stage of *Interpretation*, he only says that “perception must be *interpreted* according to our expectations” which he also describes as “trying to make sense of it” [Nor88, pages 47 and 48]. In the case of the lamp switch example, I guess this stage would consist in interpreting the change in the geometry of the switch as a motion of the part pushed, interpreting the tactile and auditory sensations as feedback that the switch has been flipped, and interpreting the light coming out of the lamp as the result of flipping the switch.

**EVALUATION STAGE** In the stage of *Evaluation*, the user has perceived how the world responded to her action and interpreted this response by giving it some meaning, and now has to compare this meaning relatively to what she expected to happen: was the intended effect produced, or did something else happen instead? In the case of the lamp switch, this stage would consist in two things: validating the sequence of actions and the intention. First, acknowledging that there is light coming out (or not) of the lamp as expected if the actions that we took were correctly executed and everything worked as expected. Second, checking that there is now enough light to read, since it was the goal that started the cycle. In other words, it consists in asserting that the intention selected was indeed a valid way to satisfy the goal. This closes the Gulf of Evaluation.

**STARTING A NEW CYCLE** After the stage of *Evaluation*, a new cycle is started with a goal that depends on the result of the Evaluation stage: If the evaluation failed, the same goal may be used for a new cycle with the constraint to do things differently. Or a new goal can be formulated as understanding why the things did not go as expected. But the goal can also simply be abandoned in favor of another activity. If the evaluation succeeded, then a new cycle can start with a new goal defined by the result of the interaction (for instance, if the previous goal was to get directions to go somewhere, then the next goal could be to follow these directions). But we also often simply resume an interrupted activity, returning to a previous goal.
Typically, in the case of the lamp switch, the goal would be to resume reading if there is enough light, and to find another way to get light otherwise, or I could decide that it is not the time to read anymore and start something else, setting a new goal. Yet, in this example, reading itself might be an activity that fulfilled a higher-level goal, such as enjoying a work of fiction or searching for an information needed to accomplish another task, and these activities can influence the formulation of the next goal.

### IN AND OUT OF THE CYCLE
Norman makes it clear that a goal can lead to sub-goals and intentions to sub-intentions, so that one stage of the cycle can trigger a full sub-cycle, and conversely, that a full cycle can sometime be seen as just one intent or action of a higher-level cycle. Not all cycles have to be completed neither: activities can be interrupted, to be resumed after the interruption or at a later time, or even to be abandoned in the end. He also mentions that a cycle can be started from any stage, not only from the formulation of a goal: people "may respond to the events of the world (in what is called data-driven behavior) rather than to seek out plans and goals. An event in the world may trigger an interpretation and a resulting response. Actions may be executed before they are fully developed" [Nor88, page 49].

## 4.2 LIMITATIONS AND AMBIGUITIES OF THE MODEL

### 4.2.1 Status of the model and nature of the stages

Norman acknowledges that "the seven stages form an *approximate model*, not a complete psychological theory. In particular, the stages are certainly not discrete entities. Most behavior do not require going through all stages in sequence, and most activities will not be satisfied by single actions" [Nor88, page 48]. Hornbæk and Oulasvirta additionally note that "whereas Norman's model provides important concepts, it has limited empirical support; we cannot think of a single empirical challenge of the model" [HO17].

One aspect of the model that is not described by Norman is the exact nature of the stages: should they be understood as cognitive processing operations that happen sequentially in time? Or as cognitive layers in the brain that process information in parallel but receive information only from the previous layer and transmit information only to the next layer? Or as more abstract descriptions of human-world interactions in terms of dependencies between cognitive processes (e.g., "a sequence of actions can only be considered if a corresponding intention has been specified first")?
4.2.2 Distinction between intentions and sequences of actions

I notably find it hard to work with the concept of intention to act, because it is hard to separate an intention such as “switch on the lamp” from corresponding actions such as “get up, walk to the switch, and flip it”. So, the line between the stage of Intention and the stage of Sequence of Action seems blurry. For instance, in [Nor88], Norman says that if I change my intention from “push the switch button on the lamp” to “ask someone passing nearby to do it for me”, then I change both my intention and the sequence of action – but in [Nor13], he provides the same example and seems to say that it would only be a change in the sequence of action, not in the intention: should the intention be only “switch on the ceiling lamp”, or can intentions also include constraints on how to act, such as in “asking someone else to do it for me”?

It is also hard to accept Intention and Sequence of Actions as succes-
sive steps: it is much more likely that when selecting an intention to
act, we also consider the sequence of actions that would be required
to fulfill this intention to act. Indeed, the amount of physical effort,
the chances to miss, or the ability to perform the sequence with little
thinking are all criterion used in the selection of an intention to act
that depend on the precise sequence of action considered.

The separation of Intention and Sequence of Actions also seems
arbitrarily in that they are both concerned with finding a way to act
that satisfies a goal specified in the previous stage (the intention, in
the case of Sequences of Actions). Instead of having two successive
stages, the model could as well have only one stage but rely more
heavily on the definition of sub-goals and sub-cycles to account for
the decomposition of goals into intentions and sequences of actions.
Other models such as GOMS<fnref n="2" /> [CNM83] rely on such a decomposition
of goals into sub-goals, which allows to consider intermediary levels
between goals and sequences of actions (for instance, getting up,
walking to the switch, and flipping the switch can be considered as
three actions in a sequence, each with their sub-sequences of actions),
as well as other cognitive tools such as strategies for dealing with
specific problems.

This new stage resulting from the fusion of Intention and Sequence
of Actions could also be defined in terms of the gibsonian affordance
framework that I presented in Chapter 3 (despite Norman's disagree-
ments with this framework): in this stage, people would identify
combinations of (perceived) affordances at all levels of a means-end
hierarchy, from a high-level affordance providing a mean to reach
their goals (instead of an intention), down to low-level affordances
for basic movements (instead of a sequence of actions). Norman's
opposition between “external” and “internal” actions is compatible

<footnote n="2">² Goals, Operators, Methods, and Selection Rules.</footnote>
with the fact that the bottom part of means-end hierarchies contains
de-contextualized affordances that are actions that the user can do,
while the upper parts contains higher-level affordances that the user
can exploit indirectly.

### 4.2.3 Focus on problem-solving rather than discovery

Norman's cycle of action suggests that we interact with the world
by solving interaction problems, starting from a goal to reach (the
problem) and finding a way to interact with the world (the solution)
so that the goal is satisfied.<fnref n="3" /> Hornbæk and Oulasvirta [HO17] observe
that HCI views on interaction in general, and Norman's model in
particular, "say little about how intentions are formed or affected by
interaction". They are concerned by the fact that "HCI, via its concepts,
has had an overwhelming tendency to understand interaction as one-
sided – as channeling and realization of human intentions through
a computer, furthermore assuming that these intentions are outside
the realm of interaction itself. This has created a 'blind spot', with
implications for our ability to address some important contemporary
topics. For designers, this means that they are not able to anticipate
how user experience changes over the course of interaction, as users
discover new opportunities for action." – which is one of the general
research questions that I have identified for this thesis (*user experience
of discovery*, Section 2.6.3).

Having the goal as a first stage in the cycle suggests we only use
a specific way to solve problems, that could be called *top-down*: by
working backward from the goal to find ways of interacting with the
world that satisfy this goal (first as intentions, and then as sequences
of actions). But the reverse approach – *bottom-up*, starting from the
action possibilities observed in the world to find which ones are likely
to get us closer to the goal – is also a valid approach, as well as
mixed strategies that would consider simultaneously both ends of the
problem. As a consequence of the lack of precision on the exact nature
of its stages, it may seem that Norman's model assumes that we solve
problems only with the top-down approach.

In a critical commentary of Norman's model, Kirsh suggests that a
bottom-up analysis of possible actions is a common user strategy to
refine an insufficiently defined goal: "Often we explore the world in
order to discover our goals. We use the possibilities and resources of
our environment to help shape our thoughts and goals, to see what is
possible, and we have no clear idea of what we want to do any more
than we always have a clear idea of what we are going to write before

<footnote n="3">³ Norman claims that his model also covers opportunistic actions triggered by the realization of an opportunity to do them rather than by their relevance to the completion of a pre-established goal, but this claim is not well supported and is often ignored by other authors when they present the model.</footnote>
we begin the process of writing." [Kir97]. Kirsh thus concludes that "the goal of an interactive interface is not merely to allow users to do what they want to do, it must also allow them to discover what they want to do" [Kir97].

One could attempt to express in Norman's model the behavior described by Kirsh, and I see two ways to do so: The first is to assume an implied goal of "finding what to do", which result will define the goal for the next iteration of the cycle (in a same way as solving a puzzle would imply to first understand the win conditions of the puzzle). The second is to assume an implied higher-level activity with its own (imprecise) goal, and to start a sub-cycle from the Perception stage, until the Evaluation stage establishes that something was relevant for the goal of the higher-level activity.

These two approaches correspond respectively to what de Bruijn and Spence call *opportunistic browsing* and *involuntary browsing*, which they distinguish from the *search browsing* that is involved in top-down problem solving [DS08]. Opportunistic browsing is a search for information and interface elements that is "intentional, but the user is unaware of any goal being pursued. The attitude underlying opportunistic browsing is 'let's see what's there'." Involuntary browsing is "unintentional, though again the user is unaware of any latent goal that might be pursued. An example is provided by a scenario in which a user's eye gaze, naturally moving rapidly between a series of fixations, serendipitously fixates on an information item that may lead to the answer to a longstanding query." Opportunistic and involuntary browsing both start from the Perception stage and go through the Gulf of Evaluation only, processing information in a way that de Bruin and Spence assume to be unconscious. While Norman does not mention opportunistic browsing behavior, he acknowledges involuntary browsing – which he associates with data-driven behavior such as encountering a friend and remembering we had a question to ask him, leading to what he calls (unfortunately for the coherence of terminology) "opportunistic actions".

Kirsh's criticism should thus not be that Norman's model does not account for behaviors of goal discovery, but rather that it can only describe this important aspect of interaction with the very generic concepts of sub-goals and sub-cycles in the generic framework of problem solving. Goal discovery behaviors rely on the types of information searches described by de Bruijn and Spence, which deserve a more explicit description in a model of user interaction with the world like the Seven Stages of Action. Involuntary and opportunistic browsing are also important aspects of *user exploration of the UI*, one of the general research questions for the study of *swhidgets* (Section 2.6.3), and are close in principle to the behaviors of accidental revelation of *swhidgets* that were described in Section 2.5.
4.2.4 Focus on information used in short-term interaction

Kirsh also claims that for designers, Norman's model puts the high-
light on the necessity of making the action possibilities and their
consequences visible, which suggests that the model may not be the
most appropriate to understand signifier-less designs as it somehow
assumes the necessity of signifiers. Indeed, Norman's model involves
specifying intentions and sequences of actions without explaining
where the knowledge of such possibilities for action comes from, and
thus assumes that this information is available, either because it is
visible or already known. This contrasts with Gibson's concept of
information pick-up (see Section 3.1.1) that explains how we are al-
ways searching for relevant action possibilities as a primary way of
interacting with the world, and how we can also pick up goals from
the environment as described by Kirsh.

The focus of Norman's model on problem-solving and information
used in short-term interaction also implies that it does not include
the discovery of affordances in any of its stages, but only accounts for
such discoveries as the result of solving "what can I do?" problems as
sub-goals, or as the result of serendipitous epiphanies like "Oh! I can
do that". As a consequence, the model can only describe discoveries
of affordances that happen in a disruptive way, since a sub-cycle has to
be started from a new (sub-)goal or from the Perception stage, which
interrupts the current interaction cycle. It is then not surprising that
Norman sees the problem of unclear relations between affordances
(see Section 3.2.2) only as a problem and never as an opportunity to
create a positive experience in the long term – which is one of the
general research questions identified in this thesis (*user experience of
discovery*, Section 2.6.3), as a possible direction to answer *what are the
benefits of not providing signifiers?* (Section 2.7.4).

Because the model only accounts for disruptive discoveries of affor-
dances, it is also not suited to investigate other types of discoveries,
such as users deducing the existence of affordances through a reflec-
tion on what they already know about the whole system of affordances
and the activity. For instance, a user who has learned recently about
the possibility to apply a style to a section of text in a word processor
might deduce that such a possibility only makes sense if there is also
a way to define styles, and might search for a way of doing so. Where
would such a deduction fit in Norman's Seven Stages of Action? Al-
though Norman provide concepts to explain such deductions, such
as his notions of mental models or logical constraints (which will be
discussed in Section 5.3.8), he does not explain when the discovery of
such constraints and the construction of such models happen in the
cycle. This is a case of the general research question on *overarching in-
terface logic* (Section 2.6.1) and a possible answer to the question of *how
can users know swhidgets despite their lack of signifiers?* (Section 2.7.3).
Finally, the last criticism from Kirsh is that by presenting interaction as a feedback loop, it is hard to account for phenomena that arise in long-term interactions. For instance, it does not account for learning and acquisition of automatic behavior. It also ignores the dependence of user behavior on the past history of interaction and its visible consequences in the current state of the world, including voluntarily setting the world in a specific state so that it will ease later interactions (including, but not restricted to the need of resuming an interrupted activity) or help the user forming habits. Indeed, the only way the Seven Stages of Action can model these behaviors is by invoking an interaction between high-level cycles dedicated to long-term activities and lower-level cycles for the sort interactions, although the model does not explain how such interactions between cycles can happen. Understanding such phenomena is however very important to model user adoption of specific interaction techniques or strategies, and generally how users change their usage patterns and habits – especially when it involves conscious decisions to invest time in improving their knowledge of the interface through practice of interaction techniques or *exploration of the UII* (Section 2.6.3).

## 4.3 CONCLUSION

The Seven Stages of Action is a model of user interaction with the world and interfaces that describes the different processing stages users go through to interact. It allows to discuss users’ informational needs in each stage, and to analyze how well a design satisfies these needs. As such, it promotes the use of signifiers as a general design solution and suggests that a lack of signifier is usually problematic.

Despite its usefulness to analyze the notion of signifiers, the model has limitations that makes it not well suited to the study of signifier-less designs: unclear definition of the stages, especially for the distinction between intentions and sequences of actions; focus on goal-oriented problem solving that insufficiently acknowledges the role of goal-discovery user behaviors, exploration of the interface, and accidental discovery; focus on information used in short-term interactions that insufficiently explains how users discover interaction possibilities, what are the consequences of such discoveries on user experience, and users’ engagement in learning about the interface in the long term.

For these reasons, I propose an alternative model of user discovery and adoption of affordances in Chapter 7, which is based on different principles: First, an equal consideration to the three dimensions that are users’ current knowledge and skills, the aspects of the interface design that can inform users, and the reasons why learning happens (including users’ motivations for acting, but not restricted to their goals). Second, the organization of user actions in a cycle is replaced
with considerations for interactions between the three dimensions that reflect evolution of user's knowledge and skills in time.
<figure><img src="image_26.png" /></figure>
In this chapter, I consider the *means* by which elements of design
in the user interface or environment can help users improve their
knowledge of the interface or of the activity supported by the system,
and become more skilled in using the interface. This chapter will be
organized accordingly to the stages of Norman's Cycle of Action (see
Section 4.1), starting from the world rather than user's goal, since the
design elements concerned by this chapter belong to the world. I will
thus discuss in the next sections how the design of the interface can
influence the stages of perception, interpretation, and evaluation.

5.1 PERCEPTION OF DESIGN MEANS

There are two ways by which design means affect perception: one
is by providing things to be perceived, as the output of the stage
of perception that will serve as an input to the following stage of
interpretation; the other is by affecting how well things are perceived,
allowing designers to control in a limited way what users are likely to
perceive and in what order.

5.1.1 *Affordances as output of perception*

As we have seen in Chapter 3, affordances are directly perceived in
Gibson's (and Norman's) view. It means that perceived affordances
(real or false) can be an input to the stage of interpretation and can
thus be interpreted. Gibson does not discuss much the possibility to
interpret affordances, but this possibility exists in his framework, as
we have seen in Section 3.6.2. Norman, however, embraces the idea,
as we have seen in Sections 3.4.3 and 3.5.1.

Djajadiningrat et al. have also suggested that designs could convey
meaning through the possible actions of users [Dja+04], and in partic-
ular that the way a controller can be manipulated could inform users
on the purpose of this controller as well – if not better – than labels
or icons. According to them, designers have thus the opportunity to
design interaction possibilities (which, in my terms, are affordances)
to convey information, and should therefore not focus their design
only on the problem of making these possibilities easy to perceive.
We will see in Section 5.3 how affordances can convey information in
practice.
1. noticeability (likelihood to be sensed when not trying to attend to it),

2. detectability (likelihood to be sensed when explicitly trying to attend to it),

3. findability (likelihood of finding it when searched among other elements),

When the design element is additionally a signifier (cognitive affordance, in Hartson's terms), these generic problems can induce more specific issues like visibility, discernability (differentiating it from other similar signifiers or from non-signifying elements, including the background), recognizability, and identifiability. Some types of media also have specific issues, like the legibility of text and audibility of sound.

To address these issues, designers can manipulate qualities of individual design elements, like color, size, location, timing of appearance, quality of graphics/sounds/haptic signals, etc. They can also manipulate the qualities of individual elements in a composition to manipulate qualities of the whole composition, like contrast, layout complexity, or redundancy (especially, the availability of multiple presentation format can help users with sensory disabilities). They should also take into consideration how the design will affect user's attention, e.g., avoiding elements competing to attract user's attention, or designs that force users to pay attention to multiple things at the same time to complete a task (divided attention).

While designers have all these means to provide/improve sensory affordances, they also have to face trade-offs between different types of affordances. A classical example is the trade-off between adding more functional affordances in the design to allow users doing more things with the designed product, and having to integrate these functional affordances in the design without compromising other types of affordances. Hartson illustrates this trade-off with a computer-assisted design (CAD) application that displays a large number of small drawing tool icons: it causes difficulties in pointing at the icons because of their small size (poor physical affordance), but can also cause sensory affordance issues with difficulties in finding a specific icon, distinguishing similar icons, and recognizing a small icon.

Different strategies can be used to deal with such trade-offs, which can be grouped broadly in two approaches: The first one is to alleviate sensory affordance issues at the cost of reduced cognitive affordances. In the above example of drawing tool icons, it could consist in reducing visual clutter by removing some signifiers like the borders of buttons (flat design), using less-efficient signifiers like icons instead of text, or abstract but easy-to-recognize icons instead of more evocative icons, etc. Since this approach affects cognitive affordances that allow users to understand how to interact with the product, it has the consequence
that it favors more experienced users who do not need these cognitive affordances anymore. The other approach is to only display a subset of the commands at a given time, relying on additional interactions to change the subset of displayed commands. Modal interfaces, menus, accordion widgets, scrollable palettes, tabs, etc. are designs that all rely on such an approach. Here, the visibility of some widgets is voluntarily decreased in order to improve the sensory affordances of the others. In Chapter 6, we will see that signifier-less designs can often be partially explained by such considerations.

### 5.1.3 Gestalt Laws

Gestalt Psychology is a descriptive theory of how humans perceive the world, based on the idea that shapes are perceived so as to form a coherent whole rather than by decomposing them into simpler shapes. In this framework, Gestalt laws are generic principles that govern the identification of such coherent wholes. Research in Gestalt Psychology has identified a few such laws, which are still commonly used today by designers as guidelines [War12], despite the Gestalt theory being now only considered as a descriptive theory.

In particular, Gestalt laws explain why some sets of objects are perceived as forming a coherent whole, by identifying existing relations between these objects such as symmetry, alignment, being inside or containing, etc. The perceived objects can become the focus of visual attention, and the relations between them can be used to move the focus of attention from one to another. A set of objects identified as a coherent whole can be attended to as a single object and interpreted in relation to its parts or in relation to the objects it is itself a part of. For instance, in Figure 5.1-a., we can perceive all four individual buttons controlling the stoves as separated objects, but we can also perceive the group of buttons as a single feature, and we are able to consider the position of each button relatively to the group.

The reason I include Gestalt laws in this review is that they can provide information to users, by creating meaning in the interface structure with no extra signifiers. They are thus useful to understand how an interface can provide “invisible” signifiers. Indeed, many of the design principles we follow arise out of gestalt laws such as closure, or common regions and good continuation in the law of Prägnanz [War12].

For instance, a frequent use in design of the law of good continuation is the alignment of widgets and fields in order to create a straightforward *completion path*: a virtual line that helps knowing in what order the fields should be filled. Another example is the use of spacing to create groups of buttons for related commands thanks to the law of proximity [Joh14, chapter 2]. The effect of this law can be seen in Figure 5.1, as we visually distinguish three groups of buttons
<figure><img src="image_27.png" /><figcaption>Figure 5.1: Three possible designs (bottom row: a, b, c) for the controllers of some stove (top). How can users know which controller should be used to change the power of a specific burner? In a., users have to read the textual labels underneath each controls. In b., the labels provide a visual representation that may require less cognitive processing than text. In c., no extra visual sign is used in labels, as the pattern of buttons matches the layout of the stoves and the position of the controls signify directly which stove they control. It is an example of *natural mappings* that will be discussed in Section 5.3.4.</figcaption></figure>

(labeled a., b., and c.), thanks to the greater distance between buttons of different groups than between buttons of a same group. In groups a. and b., the law of proximity is also what allows to associate the button labels with the closest button. Group b. also illustrates the law of *similarity*, which is often used in conjunction with proximity. It is based on the fact that the human eye tends to build a relationship between similar elements within a design when they share visual attributes such as shapes, colors, size, etc. Here, the four disks representing the stoves are perceived as a group thanks to proximity and similarity, but there is additionally two subgroups perceived where similarity breaks: one subgroup for the three white disks, and one subgroup for the black disk that represents the stove controlled by the nearby button. Group c. is an example of *natural mapping*, a design solution that will be discussed in Section 5.3.4 and cannot be explained by perception alone although perception is obviously involved in the ability to understand the similarity of patterns between the stoves and buttons.

The description of perception in Gestalt Theory also provides some mechanisms to interpret the forms and structures perceived. We just mentioned how the law of proximity allowed to group the stove buttons with their labels in Figure 5.1: this proximity relation between the buttons and labels can be interpreted as “the label informs about
<figure><img src="image_28.png" /><figcaption>Figure 5.2: Using the Gestalt law of closure as a source of information on possible interactions: In the *Message* application (iOS 12.4), a row of icons at the bottom of the screen allows to share directly in the conversation some content provided by the corresponding applications. The spacing and size of icons is such that when there are too many icons to be all shown, one icon (white, here) is made only partially visible on the right edge of the screen. Users can then understand that this bar can be scrolled to the left to reveal more application icons. This is important because users may not expect this interaction possibility, as this bar would otherwise look like the icon bars that iOS uses at the bottom of the screen in most applications, which are not scrollable.</figcaption></figure>

the nearby button”. Similarly, the alignment perceived in a completion path design is interpreted as an ordering of the input fields. The interpretation can also be made in terms of action possibilities. An example is the case of an object partially out of a scrollable view: the Gestalt law of closure allows to recognize that there is an object even if it is only partially visible, and provides the information that this object is partially outside the view boundary. Users can then use this information to infer that the view can be scrolled. This strategy is used in some designs to replace dedicated signifiers, as shown in Figure 5.2.

## 5.2 DEFINITION OF SIGNIFIERS IN PEIRCE'S SEMIOTICS

When he introduces the concept of signifiers, Norman mentions their origin in *semiotics*:

> "The term signifier has a long and illustrious career in the exotic field of semiotics, the study of sign and symbols."

[Nor13, page 14]

The theory of semiotics by Charles Sanders Peirce has been invented to describe how people can interpret signifiers, and is to my knowledge one of the very few models to account for what happens in the stage
of Interpretation without entering into cognitive theories. Norman
is right about the theory's *long and illustrious career* (it dates from the
late 19th century), and about its *exotic-ness* for HCI practitioners and
researchers: it is complex, filled with terminology that one can hardly
memorize, it aims at being an universal description of very distinct
phenomena, and it is rooted in disciplines like philosophy, logic, and
epistemology rather than experimental science. And yet, the theory is
surprisingly pertinent to better define what a signifier actually is and
how users can interpret them.

HCI researchers like Gaver [Gav89] have also noticed this pertinence
early. Other authors advocated semiotics-based approaches to HCI,
although they focused on semiotics concerns that are more in the
spirit of the works of Ferdinand de Saussure, co-founder of the field
of semiotics. Unlike Peirce who addressed semiosis, the processing of
signs by an individual, Saussure focused more on the study of systems
of signs as cultural phenomena. HCI works in this tradition have thus
focused on using semiotics to understand digital media rather than
particular interfaces [ONeo8; And01], or to analyze a design in terms
of an act of communication from a human product designer to a user
[deSo5] or between a human user and the computer [And91; Ben94].
Although semiotics have been used as the core concept in methods for
analyzing the interactions with a system or application [OBTo2], I am
not aware of any work that uses Peirce's semiosis concepts in order to
classify interaction techniques and design solutions beside Gaver's.

5.2.1 *Components of a Sign: Signifier, Object, and Interpretant*

According to Peirce, a sign is, broadly speaking, something that al-
lows an interpretation. Signifier, object, and interpretant are the only
components of a sign for Peirce.<fnref n="2" /> They are revealed by applying
respectively his metaphysical concepts of Firstness, Secondness, and
Thirdness to the sign as answers to the question “what is a sign?”:

THE SIGNIFIER is something that can be interpreted: an image, a
sound, an action, a word or sentence, a volatile object like smoke, a
country flag, a layout, an event, a pattern, a law, etc. – but considered
as a thing in itself, before any interpretation. The signifier exists by
itself outside the interpretation but provides a ground for it, i.e., the
abstract feeling that can trigger the interpretation.

THE OBJECT is what will be used in the interpretation, what the
signifier stands for or is about: what is shown in the image, the
meaning of the word, the fire that emits the smoke, the country of
the flag, the rule that governs the pattern, etc. The object determines

<footnote n="2">² Peirce has used many different terms for these concepts, and I chose these three terms for their compatibility with Norman's usage of the term *signifier*.</footnote>
the signifier, not the other way around. The object is only accessed
through its relation to the signifier, and only the qualities of the object
that are involved in this relation are relevant, as they determine the
signifier and its ground.

THE INTERPRETANT can be broadly understood as the person (or computer!) that do the interpretation, or as the process of the interpretation, but is more exactly defined as the effect that the interpretation has on the person doing it. In particular, it is the understanding that she has of the signifier-object relation. But it may also be the emotions that she feels because of the interpretation, or other types of reactions. The interpretant puts the signifier in relation with the object through its mediation.

5.2.2 Combination of Signs

In Peirce's view, signs can be combined in potentially infinite chains of interpretations. Indeed, Peirce wanted to create a framework that could describe cognition in a general way, including for instance logic and inquiry. He saw signs as the building bricks of thought, and discussed thought-signs (signs that only exist as thoughts, by opposition to the elements of signs embedded in physical objects).

For Peirce, an interpretant is also a thought-sign and can thus become a new signifier or a new object in a new interpretation. For instance, if I see a line drawn so as to form a star on a tool button in a drawing application, I can first interpret this line (signifier) as representing a star (object) and thus, being a star icon (interpretant). But I can then further interpret this star icon (previous interpretant as new signifier) as representing a drawing tool (new object), and interpret it as a tool to draw stars that will be activated by clicking the button (new interpretant).

5.2.3 *Nature of the signifier-object relation*

The following categories of signs are the most often discussed in
HCI and graphic design. They come from an interest in the relation
between the signifier and its object. There can be three different ways
to be interested in this relation:

ICON The relation between a signifier and the object of the sign ex-
ists by itself outside of any interpretation. It can be perceived directly,
and thus necessarily consists in a similarity between qualities of the
signifier and object, which "are alike". Since the object determines the
signifier, it can be said that the signifier signifies the object by virtue
of its resemblance to it.
In user interfaces, graphical icons are often semiotic icons too, as for when the concept of a "new document" is represented by a blank sheet of paper or "printing" is represented by the image of a printer. A document represented in a WYSIWYG editor is a semiotic icon, as is a file icon that is a snapshot of the document. Figure 5.1 shows how iconic signifiers can be used, first to improve on the textual labels in (a) by using iconic labels presenting a similitude between the stove pattern and the icons in (b), and then how this design can be improved in (c) to eliminate the need for button labels by configuring the buttons themselves in a pattern similar to the stoves - effectively creating a type of signifier that Norman calls a *natural mapping* [Nor88].

In the category of icons, Pierce has distinguished three subcategories: *images* that rely on a perceptual resemblance (like portraits), *diagrams* that rely on a similarity between structures so that the relations between parts of the signifier match relations between parts of the object (as for the stoves in Figure 5.1), and *metaphors* for which the similarity is arbitrarily imposed as a rule but allows to connect parts of the signifier to corresponding parts of the object (e.g., Shakespeare's "All the world's a stage, And all the men and women merely players").

**INDEX** The relation between a signifier and the object of the sign exists in virtue of an actual, factual, natural relation between the two, such as a cause-consequence relation (the smoke is an index of fire because it is produced by the fire) or a spatial relation (the finger pointing at the Moon is an index for the Moon, to which it is related by the spatial alignment of the two), or a mix thereof (the cat stands next to its empty food plate because she is hungry).

In user interfaces, it is very common to associate labels to the widgets they describe using a proximity relation, as we will discuss in Section 5.3.2. Another common type of indexes is the use of carets, cursors and other pointers to indicate an active element or position in a set, which is signified by the proximity relation between the cursor and the active element. Other examples will be provided in Section 5.3.

**SYMBOL** The relation between a signifier and the object of the sign exists by convention, habit, law, or any other entity that creates the relation through its mediation.

This is a very common type of signs in user interfaces: many graphical icons have a meaning only by convention, such as the triangle pointing to the right which means "play a media" and the two vertical bars that mean "pause the media", the × cross that means "close the window", the concentric arcs that mean "Wi-Fi", etc. Other signs may have an historical origin as icons but are now used only by convention, such as the diskette symbol that still means "save the file" despite the fact that such diskettes have been deprecated decades ago. Be-