side graphical icons, most instances of text are also symbols in that
the shape of a word is usually unrelated to its meaning (except in
onomatopoeia such as "cock-a-doodle-do" or "splash"). Symbols can
also be seen in some hotkeys, since a key combination like Ctrl-Z is
arbitrary and not related to the command "undo" that it triggers and
can thus be understood as its semiotic object.<fnref n="3" />

5.3 INTERPRETATION OF DESIGN MEANS

HCI authors have proposed concepts for the design and analysis of
signifiers, which can be discussed in a Peircean framework, notably
with the distinction between icons, indexes and symbols.

5.3.1 *Perceived affordances*

A perceived affordance is a signifier in Norman's views, and it can
also be understood as a signifier in Peirce's semiotics. Indeed, the
action possibility (the Gibsonian affordance) that has been perceived
is a signifier in a sign which semiotic object is the set of physical
objects involved in the action, and the interpretation would be an un-
derstanding that such an action can be performed with these physical
objects.

The sign is an index because the affordance is a consequence of
the qualities of these objects, notably their shapes and dimensions,
and comes neither from a resemblance or an arbitrary relation. This
classification highlights the absence of cultural, arbitrary or learned
elements in the interpretation, which is exactly why the concept of
perceived affordance was praised by Norman.

5.3.2 *Labels*

The most common type of signifiers beside perceived affordances are
labels, understood in a broad way as informative marks displayed on
or next to the element concerned by the information, whether it is
the text "push" on a door or the image of a trashcan on an on-screen
button. Such labels require a combination of signs to be interpreted:

First, the label itself has to be interpreted, independently from its
placement near another object. The label as signifier is not a sensation
or object, but is a concept (the word "push", a trashcan), and it is
interpreted as a quality of something (pushability, deletability) that
is an abstract affordance (it will be made concrete by applying it to
the door or button). The way the signifier is linked to the semiotic

<footnote n="3">³ Other hotkeys such as Ctrl-S for "Save" can been understood as icons because they share a property with the name of the command in its initial letter, but even then this is more a mnemonic and historical relation than a true one and the hotkey has still an arbitrary relation to its command.</footnote>
object, here, does not matter: the word push is a symbol (conventional) and the trashcan is an icon (looking like a real trashcan) used as a metaphor.

Second, the label has to be interpreted as informing about an interac-
tion possibility of the door or button as an index: The label is spatially
related to the object of the sign (door or button) that determines its
position, and it is thus an index.

5.3.3 *Interface Metaphors*

Metaphors – also sometime called analogies – allow users to interpret
elements of the interface by mapping them onto objects and events
of a real-world environment that is supposedly more familiar. The
most famous example is the Desktop metaphor that draws an analogy
between computer concepts such as file systems, databases, etc. and a
desk, with documents that can be opened on the desktop or stored
into drawers, etc. While the suitability of metaphors as tools to learn
the interface has been criticized [HM82], they are still used today as
core elements of the design of new products [HW10].

Peirce's definition of metaphors as a particular type of icon is com-
patible with the usage of the term in HCI [BBN04], and Gaver, based
on Peirce's works [Gav91; Gav89] additionally discusses how interface
icons can be created in interface metaphors either as semiotic symbols
(e.g., an arbitrary sound associated with an action), as metaphors them-
selves (e.g., a sound of decreasing pitch and loudness or a graphical
icon fading into transparency to represent the deletion of a document),
or as icons that use the interface metaphor indexically (looking or
sounding like an object or event implied by the interface metaphor,
such as the sound and visual animation of paper scrambled to repre-
sent the deletion of a document).

5.3.4 *Natural Mappings*

The notion of iconic signifiers can be found in Norman's *natural mappings* [Nor88], a design mean aiming at making it easier for users to identify the relationship between a set of controls and what they affect in the world. To do so, natural mappings consist in using a structural analogy between the pattern of controllers and the pattern of things controlled, or between the actions to perform on these controllers and the effects of these actions. It is believed that such a design leads to an immediate understanding of the mapping, since the analogy can be understood simply by looking at the controllers and controlled elements and does not require any external knowledge.

For instance, the controls for adjusting a car seat (Figure 5.3) look
exactly like the seat and users can move the controls corresponding to
the different parts of the seat directly in the direction they want the
<figure><img src="image_29.png" /><figcaption>Figure 5.3: Example of a natural mapping for the controls allowing to adjust a car seat.</figcaption></figure>

seat part to be adjusted. Another example is the display c. of stove controls in Figure 5.1, which leads to immediate understanding of which control affects each stove. Mapping problems are abundant, for Norman, it is one of the fundamental causes of difficulties in the interaction with everyday things. In Peircean semiotics, natural mappings would be icons, and more precisely diagrams.

An idea related to natural mappings is the concept proposed by Djajadiningrat et al. [Dja+04], consisting in designing products so that users can not only perceive an affordance (possible action) but also interpret it as an information about the purpose of this action. For instance, they propose the design of a digital camera in which the picture taken, visualized on the camera display, can be transferred on a memory card by physically pushing the display so that it connects with the visible memory card. Although there is no mapping in this case, the action of connecting the display showing the picture and the memory card is a metaphor for the transfer of the image to the card.

### 5.3.5 *Interaction Frogger*

Wensveen, Djajadiningrat, and Overbeeke proposed the *Interaction Frogger*, "a design framework to analyze person-product interaction" [WDO04]. In this framework, they propose to distinguish between inherent, augmented and functional information, conveyed either as feedback of feedforward.

**FUNCTIONAL INFORMATION** Functional feedback is defined as "the information generated by the system when performing its function, e.g., sound, light or motion" and an example would be how the TV screen lights up when the user presses the power button.
*Functional feedforward* is information “about the more general purpose of a product and its functional features”, and an example would be how “the speakers and a screen on a black box inform the user about the audiovisual functionalities of the product”.

**AUGMENTED INFORMATION** *Augmented feedback* “refers to information not coming from the action itself (which is inherent feedback), but from an additional source”, and an example with the TV would be the red light emitted by a diode next to the power button when it is pressed.

*Augmented feedforward* is “information from an additional source about the action possibilities, or the purpose of the action possibilities”, and examples would be “on-screen messages indicating what to do” and “lexical or graphical labels communicating the purpose of the action possibility”, e.g., on a button or next to it.

**INHERENT INFORMATION** *Inherent feedback* “is the information that is returned from acting on the action possibilities”, and an example would be the displacement, feel, and sound of a button when pushed.

*Inherent feedforward* “is the information that communicates what kind of action is possible (pushing, rotating, sliding) and how this action can be carried out (the amount of force that is possible, which parts of the body, etc.)” and the authors notice it can be understood “as a limited interpretation of the concept of affordance”, citing Gibson although their definition of inherent feedforward as an information is closer to Norman’s understanding of perceived affordances (see Chapter 3).

**RELATION TO PEIRCEAN SEMIOTICS** Since the six types of information introduced in this framework are conveyed by elements of a product's design (and thus, by design means), these elements can be understood as signifiers, which semiotic objects are the interaction possibilities they concern (i.e., in the terms of this thesis: affordances). Augmented feedback and feedforward are then distinct from the other types of information in that they are symbolic, but the others are indexes. Indeed, when feedback and feedforward is “augmented”, it is because they use as a signifier an element that is not involved in the interaction, and is symbolic like a label (see Section 5.3.2). On the other hand, inherent and functional feedback and feedforward are indexes in that they are directly caused by the actions and functions on which they inform. The difference between inherent and functional information then comes from the fact that they are concerned with (Gibsonian) affordances at different extremities of the mean-ends hierarchy (see Section 3.2.1).
### 5.3.6 *Natural Signals*

In his book *The Design of Future Things*, Norman criticizes the designs of appliances that use artificial and unpleasant beeps as feedback of operation [Noro9]. What he criticizes is not only the annoying character of these sounds, but also their artificiality: by being purely functional (having only qualities designed to ensure they are perceived and noticed), they miss the opportunity to positively affect users emotionally. He discusses the benefits of using in design *natural signals*, which take inspiration from signals of the natural world (notably sounds). There are two ways to understand this: one is to create signals that are similar to signals occurring in nature, such as bird chirps; the other is to use signals that are naturally emitted by the appliances.

The former type of signals relies on similarity between the appliances' signals and those perceived in a natural environment, but would not be iconic signifiers in that their object is not the sound they reproduce: a microwave oven that plays a bird sound instead of beeping when your meal is cooked informs about the cooking, not about a bird singing. The relation between the object and the signifier would still be arbitrary and the sign would thus be a symbol. However, the interpretant in Peirce's theory can also consist in an emotional response and the object of a sign can also be the feeling of peace that one can experience as she listens to the birds in a country garden: in that case, this type of natural signal could be understood as being an iconic sign.

The second approach to natural signifiers consists in having appliances communicate their state through signals that are naturally emitted as a consequence of their operation. Norman gives the example of the sound emitted by a vacuum cleaner, which may not be enjoyable but clearly informs on the operation of the device. Moreover, users can notice subtle changes in the sound and interpret them as signs of specific events, such as when a small object gets stuck in the vacuum cleaner, producing a higher pitched sound, or when the dirt bag gets full and the aspiration seems less powerful. As signs, these signals are indexes since they are event signifiers which objects are the state of the device, and the relation between the objects and the signifiers is a cause-consequence relation: the state of the device causes the signifier directly.

### 5.3.7 *Feedback and feedforward*

When design means are considered as signifiers, it can also be interesting to categorize the possible semiotic objects of the corresponding signs. As we have seen in the examples provided in the preceding sections, the semiotic object can be an interaction, and a very basic
distinction is then between signs providing feedback and feedforward: for signs that provide feedback, the semiotic object is an actual action done or being done by the user, while for feedforward it is a possibility of (future) action [DOW02; Ver+13].

This distinction is based on the timing of events in an action model relying on a loop, such as Norman's Seven Stages of Action (see Chapter 4), but a more detailed analysis can be done (especially for feedforward). For instance, in the framework *Interaction Frogger* (see Section 5.3.5), the distinction between functional and inherent information can be understood as applying to different extremities of the product's means-end hierarchy of affordances: different types of action possibilities. A similar distinction was done in [DOW02] to introduce the concept of feedforward as distinct from perceived affordances, where the former is understood to concern the purpose of an interaction and the latter to only concern the possibility of an interaction.

A related concept is the distinction between signs which object is the purpose of an interaction and those which object is the consequence of an interaction. For instance, a modern design (especially on smartphones, tablets, and the web) is to propose a generic menu accessed by taping or clicking on an icon displaying three horizontal lines stacked vertically. This icon can be understood as a signifier representing the menu as a semiotic icon: the three horizontal lines being like the entries of the menu, which also are elongated horizontal rectangles stacked vertically. In other word, this icon says nothing more than "click here to open a menu": it informs on the consequence of the action, but not on its purpose, as it says nothing about the kind of commands that can be found in this menu.

Another example of signifier conveying the consequences of an interaction rather than its purpose is provided by widgets that are disabled ("grayed") when activating them would lead to an error message. Fortunettes [Cop+19] is a technique to convey the consequences of interactions with some widgets that change their state or the state of other widgets (for instance, a disabled submit button that becomes enabled when the data provided in a form become valid), by changing the visual appearance of these widgets.

The amount of information provided about the consequences of an action can actually change the way the interaction is done. Guillon *et al.* investigate this effect in target expansion techniques [GLN15], for which they distinguish between implicit and explicit feedforward: an explicit feedforward shows the boundary of the cells where clicking has different consequences (different targets are selected) but an implicit feedforward when the user only gets the information that she entered a different cell.

Finally, signifiers can inform about the final state of the system after an interaction, in a way that is both the consequence and the purpose
of the action. For instance, in his analysis of cognitive affordances (which are like Norman's signifiers), Hartson discusses the design of a switch in his FM player to change the mono/stereo output setting [Haro3]. He argues that a good design would have used only the labels "mono" and "stereo" on each side of the switch so that users understand that moving the switch towards one label sets the system in the corresponding state. Such labels inform on the purpose of the switch (and each of its two positions), on the consequence of interacting with the switch, as well as on how to interact with it in order to get the desired setting. On the contrary, the actual design of his FM players shows the same labels but inverses their positions and adds arrows depicting the direction in which the switch must be moved in order to get to the labeled state. Harston concludes that the labels and the arrows provide conflicting cognitive affordances and that the design is confusing because of that.

It raises the question of whether it is always better to use signifiers that are about the intended system state or if it is sometimes better to use signifiers that are about the gesture that one should perform to reach this state (as communicated by the arrows in Hartson's FM player). Interaction techniques like Octopocus [BMo8] provide feedback for gestures that concern both the shape of the gesture to perform and the command it triggers (which is both the purpose and the consequence of the gesture). In this case, the information is made non-ambiguous by the fact that the labels describing the commands are placed at the end of the lines that represent the gestures, which all start from the same position, and disappear as the user performs a gesture that is not compatible with them.

### 5.3.8 Constraints

Constraints are also proposed by Norman in [Nor88]. They are properties of objects from which users can deduce possible operations. While affordances "suggest" possible actions in Norman's views, constraints "limit" the number of alternatives to consider. Properties of the objects perceived must be interpreted as constraints in the stage of Interpretation, but information about constraints can be used at two places in Norman's cycle of action: in the (future) stages of Intention and Sequence of Actions to help users determine what to do, and in the stage of Evaluation that follows to allow people to understand why an attempted action failed, as not satisfying a constraint. Norman discusses four types of constraints: physical, semantic, cultural and logical.

*Physical constraints* are action possibilities that are limited by a mechanical system. For instance, the closing line between the screen and the keyboard of a laptop is a perceived affordance that informs the laptop is made of two parts that can be moved relatively to each other.
other, the presence of a hinge constrains the direction in which users
can open the laptop. Another example, inspired by Norman's analysis
of a Lego toy assembly [Nor88, page 83-84], would be how the pieces
of a jigsaw puzzle cannot be assembled together when their shapes do
not match. Physical constraints exist also in signifier-less designs, for
instance an *item sw* like the ones in the Mail application can only
be triggered laterally, as a vertical swipe would conflict with scrolling.

*Semantic constraints* “rely upon the meaning of the situation to control the set of possible actions”. The example to open the laptop in the correct direction is also a semantic constraint in that the screen fits above the keyboard in the user’s mind, and thus, opening the laptop ends with a system in a usable configuration (which is the goal of the user for opening the laptop). In the jigsaw puzzle example, semantic constraints would be that the image formed by assembling the pieces should make sense as a picture. Semantic constraints exist also in signifier-less designs, for instance *item swhidgets* can only exist on items that would benefit from them as they can be the targets of specific commands.

*Cultural constraints* rely upon accepted cultural conventions to limit the set of acceptable behaviors. They are often meant to be read, even if they do not affect the physical or semantic operation, such as car lights: red lights placed in the rear signify the action “stop”, while white lights is the standard for headlights which go in the front.

*Logical constraints* come from the possibility to deduce what should be done from assumed rules. For instance, when there are two switches and two lights and we already know what light is controlled by the first button, we can deduce that the other switch will control the other light (assuming that all lights need a controlling switch). Similarly, Norman give the example of a scrollview with an element half visible on the view border (as in Figure 5.2), from which we can deduce that the view must be scrollable [Noro8].

5.4 EVALUATION OF DESIGN MEANS

The stage of Evaluation ends the Gulf of Evaluation that is concerned
with understanding the world. As a conclusion of this chapter, we
can see how the design means have been interpreted so as to create
expectations that can be verified in this stage.

As we have seen in the previous section, design means can provide
feedback and feedforward. Users can confront the information pro-
vided as feedback to the expectations created by feedforward before
acting. The considerations of different types of feedback and feed-
forward that I presented in Section 5.3.7 allow to better understand
these expectations, how they can be verified in the Evaluation stage,
and how users can react in case the expectations are not met. I chose
to present the information provided by feedback and feedforward
together in the stage of Interpretation because it is were this informa-
tion is obtained, but it can also be considered that this information is
used in other stages, or is required for other stages, as do [Haro3] and
[Ver+13].

The Evaluation stage could also be concerned with evaluating the practicality of a Gibsonian affordance: instead of checking the interpreted world against predefined goals, users can as well check these interpretations of design means against *possible* goals, as suggested by Hirsh (see Chapter 4.2.3). This idea can be found in McGrenere & Ho's idea to treat affordances as a continuous number rather than as possible / not possible, and the observation that people can evaluate directly the "climbability" of a staircase [MHoo]. Indeed, if we consider an ill-defined and abstract "find what to do next" goal for actions that start with the environment, we can consider that the evaluation stage is about determining if observed interaction possibilities are things worth doing next.

An important concern for the evaluation of design means is the notion of "level of detail" [BM08; Ver+13], which is related to how well the signifiers provide information required in the evaluation stage. This notion of level of detail focuses on the amount of relevant information provided, how often it is provided, etc. However, the notion of how clearly the information is provided is harder to define and equally important. I provided hints at how the semiotic theory of Peirce can be used to understand how signs are interpreted in human-computer interaction, in hope that it can provide insight about this question, notably by exposing the knowledge required to perform some interpretations and how some designs can avoid to rely on such knowledge.
<figure><img src="image_30.png" /></figure>
<figure><img src="image_31.png" /></figure>
Part II

SIGNIFIER-LESS DESIGNS
<figure><img src="image_32.png" /></figure>
<figure><img src="image_33.png" /></figure>
<figure><img src="image_34.png" /></figure>
In this chapter, I finally introduce *signifier-less designs* more formally. In Section 6.1, I first propose a simple definition of signifier-less designs, then expose the difficulties of establishing such a definition and discuss its limitations. I propose to solve these issues by defining the notion of *directly signified* affordances, building upon some subtle notions introduced in Chapter 3: instrumental affordances and affordances for mediating knowledge. In Section 6.2, I observe that signifier-less designs have been provided by the user interfaces of most systems in the past and present, review a selection of signifier-less designs and discuss how the lack of signifiers relates to the principles on which these designs are built. I will finally present in Section 6.3 three different sub-types of signifier-less designs based on which affordance is not signified, and discuss possible user reactions when they encounter them.

## 6.1 DEFINING SIGNIFIER-LESS DESIGNS

As Norman said in the quote that introduced this dissertation, from an industrial design perspective, some controls are hidden away for aesthetic reasons [Nor88]. For example, many electronic devices like desktop computer displays have a power switch at their back, avoiding accidental press on the button, but also hiding the button to users. Switching power is a function that only needs to be done once (and another one when shutting down), but it is crucial to access all other functions of these devices. From this example (and others that will be provided in section 6.2), we can propose the following broad definition:

**SIGNIFIER-LESS DESIGNS** are interface designs in which some interaction possibilities are not directly signified to the user when she needs it.

### 6.1.1 Edge cases and limitations of the definition

The expression “signifier-less design”, strictly speaking, has a natural meaning that only entails “interaction possibilities for which there is no signifier”. My definition adds two subtle requirements on top of this natural meaning, to exclude the two following cases: 1) the existence of signifiers that are not present at the moment users need them; and 2), interaction possibilities that are signified indirectly – notably, on user’s request.
The reason why these additions are required is that without them, the signifier-less aspect of some designs might be subjective, and the natural meaning of the expression is not precise enough to decide if these designs should be named "signifier-less" or not. Take for instance a door that has to be pulled to be opened, but has a classical handle that affords both pulling and pushing, and no visible hinge that could inform the user that the door has to be pulled. Such a door should certainly be called a "signifier-less design". On the other hand, if the word "PULL" is written in big letters on the door, it should certainly not be called a signifier-less design. But what about the following situations?

1. The word "PULL" is not painted on the door but printed on a poster, and the poster is pinned...

a) ...on the door itself.

b) ...on the wall next to the door.

c) ...on the wall, at the other end of the corridor, ten meters from the door.

2. The word "PULL" is absurdly painted on the other side of the door, so that you have to open the door before seeing it.

3. The word "PULL" is painted on the door, but you do not need to open it.

4. The word "PULL" is painted on the door, but the door actually does not open.

5. There is no word "PULL", but...

a) ...every person who needs to open this door has been communicated the information that it should be pulled before even facing the door.

b) ...there is always someone nearby to tell you that the door has to be pulled before you reach it.

c) ...there is always someone nearby to tell you that the door has to be pulled if they see you hesitate or try to push it.

In cases 1a) to 1c), it could be argued that the signifier (the poster) is not part of the original door design, which should thus be called "signifier-less". But there is no reason to restrict the definition to the original design of the objects/interfaces rather than applying it to the current design of objects and environments constituting the context of use. Indeed, case 2 shows that applying the definition solely to objects rather than to the context of use could lead to nonsensical use of the expression. The terms "when the user needs it" in my definition allow to precise this context of use. However, the distinction between the cases 1a) to 1c) shows that there is no way to define exactly at what
distance from the door the poster would still be visible “when the user needs it”. This issue will be addressed in section 6.1.2 below, in relation to the “directness” of the signifiers.

In addition, understanding the relation between the poster and the door gets harder as the distance between the two increases, and we now have a problem of defining how well something should be signified to not be considered as a signifier-less design. I will simply not enter into these considerations: signifier-less design are not about insufficiently signified interaction possibilities – which users could simply and rightfully dismiss as bad design – but about a total lack of signifier, which can be a voluntary design or an unavoidable one as there is no efficient way to signify this type of interaction. With an insufficiently signified design, users are still supposed to discover the interaction possibility thanks to the inefficient signifier, although for signifier-less designs we have to assume a different path of discovery. As a consequence, designers should also not expect the same reaction from users when they discover a signifier-less design than when they discover an insufficiently signified one.

Case 3 shows another limit of my definition: here, the interaction possibility is signified but “when the user needs it” does not happen. This design should not be labeled “signifier-less”, and in the definition above, “when the user needs it” should be understood as assuming the user needs it and as only providing a restriction about when the signifiers should be exposed. For the same reason, the problem of the false affordance in case 4 is not concerned by my definition of signifier-less designs, which assumes true affordances.

In cases 5a) to 5c), signifiers are replaced by a more general act of explicit communication. The goal is twice: first, to remind that material signifiers are not the only possible source of information about affordances. Second, to reinforce the idea that the problem of signifier-less designs is not a problem of having something in a design, but having an information provided or not in a context of use that requires it. The difference between 5b) and 5c) also allows to question “when the user needs it”: is it just before the user actually needs it? Or after she realized that she needs the information? If the information is only provided just before the user actually needs it, distracted users could miss the information, and in that case the interaction possibility is not signified anymore to them. So, the (un)availability of the information after the user realized she needs the information is what really matters in the definition.

### 6.1.2 Directly signified affordances

Finally, the question of the directness of the signifier has to be addressed, as when a user realizes she needs some information, she is free to try many different methods to obtain that information. In
particular, it is possible that she knows exactly where to find the in-
formation. In the case 5a), for instance, the information could have
been sent to her by mail and she forgot it, but when needing the
information she remembers it was sent to her, and she can take her
smartphone and search for the mail. Is it not “when she needs it”? An
HCI example would be the hint provided in a menu about the hotkey
that can be used to trigger the command: if a user needs to know this
hotkey, she can simply open the menu and look for it.

The problem with this approach to how users can access the infor-
mation they need is that it is extremely hard to make a distinction
between these cases and others for which the method to access the
information is not provided by the design or by the context of use.
For instance, what if the user knows exactly what to search on her
preferred web search engine to find the information as the first re-
sponse? Or what to ask to her digital assistant? In many instances, the
difference between a generic tool to access information like a search
engine and an interface element specifically designed to provide this
information is just a matter of user familiarity with the tool. The
concept of directness is introduced in the definition to reject the use
of such generic or specific search tools as “signifiers”, and to accept as
signifier-less designs interfaces that include such tools, so that we can
also question the efficiency of these tools as sources of information.

I build the concept of directness of signifiers upon Gibson's idea that human displays afford some kind of mediated knowledge (see Section 3.6.2). The idea is that signifiers have a part that is an affordance for mediating information: either the signifier is directly a perceived affordance in Norman's meaning of the expression, either it is a sign that needs to be interpreted, but then, its design relies on an affordance for mediating information. Indeed, we are not concerned here with steganography, i.e., the art of hiding messages so that only the people who know that there is a message can see it. Signifiers are normally<fnref n="1" /> recognizable as signifiers even before trying to interpret them: whatever is printed on a poster, it says “this is information” and “I am a signifier”. So does a label, a pictogram, etc. They have an affordance for conveying information, and people can perceive this affordance directly in a Gibsonian way.

The definition of direct signifiers also builds on Kaptelinin' &
Nardi's notion of handling affordance: if, from a position from which a
user could reach the handling affordance of an interaction instrument,
she can perceive an affordance for mediating information just by look-
ing around, and it mediates an information about the corresponding
instrumental affordance, then this instrumental affordance is directly
signified. Otherwise, it is a signifier-less design.

<footnote n="1">1 This is actually the result of design and conventions. However, when it does not happen and people cannot perceive a signifier as being a signifier, it is more a question of a poorly designed signifier than signifier-less design.</footnote>
## 6.2 HISTORICAL SIGNIFIER-LESS DESIGNS

Non-signified interaction possibilities have been provided by the user interfaces of most systems in the past and present. The goal of this section is not to retrace the history of signifier-less designs as an historian would do. The goal is, on one hand, to provide examples of signifier-less designs that will inform the study of this notion; and on the other hand to show that the phenomenon is not new, and not coming from the specifics of modern touch-based smartphones – although the phenomenon is quite visible with these devices. In the end, with this review, I want to ask the question: is it even possible to have interactive systems that rely on interfaces without any signifier-less design, or should we consider that signifier-less designs are an unavoidable aspect of human-computer interaction?

### 6.2.1 Command-line interfaces

Back to the time of command-line interfaces (Figure 6.1), the commands available were not signified permanently: users had to learn the list of commands recognized by the system, either from training or printed documentation. In the 70s, this documentation became digital and was included in the system so that users could access it on demand, and documentation for each command was provided when requested – both operations being realized with a dedicated command. To interact with a computer at that time, users had thus to memorize certain sets of abbreviations or hotkeys, or use cheat-sheets.

Command-line interfaces are still used today by some users for system administration, or for some activities available on all systems,

<figure><img src="image_35.png" /><figcaption>Figure 6.1: Example of command line interface.</figcaption></figure>
and they still rely on the same mechanisms. Some programming and
maintenance tasks may not have a graphical user interface and may
still use a command line.

6.2.2 *Signifier-less designs in Direct Manipulation*

Graphical user interfaces had been proposed as an alternative to
command line interfaces as early as the 60's and had been used as
the primary interface in commercial products such as the Xerox Alto
(1973) and Apple Lisa (1983), which defined the early forms of what
has later been called WIMP interfaces (for Window - Icon - Menu
- Pointer). According to Grudin [Gru06], the HCI community was
skeptical at first about graphical interfaces and only embraced the
idea that GUIs could be better than command-line interfaces after the
commercial success of Microsoft Windows in 1985. Shneiderman's
1983 introduction of the concept of *Direct Manipulation*, based on the
analysis of existing applications including text editors, spreadsheets,
data analysis tools, and video games [Shn83] became then the main
paradigm to study and design graphical interfaces and interaction
techniques.

According to Shneiderman, the concept of direct manipulation
follows four principles (quoted from [Shn83]):

1. Continuous representation of the object of interest.

2. Physical actions (movement and selection by mouse, joystick, touch screen, etc.) or labeled button presses instead of complex syntax.

3. Rapid, incremental, reversible operations whose impact on the object of interest is immediately visible.

4. Layered or spiral approach to learning that permits usage with minimal knowledge. Novices can learn a modest and useful set of commands, which they can exercise til they become an "expert" at level I of the system. After obtaining reinforcing feedback from successful operation, users can gracefully expand their knowledge of features and gain fluency.

The first and third of these principles clearly emphasize the need
for visibility of the objects that can be manipulated and the need for
visibility of the effects of these manipulations. Yet, none of the four
principles of direct manipulation requires that action possibilities be
signified to the user. This fact can be illustrated with the example of a
drag-and-drop operation such as dragging a file icon on a trashcan
icon to delete the corresponding file. This operation indeed follows
the four principles of direct manipulation:

1. Both icons represent objects of interest that are *continuously represented*.
2. The drag gesture is a simple *physical action*.

3. This action is *rapid, incremental* (as the distance between the file icon and the trashcan icon is continuously reduced), *reversible* (by canceling the drag-and-drop or by putting back the file icon at its original location), and its impact is *immediately visible* (as the file icon is moving).

4. It uses a *layered approach to learning*, as this operation builds upon the knowledge of using drag-and-drop actions to move files in the file system and extends its meaning for the trashcan.

Yet, there is often no specific design element on the file icon itself that signifies it can be moved: at best, there is a visual feedback when the user places the mouse pointer over the icon. Users have to be taught this action possibility, or have to discover it themselves by exploiting a metaphor such as the desktop metaphor. Here, the drag gesture or the desktop metaphor belong to the basic layers of knowledge that users need to possess in the “layered or spiral approach to learning” that Shneiderman presents as the fourth principle above.

### 6.2.3 WIMP interfaces

The first systems relying on a graphical user interface in the 70s' and 80s' defined a style of interface called WIMP (for Window - Icon - Menu - Pointer). Since then, and still today, most desktop computer interfaces have followed the WIMP paradigm, which kept most of its identity as it evolved slightly toward modern desktop interfaces. In addition to their reliance on Shneiderman's principles of direct manipulation, these interfaces also present many features that are not signaled to the users (and are not always in the scope of direct manipulation).

**CONTEXTUAL MENUS** provide useful commands acting on some object of the interface, and are activated by clicking on this object with a secondary mouse button or while pressing a key modifier (Figure 6.2). While most objects give access to a dedicated contextual menu, this is not systematic and there is no specific sign on the objects that have a contextual menu.

**HOTKEYS** are combinations of keys that trigger some command, although there is often no indication that some combination of keys can be used at the time users would need it. Instead, users have to learn these combinations, either by consulting a manual or the hotkey customization interface (Figure 6.3), or by noticing and remembering the hotkey hints provided with the menu item that triggers the command needed (Figure 6.2).
<figure><img src="image_36.png" /><figcaption>Figure 6.2: An example of contextual menu in Windows' File Explorer, displayed after the user right-clicked the file icon. Notice also how the menu provides hints for hotkeys on the right.</figcaption></figure>

<figure><img src="image_37.png" /><figcaption>Figure 6.3: An example of system hotkeys on Mac OS X to navigate between windows and desktop spaces. These hotkeys are only hinted in this panel of the System Preferences application, where they can be customized.</figcaption></figure>
The hints used in the later strategy may be thought of as signifiers, since users have a chance to see them when they select the command from the menu. However, from my point of view, this design is equivalent to a cheat sheet, as when the user selects the command in the menu, she does not need the hotkey anymore and she is likely not focusing her attention on the hint, and thus not learning the hotkey [Kur93, page 30]. It is more likely that the user decides to learn the hotkey at some point, and consults the menu to learn the key combination [GDB07]. Bailly et al. indeed observed that in the last uses of a command from a menu before transitioning to using hotkeys for this command, users spend more time using the menu than usually [Bai+18], which can be explained by a focus on the hotkey hint to memorize it.

In addition, the strategy of learning hotkeys from menu hints often suffers from two issues: first, not all hotkeys are hinted, even when the corresponding command is in a menu [Coc+14]. Second, some commands have no corresponding menu entries. In particular, selection and fast navigation keys are usually not shown in menus: the user may understand without hint that pressing the right arrow moves the caret to the next letter on its right in a text, but it requires more knowledge to understand that doing the same action while pressing the SHIFT key instead expands a selection to the right, or that using an ALT modifier key moves the caret to the next word instead of the next letter.

### MOUSE POINTING
Another non-signified interaction common in WIMP interfaces is the technique consisting in placing the mouse pointer in a specific area to trigger some commands or reveal some UI elements. This technique is used for instance in Mac OS to reveal the Dock, or the menu and window title bars, all three of which are hidden by default for an application in full-screen mode.

The hovering technique also allows to reveal tool-tips for specific widgets by hovering these widgets, usually with some dwelling time. Here, the visibility of the widget allows users to determine the hover area, but as for contextual menus, nothing in the design of the widgets signifies to users that a specific widget has a tool-tip attached to it.

Other techniques rely on users placing the mouse in areas where they have no reason to expect something to happen if not told beforehand. For instance, Mac OS X and many Linux systems support "hot corners", a technique allowing to trigger some system-wide commands by sending quickly the mouse pointer in one of the four corners of the screen. In Windows 8, this technique reveals the Charms bar, an always-available tool that replaced the Start menu from earlier versions of this system.
MODAL INTERFACES Modes are different global states of the inter-
face in which the system reacts differently to a same input. Modes
can be considered problematic for interaction because users can forget
in which mode the interface is, and the mode indicators are often
not visible enough, as they are not under users' focus of attention.
Yet, it is very hard to avoid the use of modes in interface design. For
instance, the drag-and-drop operation described in Section 6.2.2 is
modal: moving the mouse has not the same effect during the drag-and-
drop (moving the file icon) than usually (simply moving the mouse
pointer). For this reason, Raskin [Rasoo] advocated quasi-modes, i.e.
temporary modes that are only activated as long as the user actively
maintains them, for instance by holding a button pressed. In that view,
the information about the mode is put back "in the world" by user
action, even if it was not provided by the system.

From the point of view of signifier-less designs, the problem with
modes come from the fact that a same input can result in different
commands according to the mode – or in other words, that a same han-
dling affordance is connected to different effecter affordances through
a mode that defines the active instrument. The lack of visibility of the
mode can thus been understood as a problem resulting from the use
of inefficient signifiers to signal the current mode, which is a concern
that I discarded from my definition of signifier-less designs. However,
it could also be argued that the mode only defines the set of active
instruments, and that the problem is that these modal instruments
are not signified directly: their instrumental affordance, which results
from the connection of an handling affordance to a mode-dependent
effecter affordance, is not signified directly but only through the signi-
fiers of the mode. From this point of view, some modal interactions
can be considered as signifier-less designs.

6.2.4 *Post-WIMP interfaces*

As WIMP interfaces knew a lasting success in commercial products,
HCI research focused on establishing the concepts of so-called post-
WIMP interfaces. In a seminal paper, Nielsen lists twelve dimen-
sions across which new interfaces could diverge from WIMP [Nie93].
Among these 12 dimensions, two are particularly relevant for our
concern: first, Nielsen advocates a change of *interface control* from
being made “by user (i.e. interface is explicitly made visible)” to “by
computer (since user does not worry about the interface as such)”.
Second, he advocates a change of *object visibility* from “essential for
the use of direct manipulation” to “some objects may be implicit or
hidden”, as displaying everything prevents applications and systems
to scale in size, and some objects will need to be managed by the
system.
While Nielsen's second point concerns mainly objects of interest in an application, such as files in a file system browser, he adds in a follow-up paper [GN96] that to deal with the increase of interface controls, interfaces should use modes. He advocates to solve the problem of modes not by avoiding them, but by using richer cues, i.e., making the mode more visible.

What could now be considered as post-WIMP interfaces in commercial products are mainly the interfaces of smartphones and tablets, where Nielsen's concern of scaling the interface exists not because the interface has to contain more elements, but because it is displayed in a smaller screen. While these interfaces rely often on different panes or tabs for different types of activities and contents, and panes could technically be considered as modes, they usually have very different layouts that make the current mode quite obvious, as advised by Nielsen. However, smartphone and tablet interfaces often rely on non-signified interactions, notably navigation gestures and vocal interactions for digital assistants like Apple's Siri.

For navigation gestures, users may have to swipe to navigate between panes, to double-tap on an element to zoom on it, to pinch or spread two fingers to navigate in maps, as well as simply sliding a finger or performing a flick gesture to scroll. These gestures are believed to be part of a user's basic interaction vocabulary and to implement the system interface metaphor of manipulating physical objects, and are therefore usually not signified directly. There are sometimes indirect signifiers from which users can deduce an interaction possibility, such as little dots at the bottom of a pane to represent the different panes (one dot for each) and the active one (single dot with a different style). These signifiers are indirect in that they only represent the system state (see paragraphs on modes in Section 6.2.3), from which users can deduce the presence of an effecter affordance (to change pane), but nothing communicates the handling affordance (swiping), which users have to understand by recognizing a convention. Thus, users can learn to perceive such dots as signifiers for the navigation gestures, but the efficiency of such signifiers depends on user's expertise.

For voice interactions, a problem is similar to the one of command line interfaces in that users have no way to know what commands they can speak: these commands and their syntax are not signified. However, the problem is also different in that the assistant is supposed to be smart and understand free-form questions, so that users can use their natural communication skills to issue commands. Yet, from the point of view of instrumental affordances, the question of knowing how to communicate the command to the system is a question of handling affordance, and digital assistants also have hidden effecter affordances in that users are often unaware of what the assistant can do for them beside searching information on the web.
6.3 TYPES OF SIGNIFIER-LESS DESIGN

From the definition of signifier-less designs and the examples pre-
sented in the historical review of signifier-less design, we can observe
different types of signifier-less designs.

Since I have defined signifier-less designs in Section 6.1 on the
basis of a missing signifier for an instrumental affordance, and since
instrumental affordances result from the connection of a handling
affordance with an effecter affordance in an instrument, signifier-less
designs can be either:

PUZZLING: The handling affordance lacks signifier.

HIDDEN: Both handling and effecter affordances lack signifiers.

MYSTERIOUS: The effecter affordance lacks signifier.

In *puzzling* signifier-less designs, the effecter instrument is signified
to the user, but not the handling affordance. Users can thus know that
a possibility to do something exists but they do not know how to do
it. This is for instance the case of the handle-less cabinet mentioned
by Norman in the quote introducing Chapter 1: it is clear that there is
a cabinet with drawers, the drawers are even clearly visible, and can
thus serve as signifiers of the effecter affordance, which is that they can
be opened. But the drawers have no handle, so users do not know how
to open them: the handling affordance, which consists here in pushing
a drawer's front surface, is not signified. In a touch-based mobile
user interface, the equivalent would be, for instance, a scrollview that
can be scrolled directly by dragging its content with the finger. The
content flowing out of the visible window would be a signifier that
the view can be scrolled, and so the effecter affordance is signified
(see Section 5.1.3). But the handling affordance, consisting in dragging
the content, is not signified.<fnref n="2" />

Puzzling signifier affordances can lead users to face a wall, explore
the interface in search of the handling affordance, or call for an external
help. This can be a terrible experience, and users can blame themselves
for their incompetence instead of blaming the design. But not always.
Indeed, some people love challenges and figuring things out, and
succeeding at doing so brings an intrinsic reward in the form of self-
satisfaction. After the facts, it can also be an experience that one can
share with their belongings and make fun of it.

In *hidden* signifier-less designs, neither the handling nor the effecter
affordance are signified. Users may thus simply never get to know
that an interaction possibility exists. To expand on the example from
Norman's quote, hidden signifier-less design would now be the case

<footnote n="2">2 Note that it would not be a signifier-less design if the dragging was made with a mouse, as in some document viewers, and the mouse cursor changed for a hand that signify “dragging”.</footnote>
where "the cracks that signify the existence of a door [are] eliminated. The result can be a smooth expanse of gleaming material, with no sign of doors [...], let alone of how those doors [...] might be operated." Many hidden signifier-less design showed up in the previous section, such as command line interfaces, gestural interfaces, and vocal interfaces. *Swhidgets* are also hidden signifier-less designs, as neither the swipe gesture (handler affordance) nor the availability of further controls are signified (as for contextual menus, the effecter affordance of *swhidgets* consists in providing access to further controls).

The discovery of hidden signifier-less designs can cause mixed reactions: on one side there can be the feeling that the designers hid something useful to you. Users may thus blame the designers for missed opportunities. They may also question their competence with thoughts like "was I supposed to know that?" On the other hand, users might simply be happy to discover a new feature and appreciate how it might positively impact their life in the future. The design itself can sometime be thought of as clever, and users might also get a small boost in their feeling of being competent for discovering – and now, knowing – the hidden feature.

There is not always a clear distinction between hidden and puzzling signifier-less designs, as for some users it might be obvious that the interface must provide some feature, and so there must be a corresponding effecter affordance. Even if this effecter affordance is not signified in the interface, the application / product / system could sometime be considered as a signifier in itself. For instance, in a messaging application, you would expect to be able to send messages: there is an implied effecter affordance for sending messages, communicated to you even before you open (or even, install) the application.

Finally, in *mysterious* signifier-less designs, there is a signifier for the handling affordance but not for the effecter affordance. This situation is usually described in the HCI literature as a lack of *feedforward* (see Section 5.3.7). A typical (and exaggerated) example would be a big red button with no label on it and nothing around it: there is no way to know what it is for, except by trying to press it, with potentially dangerous results. A more common example would be an array of unlabeled light switches that control different lights: unless the switches and lights are configured to create a natural mapping (see Section 5.3.4), users cannot know which light is controlled by a given switch. However, in that case, it is also possible to consider that the placement and type of switches signify, by convention, that they control lights, and that the problem is not that the effecter affordance is not signified, but that it is imprecisely signified: we know that a switch has an effecter affordance of controlling a light, but we do not know which light. A related user interface example would be the three buttons at the top-left of every window in MacOS, which can
be used respectively to close the window, minimize it, or put it in
full screen: on recent versions of the OS, these buttons do not show
the usual cross, minus sign, and diagonal double-arrows that used
to represent the corresponding commands (they are only displayed
when the mouse hovers the group of buttons), but they are still drawn
respectively in red, orange and green. Although the placement and
color of these buttons can be considered as signifiers for users with a
minimum of knowledge about WIMP interfaces, they might as well be
mysterious signifier-less designs for new users<fnref n="3" /> – or when the function
of the button change, such as when a new version of MacOS replaced
the “maximize window” function with the current “enter full-screen
mode” one.

This type of signifier-less design causes less usability problems
than the two others, since it has the advantage of being discoverable:
as users are aware of the handling affordance, they can try it to
discover its effect. Yet, some users might be afraid of trying something
they do not understand. If the unknown effecter affordance is an
important feature for the user, she might never discover it because of
that. In other cases, trying might not provide the required information,
because the results are not visible (in addition to the problem of
feedforward, there is a feedback problem).

<footnote n="3">³ E. g., a young child familiarized with user interfaces through touch-based devices and toys, but who has never used a desktop computer with a WIMP interface.</footnote>
<figure><img src="image_38.png" /></figure>
<figure><img src="image_39.png" /></figure>
# Part III

## PROPOSITIONS FOR A MODEL OF DISCOVERY
<figure><img src="image_40.png" /></figure>
<figure><img src="image_41.png" /></figure>
<figure><img src="image_42.png" /></figure>
7.1 ABOUT THIS CHAPTER

When an affordance<fnref n="1" /> is not signified, users may not know about it, or
may not think about using it even if they know it – simply because
nothing is there to remind them about this possibility. Conversely,
when a user do not employ a non-signified interaction, is it because
she does not know about it, because she forgot about it this time, or
because she prefers another way of achieving her goals? Although this
question can be asked for any interaction technique, it is particularly
relevant for signifier-less designs: since the knowledge about them
is not “in the world” and has to be “in user’s head”, a major factor
controlling their usage is thus the user’s ability to “think” about them
when she needs it – an ability that I will call the user’s *proficiency* in this
(set of) signifier-less design(s) to distinguish it from the mere notion
of performance (indeed, the technique might actually be less efficient
than alternatives) or usage of the technique (it may be perfectly known
but not used).

This chapter concerns the mechanisms by which user proficiency in
some interaction technique can improve, in general and for signifier-
less designs in particular. To investigate this question, I propose a
model of proficiency improvements that involves three components
(as illustrated in Figure 7.1):

1. *Current knowledge and skills for the interface/task* are user attributes that affect her proficiency: the user's knowledge of the system (including the non-signified interaction possibilities); how skilled she is in using the different parts of the system; her habits and preferences; and ultimately, how she envision the tasks she's using the system for and what place she wants the system to have in her lifestyle.

2. *Design Means* by which a user can improve her proficiency by modifying some of her current knowledge and skills. This aspect of the model has already been described in Chapter 5 as consisting mostly in signifiers, and will thus only be discussed in this chapter in relation with the other two components.

<footnote n="1">1 I use “affordance” rather than “interaction possibility” in this chapter, because as we have seen in Chapter 3, affordances are not only interaction possibilities, they also have meaning as elements of a system of affordances, and this aspect might be important in their discovery.</footnote>
<figure><img src="image_43.png" /><figcaption>Figure 7.1: Proposed model of proficiency improvements, consisting in the three dimensions and their interactions (see the text for explanations).</figcaption></figure>

3. *Causes and motivations* for such changes, which explain why a user would (or would not) improve her current knowledge and skills about the interface.

I will describe the state of the art for each of these three components in the following sections, then dedicate sections to their interactions. Finally, I will present two personal contributions in these areas: *degrees of knowledge*, a simple model of how well something is known and how this knowledge evolves; and *sources of knowledge*, a generalization of the concept of design means;

## 7.2 CURRENT KNOWLEDGE AND SKILLS

Defining exactly what a user knows about an interface and how skilled she is in using it is beyond the scope of HCI as it would rather be the domain of Psychology. However, HCI researchers have proposed simple models of user knowledge and skills that are generic enough to fit any type of interface, and have used such models to understand user behavior and provide design guidelines.

The simplest of these models, and probably the most used, is a distinction between *novice* and *expert* users. These common-sense notions are however rarely defined precisely, and while it allows to raise the question of how users transition from being novice toward being expert, answering this question requires slightly more precise notions (see [Coc+14] for an example and review).

Since I am interested in understanding how users can develop their knowledge and skills, I will first focus on how the interaction with an application or device requires knowledge and skills in a variety of domains, and on how these knowledge and skills, far from being independent from each other, are connected in structures similar to the *stratified hierarchies* already discussed for affordances in Section 3.2.1. I
will thus discuss how users can exploit this structure to expand their knowledge, and develop usage habits on top of the newly acquired knowledge.

### 7.2.1 Domain of the knowledge and skills

This dimension describes what is concerned by the knowledge and skills that users need to possess to interact with a system. It is a *stratified hierarchy* inspired by some of the metrics for expertise defined by Grossman and Fitzmaurice [GF15], and particularly the metrics of *scope* (with three different sub-levels: UI, commands and task), *appropriateness of workflows*, and *domain knowledge*.<fnref n="2" />

**GENERAL KNOWLEDGE AND SKILLS** This level is about knowledge and skills developed by people to interact with the world in general – rather than for a specific interface or activity – and especially with the non-computerized aspects of their daily lives. For instance, the Reality-Based Interaction (RBI) framework [Jac+08] lists four “themes of reality” that interface designers can build upon:

* naïve physics: people have common sense knowledge about the physical world.

* body awareness and skills: people have an awareness of their own physical bodies and possess skills for controlling and coordinating their bodies.

* environment awareness and skills: people have a sense of their surroundings and possess skills for negotiating, manipulating, and navigating within their environment.

* social awareness and skills: people are generally aware of others in their environment and have skills for interacting with them.

Interfaces often rely on real-world metaphors so that users can reuse their general knowledge and skills. Some interfaces can also rely directly on some general knowledge and skills, as with the object manipulation skills involved in tangible interfaces.

**UI LEVEL** At this level, knowledge and skills concern the use of user interface elements like widgets and gestures, independently of their role in an application's support of some activity. This is the knowledge and skills that someone can use to make sense of the

<footnote n="2">² Since Grossman and Fitzmaurice are interested in the measure of user expertise from the usage of the system, they distinguish low-level expertise measures (scope and its sub-levels) and high-level ones (appropriateness of workflow and domain knowledge), as the former only concern the expertise of the interface and the latter concern more generally the activity supported by the system.</footnote>
interaction possibilities provided by an unknown application, without
ever knowing what the application is for.

For instance, skills at that level would be typing with a keyboard, pointing with a mouse, using a scroll bar, or selecting a second-level item in a menu. An example of knowledge at that level would be knowing that clicking a menu label will open the corresponding sub-menu – independently of what the menu actually contains.

This level exists because applications generally reuse interface com-
ponents provided by the operating system (or by libraries possibly
shared with other applications), which in turn often rely on some sim-
ilarities with the interfaces of other systems. It is therefore assumed
that, to the exception of some widgets specific to an application (like
Microsoft Word's Ribbon), users already possess knowledge and skills
at the UI level before even facing an application that require this
knowledge.

**COMMAND LEVEL** At this level, knowledge and skills concern the features provided by an application. Knowledge at this level would typically be the topic of application manuals or help files. For instance, knowing what font modifications (such as bold, italic and underlined) are available in a text editor and how to apply them to some text, or knowing what type of blur effects are provided by an image manipulation application and how to select the right blur parameters to obtain the desired effect. This level includes knowledge and skills that affect a user's awareness of the command and efficiency in triggering and applying the command. It would thus include the fact that the button labeled B sets the text in bold face, but would not include knowledge and skills about how to press a button, which are at the UI level. In other words, this is the knowledge about the mapping between UI elements and the effects they cause outside the UI – i.e. in the states of the world, of the documents manipulated in the application, or of the application itself – and the skills involved in manipulating the UI elements so that their effects match a desired result.

**TASK LEVEL** At this level are knowledge and skills about how to perform actual *tasks*, defined by Grossman and Fitzmaurice as a target goal that a user wishes to accomplish: "Typically the goal is to achieve a desired effect on the application content or environment, and the user must choose an appropriate method to accomplish this goal, made up of a sequence of actions" [GF15].<fnref n="3" />

Some tasks can be achieved by the use of a single command, in
which case the distinction between the task level and the command
level is blurred – although it stills exists: at the command level, the
focus is on what the command do, while at the task level the focus

<footnote n="3">³ The reader will recognize in this definition the stages of Intention and Sequence of Actions from Norman's model (see Section 4.1).</footnote>
is on how to achieve some result with the command. For instance, knowledge that a text can be made more visible by using a bold typeface would be at the task level, knowledge that the button labeled "B" (un)sets the selected text in bold face would be at the command level, and knowledge about how to press a button would be at the UI level.

Other tasks require the combination of multiple commands. For instance, a task like "printing the document edited" might involve many commands to choose what printer to print with, what portions of the document must be printed, the quality of the printing (ink, paper and speed), or options to save paper (two-sided, multiple pages per sheet of paper, etc).

Knowledge at the task level would typically be the topic of tutorials and applied lectures. It may consist in the awareness of different possible plans of action to reach the expected goal, such as using a menu entry or a shortcut to trigger a command. Skills may concern the ability to evaluate the efficiency of each plan of action in different contexts and the ability to choose the most efficient one depending on the current context. But knowledge and skills at the task level can also simply consist in the knowledge of procedures of the style "to achieve result X, do steps A, B, C" and the skills required to apply such procedures (memorizing the steps, recognizing the corresponding states of the application, etc).

**APPROPRIATENESS OF WORKFLOWS**

This level describes knowledge and cognitive skills involved in the decomposition of tasks into sub-tasks and ordering of tasks themselves.

For instance, a user can add five rows in a spreadsheet by repeating five times the task of adding one row, or she can know a way to add the five rows at once. Obviously, in this example, users can only choose the appropriate workflow if they have the task-level knowledge about how to add a single row and how to add multiple rows at once. However, even when they know how to add multiple rows at once, some users may prefer adding the rows one by one, or they might simply not think about using the more efficient method. There are thus awareness and skills involved in this choice, which is more about the appropriate way to think about the task than knowing the different ways to obtain a result.

Similarly, in their discussion of how applications can provide *support for advanced task strategies*, Cockburn *et al.* [Coc+14] provide the example of creating three identical objects made of the same parts A-B-C with a drawing application: a novice strategy might be to create the parts A, B, and then C of a first object then repeat these steps for the second object, and then for the third; but it can also be more efficient to draw three parts A, then three parts B, then three parts C, as it may require fewer changes of drawing tool; alternatively, a user
who knows about grouping shapes may prefer to create the three parts of a first object, group them, and then copy/paste the group to create the two other objects. These three strategies illustrate three different ways to think about the task: they are not as much about getting the desired result (which would be at the task level), but about organizing the workflows by realizing that operations can be reordered to minimize tool changes, or by understanding how a specific command like “grouping” can be used.

**DOMAIN KNOWLEDGE** This is the general knowledge about the activity supported by an application rather than knowledge of the application itself. For instance, it could be knowledge about how to write a research paper, by opposition to the knowledge of a specific word processor used to do it. Domain knowledge implies the awareness of specific tasks (for instance, citing another paper, adding a figure) and it can suggest to users that an application should provide some commands to accomplish these tasks, but it does not include the specific way to use these commands to accomplish these tasks (these are knowledge and skills from the previous levels).

Domain knowledge can also induce the choice of appropriate workflows, although not being directly about the workflows. For instance, a user that receives a lot of emails may have developed specific knowledge and skills such as identifying quickly spams and e-mails that need to be replied with a higher priority. This user might adapt her workflow in order to exploit these knowledge and skills. For instance, she could delete spams and flag important mails directly from the list of newly received mails, based on the information and short extracts provided by this list, before reading the flagged mails individually. On the other hand, a less experienced user might prefer to read each mail individually before deciding to delete/ignore/reply it with the widgets provided in the mail view.

Similarly, when users know things like the relative frequencies of tasks in an activity, what tasks usually follow each others, and the type of materials each task requires, they can organize their work environment on the basis of that knowledge to augment their productivity: The structures resulting from such organizations have been called *taskonomies* by the anthropologists Janet Dougherty and Charles Keller [Noro6].

### 7.2.2 Improving knowledge and skills in time

According to the influential model of Fitts and Posner [FP67], people repeatedly engaged in an activity go through three phases of acquisition of the knowledge and skills required for this activity: cognitive, then associative, and then autonomous phases [Coc+14]. As Rasmussen remarks, other authors have proposed categorizations of
human behaviors into three categories similar to the cognitive/associative/autonomous trichotomy [Ras83].

The first phase of skill improvement, the cognitive phase, is dominated by learning *what* has to be done. It involves a substantial amount of declarative knowledge (i.e., knowledge that can be expressed as sentences). It is slow, attention-demanding, serial in nature, and strongly *volitional* in that the activities can easily be avoided or interrupted.

The second phase of skill development, the associative phase, is dominated by learning *how* to do the activity and is characterized by improvements in the motor actions through subtle adjustments of execution, leading to slower but more stable performance gains than in the cognitive phase. The declarative knowledge acquired during the cognitive phase is largely unused and replaced with procedural knowledge ("know-how") that is unconscious and can hardly be expressed as sentences. For some activities (e.g., playing an instrument), this phase can last years.

Finally, the autonomous phrase, is characterized by *fast, not attention-demanding, parallel in nature, and not volitional* actions. The execution of tasks in this phase are also less demanding cognitively and physically.

The existence of these three phases of acquisition of knowledge and skills implies that user performance improves in time, and thus establishes a relation between a user's knowledge and skills and her performance. This relation will be analyzed in Section 7.3.1 as an interaction between a user's current knowledge and her ability to exploit design means, and in Section 7.5.1 as an interaction between user's current knowledge and performance-related motivations.

### 7.2.3 Dependency across levels

We can expect learning of the higher levels to build over the knowledge of lower levels. This is for instance a core concept of *direct manipulation* interfaces, as expressed in the concern of *layered learning* (see Section 6.2.2). This view integrates two concepts: the idea that users need to know an interaction technique before being able to use another one that relies on the former; and the idea that users can understand new interaction possibilities using their knowledge of other interaction techniques.

The example of deleting a file by drag-and-dropping it on the trashcan icon, already discussed in Section 6.2.2, illustrates this: First, it involves a *drag-and-drop* and this technique must be known before being able to perform a *drag-and-drop* to the *trashcan*. This is a case of dependency between two layers of knowledge, the knowledge of how to perform a drag-and-drop belonging to the UI level, while the knowledge about using a drag-and-drop to delete a file belongs to the command level. Second, it expands the domain of application of the drag-and-drop and thus allows users to re-frame their under-
standing of what a drag-and-drop is for, to see it not as a specific interaction technique to move files in the file system, but as a more general interaction technique to act on files. This generalization of the knowledge about the purpose of the drag-and-drop could then serve as an example for further generalizations (for instance, maybe drag-and-drop does not work only on file icons, but also on other types of data?).

A consequence of this second aspect of layered learning (i.e., understanding) is that users may have the knowledge of some possibility – because they understood it and remembered it – and may still not have integrated this knowledge into skills, strategies, or other forms of behavior that rely on this knowledge (i.e., they have not yet reached the associative or autonomous phases). In particular, for signifier-less designs, a problem is that users may know how to interact at the UI level despite the lack of signifier, but may not have integrated this knowledge in how to trigger some commands. A typical example is provided by hotkeys: people may know there is a hotkey or that there is likely one for some command, but not have the reflex of using the hotkey (or searching for it) when they want to trigger the command.

This is another problematic dimension of signifier-less designs: not only the lack of signifiers make them hard to discover, but it also makes it hard to remember them when they are known. As a consequence, a model of users' knowledge of signifier-less designs such as the one I will propose in Section 7.6 needs to integrate the distinction between knowing an interaction technique and adopting it.

Conversely, the role of layered learning in allowing to understand other interactions can be bypassed: tutorials and other sources of knowledge about the interfaces sometime provide “recipes” to follow to reach some result, without explaining why they work – either from the point of view of the sub-interactions and interface concepts involved, or from the point of view of how this specific way of doing things is a best practice for the activity. Users can then have an incomplete view of the relation between one piece of knowledge and other pieces of knowledge in other levels, which, if they were known, could help users adapt more easily to new situations or tasks. This is a problem that is well known in pedagogy, that has implications on signifier-less designs when their discovery depends on external sources of knowledge, but this concern will not be studied further in this thesis as the priority is to identify the possible external sources of knowledge and bring evidence of their use.
<figure><img src="image_44.png" /><figcaption>Figure 7.2: Kurtenbach’s *marking menu* is a pie menu (a) that only appears around the mouse cursor after a short delay, providing an *expert mode* (b) for users who already know well the menu and can perform the corresponding gesture before the menu appears (in the pie menu, items occupy sectors of a disk and are triggered by leaving the sector at the disk’s border). Adapted from [KB91].</figcaption></figure>

## 7.3 INTERACTION BETWEEN CURRENT KNOWLEDGE AND DESIGN MEANS

### 7.3.1 Current understanding affects the usefulness of means

The efficiency of design means such as signifiers in helping users acquire new knowledge about the interface may depend on the user's current knowledge and skills themselves. Indeed, some design features can be used as clues to improve knowledge and skills only if the user already possess some knowledge that will allow her to interpret these features as signifiers. This phenomenon is already recognized implicitly in interaction design paradigms such as the layered approach to learning of *direct manipulation* (see Sections 6.2.2 and 7.2.3), but it is worth analyzing more in details some aspects of the phenomenon and how it can be exploited in design.

Transfer of knowledge and skills between applications can rely on this phenomenon: the knowledge of some interaction contexts allows users to recognize similar contexts, and this similarity becomes a new design mean to improve the general knowledge and skills of these users. It may be an important factor in the discovery of signifier-less designs such as the *item swhidgets* in the iOS Mail application (see Section 2.1), where the list layout can act as a signifier that items in the list may hide *swhidgets*, an interpretation that a user can only make if she already knows about *item swhidgets* in other list layouts.

Rasmussen has described how a user's current knowledge and skills affects the kind of clues that she will pick up in the interface to guide her behavior [Ras83]. His model is based on the distinction between the cognitive, associative and autonomous phases of acquisition of knowledge and skills (see Section 7.2.2), and is close in principle to the Seven Stages of Actions (see Chapter 4), which was proposed later and
is simpler. In Rasmussen's model, there are three types of behaviors that he names *knowledge-based*, *rule-based*, and *skill-based*, each type of behavior being predominant in (respectively) the cognitive, associative, and autonomous phases. These behaviors and the corresponding usage of design means will be described below, taking the design of Kurtenbach's *marking menus* [KB91; Kur91; Kur93] as an example.

In *knowledge-based behaviors*, users rely on what Rasmussen calls *symbols*,<fnref n="4" /> which “represent other information, variables, relations, and properties and can be formally processed. Symbols are abstract constructs related to and defined by a formal structure of relations and processes – which by conventions can be related to features of the external world.” He provides the example of a dial’s hand displaying an abnormal value (and thus, at an abnormal position), which users interpret as a need to understand why it happens using her mental model of the system (maybe it is a pressure gauge and there is a leak in the system?), eventually updating this mental model.

Such a knowledge-based behavior can be seen in how novice users of marking menus would understand them: the appearance of the menu, with its labels and sectors in a circle (see Figure 7.2-a), would not fit their current understanding of what a marking menu is or how to use it. They would thus need to interpret the circle and its label as menu items, the radius of the menu as a distance they need to travel with the mouse to trigger the selection of an item, and the direction of the mouse movement as the way to select the desired menu item by moving the mouse in the corresponding direction. Feedback and feedforward can help them create such a model by exposing respectively its dynamic behavior and its purpose (as Rasmussen discusses [Ras83]). Once they have this model of how a marking menu works, they can start using it, but even then, they still need to know what item to select according to their needs. To do so, they must interpret the menu labels as symbols related to commands, in order to find the one that matches their needs: this identification, although simple, is an example of the kind of problem-solving behaviors that characterize the cognitive phase. At that point, the user only has to execute the item selection, relying on feedback to be sure that she does it correctly.

For signifier-less designs, the lack of “symbols” advertising an affordance is an obstacle to knowledge-based behavior. It can however still be triggered by an accidental revelation of *swmidgets*, or in case users try to manipulate the interface elements in ways that are known

<footnote n="4">4 Rasmussen does not use the words *symbol* and *sign* in the same way as Peirce does (see Section 5.2). However, Rasmussen's distinction between signals, signs, and symbols can be understood as an instance of a Peircean trichotomy in Firstness, Secondness, and Thirdness respectively. Indeed, signals are interpreted directly into calls for action, "signs" are interpreted as referring to a rule telling what action should be taken, and "symbols" are interpreted as a need to come up with a mediating explanation before determining what action should be taken.</footnote>
to be compatible with the physical metaphors used on the system (not
only for *swhidgets*). In addition, knowledge-based behavior may be
triggered by other factors than observing a symbol that does not fit
users' current understanding. For instance, it could be triggered by
realizations of an interaction need, such as "I need a command to
do X" or "that would be helpful if there was a faster way to do Y".
Exploring the interface next to find a way to satisfy these needs would
be a knowledge-based behavior (see Section 4.2.4).

In rule-based behaviors, users rely on what Rasmussen calls signs,
which "indicate a state in the environment with reference to certain
conventions for acts. Signs are related to certain features in the en-
vironment and the connected conditions for action. Signs cannot be
processed directly, they serve to activate stored patterns of behavior."
He provides the example of the dial's hand now being interpreted
differently depending on the state of the system as indicating a normal
state of operation, or a need for adjustment, or a need for re-calibrating
the system.

Rule-behavior can be seen in how users of marking menus with an
intermediate level of expertise would use them: in their knowledge
of the menu, they would have rules like "to draw a square, select
the square icon" or "if the label is grayed, the command is not avail-
able". At this point, they have integrated the knowledge of the basic
operation of the marking menu, but they still rely on the labels to be
displayed to know how to interact. They can however simply recog-
nize the label they need instead of trying to interpret it: the labels do
not need to act as feedforward anymore, but only to act as a landmark
guiding the user.

Because rule-based behavior is based on the recognition of "signs",
and any aspect of the interface can be considered as a sign, it is per-
fectly possible to have rule-based behaviors for signifier-less designs.
For instance, a list layout can be recognized as such, and activate
the rule "to delete an item, attempt a leftward swipe on it". In that
case, there is however a question of the effectiveness of the rule for
predicting the availability of *swhidgets* on the items of the list, which
is a question of consistency in the design of lists across applications.

In *skill-based behaviors*, users rely on *signals* from the interface, which
“are sensory data representing time-space variables from a dynamical
spatial configuration in the environment, and they can be processed
by the organism as continuous variables”. He provides the example
of the deviation of a dial’s hand relatively to the set point, which can
be tracked continuously and maintained close to zero with a control
loop relying only on simple psycho-motor skills.

Skill-based behavior can be seen in how expert users would use the
marking menu: they would know in what direction they should move
the mouse without even looking at the menu, and all they need is a
graphical feedback to be sure their gesture has the right direction and
length. As a consequence, they do not need to wait for the menu to be displayed and can do the "blind selection" of the expert mode shown in Figure 7.2-b.

Similar skill-based behaviors can be observed for signifier-less designs, in which continuous variables like the positions of animated elements and mouse pointers can be tracked continuously to control the precise timings and movements of a skilled manipulation. Indeed, at that point, users do not need the signifiers that are missing for novice users (which is why signifier-less designs are often considered as expert techniques, both by their designers and by users).

### 7.3.2 Learnability of the interface

The notion of interface *learnability* has been recognized as an important aspect of interface usability and can be defined broadly as "to what extent the interface supports learning (eventually for different types of users)". Learnability can thus be understood as a global measure of the efficiency of the design means. A 2009 survey about the usage of this term in the HCI community [GFA09] revealed that definitions of learnability focus on the efficiency of the design means but mostly ignore users' motivation for learning about the interface beside performance improvement (which will be discussed in Section 7.5.1). Many definitions acknowledge, however, that different populations of users with different levels of expertise have different needs in terms of learnability of the interface – which is a way of presenting the concern that actually encapsulate two aspects: 1) the fact that these populations need to learn different types of knowledge and skills, and 2) the fact that the efficiency of the design means depends on user's current knowledge and skills. Another way to say this is to observe that in the left part of Figure 7.1, there is a dependency loop between design means and current knowledge and skills, and that such a loop can create complex behaviors in terms of the time-related aspects of learnability.

The fact that learnability is a global measure of the efficiency of all the design means in the interface hides another subtlety, which is that the global learnability of the interface may be greater or lesser than the sum of the efficiencies of the individual design means it is made of. We have seen that when transfers of knowledge are possible, learning one interaction contexts can improve the learnability of another interaction context. Conversely, when multiple things must be learned, they compete for users' learning efforts (as we will discuss in Section 7.5.2). While there has been a great amount of works that concerned the efficiency of specific means, as we have seen in Chapter 5, there is much fewer works that provide insights on how to make different parts of the interface contribute together positively to the overall learnability of the interface. The guidelines in this domain seem to
fall down to “take care of consistency”, although consistency can also
be considered harmful sometimes [Gru89].

Concerning the dynamics of learnability, Kurtenbach has provided
three design principles to support learning and using gestures: reve-
lation, guidance, and rehearsal [KMB]. *Revelation* is defined as “the
system should interactively reveal information about what commands
are available and how to invoke them”. Revelation is illustrated in
how novice users can understand the whole design of a marking
menu, as we have discussed in the paragraph about knowledge-based
behaviors. *Guidance* is defined as “the way in which revelation occurs
should guide a user through the method for specifying the complete
command in any specific situation”. It is also illustrated in how novice
and intermediate users can use the labels to know in what direction
they should move the mouse to trigger the corresponding item, and
in how the system provide feedback as they do so. Finally, *rehearsal* is
defined as “the way guidance is provided should require a physical
rehearsal of the way an expert would issue the command”. Indeed,
users can perform the same gesture to select an item in the expert
mode than with the menu displayed. This form of rehearsal is believed
to help users learn the most efficient way to make the gesture and is
characteristic of the associative phase (Section 7.2.2).

Kurtenbach’s design of marking menus exploits the fact that skilled
behavior requires less design means than novice behavior (according
to Rasmussen’s model), so that visual feedforward can be removed
for expert users. A popular approach to the design of interfaces
does exactly the opposite, choosing to disclose complexity as the user
progresses toward expertise: advanced features are hidden first until
users have the necessary knowledge to understand the benefits of their
use. This approach can be seen in applications with “simple” and
“expert” interfaces, buttons labeled “advanced options” or “advanced
search”, and panels with extra commands that are hidden by default.
Customization of the interface is also related to this approach, as
far as the selection of a default configuration is concerned. Indeed,
with a simplified interface, novice users can find unknown features
more easily by problem-solving or exploration, simply because there
are fewer possibilities to consider. However, this approach cannot
be justified by considerations for learning alone, as it also has other
benefits. For instance, it can help making the product less intimidating
to new users and better expose how it is suited to their needs (see
Section 7.4.3 on User Experience). It can also make the interface
more efficient for novice users by providing less controls, but more
efficient signifiers and controls (e.g., bigger buttons that are faster to
click on, tool palettes that provide better sensory affordances – see
Section 5.1.2) – although advanced users may then need ways to regain
the loss of performance caused by switching to the advanced, less
efficient interface, such as hotkeys or gestures.
## 7.4 MOTIVATIONS

Motivation is the experience of desire or aversion: a desire to reach or to possess certain things or status, or an aversion that push you to avoid something. Here, I address the motivation for users to improve their knowledge and skills related to the interaction with a system's interface and the activities it supports. I first address the difference between intrinsic and extrinsic motivation, and then present different psychological theory that highlight different types of rewards. The last subsection will address causes for learning that cannot be linked to any motivation.

### 7.4.1 Extrinsic vs. intrinsic motivation

According to Richard Ryan and Edward L. Deci, motivation can be separated into extrinsic and intrinsic ones [RDoob].

*Extrinsic motivation* comes from external sources: you are motivated to do something because of its consequences, including the possibility to get various forms of rewards or to avoid negative effects (treats, punishments, etc.) For instance, an extrinsic motivation to have a better knowledge of some tools could be that you believe it would allow you to be more efficient in your job, and later be in position to ask for a salary rise.

*Intrinsic motivation* comes from internal sources: you are motivated to do something because you value doing this thing in itself, not because of its consequences. For instance you could play a musical instrument not because you want to make music but because you appreciate doing so: the activity that you are motivated to do is its own reward. A particular case of intrinsic motivation is the natural, inherent drive to seek out challenges and new possibilities that Self-Determination Theory (see below) associates with cognitive and social development.

There can be both intrinsic and extrinsic motivations at the same time. For instance, reading a novel before sleeping might be valuable in itself (intrinsic motivation), but can also bring the reward of a peaceful mind for a better sleep (extrinsic motivation). Similarly, practicing some sport might be fun in itself, but can also be a mean to improve your health or physical appearance.

The distinction between intrinsic and extrinsic motivation has been used to show that offering an external reward for a task that was intrinsically motivated could actually undermine the motivation instead of reinforcing it, or to observe that the efficiency of extrinsic motivators tend to decrease in time faster than the efficiency of intrinsic motivators.

The distinction is based on the source of the motivation: what is the motivator, rather than how it motivates you. Other classifications of
motivations exist on other bases, such as the distinction between push and pull motivations that are based on how you experience the motivation (it is a push if you feel that you have to push yourself toward the activity, but it is a pull if you feel attracted toward the activity or reward, such as when you act by curiosity or empathy). A complete account of all such classifications is out of the topic of this dissertation: I only presented the distinction between intrinsic and extrinsic motivations because it is pertinent in the context of learning and discovery, where external motivations such as performance improvement call for cost-benefits trade-off evaluations with consequences on the learning (as we will see in Section 7.5.1), while intrinsic motivations such as curiosity do not.

### 7.4.2 Self-determination theory

Ryan and Deci created the Self-Determination Theory [RDooa] (later simplified as SDT) in the 70s. It is a macro theory of human motivation and personality, which concerns people's inherent growth tendencies and innate psychological needs. SDT focuses on the degree to which an individual's behavior is self-motivated and self-determined. It introduced the intrinsic/extrinsic dichotomy of motivations and additionally considers three basic psychological needs, which lack of satisfaction causes negative consequences globally, while their satisfaction by an activity is associated with positive consequences of the activity:

* *Competence* is the need to seek to control the outcomes of one's actions and experience mastery.

* *Autonomy* is the need or desire to be causal agents of one's own life and act in harmony with one's integrated self.

* *Relatedness* is the need to interact with, be connected to, and experience caring for others.

Self-Determination Theory received a lot of attention in the HCI subfield of game design and gamification research [TM20], but received little attention from other subfields traditionally involved in usability and design, despite the recent availability of SDT-based questionnaires [Brü+18] and suggestions that it could help better understand the influence of input methods on users' experience with interactive products beyond games [Wat+13].

In the case of signifier-less design, I believe the SDT could help understand some aspects of the design. For instance, the needs for competency, autonomy and relatedness could cause extrinsic motivations such as: (competence) searching hidden widgets because they are efficient and will allow the user to become more competent, (autonomy) to find new ways to use the device in users' lives that better
<figure><img src="image_45.png" /><figcaption>Figure 7.3: Hassenzahl’s model of User Experience (UX), from [Haso3].</figcaption></figure>

match what they expect for their life (e.g., saving time by reading my e-mail in commutation with an interaction that is easy to perform in a crowded train), or (relatedness) be able to talk about signifier-less designs with others.

### 7.4.3 Hassenzahl’s model of UX

Hassenzahl presented a model of the User Experience (UX) of a product from both designer’s and user’s perspective [Haso3]. The model is based on the assumption that a product has some *features* that convey a particular *product character* to users, which leads to *consequences* on the user’s emotions and behaviors (see Figure 7.3). Additionally, the model distinguishes between the designer’s perspective and the user’s perspective. From the point of view of the designer, she chooses and combines the product’s features with the goal of creating an *intended character*. From the point of view of the user, she perceives the features of the product and construct her own *apparent product character* and reacts to it.<fnref n="5" /> The *consequences* on the user are not always the same and depend on the actual *situation*, and can consists in a judgment about the product’s appeal (e.g., “It is good/bad”), emotional conse-

<footnote n="5">5 The difference between the intended and apparent product characters is similar to Norman's distinction between the designer's mental model and the user's one [Nor88], although the notions of product character and mental model are quite different.</footnote>
quences (e.g., pleasure, satisfaction) and behavioral consequences (e.g.,
increased time spent with the product).

The features of a product are: what it contains (*Content*), what it
looks like (*Presentation*), how it works (*Functionality*), and how the
interaction is suggested (*Interaction*).

These four types of features give the product its own character:
a high-level description of the product, which function is to reduce
cognitive complexity and to trigger particular strategies for handling
the product. This character summarizes the attributes of the product,
such as "novel", "interesting", "useful", or "predictable". Hassen-
zahl additionally distinguishes *hedonic* attributes from *pragmatic* ones
[DHo8].

*Pragmatic attributes* concern the practical aspect of the product, the achievement of behavioral goals: tasks such as checking messages, making a call, etc. Examples of pragmatic attributes are: “simple”, “clear”, “controllable”. They are always instrumental in the sense that they concern goals that are not an end in itself. As such they have been the main focus of usability-centered approaches to design and HCI research.

*Hedonic attributes* concern the different sensations that the product
can bring to users and relate to users’ self – "Why I use this product?".
Hedonic attributes, e.g., “innovative” or “exciting”, signal the fulfill-
ment of superordinate, self-referential goals like personal growth or
self-expression. There are three categories of hedonic attributes in
Hassenzahl’s model: *evocation* concerns the memories that a product
can bring to users; *stimulation* concerns a product’s ability to address
the basic human needs of self improvement, novelty, and challenge;
and finally, *identification* addresses the human need to express one’s
self through products one owns.

A longitudinal study of the experience of users from one week
before to four week after the purchase of an iPhone [Kar+09] revealed
that the product attributes that provided positive initial experiences
were not as crucial for motivating prolonged use. According to the au-
thors of this study, "product adoption contained three distinct phases:
an initial orientation to the product dominated by the qualities of stim-
ulation and learnability, a subsequent incorporation of the product in
daily routines where usefulness and long-term usability became more
important, and finally, a phase of increased identification with the
product as it participated in users' personal and social experiences".

The *stimulation* dimension of the model could be very relevant to de-
scribe the benefits of signifier-less interaction techniques. Indeed, the
discovery of such techniques is a very stimulating experience, which
may motivate users to explore the interface in hope of such stimulating
experiences. It is also possible that the level of knowledge and skills
required to discover signifier-less designs match the user's entry in
the second phase of UX evolution, where the product get incorporated
in daily routines and its usefulness and long-term usability become relatively more important, and its learnability has become relatively less important.

### 7.4.4 *Non-motivational causes*

In addition to the motivational aspects of users' improvement of their knowledge and skills that I described above, I must mention the possibility of non-motivated causes for such improvements. In particular, improvements can result from *accidental* (i.e., unexpected) discoveries, or can be *incidental*, resulting from a practice of skills through rehearsal although the user's goal is not to become better (as it would be if the user engaged in some form of training).

## 7.5 INTERACTION BETWEEN CURRENT KNOWLEDGE AND MOTIVATION

### 7.5.1 *Knowledge motivated by performance*

**POWER LAW OF PRACTICE** A phenomenon called the *power law of practice* has been repeatedly observed when the performance of individual people involved in activities is aggregated over the group studied and plotted against time: it shows a smooth and continuous power function that starts quickly (broadly corresponding to the cognitive phase) before progressively reaching a plateau of maximal performance (corresponding to the autonomous phase) – although not all participants reach the maximal performance level and may require special training to do so.

**INTRAMODAL PERFORMANCE IMPROVEMENTS** Cockburn *et al.* use this law as a model of *intramodal performance improvement*, i.e., the performance improvement with one particular interactive method (such as pointing with the mouse) and one particular function (such as selecting the bold function in a text editor) [Coc+14]. In terms of domains of the knowledge and skills (see Section 7.2.1), it corresponds broadly to the improvement of specific UI-level knowledge and skills as support for specific command-level knowledge and skills.

**INTERMODAL PERFORMANCE IMPROVEMENTS** The notion of intermodal performance improvement is contrasted in Cockburn *et al.'s* work with *intermodal performance improvement, vocabulary extension, and task mapping at the interface* [Coc+14]. Intermodal performance improvement concerns the performance gain made possible by using a different interaction method to access a same function, when this alternative interaction method has a higher performance ceiling (i.e., the height of the plateau of maximal performance in a power law of
practice model). This notion considers performance improvements as the only motivation for using a different method although there can be other types of motivation for doing so, as we have seen in Section 7.4 (for instance, it could be motivated by a need for competence with an argument like “expert users know how to use this feature, I want to be an expert user, so I have to learn it too”, which is more a question of status than of actual performance).

Intermodal performance improvements can then be described in terms of domains of knowledge and skills as follows: first, to realize that a function can be accessed differently, there must be an improvement of the knowledge and skills for this function at the command or task level (depending on what is called a “function” of the interface); second, improvements of the knowledge and skills involved in the new interaction method must be made at the UI level to support the improvement of knowledge and skills for the function considered at the command or task level, as for intramodal performance. However, the first of these two steps is not always followed by the second. Conckburn et al. indeed consider two reasons why it could be the case: one is related to the first step, and the other is related to an opposition between short-term and long-term benefits.

### SATISFICING AND COST-BENEFIT ANALYSIS

The first reason for not switching to the new technique is an inaccurate perception of the costs and benefits of doing so, leading to a belief that it is not worthwhile. This is a problem that arises because the performance improvement is perceived as an extrinsic motivation for the learning, and might be alleviated if it was perceived as an intrinsic motivation. Users likely have an inaccurate perception of the costs and benefits of switching to the new technique, because they just learned about it and do not yet have a model of its operation that is clear enough to make such evaluations. They can thus easily overestimate the difficulty of learning the technique or underestimate its performance relatively to the “old” technique (which can also happen because she overestimates her current performance with the “old” technique). All three cases have been observed in studies (see section 5.1.2 of [Coc+14] for a review).

Investigating more deeply the question of the performance of a newly discovered technique could help users better evaluate the benefits of this technique in terms of performance (or according to other criteria than performance), but to engage in such an investigation, they must first believe that it is worth the time and effort. As a consequence, users rarely engage in such rigorous evaluations: they prefer faster, less demanding, but also less accurate evaluations of the costs and benefits of switching to the new technique, so that they can take a non-optimal but “satisficing” decision.
**PERFORMANCE DIP** The second reason to not switch to the new technique is a fear of a *performance dip*: users may understand that in the long-term, switching to the new technique may improve their performance, but in the short term, they will have to use the new technique in a way that is less efficient than the level of performance they can reach with the “old” technique. Users may then believe that now is not an appropriate time to learn the technique because they need to maintain their current level of performance. This phenomenon explains the interest of the HCI community for designs like Kurtenbach’s marking menus (see Section 7.3.2) and how their implementation of the principle of *rehearsal* allows to reduce or even cancel completely the difference of performance between the two input methods, without relying on an external motivation of the user for doing so.

**VOCABULARY EXTENSION** Cockburn *et al.'s* notion of *vocabulary extension* concerns improvements of the overall knowledge of an interface in terms of the set of commands it provides. However, despite listing works that provide users with recommendations of commands to learn, and mentioning gamified approaches to learning, Cockburn *et al.* do not mention explicitly the problem of prioritizing learning that I will discuss in Section 7.5.2. Instead, they see vocabulary extension mainly as similar to intermodal extension in that “users need to be aware of commands before using them, and they need to understand the magnitude of improvement that they will enable”.

**TASK MAPPING** Finally, the last of the four domains of interface performance improvement discussed by Cockburn *et al.* is *task mapping*. It “addresses higher level issues of the strategies that users adopt when seeking to complete their tasks with a UI. It concerns the coordination of functions to complete a task” [Coc+14]. In terms of the domains of knowledge and skills, task mapping thus concerns the improvement of user’s knowledge and skills at the task, appropriateness of workflow, or domain object levels (depending on what exactly is meat by “complete a task”), and may or may not be supported by improvements of knowledge and skills at lower levels.

What these three levels have in common in that they involve entities (tasks, workflows, domain knowledge) that are not features of the interface. At such, the knowledge and skills involved at these levels are often user self-defined strategies, and the problem is that coming up with a strategy is a slightly different process than learning about something that pre-exists as an affordance in the interface. Adopting strategies goes through the same steps of 1) getting aware of a possibility, 2) evaluating its benefits and the costs of switching to it, and then 3) doing the switch, that we discussed for intermodal performance improvements. The difference is that, like for signifier-less designs, there is often no design means in the interface to expose a possible strategy
to users. As a consequence, Cockburn *et al.* focus on design means
to expose possible strategies. They also mention another approach
suggested by Bhavnani and John [BJoo], which is to design interfaces
that allow users exploiting efficient strategies that generalize across a
variety of application domains (which would set them as belonging to
the General Knowledge and skills level), such as the Detail-Aggregate-
Manipulate strategy used in the drawing application example used
to analyze the Appropriateness of Workflows level in section 7.2.1.
In addition to raising user awareness of the strategies, this approach
additionally help them evaluating the costs and benefits of adopting
them, as they already have a clear understanding of the strategies.

7.5.2 *The Knowledge-motivation loop*

Because motivation is required to learn, but what one knows affects
one's ability to foresee the consequences of engaging in some learning
– as we have just seen for performance-motivated learning – com-
plex behaviors can emerge from the interaction loop between current
knowledge and skills and motivations (the right part of Figure 7.1). In
this section I address some aspects of this phenomenon.

**COMPETITION FOR LEARNING** A third reason that could be pro-
vided to explain why intermodal performance improvements do not
happen is by considering that a user may have many new techniques
to learn and that these techniques are competing for the user’s learn-
ing efforts (in addition to the non-learning tasks that motivate the
use of the interface in the first place). As a consequence, users may
have to prioritize the things they want to learn about the interface,
and a particular interaction technique may never been learned simply
because learning it never becomes a priority. Since this priority level is
a part of what the user “knows” about the technique, it is also possible
that a particular technique has been categorized early as “advanced,
low-priority stuff”, and that users stick to this classification later as
they do not feel the need to update it.

**RELEVANCE OF THE KNOWLEDGE AND SKILLS** Some knowledge
and skills may only become relevant for users after having understood
how to exploit them at a higher level. For instance, users of an e-mail
application who read each mail individually before deciding to delete
it or reply it (see Section 7.2.1) may dismiss the deletion widgets in
the thread list as not useful, only because they do not see the point of
using them. It is the same thing for people who prefer to format their
text with the bold/italic/underlined buttons rather than using styles:
all the style-related commands would seem useless to them and they
would not try to learn them.
Conversely, a user can discover these features and realize their potential to change the way they think about the activity supported by the application. In this case, the knowledge of these commands become relevant and can motivate users to learn them as well as the higher-level workflow and domain knowledge.

MOTIVATION FOR EXPLORATION Knowledge can also flow from an upper level to a lower one. Typically, a user's knowledge of an activity can lead her to explore the interface looking for a feature that she judges necessary for this activity, even if this feature is actually implemented with a non-signified design. This is typically the case when one looks for the power switch of a device, and other forms of *puzzling signifier-less designs* (see Section 6.3).

On the other hand, when users lack domain knowledge, they might rely on exploration of the interface in hope of acquiring such knowledge, as suggests Kirsh in his analysis of using an unknown image manipulation application to enhance the quality of a figure to be included in a research paper [Kir97] (see Section 4.2.3).

## 7.6 FIRST CONTRIBUTION: DEGREE OF KNOWLEDGE

Computer interfaces have become more and more complex to use nowadays, as some applications are designed to cover a very wide range of activities. As a consequence, on one side there can be many features of an application that are targeting activities that a user may not be interested in and do not need to know, and on the other side, there can be many different ways to interact with the application to achieve a same operation (for instance, to quit the application, users can use icons, hotkeys, or drop-down menu items).

It is then not straightforward to distinguish users as novices or experts of an application, because they may have good knowledge of the interaction possibilities in one specific area of activity but not in another area. For example, Photoshop users who know all the commands to edit photos, with the most efficient ways to trigger them, but know nothing about how to do animations in Photoshop: in some professional contexts they would be called "experts" but not in others.

Knowledge and skills about the interaction possibilities of an interface may change over time and evolve according to users' intentions, motivations, and life goals. But so do users' expectations about what they want to use an application (or system) for. More important than expertise levels, we need a measure of how close the knowledge and skills of a user are from those required to use the application "at its best". Such a measure can however not been defined, because it is too vague: how could we define what is the best way to use an application for specific users, when even them may not have a clear idea about it? But we can define measures of the knowledge and skills of users
<figure><img src="image_46.png" /><figcaption>Figure 7.4: Practical definition of the *degrees of knowledge* in terms of how users can behave relatively to an interaction technique in different situations.</figcaption></figure>

relatively to specific interaction possibilities, and see if there is room for improvement.

I propose *degrees of knowledge* (or DoK) as such a measure. Instead of trivially categorizing interaction techniques as “not known” or “known”, DoKs acknowledge a continuum of knowledge and skills that provide users with different abilities to understand interaction techniques, recognize them, perform them, or include them in higher-level usage strategies and habits.

These interaction possibilities exist in terms of menu, keyboard shortcut, icon buttons, or even mouse clicks, etc. By reviewing the degrees of knowledge at which a user stands for a set of interaction techniques, we may see different patterns of how users' strategy of their interaction vocabulary / current knowledge, it may inform designers about how users deal with the interaction techniques, or about which interaction technique “works” for which users.

I thus propose to define the *degree of knowledge* of a user concerning an *interaction technique* with the following simple ordinal scale:

1. UNKNOWN, when the user is not aware of the existence of the interaction technique, and thus cannot recognize it nor perform it.

2. RECEPTIVE, when the user is aware of the interaction technique and can recognize it if performed in front of her, but cannot use it by herself.
3. PRODUCTIVE, when the user can use the interaction technique by herself, when asked to perform a task that can be accomplished with this interaction technique.

4. PRIMARY, when the user spontaneously prefers using the inter-action technique over others when prompted to perform a task that can be accomplished with this interaction technique, or has built strategies about when to use this technique.

As seen in Figure 7.4, the degrees of knowledge can be identified experimentally by investigating some simple abilities of users – al-though the model is voluntarily simplified and merges some aspects of interaction that could be considered as separate degrees in more complex models: this limitation comes from the constraint of defining a practical measure that can be performed during experiments without spending to much efforts on trying to identify distinctions that may not be relevant.

As they get familiar with an interaction technique, users reach higher degrees of knowledge for this technique, by developing the abilities listed in Figure 7.4 for each degree. Conversely, if they do not use the technique they can forget about it and fall back to lower degrees of knowledge. To ease the discussion about users’ skills, I also define *used* for interaction techniques that can be used by the user independently of her strategies (i.e., either **PRIMARY** or **PRODUCTIVE**); and *known* for interaction techniques that the user can recognize, independently of their ability to perform it (i.e., any DoK but **UN-KNOWN**).

## 7.7 SECOND CONTRIBUTION: SOURCES OF KNOWLEDGE

If designers hide some interaction techniques, they take the risk that users may not be able to find them at all; in this case, knowing the sources of users' knowledge of these techniques may provide in-sights for design improvements. An important concept for discussing signifier-less designs is thus the *sources of knowledge*, or SoK in short.<fnref n="6" /> I propose two dimensions to categorize sources of knowledge: *distance* and *intentionality*, which are not necessarily orthogonal.

### 7.7.1 *Distance of the source of knowledge*

A first dimension to describe the sources of users' knowledge about the interface is how remote they are from the place and time this

<footnote n="6">6 Despite the proximity of the two acronyms DoK and SoK, used here for practical reasons, I invite the reader to pay attention to the fact that Degrees of Knowledge and Sources of Knowledge are very different concepts: the former is a definition of a way to quantify knowledge, while the latter is more a focus of interest that does not translate directly into a single quantification.</footnote>
<figure><img src="image_47.png" /><figcaption>Figure 7.5: Distance between a source of knowledge and the usage of this knowledge in interaction.</figcaption></figure>

knowledge is applied as users interact with the interface. There is an ill-defined but intuitive notion of distance between a source of knowledge and its application, which is depicted in Figure 7.5.

The ideal situation for users when interacting with an application is that they can get aware of its important functions by just looking at the interface, using perceived affordances, signifiers, and metaphors. When users see an icon of a trash bin, they may instantly obtain the knowledge that the function of this icon is to delete or to categorize some data as no longer useful: the interface itself is the source of this knowledge.

This is the shortest possible distance, which is epitomized by SoKs that are classical widgets and direct signifiers providing affordances and feed-forward. In this case, user's knowledge is gained by directly looking at the SoK and it happens at the same place and almost at the same time this knowledge will be used. The distance of the SoK is thus almost null, or very close to users. Sometimes, interaction possibilities are discovered by accidentally triggering them, or by explicitly trying them, which could be considered as the shortest possible distance – although it is unlikely that designers can rely on such SoKs as there is no certitude that users will indeed use such sources.

There are also SoKs that are more remote from the places the knowledge is needed, but still in the application. Examples would be how menus can provide hints for hotkeys, or Microsoft Word's *Clippy*. At the farthest, but still in the application, information boxes such as the *tips of the day* displayed at application's load time can convey some knowledge to users – although it is also usually possible to deactivate them. These sources being provided “on the way” leading to the place where the information they provide is useful (and sometimes, “in the way”), they can be considered as closer than the help pages provided by an application, because the later are only accessed explicitly by
users and require them to interrupt their activity to perform a search of the relevant page, and are less likely to be accessed.

Knowledge of an application can also be communicated outside the application but still inside the device that hosts it. For instance, with Google's Drippler or Apple's Tip, which are applications dedicated to communicating useful knowledge about other applications shipped with the system. This intermediary distance is shown in the middle of Figure 7.5.

The largest distance for a SoK is out of the device itself. A frequent case, for example, is that we learn about an interaction possibility from a colleague during the coffee break, or when searching for video tutorials on the web.<fnref n="7" /> The knowledge may also come from other external sources, such as news, social networking applications and websites, demonstrations in store, chatting between friends, or even from a glance at a stranger using her phone beside you in the subway train.

A simple categorization of distances thus considers whether the SoK is the widget or domain object itself, the application, the operating system / device, or the social environment of the user (mass media or word of mouth).

### 7.7.2 Intentionality of the discovery

There are three different situations / scenarios for users to acquaint knowledge of an interaction possibility and get in contact with the source of this knowledge, depending on the intentionality of the discovery:<fnref n="8" />

*Through explicit research:* users have a precise goal or problem to solve in using the application, and they search for information about it. This search can be performed in the interface itself (targeted exploration, use of help features provided by the application) or from more distant sources (web search, asking a friend, etc.)

*Through random exploration and information:* users may just want to explore the interface, led by curiosity. They may expect to find valuable

<footnote n="7">7 This includes the case where the tutorial video is watched on the device that is concerned by the tutorial: in that case, what matters is that the video, as a SoK, is external to the device, even if it is accessed through the device.</footnote>

<footnote n="8">8 I have described earlier some aspects of this dimension as belonging to the motivational aspects of the model proposed in this chapter, which is true as long as the distinction between design means and motivations rely on a distinction between what belongs to the interface or user. The notion of source of knowledge transcends this distinction, as it is as much a feature of the world than a particular event involving the user. A benefit of moving the aspects of intention out of the domain of motivations is also that it removes the need to discuss non-motivational aspect as a subset of motivational aspects: the notion of motivation becomes more focused in scope.</footnote>
information although they are not targeting a specific interaction need.<fnref n="9" />

*Without intentions:* the knowledge has been communicated to the user but it was not motivated by the user. For instance, knowledge can be *suggested* or *imposed* to users when they are shown a device in a physical or online store before buying it. Knowledge can also be deduced from habitual know-hows acquainted during past interaction with similar devices, systems, or applications. And finally, knowledge can also be discovered by accidentally triggering the function.

### 7.7.3 Sources of Knowledge as an extension of Design Means

The notion of a SoK can be understood as an extension of the concept of *design means* of conveying usage knowledge, such as signifiers, metaphors and constraints. It acknowledges that learning about an interface is an experience that transcends the pragmatical resolution of precise interaction problems, unlike what would suggest a model such as the Seven Stages of Action (see Section 4.2.3). The notion of a SoK also acknowledges that this experience of learning about the interface can be influenced by designers not only by the design of a product, but also by designing *around* the product (e.g., how the product is presented to customers). This understanding of design is already at the heart of design practices such as User Experience (UX) Design, and can be found in scientific approaches that take interaction as a *situated* phenomenon such as Beaudouin-Lafon's call for focusing on "interactions, not interfaces" [Beao4].

Introducing the notion of a SoK has the benefit of highlighting an important concern for designers – providing information about how to use the system – in a way that do not focus on a specific design solution (as would be terms such as signifiers or feedforward) but invite to consider all possible solutions in the design space. It also allows designers to question how their designs support behaviors like accidentally discovering an interaction technique or sharing the experience of such a discovery with relatives.

## 7.8 CONCLUSION

In this chapter, I have proposed a model of interaction technique proficiency improvement based on three types of prerequisites: a user's current knowledge and skills concerning the use of the interface and the activity it supports, a user's motivations and non-motivational

<footnote n="9">⁹ The distinction between explicit research and exploration led by curiosity is not always easy to make. For instance, when a user discovers that swiping leftward an e-mail in the iOS Mail application reveals *swhidgets*, she can wonder if swiping rightward also does something. In that case, the exploration is not random but targeted, yet users are not motivated by solving a specific problem.</footnote>
causes for improving the proficiency, and design means, the features of an interface that can help users improve their proficiency. Current knowledge and skills is the central element of this model, as it includes not only declarative knowledge about the interface and activity but also procedural knowledge about how to act, which affects directly the proficiency in an interaction technique depending on the stage at which the technique has been acquired (cognitive, associative, or autonomous). Design means and motivations can indeed be defined respectively as aspects of the interface and user's mind that are required to improve knowledge and skills.

The model however also reveals a more complex dynamic caused by the interactions between its three components, shown in Figure 7.1: the current knowledge and skills affects the efficiency of the two other components in that 1) knowledge is required to understand some design means but skilled behavior reduces the need for some design means, 2) the strength of motivations, especially extrinsic ones, is affected by users' knowledge that limits users' ability to clearly understand the consequences of their adoption of new usage patterns. As a consequence, complex behaviors emerge from the two dependency loops interconnected through the current knowledge and skills. I described shortly some of these complex behaviors, but the topic is too vast to provide a complete account of them in this dissertation.

In the last two sections, I have presented two additional contributions to the model. The first, the concept of degree of knowledge (DoK) is a measure of a user's proficiency in a technique based on four ordinal categories that attempt to capture the complexity of proficiency dynamics in a simple and low-detail model. This measure has been created to be easily integrated into experiments implying multiple interaction techniques – such as those that I have conducted on *swhidgets* which will be presented in Chapter 8.

Finally, the concept of sources of knowledge (SoK) has been introduced as an answer to the problematic of signifier-less designs, to account for the possibility of their discovery despite their lack of signifiers. As such, it provides a partial account for differences between the learnability of an interface and its actual knowledge by a population of users. It generalizes the notion of design means by acknowledging that the discovery can happen outside the interface at various levels of "distance" from the actual usage of the technique, and at various levels of intentionality. It can also be used in experiments as we will see in Chapter 8.
<figure><img src="image_48.png" /></figure>
<figure><img src="image_49.png" /></figure>
Part IV

STUDIES
<figure><img src="image_50.png" /></figure>
<figure><img src="image_51.png" /></figure>
<figure><img src="image_52.png" /></figure>
The state of the art of signifier-less designs that I presented in Chap-
ter 6 and the analysis of *swhidgets* that I presented in Chapter 2 revealed
that little is known regarding whether users rely on these controls
or not, and how they discover the *swhidgets*. In that purpose, we
conducted a first controlled experiment to observe which modalities
28 participants – all iPhone users – used when prompted to perform
a set of operations, and had them filled out 2 questionnaires to learn
the sources of their knowledge of *swhidgets*. Eager to know if this
result can apply on general *swhidgets* usage, we tried to reach more
participant by putting an online study with more detailed questions
on how they feel about using *swhidgets*. The results of this online study
provide even more interesting data about participants' appreciation of
*swhidgets*.

In this chapter, I will present the rationale and objectives of the
two experiments, detail their designs, analyze their results, and I will
finally discuss and compare the results from these two experiments.<fnref n="1" />

8.1 THE LABORATORY AND ONLINE SURVEYS

8.1.1 *Rationale and objectives*

In order to analyze the “means” and “motivations” allowing users to
use *swhidgets* to achieve tasks, it is necessary to find first how much
users know about *swhidgets*. Therefore, my first goal in designing an
experiment was to know what users currently do to solve their daily
tasks with applications that contain *swhidgets*.

The first experiment I have conducted thus aims at providing an-
swers to the research questions on *swhidgets* described in Section 2.7
by observing the participants and interacting directly with them. It
took the form of a laboratory study in order to better understand how
users interact with *swhidgets*. This study aims at investigating which
*swhidgets* users are aware of and whether they would spontaneously
use them when prompted to perform operations that can be completed
by using *swhidgets*. The goals of this experiment are more specifically:

1. to observe if participants know about *swhidgets*;

2. to find if participants discover *swhidgets* by themselves or more often by another source;

<footnote n="1">¹ This chapter has been partially published as a paper for the ISS’19 conference [PM19].</footnote>
3. to have an idea of the possible sources of knowledge from which participants generally develop their interaction vocabulary;

4. to see if participants know *swhidgets* better when they are "tech-friendly" persons;

5. to know if participants think that *swhidgets* make their life easier/more convenient;

6. to learn what participants expect from *swhidgets*.

So the research questions that I try to answer with the laboratory study were first operationalized with the following questions:

1. How many interaction techniques do users know to achieve a task?

2. What is the order/strategy when users know more than one way?

3. Where do users learn their interaction techniques?

4. How many users know about *swhidgets*?

5. Do *swhidgets* remain the favorite way for users when they know about it?

6. Do users think these *swhidgets* improve their efficiency in smart phone tasks?

After the first laboratory experiment, I had collected data that drove me to plan the second experiment online. As an online questionnaire is a better way to reach more participants, the goals of this second experiment were to:

1. have more diverse professions of participants (rather than separate them into IT and non-IT group);

2. know more precisely how the usage frequency of an application or interaction technique affects the adoption of *swhidgets*;

3. know the exact source of knowledge if participants are aware of a *swhidget*;

4. know if participants find *swhidgets* difficult to discover;

5. know if participants are willing to adopt *swhidgets* if they don't know about them;

6. know if participants will rank *swhidget* as their favorite way to achieve the task;

7. know if participants build a logic of finding *swhidgets* in the application or in other applications;
8. know if participants are capable of transferring the knowledge of using *swhidgets*;

The expected insights of these surveys belong to different categories:

* Users' original interaction vocabularies of the system.

* From where they learn these interaction vocabularies.

* Do users keep using these *swhidgets* after having acquired this knowledge.

### 8.1.2 Scope limitation

These surveys have a limited scope and focus on the iOS operating system and its most used default applications. I introduced this scope limitation to:

1. Maximize the odds that participants had a chance to use and know the *swhidgets* in these applications before the study, so that the study indeed evaluates participants' current knowledge rather than their ability to learn the interface during the experiment.

2. Ensure the set of *swhidgets* studied is coherent and stable across application versions and device models, so that this evaluation of participants' knowledge applies to a large population of iPhone users and is not biased by participants' experience with a specific device.<fnref n="2" />

3. Maximize the coherence of the set of *swhidgets* studied: Since I am interested in the possibility of knowledge transfers across applications, I need to consider applications that follow the same guidelines and share a common logic, which is more likely with applications from a same developer.

I acknowledge that *Swhidgets* also exist in other touch-based operating systems such as Android and Windows Mobile, but I believe that iOS offers a consistent interaction environment with a variety of hidden controls in it that makes it well adapted for first studies on that topic.

<footnote n="2">² The laboratory study ran from December 2017 to January 2018, and the online study ran in May-June 2019. The iPhone X was released between the two studies and introduced important changes in the iOS interface: it removed the physical main button below the screen, added a physical separation between the left and right halves of the top screen bezel, and changed the activation gesture for the two system *swhidgets*, as the Notification Center and Control Panel that were previously mapped respectively to the top and bottom of the screen became mapped respectively to the left and right parts of the top bezel. The action of pressing the physical button was replaced with a swipe across the bottom bezel, a gesture previously mapped to the Control Panel, which can be seen as a third system *swhidget* for the iPhone X.</footnote>
### 8.1.3 Applications and operations studied

The two studies involve the same four “applications” from which 14 operations have been defined in total (see Table 8.1). Since each study uses a different subset of operations but some of them are common to both studies, I describe the four applications and all their respective operations in the next paragraphs, independently of the study in which they are used.

**CONTROL PANEL OPERATIONS** Although not technically an application, I group four operations in this category, as they all use the system-wide feature that is the Control Panel. The first operation, *revealCP*, allows to assess if participants know how to reveal the Control Panel by doing a upward swipe across the bottom bezel of the screen. By asking the participants to complete the three other operations, we can assess their knowledge of the features provided by the Control Panel. These operations are: turning on/off the Wi-Fi, turning on/off the flash light, and (un)locking the screen orientation in portrait mode. For *wifi* and *flashlight*, participants can also use a *Voice control* by asking Siri to complete the operation; and for *wifi*, they can also use *Navigation Controls* in the “Settings” application.

**NOTIFICATION CENTER OPERATIONS** Although not technically an application, I group three operations in this category, as they all use the system-wide feature that is the Notification Center. The first operation, *revealNC*, allows to assess if participants know how to reveal the Notification Center by doing a downward swipe across the top bezel of the screen. By asking the participants to complete the two other operations, we can assess their knowledge of the features provided by the Notification Center. These operations were checking the current weather and responding to missed notifications. The latter can be performed explicitly by touching the notifications listed in the Notification Center, or implicitly by opening the application associated to the notification and using *Navigation Controls* in this application to access the content notified. Participants could get the weather information with *Swhidgets* by reading it in the Information Panel, with *Navigation Controls* by opening the Weather application, or with *Voice Controls* by asking Siri. The Information Panel is a sub-panel of the Notification Center that can be accessed with a rightward swipe in the Notification Center, on the lock screen, or on the main panel of the Home Screen.<fnref n="3" />

<footnote n="3">³ Technically, this rightward swipe is not a *swhidget* but rather a gesture to navigate between panels. There is indeed a very subtle signifier for this gesture at the bottom of the screen, in the form of small dots – one for each panel, the current one being brighter.</footnote>
<table><caption>Table 8.1: List of the possible input methods to complete the operations that participants were asked to perform during the experiments. Depending on the operation, input methods are either a *Swhidget*, a *Navigation control*, or a *Voice control* (Ø indicates an input method that is not available). Tasks in blue and orange were respectively used in the laboratory study and online study.</caption><thead><tr><th rowspan="2">APPLICATIONS</th><th rowspan="2">OPERATIONS</th><th colspan="2">STUDIES</th><th colspan="3">INPUT METHODS</th><th rowspan="2">INSTRUCTIONS</th></tr><tr><th>lab</th><th>web</th><th>Swhidgets</th><th>Navigation Controls</th><th>Voice Controls</th></tr></thead><tbody><tr><td rowspan="4">Control Panel</td><td>revealCP</td><td></td><td></td><td>system</td><td>Ø</td><td></td><td>Do you know how to reveal the Control Panel on your touch-based device?</td></tr><tr><td>flashlight</td><td></td><td></td><td>system</td><td>Ø</td><td>✓</td><td>Do you know how to turn on/off the Flashlight on your iOS touch-based device?</td></tr><tr><td>lockPortrait</td><td></td><td></td><td>system</td><td>Ø</td><td></td><td>Do you know how to lock the orientation of the screen on your touch-based device?</td></tr><tr><td>wifi</td><td></td><td></td><td>system</td><td>Settings application</td><td>✓</td><td>(lab) Can you turn off the Wi-Fi?<br>(web) Do you know how to enable/disable Wifi on your iOS touch-based device?</td></tr><tr><td rowspan="4">Notification Center</td><td>revealNG</td><td></td><td></td><td>system</td><td>Ø</td><td></td><td>Do you know how to reveal the Notification Center on your touch-based device?</td></tr><tr><td>weather</td><td></td><td></td><td>system</td><td>Weather application</td><td>✓</td><td>Can you tell me what is the current temperature outside?</td></tr><tr><td>notification</td><td></td><td></td><td>system</td><td>directly open apps</td><td>✓</td><td>Do you know how to respond to the notifications that you missed on your touch-based device?</td></tr><tr><td>SMS.delete</td><td></td><td></td><td>item</td><td>edit mode (long press)</td><td></td><td>(lab) Can you delete this message, please?<br>(web) Do you know how to delete a conversation from a specific person in the Message application?</td></tr><tr><td rowspan="3">Messages</td><td>SMS.search</td><td></td><td></td><td>view</td><td>scroll to find</td><td>✓</td><td>(lab) Can you search the SMS chat with John?<br>(web) Do you know how to search messages from a specific person in the Message application?</td></tr><tr><td>SMS.time</td><td></td><td></td><td>view</td><td>read time</td><td></td><td>(lab) Can you tell me at what time this message was received?<br>(web) Do you know how to see the timing of messages in a conversation in the Message application?</td></tr><tr><td>Mail.delete</td><td></td><td></td><td>item (+long)</td><td>edit mode (edit button)</td><td></td><td>(lab) Can you delete this e-mail?<br>(web) Do you know how to delete e-mails in iOS Mail?</td></tr><tr><td rowspan="4">Mail</td><td>Mail.search</td><td></td><td></td><td>view</td><td>scroll to find</td><td>✓</td><td>(lab) Can you search the e-mails from John?<br>(web) Do you know how to search an e-mail on your touch-based device?</td></tr><tr><td>Mail.markAs</td><td></td><td></td><td>item (+long)</td><td>flag button</td><td></td><td>(lab) Can you mark this e-mail as read?<br>(web) Do you know how to mark an e-mail as read/unread in iOS Mail?</td></tr><tr><td>Mail</td><td></td><td></td><td>item</td><td>reply button</td><td></td><td>Can you reply this e-mail?</td></tr></tbody></table>
**MESSAGES OPERATIONS** Three operations concern the Messages application, shipped with iOS to provide access to many messaging services, most notably SMS.

The first operation consists in deleting the entire conversation with someone. I included this operation in the studies because it represents a risk of data loss and its design might thus have benefited from special attention, and also because it is likely that many users never or rarely perform it. The operation can be done with an *item swhidget* by swiping leftward on the appropriate item in the conversation list to reveal buttons, then pressing the Delete button and finally confirming the deletion. Participants can also use *Navigation Controls* by performing a long press on any message in the conversation view, which displays a menu, then selecting the “More…” option in this menu to enter the edition mode, and finally press the "Delete All" button on the top-left of the screen.<fnref n="4" />

The second operation consists in searching for the messages sent by a person identified by her name. This is an operation that presents no risk and might be performed more or less frequently depending on the users' chat habits and number of contacts. Participants can use a *view swhidget* by performing a downward swipe to reveal a search bar at the top of the discussion list, they can use *Navigation Controls* by scrolling up or down the conversation list until they find the one searched, or they can use *Voice Controls* by asking Siri.

The third operation consists in looking for the time a specific message was sent. Participants could only get the exact information by using a *view swhidget*, swiping leftward from anywhere in the conversation view in order to reveal the information. Alternatively, they could get an approximate timing information with *Navigation Controls* by scrolling the conversation until they find the nearest time information displayed in the view between messages. Such timestamps are inserted in the discussion above a message only when the previous message was exchanged more than one hour earlier.

**MAIL OPERATIONS** Four operations concern the Mail application provided with iOS as the default application to read and send emails. I have included the deletion and search of mails so that comparisons between the two applications can be performed. I also included the two operations consisting in marking e-mails as read and replying to emails, as they use variants of a similar *swhidget* and differ by their frequency of use and users' way of managing e-mails.

Deleting e-mails can be performed with *Swhidgets* by performing a short leftward swipe on a mail in the thread list, then selecting the Delete button that it revealed. Interestingly, an “expert” version of this

<footnote n="4">4 There is actually a third way to do it, which is to press the “Edit” button on the top left of the conversation list, then select the conversation(s) to delete, and finally press the “Delete” button on the bottom-right corner of the screen. But nobody used it in the lab study.</footnote>
method exists, consisting in performing a longer swipe, which will
directly delete the e-mail. Participants could also use two Navigation
Controls, as they could press the trashcan icons in the mail view or
thread view, or they could enter the edition mode in the thread list by
pressing the Edit button in the top-right corner, selecting the threads
to delete, and finally press the Delete button in the bottom-right corner
of the screen.

Searching emails can be performed with *Swhidgets*, as it is again
possible to use the search bar that hides under the upper navigation
bar, and that participant can reveal with a downward swipe in the
thread list. It can also be performed with *Navigation Controls* by
rolling the thread list and opening threads until the searched one is
found, or with *Voice Controls* by asking Siri.

Marking e-mails as (un)read can be done with a *Swhidget* by swiping
rightward on a thread (in the thread list) or on a mail (in the thread
view) to reveal an unique button representing a closed or open mail.
Like for deleting mails, an expert mode exists for this operation with a
longer swipe. Alternatively, participants could use *Navigation Controls*,
by directly opening a thread view and scrolling – which marks mails
as read as they appear into view, or by pressing the flag icon in the
bottom-left of the thread view to open a menu containing a “Mark as
Unread” option.

Finally, replying an email could be performed with *Swhidgets* by
performing a short leftward swipe on a mail in the thread view (but
not in the thread list), to reveal a blue button that give access to a
menu containing a "Reply" option. Here, the alternative *Navigation
Controls* seem more practical, as there is a leftward arrow icon at the
bottom of the thread view and mail view that can be touched to reply
to the mail (or whole thread).

8.2 LABORATORY STUDY

The laboratory study focuses on direct observation of and interaction
with 28 participants, including notably phases in which we observe
how they interact with a device in response to a request, and debriefing
interviews.

8.2.1 *Factorial design of the experimental tasks*

The study follows a within-subject factorial design with two indepen-
dent variables and two dependent variables. The two independent
variables are the *input method* (three levels: *Swhidgets*, *navigation controls*, and *voice controls*) and the operation (9 levels, see Table 8.1).
The two dependent variables are *primary-or-uncompleted* and *degree of knowledge* (their levels will be described in Section 8.2.6). The factorial
design is complete if we ignore the voice controls: each participant
faces exactly one *swhidget* and one *navigation control* for each level of
*operation*, but also one *voice control* in some operations.

8.2.2 *Experimental tasks*

Participants are invited to accomplish different “operations” designed
to test their knowledge of four “applications” (see Table 8.1): the
control panel, the information panel, and hidden commands in the
text messaging and email applications. All operations can be achieved
with various *swhidgets* triggered by swipes, but other interaction possi-
bilities are also available for the participants. For instance, participants
can invoke Siri or use visible buttons in the navigation bar, or 3D-
touch some interface elements. The four applications are divided in
operations, which will be described in the next paragraphs.

By running through the operations of all four applications, we
intend to evaluate the proportion of *swhidgets* used –or known but
not used– by each participant, as well as to evaluate the proportion of
users who use –or know but do not use– a specific control. To control
learning and fatigue effects, the four applications are presented to
participants in one of four possible orders according to a Latin square
matrix.

8.2.3 *Participants*

Twenty-eight persons (sixteen males and twelve females) have partici-
pated in the laboratory survey. The average age is 39.5 year-old. As
we focused on iOS *swhidgets* in this study, we only recruited iPhone
users. Eighteen participants come from the *Inria Lille - Nord Europe
research center* and from the University of Lille, although only 13 had a
technical background. Elder persons and youngsters were recruited
among some of our participants' relatives, although we could not
recruit enough youngsters and did not use their data in the results
reported here. A binary AGEGROUP factor was created, distinguishing
five participants aged 45 or more. We also considered a binary COM-
PUTERJOB factor distinguishing the 13 participants who work in the
IT industry or academic research at positions that require a technical
background, from those who are not.

8.2.4 *Material and Apparatus*

The survey is equipped with one consent form, two questionnaires, one
camera on tripod, pens to fill out the forms. We also provide an iPhone
6S running iOS 11, with a fictive account for all participants so that
we exclude the risk of possible invasion concerning personal privacy
in participants' own phone. To participate in this survey, participants
are informed that the interaction section may be videotaped, if they
agree, only with their hand movement when they interact with the
iPhone. They have, therefore, an agreement to sign before beginning
this survey. We set up a camera when the participants are ready
to interact with the iPhone. The study obtained an agreement from
Inria's Ethic Committee and Data Protection Officer.

8.2.5 Protocol

**PROCEDURE** Participants were invited to our team building in a dedicated room. As the main goal of the study was to assess usage of *swhidgets*, I only specified that the goal of the study was to observe interaction on smartphones without further details, to avoid making participants take the participation as a competition. Participants first read and signed a consent form, and answered various questions regarding their demographics and usage of touch-based devices. Then, they were presented with an iPhone in its default configuration and populated with artificial text messages and e-mails, before moving to the main phase of the experiment.

After the first questionnaire is filled out, we set up the camera and
let the participant interact with the iPhone provided by Inria with
fictive information such as mails and text messages. In the interest
of collecting all the interactions that participants can do during the
study, we also assure our participants that there is no pressure to be
competitive, we may continuously ask if they know other possible
way to do the task even if there are no other methods left. The four
applications listed in Table 8.1 were administrated in a Latin Square
order. After finishing all operations, participants are asked to fill out
the second questionnaire. After the second questionnaire is filled out,
we start debriefing all the interaction techniques that participants *did*
*not* use in the study. The detail of important elements in this study
are listed below:

**FIRST QUESTIONNAIRE** In the first section, participants are asked to fill out the first questionnaire (provided in Appendix J), where we collect their age, professional and personal background. There are also questions aiming at knowing how the participants interact with their iOS system in different activities such as to play games, or to listen to music. There are also questions asking participants to estimate their knowledge of different smartphone operating systems on a Likert-type scale. At the end of this questionnaire, we also ask participants if they are interested in new technologies and to evaluate if they think themselves as a person who is highly interested in new technology and if they recognize themselves in eleven description with five-level Likert-type scales.
**OPERATIONS SECTION** is the moment I start to videotape participants and to observe their interaction vocabulary while dealing with given operations. Operations are divided into four different applications: Control Panel, Notification Panel, SMS and Mail, (see Table 8.1). The order of applications was counterbalanced across participants using a full Latin square, but the order of *Operations* in an application was the same for all participants. Participants were prompted to perform all the operations, which were grouped into *Applications*. For each operation, I set the smartphone to the home screen and asked the participant *Could you please perform <operation>?* (for instance, *Could you please enable/disable WiFi?*). If participants did not know how to perform the operation, they were invited to reply *I do not know how to do it* and the operation was marked as *uncompleted*. If the participant was able to perform the operation, I logged which input method she used as *Primary*, and then reset the smartphone to the home screen and asked the participant *Do you know if this operation can be performed with an alternative method? If yes, could you please do it?* This could result in two new situations: either the participant knew an alternative and did it, or the participant replied *I do not know any alternative*. If participants knew an alternative method, I logged which alternative input method had been used as *Productive*, and then prompted participants again for alternative methods until the participants replied that they do not know any alternative, in which case the experiment moved to the next *operation*. Once the participant had completed all operations, I would pass to the second questionnaire.

**THE SECOND QUESTIONNAIRE** is provided in Appendix K. It was given after participants interacted with the iPhone. It aims at knowing firstly if participants think that these *swhidgets* help them in their daily tasks, then, to know the sources of knowledge from which participants learned about *swhidgets*.

**DEBRIEFING SECTION** Finally, the experiment concluded with a directed interview. In this section, I present all the input methods that the participant did not use during the experiment during the debriefing section, each time asking if the participant was aware of the method. An open discussion regarding the experiment ended the session, in which I investigate participants general feelings about *swhidgets*, if there are particular gestures that the participants suggest to trigger these *swhidgets*, and if participants have other expectations about what these *swhidgets* could do.

### 8.2.6 Data Collection

During the survey, participants could fail or succeed to complete an operation, and a unique **PRIMARY** input method was recorded for
this operation only in case of success. We have thus computed for each *Operation* and participant a *primary-or-uncompleted* categorical dependent variable with four levels: *Swhidget*, *Navigation*, *Voice*, and *Uncompleted*. The within-subjects categorical independent variable is *Operation* (9 levels, see Table 8.1), and the factorial design is complete.

An ordinal *DoK* dependent variable was defined with values collected for each participant and interaction method (i. e., for each *Participant* × *Operation* × *Input method* triple), with four levels defined as follow (in decreasing order): The degree *UNKNOWN* was given when a participant had no idea that the interaction method could be used to accomplish the operation. The degree *RECEPTIVE* was given when the participant did not think about using the interaction method by herself, but acknowledged she knew about the method when asked after the operation. The degree *USED* was given when the participant effectively used the interaction method spontaneously or when she used it after the examiner asked her if she knew more ways to accomplish the operation. In addition, we recorded which interaction technique was used first to accomplish the operation (if any), providing data about the *PRIMARY* degree of knowledge. This last degree has however to be treated with care, since at most one interaction technique can have this degree for each operation-participant pair, and the observations for the different interaction techniques would thus not be independent. We thus exclude data relating to the *PRIMARY* degree when we want to compare interaction methods in a same operation, falling back to a three-level *DoK* ordinal dependent variable. We however use *PRIMARY* as a fourth ordinal level when comparing input methods across operations.

### 8.2.7 Data processing

Concerning the degrees of knowledge as ordinal data: I followed the advice of Kaptein et al. [KNM10], and analyzed these data with Brunner and Puri's non-parametric tests for factorial plans [BP01] provided by the R package *nparLD* [Nog+12]. I report p-values computed with the ANOVA-type statistic, which is more appropriate for our number of participants (see [BP01]).

Concerning the *primary-or-uncompleted* categorical dependent variable, I analyze the dataset with two Cochran's Q test, one considering if the participants completed or failed each operation, and the other considering if the participants used *swhidgets* as *PRIMARY* or not. Since both tests are performed on the same dataset, I report the Bonferroni adjustments of the p-values.
Participants' usage of smartphones

<figure><img src="image_53.png" /><figcaption>Figure 8.1: Proportion of the 28 participants of the laboratory study who declared to use their smartphone for various activities (See Appendix J for the corresponding questions).</figcaption></figure>

## 8.3 RESULTS OF THE LABORATORY STUDY

### 8.3.1 Participants

**EXPERIENCE WITH SMARTPHONES AND IOS** Participants were relatively experienced in interacting with a touch-based devices: all 28 participants owned an iPhone for at least 3 months.

**BACKGROUND** Participants had various backgrounds although 13 had a professional activity in IT.

**ACTIVITIES** Figure 8.1 shows the activities that participants reported to practice with their smartphones. Four activities (text messaging, web browsing, taking photos/videos, and receiving calls) were reported to be practiced frequently by all the participants. Eight other activities (among which: checking mails, chatting, playing games and taking notes) were reported to be practiced frequently by at least 19 participants (67.9%). As expected, most participants use their iPhones for text messaging and checking e-mails, which are two activities implying operations included in the survey.

### 8.3.2 Completion of Operations

**GENERAL OBSERVATIONS** Average operation completion rate was 86.9% (33 operations uncompleted over the 252=9 operations × 28
<figure><img src="image_54.png" /><figcaption>Figure 8.2: Distribution of participants by the number of operations in which they used *Navigation* or *Swhidgets* as PRIMARY, or that they did not complete.</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>Figure 8.3: Distribution of primary answer across participants globally (left) and for each operation (right).</figcaption></figure>

participants). However, the distribution of participants by the number of operations they did not complete, shown in gray in Figure 8.2, reveals that it was common for a participant to not be able to complete at least one operation. Indeed, only 8 out of 28 participants (28.6%) were able to complete all the operations. On the other hand of the spectrum, two participants (7.1%) could not complete as many as 4 of the 9 operations (44.4%). On average, each participant ended up with 1.18 operations uncompleted (13.1% of the 9 operations).

**EFFECTS OF OPERATION ON COMPLETION** *Operation* had a statistically significant effect on *Uncompleted* frequency ($Q=40.5, df=8, p<10^{-5}$), as could be guessed from Figure 8.3. Post hoc pairwise comparisons between operations with Wilcoxon signed-rank tests revealed statistically significant differences in operation completion rate only between SMS.FindTime and each of mail.delete, mail, weather.consult, and wifi.set. Three operations were completed by all participants: deleting mails, consulting the weather and setting the wifi. The least often completed operation was finding the time of an
<table><caption>Table 8.2: Distribution of primary usages by input method.</caption><thead><tr><th>Input method</th><th>Number of instances</th><th>Number of possible uses</th><th>% of completed operations</th><th>% of possible uses</th></tr></thead><tbody><tr><td><i>Swhidgets</i></td><td>136</td><td>252</td><td>62.1%</td><td>54.0%</td></tr><tr><td><i>Navigation</i></td><td>80</td><td>252</td><td>36.5%</td><td>31.7%</td></tr><tr><td><i>Voice</i></td><td>3</td><td>84</td><td>1.4%</td><td>3.6%</td></tr></tbody></table>

SMS (uncompleted by 42.9% of participants), followed by marking an e-mail as (un)read (25.0%).

### 8.3.3 PRIMARY usage of input methods

**GENERAL OBSERVATIONS** Table 8.2 gives the distribution of primary usages by input method and shows that *swhidgets* were globally the most frequent PRIMARY input method, with 136 of the 219 completed trials (62.1% of completed operations, 54.0% of the 252 possible usages).

The distribution of participants by the number of tasks they completed with a *swhidget* as PRIMARY, shown in blue in Figure 8.2, is rather wide and balanced. This suggests that knowledge of (or preference for) *swhidgets* was quite uneven among the participants.

**EFFECTS OF OPERATION** Figure 8.3 shows the distribution of PRIMARY input methods by operation. *Operation* had a statistically significant effect on the rate of PRIMARY usage of *swhidgets* over other input methods or non-completion of the operation ($Q=37.7, df=8, p<10^{-4}$). Post hoc pairwise comparisons between operations with Wilcoxon signed-rank tests revealed statistically significant differences only between 5 pairs of operations, which are shown in Figure 8.3.

### 8.3.4 Degree of knowledge of Swhidgets vs. Navigation

**DATA PROCESSING AND ANALYSIS** To investigate the difference in participants' knowledge of *swhidgets* and navigation controls, we perform a statistical analysis with two categorical within-subjects independent variables and one ordinal dependent variable in a complete factorial design. The independent variables are *Operation* (9 levels, see Table 8.1) and *Input method* (2 levels: *Swhidgets* and *Navigation*, since we are not interested in *Voice*, here). The ordinal dependent variable is the *degree of knowledge*, with only three levels (*PRODUCTIVE*, *RECEPTIVE*, and *UNKNOWN*) since we treat PRIMARY as simply PRODUCTIVE to ensure the hypothesis of independence between the two levels of
<figure><img src="image_56.png" /><figcaption>Figure 8.4: Degree of knowledge of *swhidgets* by operation, with an additional level to differentiate the use of a *swhidget* as the interaction method spontaneously used by the participant from a later use.</figcaption></figure>

*Input method* (for a participant and task, there can only be one input method labeled PRIMARY).

Following Kaptein et al. recommendation to the HCI community [KNM10], statistical analysis was performed with Brunner' and Puri's non-parametric test LD.F2 from the *nparLD* R package [BP01; Nog+12]. We report test results for the ANOVA-type statistic *ATS*<sub>df</sub>, which can be approximated by a *F*<sub>df,∞</sub> distribution for small and moderate sample sizes [BP01, p. 36].

**FACTORS AFFECTING OVERALL DEGREE OF KNOWLEDGE** Figure 8.4 shows the degrees of knowledge reached by participants for *Swhidgets* in each *Operation*. Statistically significant effects of *Operation* on *degree-of-knowledge* were observed (*ATS*<sub>5.89</sub>=7.87, *p*<10<sup>-7</sup>), but no effect of *Input method* could be found (*ATS*<sub>1</sub>=1.73, *p*=.18). However, interaction between these two factors was statistically significant (*ATS*<sub>6.13</sub>=9.443, *p*<10<sup>-9</sup>), suggesting that *Input method* had different effect directions depending on the *Operation*. Post-hoc tests conducted with the LD.F1 test and Holm-Bonferroni adjustments indeed revealed a statistically significant effect of *Input method* for *SMS.findTime* (*ATS*<sub>1</sub>=18.5, *p*<10<sup>-4</sup>, *p*<10<sup>-3</sup>), *SMS.delete* (*ATS*<sub>1</sub>=18.4, *p*<10<sup>-4</sup>, *p*<10<sup>-3</sup>), *mail* (*ATS*<sub>1</sub>=16.1, *p*<10<sup>-4</sup>, *p*<10<sup>-3</sup>), and *weather.consult* (*ATS*<sub>1</sub>=16.1, *p*<10<sup>-4</sup>, *p*<10<sup>-3</sup>). Direction of effect was different in these four operations, with a better knowledge of *swhidgets* for *SMS.findTime* and *SMS.delete*, but the opposite for *mail reply* and *weather.consult*.

**DEMOGRAPHIC FACTORS** We could not observe any statistically significant effect of *TASKORDER*, *GENDER*, *AGEGROUP*, or – more surprisingly – *COMPUTERJOB* on degrees of knowledge in general (see Section 8.2.3 for the definition of these groups). We could neither
<figure><img src="image_57.png" /><figcaption>Figure 8.5: Distribution of levels of usage by input method.</figcaption></figure>

find a summary statistics of participant's interest in new technologies (based on the questions from the first questionnaire, see Appendix J) that would correlate with participants' degrees of knowledge.

### 8.3.5 PRODUCTIVE *knowledge*

**GLOBAL PRODUCTIVE KNOWLEDGE** I define as a *level of usage* the proportion of input methods that were *used* by a participant during the experiment, out of the available input methods of a given type. On average, the *global* level of usage (proportion of input methods *used* out of the 21 available) of our participants was of 42.5% (sd=28.8), which means that they used less than half of all the available input methods.

**METHODOLOGY** I investigate the effects of *input method* and *operation* on participants' ability to effectively use an input method to perform an operation, by merging the *RECEPTIVE* and *UNKNOWN* levels of the dependent variable *degree of knowledge* into a single *unused*. When performing statistical tests for this analysis, I ignore the *Voice* input methods – since they were not available in all operations – and thus consider only exactly one *Swhidden* and one *Navigation* input method by *operation*.

**EFFECTS OF INPUT METHOD** The level of usage was of 56.0% (sd=24.7%) for *Swhidgets*, 40.9% (sd=16.9%) for *Navigation* and 7.1% (sd=18.9%) for *Voice* (Figure 8.5). Figure 8.6 shows the distribution of productive knowledge across participants and input methods, suggesting that usage of *swhidgets* (median=5.6 operations) is better known than usage of navigation controls (median=3.8 operations). Statistically significant effects were observed for *Input method* ($\text{ATS}_1$=4.70,
<figure><img src="image_58.png" /><figcaption>Figure 8.6: Distribution of participants by the number of operations in which they used *swhidgets* and navigation controls (USED).</figcaption></figure>

$p=.030$) in direction of a better productive knowledge of *swhidgets* than for navigation controls.

### EFFECTS OF OPERATION
Statistically significant effect of Operation ($ATS_{6.19}=5.44, p<10^{-5}$) and of the interaction Operation × Input method ($ATS_{6.16}=10.42, p<10^{-11}$) were also observed. Post hoc tests confirmed the effect direction and statistical significance of Input method for the four operations identified in section 8.3.4, and also revealed an effect toward *Swhidgets* for SMS.search ($ATS_1=8.24, p=.0041, p^*=.021$).

This result suggests that participants tend to use in priority *swhidgets* (assuming they know them) over navigation controls, which is not so surprising considering that *swhidgets* often acts as shortcuts (e.g. for Wifi or for deleting an e-mail) or provide a more detailed information (typically for finding the time a sms message was sent at). Moreover, 22 of the 28 participants (78,6%) also reported during the post-experiment interview that *swhidgets* improve their everyday interaction, which can also explain why participants preferably use *swhidgets* when they know about them. While the priority of usage of Voice control is extremely low, it is important to note that this result is artificially amplified by our experimental design since only 3 operations could be performed using voice control. Therefore, it was virtually impossible for a participant to systematically use voice control as Primary, except if that participant completed only the 3 tasks where voice control was available. That being said, we must stress that 26 of our 28 participants never used voice control as primary.

### 8.3.6 Awareness and RECEPTIVE knowledge

### METHODOLOGY
I investigate the effects of *Input method* and *Operation* on participants' awareness of the existence of an input method to
<figure><img src="image_59.png" /><figcaption>Figure 8.7: Distribution of participants by the number of operations in which they were aware they could use *swhidgets* and navigation controls (KNOWN).</figcaption></figure>

perform an operation, by merging the PRODUCTIVE and RECEPTIVE levels of the dependent variable *degree of knowledge* into a single KNOWN.

**GLOBAL RECEPTIVE KNOWLEDGE** On average, 20.6% of all input methods were classified as RECEPTIVE by our participants (121 out of the 28 × 21 = 588 input methods). In other words, there were around one fifth of the input methods demonstrated to them that participants could recognize but would not think about using. Together with the 42.5% of input methods classified as USED, it means that participants were aware of 63.1% of all the input methods considered in this study.

**EFFECTS OF INPUT METHOD** Input methods classified as RECEPTIVE make 11.1% of all *Swhidgets*, 29.8% of all *Navigation* controls, and 21.4% of all *Voice* controls. However, these differences seem to disappear at the KNOWN level: Figure 8.7 shows the distribution of participants by the number of *Swhidgets* and *Navigation* input methods they are aware of, suggesting that awareness of *swhidgets* (median=6.50 operations) is similar to the awareness of *Navigation* controls (median=6.07 operations). Indeed, no statistically significant effect of *Input method* on KNOWN could be found ($ATS_1=.70, p=.40$)

**EFFECTS OF OPERATION** Statistically significant effect of *Operation* ($ATS_{5.54}=8.02, p<10^{-7}$) and of the interaction *Operation* × *Input method* ($ATS_{6.07}=6.13, p<10^{-5}$) were observed using the methodology described above. Post hoc tests confirmed the effect direction and statistical significance of *Input method* found in section 8.3.4 for mail, SMS.findTime, and weather.consult.
<figure><img src="image_60.png" /><figcaption>Figure 8.8: Distribution of Degrees of Knowledge by interaction method, sorted by decreasing proportion of primary usage.</figcaption></figure>

### 8.3.7 Preference for *swhidgets*

When participants knew how to use a *swhidget* to complete an operation (i.e., at the USED level), they generally used it first (i.e., at the PRIMARY level). Indeed, the ratio of PRIMARY instances over USED instances is 96.5% (136/141) for *Swhidgets*, 77.7% (80/103) for *Navigation* controls, and 50% (3/6) for *Voice* – although there is not enough data in this last case to draw a conclusion. Figure 8.8 suggests that this preference for *swhidgets* holds for all three types of *swhidgets*, but do not hold for any of the other input methods.

### 8.3.8 How participants discovered *swhidgets*

Fifteen participants reported that at least one of the *swhidgets* they used during the experiment had been discovered while consulting an on-line resource (either a Website, a Blog or a video tutorial). Participants also reported to heavily rely on their social network to discover *swhidgets*, as 23 of our 28 participants reported to have discovered at least one *swhidget* following a demonstration by a friend or colleague. Finally, the main knowledge source that 27 of our 28 participants reported to rely on is through self-discovery while interacting with the device, either by accident, or by trying to apply gestures that they know from a different touch-based operating system (typically Android in this case). Figure 8.9 shows for each possible source of knowledge the proportion of participants who acknowledged that they used such a source, with 95%-level confidence intervals.

It is important to note that these results report only *how* participants discovered at least one *swhidget* they used in the experiment, but this must still be put in perspective with our participants' use of *swhidgets*. On average, participants still ignored more than 4 *swhidgets* out of 9, motivating the importance to help users discovering these widgets,
<figure><img src="image_61.png" /><figcaption>Figure 8.9: Proportion of participants who cited each source of knowledge, with 95%-level confidence intervals, sorted by increasing proportion.</figcaption></figure>

either through alternative graphical design or by fostering exploration and social exchanges about touch-based user interfaces.

Although the large overlapping of the confidence intervals does not allow to conclude with certitude, the trend seems to be that the applications and devices used are the most frequent sources of knowledge, above other people and Internet. The latter sources of knowledge can however not be neglected, as e.g. blogs are cited by at least 19.3% of the participants. The high rank of "this_experiment.experimenter" is another hint that participants have a low level of knowledge for the interaction methods used in this experiment.

### 8.3.9 Participants' reaction about swhidgets

In the final part of the survey, 22 of the 28 participants (78,6%) reported that *swhidgets* "improve their everyday interactions", which can also explain why participants preferably use hidden controls when they know about them. The 95%-level confidence interval for this proportion is [58.5,91.0], which is clearly above the status-quo, but also leaves some room for a non-negligible proportion of people who disagree with this statement. During the post-experiment interview, these 22 participants shared their intention to not only keep using the *swhidgets* they already knew before the survey, but to also consider using the ones they learned at the end of the survey.
## 8.4 DISCUSSION OF THE RESULTS OF THE LABORATORY STUDY

### 8.4.1 Reflection on the study

*Summary.* I conducted a controlled experiment investigating the usage of *swhidgets* on iOS by prompting 28 participants in majority from a IT background to perform various operations on the smartphone, and observing which input method they used. This experiments showed that participants have a moderate level of knowledge of *swhidgets* as they used on average slightly more than half of the *swhidgets* available in our experiment. The results also suggest that users tend to prioritize *swhidgets* over alternative input methods (assuming that they are aware of them).

*Usage of Navigation controls.* One surprising result of this experiment was the relatively low level of usage of Navigation controls, which was significantly lower than for *swhidgets*. While surprising at first, it can be explained by the fact that three operations (*search SMS*, *find time SMS*, and *Search Mail*) were perceived as unclear by the participants. Indeed, unlike the other operations, the *Navigation control* alternative for these operations consisted in browsing the corresponding SMS/E-mails rather than activating a command by tapping on specific widgets. Therefore, *find time SMS* was the least completed operation, which in the end also resulted in participants reporting significantly more *Navigation controls* than *swhidgets* as RECEPTIVE.

*Experimental design decisions.* I investigated the problem of *swhidgets* usage with a controlled experiment because I believe this methodology offered the best trade-off between applicability and validity of results. Ideally, *swhidgets* usage would be investigated by instrumenting users' device in order to monitor everyday interaction and precisely quantify the proportion of *swhidget* usage. Unfortunately, accessing that type of information on the iOS operating system is not possible without asking the participant to *jailbreak* his or her device. An alternative method would be to record the screen of the device of the user and post-annotate her interaction but it would raise obvious privacy issues.

### 8.4.2 When participants did not use swhidgets

A large majority of participants (78.6%) agreed that *swhidgets* do "help"^5 them with their everyday tasks on touch-based devices, which is understandable considering that most *swhidgets* provide shortcuts to frequently used features, make it possible to achieve faster a daily task, or simply allow to access information that would not be accessible otherwise (typically, the exact time a text message was received). That being said, this result also means that 21.4% did not consider that

<footnote>^5 The wording of the question was voluntarily vague, in order to stimulate discussion about how *swhidgets* are useful or not in the open interview after the questionnaire.</footnote>
swhidgets are helpful for their everyday interaction. I review here some reasons given by the participants not to use swhidgets.

Obviously, swhidgets can bring benefits if users know how to use them. The results of this survey show that even if most participants knew at least a few interaction methods based on swhidgets, the knowledge of swhidgets is not evenly spread. In particular, the knowledge of one interaction method does not seem to transfer well to similar interaction methods in other contexts. It often occurred, in the debriefing part of the survey, that after being introduced with such a hidden-control-based interaction method, participants reacted with a reflection such as "I've heard about it from somewhere but I didn't knew I can also use it here" or "I wasn't aware of that, but I will do it this way from now."

When users don't know how to use swhidgets, it does not only affect their overall performance: it can actually bring confusion to users. Several participants reported that because some types of swhidgets cannot be used everywhere the design would technically allow it (e.g. there is no swipe from the left in the SMS application), they had no clue about when to use them or not, so they rather sticked to using the navigation controls (visible widgets) instead.

Another interesting observation is that users can have a wrong perception of the performance gain provided by swhidgets. For instance, several participants who knew how to delete emails with a hidden button still preferred to tap on the visible button, because they believed it is faster to select all the emails that they want to delete and then put them into the trashcan together.

### 8.4.3 Task-based vs. System-based Interaction Models

The analysis of the interaction vocabularies for swhidgets, shown in Figure 8.7, reveals that it is not a bimodal distribution with peaks at the extreme values, which should be the case if users were perfectly able to transfer their knowledge from one interaction method to another of the same type: they would know all of them or none of them. A possible interpretation would be that users tend to build interaction models for each task, rather than build global interaction models to match the system's overall design: They try to find interaction possibilities on the basis of the task they want to achieve rather than on general interaction models that they have learned. The rather low score of "other systems" as a source of knowledge in figure 8.9 tends to support this hypothesis. If this interpretation holds, it means that the phenomenon of "interface layouts" that we have observed in the analysis of swhidgets in Chapter 6 is not successful as a way to ease transfers of knowledge between applications.
8.5 STUDY 2: ONLINE SURVEY ON AWARENESS, USAGE AND DISCOVERY OF SWHIDGETS

The results of our online study suggest that users have an uneven
knowledge of *swhidgets*, and that this input method tend to be pre-
ferred to other types of interactions. We then conducted a second,
larger scale, online study in order to confront the results of the pre-
vious study in terms of awareness and usage of *swhidgets*, but also
gain more insights on their perceived efficiency, as well as better
understand how users discover them.

8.5.1 *Design and Procedure*

We designed an online survey in order to investigate usage of various *swhidgets*. It started by asking demographic questions, and general questions regarding smartphone/tablet usage and interest in new technologies. Then, for each operation, the respondent was first asked whether she knows how to perform it or not, and if yes, she was asked how frequently this operation was performed. Then, for each associated input method, she was first shown a video demonstrating this input method and then asked whether this input method was new to her. If the input method was unknown, she was asked to report on a 1-7 scale if the input method looked *easy to discover, more convenient than the way usually used to perform the operation*, and if she *would consider adopting this input method in the near future*. If she already knew the input method, she was asked if using it is part of her everyday interaction, if the technique was *easy to discover* (1-7 scale), and if she remembered how she discovered it (for *swhidgets* only). Finally, regardless of the input method, we asked if the respondent would have considered any reasons not to adopt it. This sequence of video and questions was then repeated for each input method corresponding to this operation. Before to move on to the next operation, the respondent was asked to rank all possible input methods by order of preference. Having run through all the *operations*, respondents reach the final part of the survey asking various questions regarding *swhidgets*, among others about their perceived performance and assessing if the logic behind the physical metaphor was understood.

The survey investigated 13 *operations* across 4 *applications* (see Table 8.1, orange). Order of the application was counterbalanced across participants, and order of operations was randomized within an application with two exceptions<fnref n="6" />. Order of input methods was the same for all operations. Similarly to study 1, *Voice* control was included for ecological validity but often ignored in the following analysis.

<footnote n="6">6 *Reveal* for Notification Center and Control Panel which were always presented first as they are needed to achieve the other operations.</footnote>
### 8.5.2 Apparatus and Participants

The survey was created using the LimeSurvey platform [19] and the RemoteControl 2 API. 144 participants were recruited via calls for participation on social networks. All were daily iPhone users, with an average age of 36 year old (sd=9.54). All data were collected anonymously with a limitation of legal age, participation was entirely voluntary and not compensated. The study obtained an agreement from Inria's Ethic Committee and Data Protection Officer.

## 8.6 RESULTS OF ONLINE STUDY

### 8.6.1 Participants Background and Experience

The survey was completed by 144 respondents (114 females, 29 males, 1 other), aged 18-75 (mean=36.4, SD=10.0). They were relatively educated, with 82.6% having at least an undergraduate-level degree, 74.3% having graduated, and 22.9% owning a master's degree or higher. Participants had various backgrounds, as they described their main working industry using 27 of the 30 different industrial categories. Overall, participants reported to be interested in "collecting the latest technology news", the median response on a 1-7 scale<fnref n="7" /> being 5 (Q1=4, Q3=6), with 59.0% of respondents answering 5 or more, and only 4.9% answering 3 or less. Participants reported to be quite experienced with iOS devices, as 93% used one for at least two years (over 6 years for 48.6%). 93.1% of them declared spending at least one hour a day on an iOS device, 67.4% over three hours, and 43.8% over five hours. On a scale from 1="I know nothing" to 7="I know everything", they all evaluated their knowledge of how to interact with their iOS device at 3 or higher, strictly above 3 for 93.1%, and at 5 or more for 74.8% of them (Q1=4, median=5, Q3=6). This score positively correlates with the agreement to "I am highly interested in collecting the latest technology news" (ρ = .486, *p* < 10⁻⁹), which suggests that following the technology news contributes to participants' knowledge of iOS, or at least to their confidence in knowing it.

### 8.6.2 General Usage

*Overall knowledge of input methods.* Of all 28 input methods presented in this study, participants were aware of 19.7 (70.4%) in average (Q1=15, median=21, Q3=25), knew how to use 18.1 (64.6%) (Q1=14, median=19, Q3=23), and reported to effectively use 13.9 (49.6%) (Q1=10, median=15, Q3=17).

*Application and task completion.* The four applications used in this study were well known, evaluated by the awareness of at least one

<footnote n="7">⁷ with 1 being "totally disagree" and 7 "totally agree"</footnote>
input operation to complete them: 96.5% of participants for the Con-
trol Panel, 92.4% for Notification Center (92.4%), 91.0% for Messages
(91.0%) and 84.0% for Mail (84.0%). Among all 12 tasks, searching a
mail is the one with the smallest proportion of participants who know
at least one input method to do it (67.4%), while activating Wi-Fi is the
most frequently known task (95.8%). Lack of knowledge of the tasks
can be explained only partially by a lack of knowledge of the appli-
cations: the proportions of participants who know at least one input
method to perform the task among those who know the application
range from 77.9% for seeing SMS timings to 99.3% for setting the Wi-Fi
(Q1=84.1%, median=92.3%, Q3=94.7%). The 12 tasks had statistically
significant differences in reported usage frequency (Skillings-Mack
statistic = 307.1, *p* < 10<sup>-6</sup>). Based on reported frequency, the most
frequent tasks were all notification-related tasks and revealing the
Control Center (several times a day), followed by turning the Wi-Fi on
or off, (un)locking the screen orientation, and all Mail-related tasks
(several times a week), and finally, flashlight and Messages-related
tasks (once a week or less).

8.6.3 Awareness and Usage of Swhidgets

*Awareness.* We evaluate user awareness of a *swhidget* by the proportion of participants stating that the input method was not new to them. Awareness rates of *swhidgets* range from 35.4% (see *SMS times*) to 95.1% (set Wi-Fi), with a median of 80.2% (Q1=64.0%, Q3=89.1%). Considering only participants who actually knew how to complete the operation, awareness rates of *swhidgets* ranges from 50% to 100% with a median of 95.0% (Q1=89.8%, Q3=99.8%). A repeated measures logistic regression was used to model participants’ odds of being aware of an input method (when they know the task it solves), and revealed statistically significant effects of task (*p* < 10<sup>-15</sup>), input method (*p* < 10<sup>-15</sup>), and interactions between task and input method type (*p* < 10<sup>-12</sup>). Post-hoc Tukey tests show statistically significant differences between all types of input methods, with average awareness rates for voice (64.4%) being lower (*p* < .001) than for *swhidgets* (88.6%), themselves lower (*p* = .020) than for navigation controls (96.6%). A logistic regression on the *swhidgets* awareness odds alone showed that awareness rates for the three types of *swhidgets* were different (*p* < 10<sup>-15</sup>). Post-hoc Tukey tests show that awareness rates for view *swhidgets* (78.4%) are lower (*p* = .009) than those for item *swhidgets* (80.8%), which are themselves lower (*p* < .001) than those for system *swhidgets* (98.3%).

*Usage.* We evaluate users' usage of a *swhidget* by the proportion of participants stating that they use the corresponding input method or used it in the past. Usage rates are high for most *swhidgets*, ranging from 28.5% (see *SMS times*) to 84.7% (set Wi-Fi), with a median of
66.0% ($Q_1=54.3\%$, $Q_3=81.6\%$)<fnref n="8" />. When only considering users who were aware of the *swhidget*, the proportion ranges from 66.3% to 94.6% with a median of 83.1% ($Q_1=81.2\%$, $Q_3=89.4\%$). A repeated measures logistic regression modeling participants' odds of knowing an input method they are aware of revealed statistically significant effects of task ($p<10^{-15}$), input method type ($p<10^{-15}$), and interactions between task and input method ($p<10^{-4}$). Post-hoc Tukey tests revealed that usage of voice control (29.8%) is significantly lower ($p<10^{-5}$) for both) than navigation controls (83.4%) and *swhidgets* (86.5%), but the difference between *swhidgets* and navigation control is not significant. A logistic regression on the *swhidgets* alone showed that usage rates for the three types of *swhidgets* were different ($p<10^{-15}$). Post-hoc Tukey tests show that usage rates for view *swhidgets* (85.1%) are lower ($p=.0084$) than those for item *swhidgets* (80.7%), which are themselves lower ($p<.001$) than those for system *swhidgets* (90.5%).<fnref n="9" />

*Preferred input method.* Unlike in previous study, we could not query which input method is spontaneously used first for performing an operation. Instead, we asked participants to rank by order of preference all possible input methods for each operation. The proportions of participants aware of a *swhidget* for an operation who ranked it as their preferred input method range from 39.5% (delete mails) to 91.5% (flashlight), with a median of 68.6% ($Q_1=53.6\%$, $Q_3=72.3\%$). These results suggest that users tend to prefer *swhidgets* when they are aware of them, but that it also depends on the operation. For instance, several respondents reported to use conventional buttons for deleting e-mails because it is more convenient, as they often delete them right after reading and using the *swhidget* would require to come back to the list first. Note that among respondents who were not aware of the *swhidget* for an operation, proportions range from 18.2% to 85.7% (median=40.0%, $Q_1=25.0\%$, $Q_3=53.6\%$), showing that participants could also be highly interested in discovering some of the *swhidget* they do not know. As illustrated on Figure 8.10-Left, participants preferred *swhidgets* for 6 tasks out of 9. It was preferred over *Navigation* methods 5 times out of 8 when both input methods were available. *Voice* was by far the less preferred input method, which is not surprising given that respondents were also much less aware of it.

<footnote n="8">⁸ Note that for 22 respondents out of 144, a database issue prevented us to retrieve the answer to this question for the *swhidget* of the Messages search operation.</footnote>

<footnote n="9">⁹ These proportions are not in the order revealed by the logistic regression because the latter accounts for variations in the knowledge of participants, and therefore reduces the weight of those who know more input methods. On the other hand, the proportions given are biased toward the behavior of the most knowledgeable participants.</footnote>
<figure><img src="image_62.png" /><figcaption>Figure 8.10: Top: Distribution of preferred input techniques to perform operations, when alternatives to *swhidgets* exist; Bottom: Distribution of usage frequency of *swhidgets*.</figcaption></figure>
### 8.6.4 *Discovery of Swhidgets*

**SOURCES OF KNOWLEDGE** On all participants and operations, exploration of the interface was the most frequently reported source of discovery of *swhidgets* (42.6%), followed by accidental activation (15.9%), transfer of knowledge from another application or system (8.8%), explicitly looking or asking for it (8.6%), informing herself generally about the interface or application (6.3%), being demonstrated the input method (4.9%), remembering seeing someone else using it (3.1%), and others (1.0%). Surprisingly, respondents replied "I cannot remember" for only 8.8% of the *swhidgets*.

The three most cited sources of knowledge (exploration, accidental discovery, and transfer of knowledge) all concern knowledge acquired from the usage of the interface and systems themselves rather than from external sources, and account together for 65.3% of participants' answers, almost three time more than for external sources of knowledge – which still account for 22.9% or answers.

That being said, it would be presumptuous to over-interpret these results, as it remains uncertain that participants remember well how they learned an input method, especially since our participants have a long experience of iOS and probably learned the *swhidgets* a long time ago. As such, it is likely that they assumed how they discovered them, rather than how they actually did discover them.

**SWHIDGETS' PHYSICAL METAPHOR** There was generally no consensus on the questions I7 "I can tell from the interface which on-screen objects could be moved to reveal *swhidgets*" (median=4, Q1=2, Q3=5) and I8 "I can tell from the interface in which direction I should move objects to reveal *swhidgets*" (median=4, Q1=3, Q3=5). These two ratings were strongly correlated ($\rho$ = .709, $p$ < 10⁻²²), which suggests that the design of *swhidgets* in iOS allows users to integrate these two aspects of the interaction in a single concern. The consistence of *swhidgets'* design in iOS interactions was better acknowledged, however, with 61.1% of participants agreeing (scored 5-7) and only 8.3% disagreeing (scored 1-3) on I9.

Although there was no consensus on the statement I6 "I do not expect controls to hide under interface elements or bezels", 45.8% respondents disagreed (scored 1-3) whereas 22.2% agreed (scored 5-7). Agreement with this statement was slightly and negatively correlated with the interest for technologies ($\rho$ = -.226, $p$ = .0064), which suggest that despite being rooted in real world physics, additional knowledge about technological product interfaces might be necessary to grasp this aspect of the metaphor.

**TRANSFER OF KNOWLEDGE THROUGH APPLICATIONS** There was a clear consensus among participants to agree with the statement