# Proxemic Flow: Dynamic Peripheral Floor Visualizations for Revealing and Mediating Large Surface Interactions

Jo Vermeulen, Kris Luyten, Karin Coninx, Nicolai Marquardt, Jon Bird

► To cite this version:

Jo Vermeulen, Kris Luyten, Karin Coninx, Nicolai Marquardt, Jon Bird. Proxemic Flow: Dynamic Peripheral Floor Visualizations for Revealing and Mediating Large Surface Interactions. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.264-281, 10.1007/978-3-319-22723-8_22. hal-01610802

HAL Id: hal-01610802

https://inria.hal.science/hal-01610802

Submitted on 5 Oct 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Proxemic Flow: Dynamic Peripheral Floor Visualizations for Revealing and Mediating Large Surface Interactions

Jo Vermeulen<fnref n="1" />, Kris Luyten<fnref n="2" />, Karin Coninx<fnref n="2" />, Nicolai Marquardt<fnref n="3" />, and Jon Bird<fnref n="4" />

**Abstract.** Interactive large surfaces have recently become commonplace for interactions in public settings. The fact that people can engage with them and the spectrum of possible interactions, however, often remain invisible and can be confusing or ambiguous to passersby. In this paper, we explore the design of dynamic peripheral floor visualizations for revealing and mediating large surface interactions. Extending earlier work on interactive illuminated floors, we introduce a novel approach for leveraging floor displays in a secondary, assisting role to aid users in interacting with the primary display. We illustrate a series of visualizations with the illuminated floor of the Proxemic Flow system. In particular, we contribute a *design space for peripheral floor visualizations* that (a) provides peripheral information about tracking fidelity with personal halos, (b) makes interaction zones and borders explicit for easy opt-in and opt-out, and (c) gives cues inviting for spatial movement or possible next interaction steps through wave, trail, and footstep animations. We demonstrate our proposed techniques in the context of a large surface application and discuss important design considerations for assistive floor visualizations.

**Keywords.** Feedback; proxemic interactions; implicit interaction; discoverability; intelligibility; spatial feedback.

## 1 Introduction

Large interactive surfaces, such as interactive vertical displays or tabletops, have become commonplace in many public settings. These displays often react to the presence and proximity of people [1] or support other interaction modalities such as mid-air gestures, body posture or touch [2].

Several studies, however, report on problems encountered by users while interacting with these large surfaces. People can be prevented from engaging in interactions with displays because they tend to ignore them (*display blindness* [3, 4]) or fail to recognize that they are interactive (*interaction blindness* [5]). Additionally, people could be uncertain about the available possibilities for interaction [4, 6], or be hesitant to interact due to social embarrassment [2, 5, 7]. Finally, people are often unaware of ways to recover from mistakes such as accidental interactions [5, 7]. It is difficult for

<footnote n="1">¹HCI Centre, University of Birmingham</footnote>

<footnote n="2">²Hasselt University – tUL – iMinds, Expertise Centre for Digital Media</footnote>

<footnote n="3">³University College London, UCL Interaction Centre / ICRI Cities</footnote>

<footnote n="4">⁴City University London</footnote>
people to know how to address (or how *not* to address) displays and what they can do when the surface is reacting to their input, as these displays typically rely on implicit interaction using sensors [8]. These problems—users abandoning the surface or not interacting with it because of confusion, frustration or fear of being embarrassed— pose important challenges for designers of applications on large interactive surfaces.

In this paper, we explore the design of *dynamic, peripheral floor visualizations* to help address interaction challenges with large interactive surfaces. The core of our work is combining vertical interactive displays with a *secondary, peripheral floor display*. We apply the term ‘peripheral’ similar to its use for public ambient displays [9], as defined by Weiser and Brown [10] to describe, “*what we are attuned to without attending to explicitly*”. Information shown in the periphery can seamlessly become the center of the attention and move back to the periphery in fluent transitions [10]. Extending the foundations of existing work on interactive illuminated floors

(e.g., [11, 12]), the floor display is not used as a primary interaction space, but serves a *secondary, assisting role* to aid users in interacting with the main display. The floor display can be used to inform users about the tracking status, indicate action possibilities, show interaction zones, and invite and guide users throughout their interaction with the primary interactive display. In the future, we imagine these floor displays being integrated in public spaces to accompany vertical public displays. Due to their relatively low cost, LED floor displays can be frequently found in urban spaces, such as the flashing LED lights in Washington DC metro stations to show arriving trains (Fig. 1).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** An LED floor display used in the Washington metro (YouTube ID: DppgBi0ZMc8).</figcaption></figure>

We illustrate how floor displays can provide additional feedback and guidance through a series of visualizations on the illuminated floor of the *Proxemic Flow* system. As shown in Fig. 2, we focus on leveraging floor visualizations to mitigate interactions with a proxemic-aware vertical display [1] that reacts to people’s presence, approach, and movements (inspired by Ballendat et al. [1]). For example, the floor shows personal halos indicating when people are sensed by the system (circles in Fig. 2) and provides information about the quality of tracking. Furthermore, the floor reveals the boundaries of the interaction area (red line at the bottom of Fig. 2) and invites moving closer to interact (footsteps at the top of Fig. 2).

Our aim with this work is a *design space exploration*, and in the remainder of this paper we provide the following contributions:

* We propose an assistive role for floor displays, where they serve as secondary, peripheral displays that help users in interacting with the primary proxemic-aware surface.

* We demonstrate the expressive power of this new design space with a vocabulary of in-situ floor visualization strategies, and explain in detail how we implemented these visualizations for a low-resolution large floor display.
# 2 Related Work

## 2.1 Feedback, Discoverability and Guidance for Large Interactive Surfaces

Earlier work has explored techniques to address interaction challenges for large interactive surfaces. We provide a brief overview of related techniques, categorized by the challenges that they address.

### Attracting Attention to Overcome Display Blindness and Interaction Blindness

Displays can be designed to attract attention and motivate users to interact with a display, and thus overcome *display blindness* [4] and *interaction blindness* [5]. Earlier work identified barriers that prevent people from engaging with a large display when passing by (e.g., [7, 13]). With a few exceptions (e.g., [14]), most techniques attempt to convey interactivity and attract users by using visualizations on the interactive display (e.g., [13, 15, 16]). In contrast, with Proxemic Flow we primarily focus on the floor display to convey interactivity. One of the advantages of providing visualizations on a secondary display is that they do not occlude or distract from existing content on the primary display. The floor can reveal the interaction area through borders and zones, and show halos when people are recognized by the display. Our floor visualizations are designed to indicate to passersby that they *can* interact, making them aware that they are tracked, and also allowing people to avoid interacting with the display. This addresses a common issue with interactive surfaces in semi-public settings, i.e., that they lack option and opt-out choices [17].

**Fig. 2.** Proxemic Flow providing awareness of tracking and fidelity, zones of interaction, and invitations for interactions.

<figure><img src="image_3.png" /></figure>

### Revealing Action Possibilities and Providing Guidance

Commonly used interaction modalities for public displays (e.g., proximity, body posture, mid-air gestures) are often hard to understand by passersby at first glance [2]. A number of systems suggest *action possibilities* and input gestures by visualizing sensor data, such as depth camera images, detected user skeletons [18, 19] or mirrored
images [9, 20]. Early work by Vogel and Balakrishnan [9] included a self-revealing help feature using a mirror image video sequence. Walter et al. [6] studied different visualizations to reveal a ‘teapot gesture’ that allows users to indicate that they would like to start interacting with the display.

These techniques work well in particular for revealing action possibilities (e.g., mid-air gestures and body postures) on the display itself. Our in-situ techniques introduce a new vocabulary of visualizations potentially more appropriate for proxemic-aware surfaces that often use people's and devices' spatial movements as implicit input. Such implicit application input might be surprising and possibly disturbing in walk-up-and use scenarios, as in the Proxemic Media Player [1] where videos are automatically paused when users are not facing the display. Techniques that reveal action possibilities on the primary display often focus on particular proxemic dimensions [1]. For example, cross-device interaction techniques typically use *orientation* to show possible targets around the user's device, as in the Gradual Engagement pattern [21] or the RELATE interaction model [22].

## Providing Tracking Feedback

A common problem users experience while interacting with public displays is a lack of feedback about how the system is currently recognizing and interpreting their input, and also how reliably this input is being sensed. In crowded spaces, people can be unsure about the level of control they have over the display [19]. A number of systems reveal tracking feedback for proxemic interactions to convey what the system sees of the user, e.g., by visualizing detected skeletons [18] or mirror images [20]. With their interactive whiteboard, Ju et al. [8] showed a dot pattern to indicate in which proximity zone the user was recognized. In the Medusa proximity-aware tabletop [23], the user's proximity is shown using an orb visualization. Both in the Proxemic Media Player [1] and the Gradual Engagement pattern [21], tracked devices are visualized on the large surface with their relative size mapped to their proximity to the large display.

With Proxemic Flow we provide in-place tracking feedback on the floor, in the space where users are tracked. A number of earlier systems provided projected tracking feedback on the floor in the form of halos such as the Solstice LAMP [24] and Proximity Lab [25].

## In-Situ Feedback and Guidance

Proxemic-aware systems typically take different actions based on the interaction zone in which the user is located [1, 8, 9], which may be unintelligible to users. Rehman et al. [26] used augmented reality to visualize interaction zones in-place, but this technique required users to wear head-mounted displays. Researchers have also explored the possibilities of providing in-situ feedback and guidance using spatial augmented reality, which uses a combination of projectors and depth cameras and thus eliminates the need for users to wear additional apparel. For example, LightSpace [27] shows when users are tracked by the system by projecting colored highlights on the user's body. LightGuide [28] uses a projector and depth cameras to project visualizations on
the user's body that provide movement guidance. In a public art context, Ozturk et al. [29] explored people's reactions to projections of their 'future footsteps' in an airport terminal.

Although the use of projectors allows for high-resolution visualizations and more flexibility, projectors often require low-lighting conditions, which makes these techniques less suitable for large interactive surfaces in urban spaces (especially during daytime). Regarding guidance using LED floors, Rogers et al. [30] explored the use of LEDs embedded in carpet tiles to motivate people to use the stairs more often. They observed that the LED lights had a significant effect on people's behavior, which illustrates the power of in-situ visualizations and guidance.

## 2.2 Related Work on Interactive Illuminated Floors

With Proxemic Flow, we propose the use of graphical information shown directly on the floor of the interactive space, around the people who are engaging in the interaction, for providing feedback about the system status or informing users of action possibilities and consequences.

Interactive illuminated floors have been used in different contexts, such as interactive art [24, 25] or games [31], and have recently seen increasing exploration as a primary interaction space [11, 12]. A variety of input and output technologies have been used for these interactive floors, such as tracking users through computer vision techniques [31] or pressure sensing [12], and showing output using projectors [12, 31], LED illumination [32] or vibrotactile feedback [33].

Our work extends this earlier research by (a) proposing the use of the floor as a peripheral/secondary output device that can help to mediate interactions with a different, primary interaction device, and (b) providing a vocabulary of strategies to provide in-situ feedback about current and future interactions with the system.

# 3 In-Situ Floor Visualization Strategies

In order to mitigate the previously mentioned interaction challenges, we introduce a series of interaction techniques and in-situ visualizations on the floor. These are categorized into three phases, progressing from:

- a) *in-situ tracking feedback* (answering the questions: What does the system see? How well does the tracking work?), and

- b) *revealing interaction possibilities* (answering: What possible interactions are available?), to

- c) *inviting for and guiding interactions* (answering: What can I do next?).
## 3.1 Walkthrough with Photo Gallery Application

We will illustrate all our in-situ visualization strategies with a running example application, inspired by the design of the Proxemic Media Player [1]. Our photo gallery application shows photos collections on a large public display. A series of interactions are possible with this gallery application: it shows photo thumbnails when in idle mode (Fig. 3-a), reveals more content when a person approaches the display (Fig. 3-b), shows full screen photos when people stand directly in front of the display (or sit down), and allows mid-air gestures to navigate the photo collection (e.g., waving left or right to browse through the timeline of photos). While limited in scope, we believe this example application captures the essence of many proxemic interactions applications and works best for demonstrating our in-situ floor visualization strategies. Throughout this paper, however, we will also refer to the use of our visualization strategies in other application contexts.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Walkthrough Photo Gallery Application: photo thumbnails at a distance (a), revealing more content when moving closer (b).</figcaption></figure>

## 3.2 Design Space for Floor Visualization Strategies

Our visualization strategies can be categorized in a design space for in-situ floor visualizations. Table 1 shows an overview of the different strategies and indicates to which of the three phases (tracking feedback, action possibilities or guidance) they correspond. The table compares our floor visualization strategies based on three different aspects: *perspective*, *position*, and *temporal relevance*. Regarding perspective, we distinguish between *egocentric* and *exocentric* visualizations. For example, tracking halos are targeted towards being viewed from the user's own perspective (*egocentric*), while zones and borders are mostly useful from an external perspective (*exocentric*). Additionally, a few visualizations have a *static* position on the floor, while others can move *dynamically* (e.g., together with the user). Finally, visualizations can be relevant to the user's current (or *present*) interactions with the primary display (e.g., quality of tracking), or can alternatively provide clues about *past* or *future* actions.

<table><caption>**Table 1.** An overview of the design space for in-situ floor visualization strategies.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Phase</th><th rowspan="2">Purpose</th><th colspan="2">Perspective</th><th colspan="2">Position</th><th colspan="3">Temporal Relevance</th></tr><tr><th>Egocentric</th><th>Exocentric</th><th>Static</th><th>Dynamic</th><th>Past</th><th>Present</th><th>Future</th></tr></thead><tbody><tr><td>Halos</td><td>1</td><td>Tracking feedback</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td>✓</td><td></td></tr><tr><td>Trails</td><td>1, 3</td><td>Historic traces of action</td><td>✓</td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓(a)</td><td>✓(b)</td></tr><tr><td>Zones</td><td>2</td><td>Interaction zones<br>Action possibilities</td><td>✓(c)</td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td>✓(d)</td></tr><tr><td>Borders</td><td>2</td><td>Opt-in & opt-out</td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td>✓(e)</td></tr><tr><td>Waves</td><td>3</td><td>Inviting for interaction</td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td>✓</td></tr><tr><td>Steps</td><td>3</td><td>Guiding spatial movement</td><td>✓</td><td></td><td></td><td>✓</td><td></td><td></td><td>✓</td></tr></tbody></table>
We see this design space as a starting point for characterizing in-situ floor visualiza-
tions for mediating large surface interactions, and foresee possible future extensions.
It also functions as an analytical tool to reflect on the set of floor visualization strate-
gies that we propose in this paper, and can help to further explore alternative floor
visualizations. We will now go over the three phases, and will later come back to
these different aspects while discussing our floor visualization strategies.

3.3 Phase 1. In-situ Personal Tracking Feedback with Halos

A fundamental challenge for interaction with large surfaces is providing a person with
immediate feedback about how the system is currently recognizing and interpreting
gestures or other input from the user. In this section, we introduce visualization strat-
egies to provide this feedback directly in the physical space where the person is mov-
ing in front of the display.

Personal Halos

The personal halo provides immediate feedback on the floor display about the track-
ing of a person in space. When the
person enters the area in front of the
public display, a green halo (an area
of approximately 1m diameter) ap-
pears underneath the person's feet
(Fig. 4-a). The halo moves with them
when moving in the tracking area,
and therefore gives continuous feed-
back about the fact that the person is
being recognized and tracked by the
system.
a

<figure><img src="image_5.png" /><figcaption>Fig. 4. Halos: (a) providing feedback about active tracking and (b) the tracking quality.</figcaption></figure>

Another important part of infor-
mation (besides information about
the fact that a person *is* tracked) is
the actual *quality* of tracking. Most
computer vision based tracking sys-
tems (RGB, depth, or other tracking)
have situations where tracking works
well, where it does not work well, or where it does not work at all (e.g., due to light-
ing conditions, occlusion, limited field of view). Therefore, our personal halo visuali-
zation encodes the quality of tracking in the color of the halo. To indicate tracking
quality, we use three different colors (Fig. 4-b). A green halo indicates optimal track-
ing of the person in space. Its color changes to yellow when the quality of tracking
decreases, for example when the person moves to the limits of the field of view or
when partially occluded by another person or furniture. Finally, a red halo color is
shown when the tracking of the person is lost, such as when moving too far away
from the camera, or if the occlusion is hiding the person completely. For this last case,
since the person is now not tracked anymore, the red halo visualization remains static at the last known location of the person, fades in and out twice, and then disappears (the duration of that animation is approximately 4 seconds). If the person moves back into the field of view of the camera and the tracked region, the halo color changes back accordingly to green or yellow.

The immediate feedback of tracking through halos can provide people with more control over their interaction with the system. For example, when noticing that they are being tracked, the user could decide to opt out of interaction with the system by moving back out of the active tracking area.

## Alternative Halo Visualization Strategies

Although this is a crude mapping of tracking accuracy to different colors, we found it to be an effective in-situ form of feedback about tracking activity *and* fidelity. Other applications might require different levels of granularity. For example, for interactive proxemic game experiences [34], tracking accuracy per body part could be helpful information for players, which could be visualized using more fine-grained halos. Tracking accuracy of different body parts could be mapped to different areas of the halo; e.g., front left corresponds to left arm, back left to the left leg. Alternatively, halos could change their size depending on the area covered by the player. There is a limit to the amount of information that can be conveyed using our low-resolution floor display. Revealing precise details about the tracking quality for different body parts (or showing text for instructions) would require higher-resolution floor displays. For the remainder of this paper, however, we focus on the expressive potential of low-resolution peripheral floor visualizations.

## Multi-User Halos

Interaction around interactive surfaces is often not limited to a single person, but can involve multiple people present in the space and interacting with the display. With

multiple people, information about active tracking and its fidelity becomes even more important, because tracking problems increase with the likelihood for occlusions.

If multiple people are present in front of the screen, each person's individual position that the system currently tracks is shown with a

colored halo (Fig. 5-a). Color changes indicate a change in how well the user is tracked. For example, in case another person walking in interrupts the tracking cam-

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Halos for multi-user interaction: (a) both people are visible to the system; (b) one person is occluding the camera's view of the other person, indicated by the red halo.</figcaption></figure>
era's view of a person, the changing color of the halo from yellow to red tells the person that they are not tracked anymore (Fig. 5-b). Similarly, if two people stand very close to another, making it difficult for the computer vision algorithm to separate the two, the halo color changes to yellow.

## Trails: Revealing Interaction History

As a variation of the halo technique, the *spatial trail* feedback visualizes the past spatial movements of a person in the interaction area. The trails are shown as illuminated lines on the floor that light up when a person passes that particular area (Fig. 6). The illumination fades out after a given time (in our application after five

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Trails, visualizing the history of spatial movements of a person.</figcaption></figure>

seconds), thus giving the impression of a comet-like trail. The colors that are used to light up the floor are identical to those of the person's halo (i.e., green, yellow, red), and therefore still provide information about the tracking quality. Because the trail visualization remains visible for a longer time, it provides information about the past movements of the people interacting with the system. Potentially, the trails could help to amplify the *honeypot effect* [7] by showing the past trails of other people moving towards the interactive display, and thus inviting other bystanders and passersby to approach the display as well—which is why they are categorized in both phase 1 and 3 (Table 1).

## Reflecting on Phase 1 in the Design Space

Halos are an example of an *egocentric* strategy (Table 1). They are primarily designed to be viewed from the user's perspective, providing feedback about the tracking status. The trails variation, however, is a mostly *exocentric* technique that shows information about past interactions from the perspective of other users. However, since the trails are still shown underneath the user's feet, and change color depending on the user's tracking accuracy, they are simultaneously *egocentric* and inform the user about their *present* interactions (Table 1-a). In addition, as they potentially invite bystanders to interact with the display, the trails can serve as an invitation for *future* interactions (Table 1-b). We can also imagine other *exocentric* halo visualizations. For example, pulsating exocentric halos could indicate open spots where users could move towards, e.g., to form teams in proxemic gaming scenarios [34].
3.4 Phase 2. Zones and Borders: Entries and Exits for Interaction

As mentioned earlier, people often have difficulties knowing when and how they can
interact with a large public display [2, 19]. To mitigate this problem and to reveal
interaction possibilities, we explicitly visualize the spatial zones for interaction and
the borders of the interaction space (Fig. 7).

Opting-in: Proxemic Interaction Zones

Many designs of large interactive displays use spatial zones around the display to
allow different kinds of interaction [9] or change the displayed content depending on
which zone a person is currently in. These zones, however, are not always immediate-
ly understandable or perceivable by a person interacting with the display. Our floor-
visualizations explicitly reveal zones of interaction, allowing a person to see where
interaction is possible, and make deliberate decisions about opting in for an interac-
tion with the display by entering any of the zones.

We demonstrate the use of zone visualizations with the Proxemic Flow photo gal-
lery application. Similar to earlier examples of proxemic-aware displays [1, 9], our
application uses discrete spatial zones around the display that are mapped to the inter-
active behavior of the application on the large display. When no users are interacting
with the system, a large red rectangular zone indicates the area furthest away from the
display that triggers the initial interaction with the display (Fig. 7-a). This serves as an
entry zone for interaction, i.e., an area to opt-in for interaction with the system. In our
current implementation, we use a 3s pulsating luminosity animation, fading the color
in and out, in our approach of balancing the goal of attracting attention while not be-
ing too intrusive. While a static color would be possible, identifying it as part of an
interactive system is potentially more difficult. Once a person enters this zone, the
large display recognizes the presence of the person, tracks the person's movement,
and their halo is shown. The first zone now disappears and a second zone appears—
an area to interact with the display when in front of is (visible as the blue rectangle in
Fig. 7-b). When the person begins approaching the display, the content gradually
reveals more of the photo collection on the display. The closer the person gets, the
more images become revealed (this is a behavior identical to the Proxemic Media
Player [1]). Once entering the second zone, the person can now use hand gestures in
front of the display to more precisely navigate the temporally ordered photo gallery
(e.g., grabbing photos, sliding left or right to move forward or back in time). Again,
once the person entered that close-interaction zone in front of the display, the visuali-
zation disappears.

Opting-out and Exit Interaction: Borders

While we envision zones primarily as explicit visualizations of the zones to interact,
and for allowing a person to deliberately engage and “opt-in” for an interaction with
the system, we can also consider visualizations that help a person to leave the interac-
tion area (i.e., opting out). We illustrate this concept with borders shown in the Prox-
emic Flow application. In continuation of the application example from before, once
the person entered the interaction zone (blue) directly in front of the display and interacts with the display content through explicit gestures, a red border around the actively tracked interaction area surrounding the display is shown to make the boundaries of that interaction space explicit and visible (Fig. 7-c). While we decided to dynamically show the border only in situations when a person engaged with the system, alternatively it could remain a fixed feature of the visualizations shown on the floor. A reason for showing a fixed visualization of the interaction boundaries with borders could be to always clearly indicate where a person can both enter but also leave the interaction area (Fig. 7-d).

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** The interaction areas in front of the display represented as (a) red and (b) blue rectangular zones; (c) borders indicate thresholds to cross for (d) leaving the interaction space in front of the display.</figcaption></figure>

## Using Zones and Borders with Multiple Users

We can consider alternative design aspects when using zone and border visualizations with multiple users. For example, we can consider whether area visualizations are only shown to the first user entering the space and disappear once that person entered the zone, or whether the visualizations remain persistent. Showing visualizations for the first person entering a space seem most critical, and hiding the zone visualizations after the person enters a particular zone has the advantage of a floor that is less visually cluttered and therefore can help emphasizing certain parts of the visualizations (for example, make the halos stand out).

## Reflecting on Phase 2 in the Design Space

In contrast to halos and trails, zones and borders are *static* visualizations. They are fixed at a certain position, and although they might only be shown at certain times, they do not follow the user. Zones and borders are also mostly *exocentric*, as they are designed for observations from an external point of view. Nevertheless, zones can also be used from an *egocentric* perspective, when the user is inside the interaction zone (Table 1-c). Finally, they convey cues relevant to the user's current interactions (*present*), such as borders around the actively tracked interaction area. However, zones and borders can also provide cues for *future* interactions, such as possible next areas to move to, or where to go to opt-out of the interaction (Table 1-d-e).
## 3.5 Phase 3. Waves and Footsteps: Inviting for Approach, Spatial Movement, or Next Interaction Steps

The last set of floor visualization strategies we introduce is designed to invite for approach, encourage a person's movement to a new location, and suggest possible next interaction steps. In particular, in this category of visualizations we introduce two strategies: waves and footsteps.

### Waves: Encouraging Approach

Our first strategy is intended for inviting people to move closer to the large display for interaction. Several strategies for encouraging approach of people have been proposed in the past, including showing text labels, animations, graphic icon representations or using sound (e.g., strategies in [15]). With our waves

technique, we leverage the output capabilities of the illuminated floor for showing looped animations of lights fading in and out, with the effect of a wave of light going towards the large screen (Fig. 8-a). Alternatively, different visual designs of the wave effect are possible, for example a circular wave effect with the large display at the center, starting with circles having a large radius and continuously decreasing the circle radius.

<figure><img src="image_9.png" /><figcaption>Fig. 8. (a) Waves inviting for interaction and (b) footsteps suggesting action possibilities.</figcaption></figure>

### Footsteps: Suggesting Next Action Possibilities

The footsteps visualization is designed to offer a person clues about possible next interaction steps (directly addressing *discoverability*), in particular for encouraging spatial movements in the environment. The visualization shows animated footsteps (in our case these are represented through glowing circles) beginning at one location on the floor and leading to another location. This technique is inspired by earlier work of the *Follow-the-light* [30] design that uses animated patterns of lights embedded in a carpet to encourage different movement behaviors by luring people away from an elevator towards the stairs.

To illustrate this technique, we again revisit our Proxemic Flow example application with the large display photo gallery viewer. When a person entered the interactive (i.e., tracked) space in front of the display and stands still for over 5 seconds, the floor begins the footstep animation (Fig. 8-b) to invite the person to move closer to the display—in particular, moving to the interaction zone in front of the display allowing the person to use mid-air gestures to further explore the image collection. The
footstep animation begins directly in front of the person and leads towards the blue rectangular area highlighted in front of the display (Fig. 8-b). The footsteps visualization strategy can be used to reveal interaction possibilities—in particular those involving spatial movements of the person. The strategy can be used in many other contexts for guiding or directing a user in the environment, and for encouraging movements in space.

## Reflecting on Phase 3 in the Design Space

The visualization strategies for phase three provide cues that invite users to *future* interactions. The waves strategy is *exocentric*, as it invites bystanders to interact with the primary display. It is a *static* visualization, as people's movements do not influence its position. The waves pattern could be shown across the full floor display or be centralized around the primary display. The steps strategy, on the other hand, is a *dynamic* and *egocentric* visualization that starts from underneath the person's feet, and guides them towards a certain position.

## 3.6 Reflection on In-Situ Visualization Strategies

We discussed and demonstrated a set of in-situ floor visualizations that provide peripheral tracking and fidelity information with *personal halos*, make interaction *zones and borders* explicit for easy opt-in and opt-out, and provide cues inviting for spatial movement or possible next interaction steps through *wave, trail*, and *footstep* animations. This set of floor visualization strategies targets important interaction issues with large interactive surfaces that were identified in earlier research. During informal observations of people interacting with our floor display, we noticed that essential concepts such as halos and zones were easy to understand. Future studies are necessary, however, to confirm these early observations. The strategies we presented here are a starting point for a collection of building blocks for how to provide in-situ visual feedback on the floor to mediate spatial interactions. In the next section, we present the Proxemic Flow software architecture and explain how we implemented the floor visualizations.

# 4 Implementation

The Proxemic Flow architecture consists of three major technical components: (1) the *hardware setup* of the illuminated floor, (2) the *user tracker* and (3) the *floor renderer*. The user tracker is responsible for tracking users in the space in front of the display, and for mapping these positions to positions on the floor. The floor renderer consists of a .NET client that draws visuals to a bitmap and sends display updates over the network to a Processing sketch connected to the Arduino board controlling the different light units. We will now explain these components in more detail.
## 4.1 Hardware Setup of the Interactive Floor Display

The foyer floor that we use for our setup comprises 288 light wells set in concrete, of which 216 of these wells are fitted with a custom light unit [35]. The custom light units in each of the 216 light wells consist of four RGB LEDs cut from an LPD8806 LED strip, joined together and mounted onto a plastic cap which fits neatly into the concrete surface from the floor below. The light units are connected in series, with three modified ATX power supplies providing power. A single Arduino Mega with the ATmega1280 microcontroller is controlling the floor display. Each of the light units can be set to one of around 2 million colors and the whole array can be updated at a rate of up to 25 fps—effectively turning the floor into a large display with a resolution of 12x18 pixels.

<figure><img src="image_10.png" /><figcaption>**Fig. 9.** The Proxemic Flow rendering pipeline: Visualizations on the floor display are abstracted in a floor scene (a). This floor scene is processed by the floor renderer (b), resulting in (c) a floor bitmap (an abstraction of a floor display update) that is sent over the network to the connected floor displays that implement the IFloor interface (d). We also implemented a projected floor display (f).</figcaption></figure>

## 4.2 Tracking Users

Users are tracked across the floor using a single Microsoft Kinect depth camera and the Kinect SDK, which allows us to track up to six simultaneous users (with skeleton data available for two users). As users positions can be represented in a 2D (x, z) plane (we ignore the user's vertical position), a simple affine matrix transformation suffices to map the (x, z) coordinates as given by the Kinect camera to a position on the floor. To set up the system, a four-point calibration is performed to map positions seen by the Kinect to the corresponding floor positions, after which the corresponding transformation matrix is calculated.

Each user's tracking accuracy—as used for determining the color of their personal halo—is specified as a value in the range [0,1]. We calculate the tracking accuracy by performing an arithmetic average over the accuracy of the skeleton joints. Skeleton joints have one of three states: *tracked*, *inferred*, or *not tracked*. We currently assign
the value 1.0 to *tracked* joints, 0.3 to *inferred* joints and 0.0 to joints that are *not tracked*. Green halos are shown for accuracies over 0.7; yellow halos for accuracies between 0.3 and 0.7; and halos turn red when the accuracy drops below 0.3. These specific thresholds have been selected based on empirical observations, but can be easily changed.

## 4.3 Floor Renderer

The Proxemic Flow renderer provides a set of reusable rendering primitives that react on user tracking updates. All graphics and animations can be translated into a *floor bitmap*, which allows the *rendering pipeline* to be agnostic to the specifics of the graphics being shown on the floor. This *rendering pipeline* is the central hub of the architecture and handles updates to the floor display (Fig. 9).

The *floor renderer* (Fig. 9-b) uses a timer to allow sending update messages to the floor at a fixed rate. A floor update message is represented by a *FloorBitmap* object (Fig. 9-c), which is an 18x12 grid of color values for each of the light wells in the grid. Every tick, the rendering pipeline sends a floor update message to the connected instances of the *IFloor* interface (Fig. 9-d). The default IFloor implementation (Fig. 9-e) sends messages to a Processing sketch that is connected to the Arduino board that runs the floor, which then renders the floor bitmap to the physical floor display.

## 4.4 Alternative Implementations

The concept of Proxemic Flow goes beyond the specifics of our floor setup. The illuminated floor could be implemented using different floor displays (e.g., using projectors or FTIR floor displays [11]) and tracking solutions (e.g., 2D cameras with markers, other depth cameras, or optical trackers such as VICON). Fig. 10 shows an alternative render-

<figure><img src="image_11.png" /><figcaption>**Fig. 10.** Alternative floor display using a ceiling-mounted short-throw projector.</figcaption></figure>

ing solution we implemented in order to show visuals on arbitrary surfaces, based on an overhead projector mounted to the ceiling. It connects another IFloor instance to the same rendering pipeline (Fig. 9-f), so that applications written once run without modification. In this implementation, a separate Windows Presentation Foundation (WPF) window that renders the floor grid is projected onto the floor. Our projection-based floor responds to floor update messages by changing an internal model of the floor grid, which is then also updated in the WPF view. Higher resolution floor displays (e.g., with a circular halo visualization) would be possible by making rendering primitives adapt to different resolution floor bitmaps, specific to each IFloor instance.
# 5 Discussion

We presented Proxemic Flow, dynamic in-situ floor visualizations for revealing and mediating large surface interactions. Based on previously identified interaction challenges with large interactive surfaces, we demonstrate the expressive potential of the floor as a peripheral/secondary output device for showing in-situ feedback using three categories of visualizations: (1) personal halos and trails that provide peripheral information about current tracking and tracking fidelity; (2) interaction zones and borders for easy opt-in and opt-out; and (3) wave and footstep cues that invite users for movement across the space or possible next interaction steps.

Our approach is intentionally minimalistic: we reduced the visualizations to essential cues that require minimal visual bandwidth. These can be extended—for example with more fine-grained spatial movements cues—but we believe it is important to avoid a visually cluttered floor with (perhaps even animated) visualizations that detract the user. Ideally, the visualizations should be shown when needed, but not unnecessarily draw the user's attention and detract from interacting with the primary display, as the floor serves a secondary, assisting role. We plan studies further investigating the balancing of showing information while avoiding distractions.

During initial observations, we noticed that people became aware of the floor being a display as they approached the tracking zone. Users noticed their personal tracking halos when they entered space in front of the display. Due to their low visual complexity, a quick glance at the visualizations is often sufficient, e.g., when users are unsure about action possibilities. An interesting opportunity for future work is to investigate how user's peripheral view, which is very sensitive to motion [36], can be used to draw their attention when needed.

# Acknowledgements

This work was supported by ICRI Cities.

# References

1. Ballendat, T., Marquardt, N., Greenberg, S.: Proxemic interaction: designing for a proximity and orientation-aware environment. Proc. ITS '10. pp. 121–130. ACM, New York, NY, USA (2010).

2. Müller, J., Alt, F., Michelis, D., Schmidt, A.: Requirements and Design Space for Interactive Public Displays. Proc. MM '10. pp. 1285–1294. ACM (2010).

3. Huang, E.M., Koster, A., Borchers, J.: Overcoming Assumptions and Uncovering Practices: When Does the Public Really Look at Public Displays? Proc. Pervasive '08. pp. 228–243. Springer-Verlag, Berlin, Heidelberg (2008).

4. Müller, J., Wilmsmann, D., Exeler, J., Buzeck, M., Schmidt, A., Jay, T., Krüger, A.: Display Blindness: The Effect of Expectations on Attention Towards Digital Signage. Proc. Pervasive '09. pp. 1–8. Springer-Verlag (2009).
5. Ojala, T., Kostakos, V., Kukka, H., Heikkinen, T., Linden, T., Jurmu, M., Hosio, S., Kruger, F., Zanni, D.: Multipurpose Interactive Public Displays in the Wild: Three Years Later. Computer. 45, 42–49 (2012).

6. Walter, R., Bailly, G., Müller, J.: StrikeAPose: Revealing Mid-air Gestures on Public Displays. Proc. CHI '13. pp. 841–850. ACM, New York, NY, USA (2013).

7. Brignull, H., Rogers, Y.: Enticing People to Interact with Large Public Displays in Public Spaces. Proc. INTERACT '03 (2003).

8. Ju, W., Lee, B.A., Klemmer, S.R.: Range: exploring implicit interaction through electronic whiteboard design. Proc. CSCW '08. pp. 17–26. ACM (2008).

9. Vogel, D., Balakrishnan, R.: Interactive Public Ambient Displays: Transitioning from Implicit to Explicit, Public to Personal, Interaction with Multiple Users. Proc. UIST '04. pp. 137–146. ACM, New York, NY, USA (2004).

10. Weiser, M., Brown, J.S.: Designing calm technology. PowerGrid J. 1, 75–85 (1996).

11. Augsten, T., Kaefer, K., Meusel, R., Fetzer, C., Kanitz, D., Stoff, T., Becker, T., Holz, C., Baudisch, P.: Multitoe: High-precision Interaction with Back-projected Floors Based on High-resolution Multi-touch Input. Proc. UIST '10. pp. 209–218. ACM, New York, NY, USA (2010).

12. Schmidt, D., Ramakers, R., Pedersen, E.W., Jasper, J., Köhler, S., Pohl, A., Rantzsch, H., Rau, A., Schmidt, P., Sterz, C., Yurchenko, Y., Baudisch, P.: Kick-ables: Tangibles for Feet. Proc. CHI '14. pp. 3143–3152. ACM, USA (2014).

13. Michelis, D., Müller, J.: The Audience Funnel: Observations of Gesture Based Interaction With Multiple Large Displays in a City Center. Int. J. Hum.-Comput. Interact. 27, 562–579 (2011).

14. Houben, S., Weichel, C.: Overcoming Interaction Blindness Through Curiosity Objects. Proc. CHI EA '13. pp. 1539–1544. ACM, New York, NY, USA (2013).

15. Cheung, V., Scott, S.D.: Investigating Attraction and Engagement of Animation on Large Interactive Walls in Public Settings. Proc. ITS '13. pp. 381–384. ACM, New York, NY, USA (2013).

16. Wang, M., Boring, S., Greenberg, S.: Proxemic Peddler: A Public Advertising Display That Captures and Preserves the Attention of a Passenger. Proc. PerDis '12. pp. 3:1–3:6. ACM, New York, NY, USA (2012).

17. Greenberg, S., Boring, S., Vermeulen, J., Dostal, J.: Dark Patterns in Proxemic Interactions: A Critical Perspective. Proc. DIS '14. pp. 523–532. ACM, New York, NY, USA (2014).

18. Beyer, G., Binder, V., Jäger, N., Butz, A.: The Puppeteer Display: Attracting and Actively Shaping the Audience with an Interactive Public Banner Display. Proc. DIS '14. pp. 935–944. ACM, New York, NY, USA (2014).

19. Jurmu, M., Ogawa, M., Boring, S., Riekki, J., Tokuda, H.: Waving to a Touch Interface: Descriptive Field Study of a Multipurpose Multimodal Public Display. Proc. PerDis '13. pp. 7–12. ACM, New York, NY, USA (2013).

20. Müller, J., Walter, R., Bailly, G., Nischt, M., Alt, F.: Looking Glass: A Field Study on Noticing Interactivity of a Shop Window. Proc. CHI '12. pp. 297–306. ACM, New York, NY, USA (2012).
21. Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., Hinckley, K.: Gradual Engagement: Facilitating Information Exchange Between Digital Devices As a Function of Proximity. Proc. ITS '12. pp. 31–40. ACM, New York, NY, USA (2012).

22. Gellersen, H., Fischer, C., Guinard, D., Gostner, R., Kortuem, G., Kray, C., Rukzio, E., Streng, S.: Supporting Device Discovery and Spontaneous Interaction with Spatial References. Pers. Ubiquitous Comput. 13, 255–264 (2009).

23. Annett, M., Grossman, T., Wigdor, D., Fitzmaurice, G.: Medusa: A Proximity-aware Multi-touch Tabletop. Proc. UIST '11. pp. 337–346. ACM, USA (2011).

24. Hespanhol, L., Tomitsch, M., Bown, O., Young, M.: Using Embodied Audio-visual Interaction to Promote Social Encounters Around Large Media Façades. Proc. DIS '14. pp. 945–954. ACM, New York, NY, USA (2014).

25. Karatzas, E.: Proximity Lab: Studies in Physical-Computational Interface and Self-Directed User Experience, (2005).

26. Rehman, K., Stajano, F., Coulouris, G.: Visually Interactive Location-Aware Computing. Proc. Ubicomp '05. pp. 177–194 (2005).

27. Wilson, A.D., Benko, H.: Combining Multiple Depth Cameras and Projectors for Interactions on, Above and Between Surfaces. Proc. UIST 2010. pp. 273–282. ACM, New York, NY, USA (2010).

28. Sodhi, R., Benko, H., Wilson, A.: LightGuide: Projected Visualizations for Hand Movement Guidance. Proc. CHI '12. pp. 179–188. ACM (2012).

29. Ozturk, O., Matsunami, T., Suzuki, Y., Yamasaki, T., Aizawa, K.: Real-time Tracking of Humans and Visualization of Their Future Footsteps in Public Indoor Environments. Multimed. Tools Appl. 59, 65–88 (2012).

30. Rogers, Y., Hazlewood, W.R., Marshall, P., Dalton, N., Hertrich, S.: Ambient Influence: Can Twinkly Lights Lure and Abstract Representations Trigger Behavioral Change? Proc. Ubicomp '10. pp. 261–270. ACM, New York, NY, USA (2010).

31. Grønbæk, K., Iversen, O.S., Kortbek, K.J., Nielsen, K.R., Aagaard, L.: IG-ameFloor: A Platform for Co-located Collaborative Games. Proc. ACE '07. pp. 64–71. ACM, New York, NY, USA (2007).

32. Dalton, N.S.: TapTiles: LED-based Floor Interaction. Proc. ITS '13. pp. 165–174. ACM, New York, NY, USA (2013).

33. Visell, Y., Law, A., Cooperstock, J.R.: Touch Is Everywhere: Floor Surfaces as Ambient Haptic Interfaces. IEEE Trans. Haptics. 2, 148–159 (2009).

34. Mueller, F., Stellmach, S., Greenberg, S., Dippon, A., Boll, S., Garner, J., Khot, R., Naseem, A., Altimira, D.: Proxemics Play: Understanding Proxemics for Designing Digital Play Experiences. Proc. DIS '14. pp. 533–542. ACM (2014).

35. Bird, J., Harrison, D., Marshall, P.: The Challenge of Maintaining Interest in a Large-Scale Public Floor Display. Proc. EIPS '13 workshop (2013).

36. Heun, V., von Kapri, A., Maes, P.: Perifoveal Display: Combining Foveal and Peripheral Vision in One Visualization. Proc. Ubicomp '12. pp. 1150–1155. ACM, New York, NY, USA (2012).