Contact Sensing and Interaction Techniques for a Distributed,
Multimodal Floor Display
Yon Visell, Severin Smith, Alvin Law, Rishi Rajalingham, Jeremy R. Cooperstock
McGill University, Montreal, Canada
A BSTRACTThis paper presents a novel interface and set of techniques enabling users to interact via the feet with augmented floor surfaces.
The interface consists of an array of instrumented floor tiles distributed over an area of several square meters. Intrinsic force sensing is used to capture foot-floor contact at resolutions as fine as 1
cm, for use with floor-based multimodal touch surface interfaces.
We present the results of a preliminary evaluation of the usabilityof such a display.
Keywords: Foot interaction, Multimodal display
1

I NTRODUCTION

Arguably, one factor that has limited the use of foot-based interaction for computationally augmented environments is the lackof efficient interfaces and interaction techniques capable of capturing touch via the feet over a distributed display. In this paper, we present the design of an interface based on a distributed
network of low-cost, rigid floor tile components, with integrated
sensing and actuation capabilities. In order to make good use ofthis system, we draw on contact sensing techniques that are able
to capture foot-floor interactions with much finer resolution thanis achieved if the tile is regarded as the smallest relevant spatial
unit.
2

F OOT I NPUT IN H UMAN -C OMPUTER I NTERACTION

Examples of the use of foot-controlled input in HCI, interactive
arts and video gaming date at least as early as Amiga’s Joypad(1983) [13]. Pearson and Weiser later introduced a foot input device for a desktop PC [10]. Despite the sustained interest in touch
screens for the hands, less research has addressed the design andusability of similar interfaces for the feet. Companies such as
Gesturetek and Reactrix have developed interactive floor-based
visual displays using video sensing technology, but such sensors
provide no direct information about foot-floor contact forces andpositions. Such information is arguably essential for renderinginteractions with virtual objects or controls. Floor-based multimodal (visual, auditory, tactile) information displays have only recently begun to be investigated [14]. In the domain of immersive
virtual environments (VEs), devices for enabling omnidirectional
in-place locomotion in VEs exist [4], but are complex and costly.
Lower cost methods for navigation and interaction in VEs, such as
the shoe-based Step WIM interface of LaViola et al. [7], require
special apparel and provide limited feedback. Most prior workon tactile interaction with floor surfaces utilizes surface sensing
arrays [9, 12], for applications such as person tracking, activity
tracking, or musical performance. Although similar sensing interfaces are now commercially available, costs remain high. Further

differences between tactile sensing and our approach are noted inSec. 4.
3

S YSTEM D ESCRIPTION AND C OMPONENTS

The floor interface (Fig. 1) consists of a square array of 36 rigid
tiles, each of which is instrumented with force sensors (four pertile) and a vibrotactile actuator. The floor is coated in gray projection paint. A pair of overhead video projectors is used forvisual display, in order to reduce the impact of shadows castby users. The tiles are rigid, composite plates with dimensions
30.5 × 30.5 × 2 cm, supported by elastic vibration mounts, andcoupled to a vibrotactile (VT) actuator (Clark Synthesis, model
TST229) beneath the plate. Actuator signals are generated on
personal computers, output via digital audio interfaces, and amplified. The floor tile display achieves a VT passband from 50 Hz
to 750 Hz, and is capable of reproducing the largest forces needed
for interaction with virtual ground surface objects or properties(more than 30 N across the indicated frequency band).
Normal forces are sensed at locations below the corner vibration supports of each tile using a total of four resistive force sensors (Interlink model 402 FSR). Analog data from the force sensors is conditioned, amplified, and digitized via a 32-channel, 16bit data acquisition board. Each sensor is sampled at a rate of up
to 1 kHz transmitted over a low-latency Ethernet link. An array ofsix small form factor computers is used for force data processing
and audio-VT rendering. A separate, networked server is responsible for rendering visual feedback and managing user input.
4

I NTRINSIC CONTACT SENSING

Intrinsic contact based sensing aims to resolve the locations ofcontact, the forces at the interface, and the moment about the contact normals using internal force and torque measurements [2].
It is assumed to involve contact between a rigid apparatus andan object (here, a foot). This approach has mainly been applied
to problems in robotic manipulation, but we have adapted it tofoot-ground interaction sensing. It can be viewed as an efficient
alternative to sensing via dense surface mounted arrays, as it requires far fewer sensors. The method resolves the contact centroid
xc associated with a pressure distribution pR (x) that is distributed
over an area R. xc is a contact point such that a point force Fcthat gives rise to the same intrinsic force measurements as pR (x)does [2]. The sensing problem is simple to formulate for a single
floor tile (Fig. 2), with force sensor locations x j where internal
force measurements f j are taken and j indexes the tile sensors.
We ignore friction effects for the moment. The contact centroid
xc and normal force Fc = (0, 0, Fc ) can be recovered from scalar
force measurements F j = (0, 0, f j ) via force and torque equilibrium equations,4∑ f j + Fc + f p

4=∑ x j × F j + xc × Fc + x p × F p

j=1

0

(1)

j=1

=

0.

(2)

Force sensor
locationsTilesurface

30 cm

30 cm

Figure 1: Left: The floor interface is situated within an immersive, rear projected virtual environment simulator. Middle: Sensing and actuating
components are integrated beneath the floor. Right: View from above showing sensor locations.

F p = (0, 0, f p ) is the weight of the the plate and actuator at the
tile’s center x p . The three nontrivial scalar equalities (1, 2) can besolved for contact centroid parameters:
!
4
4
1
(3)
Fc = ∑ fi − f p , xc =∑ (xi − x p ) fi + fc x p
Fc i=1i=1
The contact centroid lies within the convex hull of the contact
area (dashed line, Fig. 2) at the centroid of the pressure distribution [2], and thus provides a concise summary of the foot-floor
contact locus, but not about shape or orientation. When the footfloor contact area overlaps two or more tiles, the pressure centroid xR for the entire contact area can be computed from contact
centroids xck for each tile (computed from Eq. (3)). It is givenby the weighted average xc = w1 xc1 + w2 xc2 , where wk = Fi /F.
The domain-independence of this result thus makes it possible to
track these points as they move across tile boundaries. Figure 3
represents results of real measurements of 50 estimated contact
positions determined by the method of Eq. (3), using a single calibrated floor tile. Despite distortion near tile edges, contacts werelocalized with a typical accuracy of 1.5 cm, and worst-case valuesof ≈ 3 cm, smaller than the linear dimensions of the tile (30 cm)
or the typical width of a shoe.
A PPLICATIONS : F LOOR TOUCH UI S

These sensing methods can be employed to implement virtual
floor-based touch interfaces. One set of examples we have createdconsist of array of standard UI widgets to be controlled with thefeet (Fig. 4). Input is based on a multi-touch screen metaphor mediated by a set of interaction points (cursor locations), which aredefined as the contact centroids xc with the largest forces. Forcethresholds associated with a control are used to determine selection. The controls provide positive tactile feedback supplied by
the actuators, in the form of synthesized click-like transient vibrations or sliding (friction) vibrations.
f11

5.1 Preliminary User Evaluation
A question we encountered when beginning to design such touchsurface applications concerned the appropriate size of virtual controls. The required size depends on factors including sensing
limitations, users’ motor abilities, target parameters, and feedback modalities, as has been extensively studied and modeled in
the HCI literature [8, 5]. The size appropriate for touch screen
controls has been shown to depend on the interaction techniqueadopted. Precision control strategies can enable single pixel accuracy in finger-based touch screen interaction [1, 11], and related
techniques may prove effective for use with the feet. Limited research has addressed floor interfaces (Sec. 5), so we focused here
on a basic task requiring the selection of controls presented at various locations and sizes to a stationary user.
Human movement research has investigated foot movement
control in diverse settings. Visually guided targeting with the foothas been found to be effectively modeled by a similar version of
Fitts’ law as is employed for modeling hand movements, with an
execution time about twice as long for a similar hand movement[3]. However, the present, preliminary, investigation addresses asituation in which usability is manifestly co-determined by both
operator and device limitations, providing a window on both.
Apparatus The apparatus is the floor interface presented
above. Due to the floor size, the sensor calibration was less accurate by a factor of two than that which yielded the position estimates noted above, but was sufficient for interaction points to be
Positions (x,y) for contact centroids
30

f12

x11

25

x12
Fc

R
xc

pR(x)

f13

f14

x13

x14

Figure 2: A normal force distribution pR (x) and associated contact
centroid position xc .

Position, cm

5Interface design toolkit Interface design is facilitated by asoftware layer and network protocol that abstracts the hardware
systems, which are accessed over a local Ethernet network, andconnects them to the user interface. This software layer processes
the sensor data to extract interaction points, and provides themwith IDs that persist throughout contact. Second, it allows to remotely cue and present VT feedback localized to the area definedby each interface object on the floor. The protocol design is basedin part on the TUIO protocol for table-top touch interfaces [6].

20
15
10
5
0
0

5

10

15
20
Position, cm

25

30

Figure 3: Results of 50 measurements comparing true normal force
positions (circles) with contact centroid estimates (Xs).

Success rate vs distance1

Successrate
vs angle
Success
vs distance
)
1

0.8

$%(
0.8

0.6

$%'
0.6

0.4

$%&
0.4

0.2

$%#
0.2

0

Figure 4: Still image from video of a user interacting with floorbased interface widgets.

effectively tracked over extended distances on the floor, as shownin the supplementary video.
Stimuli and method The stimuli consist of round virtual buttons to be selected by users, who began each trial with their feetin locations marked by white rectangles. Users could activate abutton by pressing it in a way that resulted in a contact centroid
within the area of the button exceeding a force threshold of about
35 N. The buttons ranged in diameter from 4.5 to 16.5 cm, andwere presented at four distances, on lines radiating from between
their feet, oriented at one of two angles, as shown in Fig. 5. Upon
selection, the buttons provided visual feedback in the form of a
20 cm white disc centered in place of the original appearance. All
buttons provided the same feedback. Only the buttons and foot locations were displayed. No audio or VT feedback was provided.
Hypothesis We expected user performance to improve withtarget size and degrade with target distance. Interaction between
target distance and width might be anticipated, but we do not attempt to validate a model. We expected good performance fortargets that are at least as wide as the foot.
Participants Eight participants, ranging in age between 21
and 38, volunteered for this study. All of them were research staff
or students in the Faculty of Engineering at McGill University.
Procedure Participants wore their own shoes during the experiment, and selected targets with their preferred, dominant foot.
They were instructed to activate the buttons precisely and quickly.
The non-preferred foot was not constrained, but participants wererequired to return both feet to the two rectangular regions shownin Fig. 5 between stimuli. Most chose to leave their non-preferredfoot in place throughout each session.
The experiment began with a practice period lasting three minutes, followed by the main experiment. The latter consisted of two
sessions of 12 minutes, with a short pause in between. A total of
240 stimuli were presented to each participant. Stimuli were presented in sequential, randomized order. Each button appeared andremained visible and active for two seconds during which users
were able to select it. A three second pause followed, after whichw
4.5 cm
7.5 cm10.5 cm

d = 15 cm
25 cm

13.5 cm

35 cm
45 cm
θ=π/3

16.5 cm
2π/3

Figure 5: Configuration and stimuli from the experiment.

$0

15 25 35 45
Target distance, cm

Success rate vs width1

Success rate vs width, d<45 cm
1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

4.5 7.5 10.5 13.5 16.5
Target width, cm

15/!"25#/!"35 45
Target radians
distance, cm
Angle,

0

4.5 7.5 10.5 13.5 16.5
Target width, cm

Figure 6: Top Left: Successful target selection rate vs. distance, averaged across the other conditions. Top Right: Success rate vs. angle of presentation (measured away from preferred foot). Lower
Left: Successful target selection rate vs. button width, averaged
across the other conditions. Lower Right: The same measure, excluding the farthest targets.

the next button appeared. The success of selection and time required were recorded. Participants completed a response questionnaire and provided verbal comments afterward.
Analysis Summaries of the success frequencies are presented
in Figure 6. Using a logistic regression analysis, we determinedthat the main factors of width w, distance d, and bearing angle θsignificantly affected success of selection (p < 0.001). The fitted
logit is z = 1.4 + 0.071w − 0.062d − 0.6θ (with t-values > 7.8).
θ is in radians, increasing away from the preferred foot; d andw are measured in cm. The model correctly predicts 86% of the
responses.
Discussion Users selected larger targets within the allotted
two-second interval at a higher rate of success than smaller ones.
Performance with the largest was very high (98%), and that for thesmallest was low (44%). Small targets pose two potential problems. First, as with other touch surface displays, targets can be
occluded by the foot during selection. This problem appeared tobe mitigated because targets could be seen before selection, whileduring selection they were projected on the top of the foot. Second, limitations on precise control can arise from factors such asshoe width, human motor abilities, and sensor positioning errors.
Six out of eight participants reported finding a strategy to activate the small buttons by using a feature of the shoe or changing
the applied force. Software interaction techniques for improving
precise touch screen control are known [1, 11], and we intend
to investigate these in future work. Nearby targets (distances of
D =15 to 25 cm) were selected at a higher rate. However, performance was better at 25 cm than at the nearest distance of 15
cm (98.5% vs. 84%, with p < 0.001 using Fisher’s exact test,
two-tailed). One likely reason is that if an interface element isbeneath a standing user, it can be occluded from view by thebody, or present a difficult viewing angle. Due to such effects
we would not expect selection time T to follow a Fitts’ Law relation, T = a + b log2 (D/W ), but this was not tested here. Althougha mobile user may be able to avoid visibility problems, they seemto be an important consideration. For our device, position sensing
is most accurate near the centers of the tiles, as indicated in thepreceding section. This was noticed by users of the system, two

of whom volunteered that they had learned to better activate small
buttons that were close to edges by pressing them off-center. In
ongoing work, we are developing algorithms for correcting such
positioning distortions. Participants consistently reported difficulty in selecting targets that were oriented away from their active,
selecting foot, however the effect of bearing angle on performance
was small (Fig. 6). It is possible that these responses were indicative of a larger motor effort. Neck fatigue was most frequently
cited by participants as a source of discomfort.
Future work Although these results are suggestive, further
work is needed in order to characterize the usability aspects of thisdisplay, and others like it. A greater understanding of factors suchas control element size, display scale, motor abilities, modalities,
and other aspects salient to the use of such a device will certainly
be needed.
One notable question not addressed by this study concerns theinterplay between users’ movements on foot and their interactions
with the touch surface. A novel aspect is that, implicitly, both
feet are involved, due to requirements of movement and of maintaining balance. In everyday actions, like striking a soccer ball,weight is often shifted onto one foot, which specifies an anchoredlocation, while the opposite is used to perform an action. Floorinterfaces that involve movement may thus be expected to havesomething of the flavor of bimanual interaction in HCI, a connection we intend to explore in future work.
5.2 Floor UIs: Potential application space
Virtual floor controls could be advantageous in some areas ofman-machine interaction in which foot interfaces are common,such as manufacturing, mass transportation, or dentistry. Certain
applications areas, such as those related to pedestrian navigationor map-based visualization, may emerge as particularly salient.
In domains such as medicine, documented problems with existing foot controls might be overcome [15]. Other fields, including
entertainment, gaming, and marketing, were noted in the introduction.
6 C ONCLUSIONSThis paper presented interaction techniques based on intrinsic
contact based force sensing via a novel distributed floor interface.
Such foot-floor contact information is not usually available
through optical sensing channels such as motion capture. The
system is low in cost and complexity, and can be employed bymultiple simultaneous users, without any specialized apparel.
In addition, this paper demonstrates the integration of theseinteraction techniques within multimodal displays implementing
virtual ground surface simulations or floor-based control interfaces. Despite the promising nature of these results, there areseveral respects in which the present system might be improved
or extended:– Our system senses three DOF per tile (the normal force fc andposition xc ). To solve the sensing problem required assuming
frictionless contact (Sec. 4). A future interface for sensing thefull six rigid DOF per tile (via additional sensors) would remove
this assumption.
– A floor interface with a denser array of tiles would be capableof capturing more information about foot-ground contact shape.
– During multi-tile foot-floor contact, a contact-based sensing
approach results in clusters of contact centroids. New techniques
are needed in order to acquire the information arising from suchfeatures.
– Interaction points can be followed only as long as foot-floor

contact is sustained. Methods for tracking users’ feet between
gestures or actions would be desirable.
– As noted more extensively in Sec. 5.1, above, further work
is needed in many areas of usability in order to develop design
guidelines and strategies for floor-based interfaces.
It is nonetheless hoped that this contribution succeeds in highlighting the potential of floor-based touch surfaces.
ACKNOWLEDGEMENTS
The authors acknowledge support from the MDEIE of Quebec for
the EU FP7 project NIW (no. 222107).
R EFERENCES[1] P. Albinsson and S. Zhai. High precision touch screen interaction.
In Proc of the ACM SIGCHI conference on Human factors in computing systems, pages 105–112. ACM New York, NY, USA, 2003.
[2] A. Bicchi, J. Salisbury, and D. Brock. Contact sensing from force
measurements. The Int J of Robotics Research, 12(3):249, 1993.
[3] E. Hoffmann. A comparison of hand and foot movement times. Ergonomics, 34(4):397, 1991.
[4] J. Hollerbach. Locomotion interfaces and rendering. Haptic Rendering: Foundations, Algorithms and Applications. AK Peters, Ltd,
2008.
[5] J. Jacko and A. Sears. The human-computer interaction handbook:fundamentals, evolving technologies, and emerging applications.
Lawrence Erlbaum Assoc Inc, 2003.
[6] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza.
Tuio: A protocol for table-top tangible user interfaces. In Proc of
Gesture Workshop 2005. Gesture Workshop, 2005.
[7] J. LaViola Jr, D. Feliz, D. Keefe, and R. Zeleznik. Hands-free multiscale navigation in virtual environments. In Proc of the 2001 symposium on Interactive 3D graphics, pages 9–15. ACM New York, NY,
USA, 2001.
[8] I. MacKenzie. Fitts’ law as a research and design tool in humancomputer interaction. Human-Computer Interaction, 7(1):91–139,
1992.
[9] J. Paradiso, C. Abler, K.-y. Hsiao, and M. Reynolds. The magic
carpet: physical sensing for immersive environments. In CHI ’97:
CHI ’97 extended abstracts on Human factors in computing systems,
pages 277–278, New York, NY, USA, 1997. ACM.
[10] G. Pearson and M. Weiser. Of moles and men: the design of footcontrols for workstations. In Proc of the ACM SIGCHI conf on Human factors in Comp Sys, pages 333–339. ACM New York, NY,
USA, 1986.
[11] R. L. Potter, L. J. Weldon, and B. Shneiderman. Improving the accuracy of touch screens: an experimental evaluation of three strategies. In CHI ’88: Proc of the SIGCHI conference on Human factors
in computing systems, pages 27–32, New York, NY, USA, 1988.
ACM.
[12] A. Schmidt, M. Strohbach, K. v. Laerhoven, A. Friday, and H.-W.
Gellersen. Context acquisition based on load sensing. In UbiComp’02: Proc of the 4th international conference on Ubiquitous Computing, pages 333–350, London, UK, 2002. Springer-Verlag.
[13] J. Sinclair, P. Hingston, and M. Masek. Considerations for the designof exergames. In Proc of the 5th Int ACM Conf on Comp Graphics
and Interactive Techniques, Southeast Asia, page 295, 2007.
[14] Y. Visell, A. Law, and J. Cooperstock. Touch Is Everywhere: Floor
Surfaces as Ambient Haptic Interfaces. IEEE Transactions on Haptics, 2009.
[15] L. Wauben, M. van Veelen, D. Gossot, and R. Goossens. Applicationof ergonomic guidelines during minimally invasive surgery: a questionnaire survey of 284 surgeons. Surgical endoscopy, 20(8):1268–
1274, 2006.