Spatial Computing
Simon Greenwold
B.S., English & Applied Math
Yale University, June 1995

Submitted to the Program in Media Arts and Sciences,
School of Architecture and Planning, in partial
fulfillment of the requirements for the degree ofMaster of Science in Media Arts and Sciencesat theMassachusetts Institute of Technology
June 2003© Massachusetts Institute of Technology
All rights reserved

Author: Simon Greenwold
Program in Media Arts and Sciences
May 16, 2003

Certified by: John Maeda
Associate Professor of Media Arts and Sciences
Thesis Supervisor

Accepted by: Dr. Andew B. Lippman
Chair, Departmental Committee on Graduate Studies
Program in Media Arts and Sciences

Spatial Computing
Simon Greenwold
Submitted to the Program in Media Arts and Sciences, Schoolof Architecture and Planning, on May 16, 2003, in partial
fulfillment of the requirements for the degree of Master of
Science in Media Arts and Sciences

Abstract
Spatial computing is human interaction witha machine in which the machine retains andmanipulates referents to real objects and spaces. It
is an essential component for making our machines
fuller partners in our work and play. This thesispresents a series of experiments in the discipline
and analysis of its fundamental properties.

Spatial Computing
Simon Greenwold

Joseph A. Paradiso
Associate Professor
Co-Director, Things That Think Consortium
MIT Media Laboratory
thesis reader

Chris Csikszentmihályi
Assistant Professor of Media Arts and Sciences
MIT Program in Media Arts and Sciences
thesis reader

John Maeda
thesis advisor

Associate Professor
MIT Program in Media Arts and Sciences

Acknowledgements
A lot of people helped me write this.
Ben expounded,
Tom explained,
Megan offered,James thought,
Justin twisted,
Elizabeth read,Michael invented,
Missy laughed,
Ollie wondered,
Stephanie reminded,
Axel agreed,
Chris connected,
Joe knew,
John reacted,
Jeanna fed,
Craig prodded,
Tim extrapolated,
Jeana worried,
Ryan subverted,
Michelle gave,
Tad questioned,
Eric responded,
Jake reflected,
Thea felt,
Jess walked,
Sharon painted,
Joanne lavished,
Richard played,
Rebecca conspired,
Jeremy argued,
Mom checked,
Dad worked,
Diana played,
Amanda asked,
Roger understood,
Charlotte smiled,
Jenny did a million things every day.

Contents
1. Preliminaries......................................... 8
1.1 Introduction
1.2 Definition

Part I: Framework811

2. Background ........................................ 13
2.1 History
2.1.1 The Machine in Space
2.1.2 Space in the Machine
2.1.3 Networked Space
2.1.4 The Denial of Space
2.3 The Problems With Realism
2.4 The Problems With Interactivity

13
13
16
20
21
23
30

3. Motivation.......................................... 32
4. Enter Spatial Computing ................... 33
5. Methodology...................................... 34
6. Precedents .......................................... 35
7. Roadmap of Experiments .................. 40
7.1 Installation
7.2 Internaut
7.3 Stomping Ground
7.4 Hotpants/LittleVision
7.5 Pointable Computing
7.6 EyeBox

Part II: Experiments

40
41
41
42
43
44

8. Installation.......................................... 47
8.1 Introduction
8.2 System Description
8.3 Technical Details
8.4 Precedents
8.5 Evaluation and Critique
8.6 Future Work

47
48
51
52
54
60

5

9. Internaut............................................. 62
9.1 Introduction
9.2 Technical Description
9.3 Precedents
9.4 Evaluation and Critique
9.5 Future Work

62
63
65
67
70

10. Stomping Ground ................................ 71
10.1 Introduction
71
10.2 System Description
71
10.3 Precedents
72
10.4 Evaluation and Critique72
11. Hotpants/LittleVision ..........................74
11.1 Introduction
74
11.2 System Description
74
11.3 Technical Details
75
11.4 Precedents
77
11.5 Evaluation and Critique78
12. Pointable Computing .......................... 80
12.1 Introduction80
12.2 System Description
82
12.3 Precedents
83
12.4 Use Scenarios
85
12.5.1 Universal Remote85
12.5.2 Active Tagging
85
12.5.3 Getting and Putting
85
12.5.4 Instant Wiring
86
12.5.5 Reactive Surfaces
86
12.5 Evaluation and Critique87
13. EyeBox................................................ 90
13.1 Introduction90
13.2 System Description
91
13.3 Motivation
92
13.4 Technical Description
92
13.5 Precedents
95
13.6 Design and Operation
97
13.7 Evaluation and Critique
100

6Part III: Evaluation

14. Summary Conclusions.................... 102
14.1 Editing
14.1.1 Suggestion vs. realism
14.1.2 Literalness vs. metaphor
14.1.3 Design vs. engineering
14.2 Qualities of Interactions
14.2.1 Full vs. limited
14.2.2 Feedback vs. one-way control
14.2.3 Predictablity vs. inconsistency
14.3 Shortcuts
14.3.1 Relativity of perception
14.3.2 Opacity
14.4 The Future of Spatial Computing
14.4.1 Technologies
14.4.2 Applications

102
102
103
104
104
105
105
105
106
106
107
108
108
109

15. References ...................................... 111

Appendix A. Other Experiments ............117
Appendix B. SBalls code ........................120
Appendix C. Hotpants tech. docs ........... 129

7

1.1 Introduction
We have arrived at a critical point in the historyof the machine in space. Engineers are rapidly
banishing the last moving parts from consumer
electronics, allowing the machines to shrink intonear invisibility. Bulky CRTs are yielding to flat
panels, allowing us to embed them into the surfaces
we use daily and to free up valuable “real estate” onour desks. The businesses of computer vision andgraphics have pushed our abilities to recover spatial
information from the world at large and represent
it recognizably to the eye. The long-standing divide
between the idealized spaces of computer scienceand the heavy, cluttered spaces of real-world
engineering are wider than ever, polarizing researcharound the world. Now that computation’s denialof physicality has gone about as far as it can, it istime for a reclamation of space as a computational
medium.
The sublimation of the embodied world has notgone unremarked or unchallenged. Criticism suchas Katerine Hayles’ How We Became Posthuman,[Hayles, 1999] explores the intellectual andcultural roots of our sloughing the physical. In thelaboratory, Rodney Brooks has shifted the focusof artificial intelligence research at MIT awayfrom software systems to robots with “embodied
intelligence,” theorizing that intelligence exists only
in relation to a physical world [Brooks, 2001].
Recognizing the benefits of integrating systems into
existing places and practices, research in virtualenvironments is growing to include facets of thereal, albeit often heavily filtered and digitized.
A recent survey of this trend, Mixed Reality:
Merging Real and Virtual Worlds, organizes
these efforts along an axis of “Extent of World
Knowledge” [Milgram, 1999]. According to thisheuristic, a computational system inside a purely
real environment has no need to model the world at
8

all, whereas an entirely virtual environment mustmodel the world in every particular. The systems
organized on this scale and the principles derived
from them properly belong to the field of HumanComputer Interaction (HCI), whose literature hasballooned beyond manageability.
In the middle of all this activity I place my own
research agenda, spatial computing, which borrows
from cultural criticism, theories of HCI, andsystems of mixed reality, but belongs to none ofthem exclusively. This mongrel nature complicatesthe dialectic of its presentation. There is both arhetorical and technical agenda to serve here.
Therefore, in the interest of clarity, I organize mythesis into three parts.
Part I: Framework
I begin with a definition of spatial computing andthen sketch the conditions that give it currency.
Finally I give a brief chronology of the experiments
I have undertaken in the field.
Part II: Experiments
This section fully presents the six completed
projects. They are described roughly as they would
be for technical publication. Each system had its
own local set of concerns and conditions apartfrom the larger context of spatial computing.
The analyses and critiques of each system in thissection, is with regard to its particular objectives.
An experiment (such as Internaut) that was not
an unqualified success in achieving its own goals
sometimes had a significant contribution to thebroader study of spatial computing.
Part III: Analysis
This section presents a combination of technicaland theoretical results. First I present an
incomplete list of guiding principles for the9successful implementation of projects in spatial
computing, synthesized from the technical analysesof each of the projects. Next I offer a guesses about
the futures of the technologies involved in spatial
computing. Finally I describe a sample applicationof the field.
But first we need a definition…

10

1.2 Definition

Figure 1.1. This project, Installation,
allows users to place virtual objects inreal space. It is a good example of spatial
computing. (Discussed in detail below.)

I define spatial computing as human interaction
with a machine in which the machine retains andmanipulates referents to real objects and spaces.
Ideally, these real objects and spaces have priorsignificance to the user. For instance, a system thatallows users to create virtual forms and install theminto the actual space surrounding them is spatial
computing. A system that allows users to place
objects from their environments into a machine for
digitization is spatial computing. Spatial computing
differs from related fields such as 3D modelingand digital design in that it requires the forms andspaces it deals with to pre-exist and have real-world
valence. It is not enough that the screen be used torepresent a virtual space—it must be meaningfully
related to an actual place.
I use “virtual space” broadly here not just to referto three-dimensional Cartesian worlds, but any
space maintained by a computer and supposed
to appeal to a human sense of space. By thisdefinition a “desktop” in a graphical user interface
is a virtual space. Similarly spatial computing does
not necessarily take place in a three-dimensional
representation. For many human purposes a pieceof paper is better understood as a two-dimensionalsurface than a three-dimensional object. In fact,
spatial computing may not present a space to theuser at all. It necessarily maintains an internal
representation of space, even if it is only implicit incollected data, but its interaction with a user neednot be visual or spatial. The simplest example may
be an auto-flushing toilet that senses the user’s
movement away to trigger a flush. This is trivial
spatial computing, but it qualifies. The space of thesystem’s engagement is a real human space.

Figure 1.2. The desktop is a virtual
space. Notice here shading and occlusion.

11

The criterion that the objects and places in spatial
computing have physical instantiation is not an
arbitrary or trivial distinction. There are specificcharacteristics that make the production andanalysis of spatial computing systems different
from purely synthetic virtual systems. Thisdistinction does not imply a value judgment—
virtual systems have their place. However there aremany cases, some discussed below, in which spatial
computing could significantly benefit existing
virtual systems.
It may seem that the category of computational
systems that engage true space is too broad totackle in a single thesis. That is likely true, and I
wish to be careful with the generality of the claims
I make. But I do not think that the diversity inside
the topic defeats the purpose of considering it as awhole. Instead, I think it may be useful to do so inorder to upset a traditional taxonomy, one which
would not allow the analysis of physical systems
next to software systems. In presenting spatial
computing as an organizing principle, I allow
several systems I have engineered to be brought
into analysis together closely enough that they canshed light on one another.

122.1 History
Spatial computing proposes a tight linkage ofthe space in the machine (the space of digital
representation) and the machine in space (physicalpresence). The starkness of the divide owes
something to the fundamental differences of themedia but a great deal also to social and historicalconstruction. In order to understand our present
dilemma it is necessary to examine a history ofcomputation in physical space.

2.1.1 The Machine in SpaceThe earliest machines designed as engines forcalculation did not try to deny their physicality.
They wouldn’t have because they were purely
mechanical devices. The abacus, from about
600 BC, for example, encodes numbers entirely
spatially. It is programmed as it is read, in position.
Here there is absolutely no abstraction of space.
Data space is physical space [Ishii, 1997].

Figure 2.1. The abacus is a physicalcomputer not only in its computation,
but also in its input and output.

Early computers couldn’t help but be spatial.
They took up space, and they used the nature andqualities of the physical world to perform theirwork. This continued to be true as the calculatingmachines abstracted their input and output awayfrom physical configuration to digital displays, as in
Blaise Pascal’s mechanical adder of 1640.
The critical shift did not occur until electrical logic
became cheaper, smaller, faster, and more reliable
than physical switching. Motors and gears gave way
to tubes and wires. Suddenly physics, which had
been a computational medium became an enemy
to be conquered. Computers were too big and too
heavy, and things needed to get denser. Initially,
computers were made into furniture as in 1957’s

13IBM 705, which doubled as a desk in order to makeits outlandish size more palatable.
Transistors, of course, proved to be the vehicle
for shrinkage. As they replaced tubes, computers
became objects in space as opposed to defining
their own spaces. The rest of this history is common
knowledge, how the computer shrank and shrank
until we began to fold them up and put them inour pockets. But what does this neutron-star-likecompression imply?
First, it puts a clear value-system in place: forcomputation smaller is better. This seems obvious,
but it is not the case for many things—houses andsnack food, for instance. There is a clear advantage
to a computer that is small enough to carry. And
physical space has emerged as perhaps world’s
primary limited resource. But we never seem tostop the furious miniaturizing, and that has to dowith computing power. The outsides of electronics
have on whole stopped getting smaller. We havealready seen cellular phones hit an uncomfortablelevel of tininess and bounce back somewhat insize. Things that are of the body must remain
proportionate to it, but the computational cores ofelectronic objects are not bound to the body. If they
are, it is only as added weight to be minimized. The
parts of computation that are necessarily humanscale are the points at which the machine meetsthe user—input and output. So there is a tension
introduced as the limits of human physiology keep
computers from spiraling into nothingness, but at
the same time we must keep making the insides
smaller so that the objects themselves can become
more powerful.
No one feels this tension more acutely than theelectronics hobbyist. Traditionally integrated
circuits, the bread and butter of any reasonably

14

Figure 2.2. The IBM 705 from 1957.
The computer as furniture.

Figure 2.3. Not so long ago computers
made their own spaces. The 1951 Whirlwind computer.

Figure 2.4. The SG22oo from Sewon
claims to be the smallest cell phone.

complicated electronics project, have been available
in packages of sufficient size to allow them to behandled with fingers and soldered by hand—DIP“dual inline packages,” for instance. But manyof today’s technologies such as BlueTooth areavailable for use only in packages with leads somany and so small that no human being could
reasonably expect to manipulate them. Thesetypes of chips are produced for companies who
design circuits on a computer and then have themassembled by robots. This happens, of course,
because the economics of serving a hobbyist
population doesn’t justify the expenditure. But
there is the feeling that consumer electronics
technologies are shrinking away from accessibility
to individual experimenters.
The physical shrinkage of the machine manifests
itself as an embarrassment of the flesh. The
thinner the notebook computer, the better.
Computation is an anorexic industry. As Katherine
Hayles understands it, we owe this desire for theeradication of the body to

Figure 2.5. Current phones are larger
than they were. Now they hide behindlarge color displays.

a conceptualization that sees information andmateriality as distinct entities. This separation
allows the construction of a hierarchy in which
information is given the dominant position andmateriality runs a distant second... embodimentcontinues to be discussed as if it were a
supplement to be purged from the dominantterm of information, an accident of evolution weare now in a position to correct. [Hayles, 1999;
12]
Spatial computing proposes to celebrate
corporeality of data rather than trying to deny it.

Figure 2.6. Cygnal proudly offers us
the C8051xxx microcontroller family.
Good luck soldering that one. [http://www.cygnal.com/]15

2.1.2 Space in the Machine

Figure 2.7. The 1982 film Tron demonstrated the cultural fascination with andfear of being swallowed by the machine.

Our fascination with the space inside the machine
is not new. The Aristotelian universe was essentiallya mechanical system that described planetary
motions as part of a giant machine. Describing life
inside space stations and bubbles large enoughto hold populations has been the bedrock ofscience fiction for as long as we’ve had it. In 1964
Archigram even reimagined the city as a huge
walking robot that could dock with other cities.
At least as far back as the Renaissance, artists suchas Durer used machines to help them represent
space. In the second half of the twentieth, however,
the growing internal power of machines began toallow them to represent spaces and objects directly
to our eyes, often spaces and objects with noreferent in the real world. Computers turned out
to be masters of perspective and simple shading,a few of the artist’s simplest tricks for conveying
depth. Suddenly there appeared to be whole open
landscapes inside the machine.
And as the outsides of the machines shrank and the“space” of memory and storage inside exploded, it
became possible to popularize the idea of moving
16

Figure 2.8. In 1959, the DAC-1 (Design
Augmented by Computers), developedby General Motors and IBM, became thefirst interactive 3D computer graphics
system.

Figure 2.9. Archigram’s Walking City,
1964. [Herron, 1964]

ourselves wholesale out of messy old real space
and into virtual space. A magnetic core memory of
4,000 bits weighed tons in 1951, but now (April 9,2003), we store a billion bits on a chip the size of afingernail. The scarcity, expense, and imperfectionof real land made the idea of a boundless internallandscape too tempting to resist. This notion was
also greeted with anxiety as demonstrated bymovies such as Tron and Lawnmower Man, in
which humans are sucked into and trapped inside avirtual environment.

Figure 2.10. 4,000 bytes of memory
from the 1951 Whirlwind computer,standing roughly 9 feet tall. Today weput 1,000,000,000 bytes on a single
chip.

Figure 2.11. A typical representation ofa machine-generated “space” from thesoon to be released online environment,
Second Life [Linden Labs, 2003].

Early computer-generated spaces tended to be(and still often are) rigidly planar expanses of
exaggerated linear perspective. Lines are straight,
corners are perfect, and ornamentation is minimal.
Interestingly this represents something of a returnto Modernist form. In 1908 in an essay entitled“Ornament and Crime,” the architect Adolf Loos
wrote, “The evolution of culture is synonymouswith the removal of ornament from utilitarian
objects” [Loos, 1908]. This spare, functionalist
notion cast a long shadow over architecture. Miesvan der Rohe’s architecture, for instance, exhibits
what he called “universal space” and the “openplan.” It results in floating planes and broad

Figure #: The Barcelona Pavilion by Mies Van der Rohe. One of
the few real spaces that looks virtual.

17

gridded plazas. Lev Manovich also finds a return toa kind of “soft modernism” in the aesthetics of the
web [Manovich, 2002].
The functionalist rationalization of pure forms(“Form follows function” [Sullivan, 1918]) paved
the way for Le Corbusier, in many ways thefather of Modernist architecture, who famously
called the house, “a machine for living in” [LeCorbusier, 1923]. This was possible to espouseat the turn of the century, but could not survive
the deconstructive assault of Derrida and others
that followed World War II: “It is now a familiar
story how deconstruction exposed the inabilityof systems to posit their own origins, thus
ungrounding signification and rendering meaning
indeterminate” [Hayles, 1999; 285].

Figure 2.12. Villa Savoye a Poissy by Le
Corbusier, who famously called a house“a machine for living in.”Further, these “functionalist” spaces, pristine intheir first realization, did not stand up well toweather and time. They were extremely difficult tobuild and maintain. Interestingly it is exactly theirease of production and maintenance in machinesthat keeps them present as virtual architecture
although they had faded from prominence inphysical architecture before the first computer
graphics arrived.
What this really serves to demonstrate is thatform follows economics of production. Computers
make it cheap and easy to make clean corners,
so that’s what we see. Baseboards help cover up
irregularities in physical meetings of wall and floor,
so most real buildings have them. That virtualenvironments are becoming more detailed andmore topographically complex is due to improved
tools for their construction and deployment. Thereseems to be little ideology driving the developmentof a virtual “style” except for the quest to do
whatever technology has made newly possible. See

18

Figure 2.13. It is obvious the floating
object is artificial because its colors are
too consistent, its lines and corners too
sharp.

for example the web sites of makers of graphics
hardware, such as nVidia [www.nvidia.com]. The
features of the cards drive the aesthetics of virtualenvironments.

Figure 2.14. A bronze Buddah rendered with a procedurally-generated
patina. [Dorsey, 1996]

Figure 2.15. Solutions such as this may
have difficulty catching on.

Figure 2.16. Robert Venturi and
Denise Scott Brown see the world as alayering of signs and symbols. [Venturi,
2001]

One of the hallmarks of the unreality of virtual
spaces is their over-perfection. On a computer
screen, things look like their ideals (or at least
anything with corners and flat faces does). A line
is a line and a cube is a cube. These images areunconvincing because we know that there is noreal substance looks so perfect. Much time andattention in computer graphics nowadays goes
toward making things look imperfect enough to beconvincing [Dorsey, 1996], [Paquette, 2001]. It is a
hard problem, and it isn’t yet solved.
Computer graphics’ primitive appeal to an
impossible purity makes the idea of virtual
space feel somewhat immature and naive, andits throwback to long outgrown architectural
ideologies doesn’t help either. The proponents ofvirtual environments have suggested without irony
that we use systems that make us look like cyborg
monsters. There really isn’t anything appealing
about this vision to many important sectors ofculture. All of this leads to some deserved ridicule
surrounding the field of virtual reality.
Where computer graphics diverges completely fromspare modern spaces is in matters of graphicalinterface. The collapse of Modernism brought
forward the dominance of the symbol. Architecture
blossomed with overt historical quotations andreferences. Robert Venturi and others recognizedthat there is no form that does not carry infinite
layers of meaning. What is suggested is as real as
what is physically present. This is the languageof graphical user interface, where the icon reigns
supreme, and language is larded over the top of
everything.
19

This mess of signifiers is pretty much where
software spaces remain today. Spatial computing
does away with icons, lists, and menus as much as
possible, to allow things to stand for themselves.

2.1.3 Networked SpaceThe advent of the Internet considerablycomplicated the relationship of computation to
space. Suddenly connections made inside themachine had the potential actually to span half the
globe. Every screen became a portal onto the same
shared, parallel world.
The bright side was the promise of an end tosolitary virtual existence, replaced by virtual
networked communities. And it is true that muchof Internet traffic consists of e-mail and instantmessages. However, one of the strange qualities
of web space is that the user is always alone init. No matter how many other people are looking
at the same information, no one sees each other.
Everyone has the feeling of having the entire vast
Internet to themselves.
People saw the expanding World Wide Web itself asa kind of virtual space, and it did take root even if
it didn’t replace the physical world as many feared.
It seemed that the Internet could act as a kind of
spatial prosthesis, a vastly enhanced telephone.
Everything on the web is a single address away—aclick, maybe two. (Advertisers consider three clicks
to be an unacceptably long “distance.”) But whatis the product of total equidistance if not collapse
into singularity? It’s not a new spatiality, it’s a nonspatiality.
The spatial analog of the hyperlink would be theteleportation terminal. Such a terminal opens a

20

Figure 2.17. Venturi and Scott Brown’svision of architecture makes the world
look a lot like software interface.

hole in space, a discontinuity which we cannot
contextualize in actual space specifically because
it is not of real space. If the essence of spatialconnectedness in a network is supposed to be thehyperlink, we are building a space out of holes inspace, which is like trying to constitute a sponge outof its voids.
A further problem of networked space results fromits devaluation to the point that it can be minted
practically for free. Space becomes valueless. As
soon as some space becomes ruined, we can justmake another new one twice its size. There isno limiting condition to generate value. Perhaps
what makes the Internet valuable is that it is nonspatial and attempts to introduce space to it may befundamentally flawed. (I will have more to say onthat in my analysis of my own attempt to do this,Internaut.)

2.1.4 The Denial of Space
The Internet is not the only agent of spatial denialin computer science. The dream of escaping theimperfect and unpredictable real world is theengineer’s heaven. It is a denial of heaviness,friction, death, and decay. The memory spacesof computer science are the site of huge projects
in idealized engineering—where programmers
construct machines of astonishing complexity inthe absence of gravity and corrosion.
Escape from the uncontrollable and capricious realworld into a perfect world of rules, where every
consequence has a cause if one knows enough todiscover it helps explain the motives of strange
hackers and virus writers who measure theirsuccess by the quantity of their access and breadthof their spread. These people, powerless in the realworld, are masters of the machine, perfect in its

21

willingness to do as it’s told. The author can attest
that this kind of power can be very compelling toa young person who longs for a ordered world inwhich every problem eventually yields to logic.
It also helps explain why virtual spaces have had
only one resounding area of success—violent firstperson games in which players shoot each other atwill. These scenarios appeal to the same crowd of
teenage boys.
Absurdity grows like a barnacle at sites of cultural
tension. All it takes is a look at the size andcomplexity of the heatsinks that accompany any
modern microprocessor to know that engineering
is engaged in a fight with physics. We are poised ata point of extreme tension in the spatial relationsof computation. I propose a computation thatembraces the machine as a spatial object at thesame time integrating it with the space insideitself. Spatial computing shares an aspiration withKatherine Hayles’s vision for “posthumanity:”
my dream is a version of the posthuman thatembraces the possibilities of information
technologies without being seduced byfantasies of unlimited power and disembodied
immortality, that recognizes and celebrates
finitude as a condition of human being, andthat understands human life is embedded ina material world of great complexity, one on
which we depend for our continued survival[Hayles, 1999; 5].

22

Figure 2.18. The size of this heatsink
relative to its host is a sign of the heroic
struggle of technology against the physical world.

2.3 The Problems With Realism
Something that tends to go unchallenged is therealism of virtual spaces. The increasing power ofprocessors and graphics cards enables more andmore accurate modeling of the physics of light andthe mathematics of surfaces. As Lev Manovich
understands it, realism has become a commodity
we can pay more to buy more of [Manovich, 1996].
But there is a subtlety that is missing from all of themarketing and analysis of virtual systems.

Figure 2.19. The Cornell box is thebenchmark for photorealism. Rendered images are compared against
pictures taken inside a real box.
(This one is rendered.) [http://
www.graphics.cornell.edu/online/box/
compare.html]

There is a tremendous disconnect between screenbased representations of reality and experiential
reality that makes increasingly accurate physicalsimulations somehow less engaging than it seems
they ought to be. The computer graphics term for
rendered realism is “photorealism,” and that hints
at the problem. The realism that computation tends
to aspire toward is the realism of a photograph.
Human beings do not experience a photograph as
an instantaneous and engaging reality in which they
are part. They do not imagine the camera’s eye tobe their own. They remain firmly outside the image,and understands it usually as a captured momentof objective representation. It is undeniable that
there is something compelling about the asymptotic
approach to photorealism. Increasingly accurate
renderings continue to inspire wonder even now
that the game of chasing reality has grown old.
But the wonder masks an important distinction that
virtual reality denies. The wonder is the wonderthat the image was not produced by a camera,
not the wonder that the viewer was not present as
the perceiver of the scene. There hangs above thediscipline a notion that we are just a breath awayfrom producing representations that are sufficiently
accurate to fool the viewer into total engagement. It
can’t happen that way.
23

This confusion of “realism” is apparent fromlooking at the use of the term “realistic” as it isapplied to computer simulations such as games.
Sega’s basketball game NBA 2K3 is hailed all overthe Internet as the most “realistic” basketball gameever to be produced. What this seems to mean isthat the players bodies and faces are taken fromreal NBA players and the camera shots look like
television coverage of basketball. The view is not
first-person from a player in the game, and not evenfrom a fan. Instead “realistic” here means creating
television with your thumbs. This could hardly be
farther from the reality of a player in the game.
This is again evident in the popular, “behind yourown back” view in first-person games. It is often
possible to switch the first-person viewpoint, which
is supposed to correspond to the player’s eyesight
to a view that is over the player’s own shoulder
or behind them. This is often more convenientfor game-play because it shows the player in thecontext of the scene. The movement of the camera
from the virtual eye—the most privileged viewpoint
imaginable—to some arbitrary but convenient
location outside the virtual body ought to induce
extreme disorientation and shatter the vital illusion.
But it doesn’t. It is surprisingly easy to digest, andrequires no special stretch of the imagination toaccommodate. The ease with which this translationis accepted indicates that the egocentric view does
not lay special claim to the engagement of players
as if it were their own eyesight.
This has everything to do with the nature ofperception. The fundamental discovery of art andthe physiology of perception since the Renaissanceis that the eye is not a camera. Vision is aconstructed sense. We have a tiny area of acuitywith which we constantly and actively scan the24

Figure 2.20. Sega’s NBA 2K3. Widely
touted as “the most realistic basketballgame ever.” [http://www.epinions.com/
content_85992509060#]

Figure 2.21. The view from behind
yourself in One Must Fall Battlegrounds.
[http://thegamebd.tripod.com/
previews/OMF/Omf.htm]

world. Any notion of a photographic experience ofa real scene is one constructed by the brain. Thisis different from the experience of a photograph,
which appears as a small colored patch in ourfield of view. We can understand it as it relates toour experience of the visual world, but it does not
mimic our experience of it.
There is nothing “natural” about a rendered
perspective projection. It is intelligible, but it isn’t
how we see things. In some cases, increasingly“realistic” representations only serve to alienate
us from what we are seeing. For instance, in
the Quake II engine from Id Software, as theprotagonist walks, the view bounces up and down.
It is particularly noticeable when the character iswalking close and parallel to a textured wall. It isa bizarre sensation to watch the representationof space bob up and down as the player moves
forward. But if one looks closely at walls while
walking in the real world, it actually does the samething. We just filter it out so we don’t even noticeit. In our minds, walls don’t bounce. So which is
the more “realistic” representation? There is aperfectly valid argument that whatever alienates
the viewer less is the more realistic. Game players
say that after a while one ceases to notice thebouncing, just as presumably, we cease to notice
it in the world because it is always present. But I
expect that learning to ignore this effect is the samekind of learning that allows players to meld their
being with a paddle in Pong. They simply ignore theclear signals that tell them there is an other reality
outside of this small area of focus as if it were notthe case.

Figure 2.22. Eye-movement traces as a
subject explores a picture of the bust of
Nefertiti. [Yarbus, 1967]

E. H. Gombrich points out that vision proceeds
not as construction of a literal image but asprogressive hypothesis testing against actively
acquired percepts [Gombrich, 1969]. We refine25

our understanding of the world by actively testing
it with our eyes, which are attached to our heads.
That means if there is an uncertain condition
to our right, we may turn our heads. Any visual
information we gather there is added to our mental
image of the scene in front of us, but the image is
as much constructed in reverse from beliefs andmemories as it is from light hitting our retinas.
In fact patients with lesions in their visual cortex
called scotomas, which make it impossible toperceive anything from large patches of their visual
fields, often fail to notice they have a problem[Pessoa, 2003]. The process by which the brain
stitches the hole with pieces of what it thinks ought
to be there is called “filling-in,” and it is an areaof active research. In fact we all such a blind spot10˚ off the center of our visual focus, but we nevernotice it unless we design a specific test to discover
it. A photographic image may well describe ourmental construction of what we perceive, but it
vision requires quite a bit of editing to produce
what we “see.”A photograph has done all the work of perception
for us. In its pure instrumentality, it supersedes theeye. We are often inclined to trust it more than our
vision or our memories. It sucks the subjectivity
out of seeing, and allows us to revisit an instant (afiner time slice, by the way, than was ever possible
for our eye) at our leisure. We scan the photographusing the same mechanisms of active perception weuse in the world, and certainly this helps us focus
on different parts of it, but it does not help with any
ambiguity that is frozen into the photograph.
There are many conditions that appear visuallyconfusing in photographs that could never be inreality. Sometimes a tree appears to be growing
out of a person’s head. We almost never getthat impression in reality. The active quality of

26

Figure 2.23. Every eye has a blindspot where the retina is joined tothe optic nerve. We don’t notice it.
[www.nidek.com/blind.html ]perception disambiguates the situation before it
even becomes questionable in reality. For instance,
there is always motion in the real world, andthere will be differences in the relative speedsof motion of the tree and the head in the visual
field. This effect, called head-motion parallax, ismore important to our perception of depth thanstereopsis [Arthur, 1993]. Our ability to perceive
is distinctly limited in virtual realms because
the system cannot possibly respond to all thetechniques for active perception that we use. Somesystems try to allow for it by using gaze or headposition tracking, but the instrumentality of thesystem always acts to flatten some of the nuancethat we rely on in our sensing dialog with the world.

Figure 2.24. A VR “CAVE” projects
10’ X 10’ images on four sides. [http://www.eds.com/services_offerings/vr/
centers_the_way.shtml]

Systems that use specialized hardware to try toreplace as much of a subject’s sensory input withsynthetic representations of information are
called “immersive,” and they all suffer the same
unavoidable problem. We have no experience of“immersion” in our real existence. We are partof it, and it is part of us. There aren’t even clear
boundaries between the self and environment—it
has been contested for centuries. When subjects
are “immersed” in a virtual simulation such asa “CAVE,” which projects images on 10’ squarewalls all around a subject, they have an experienceof immersion, which is distinctly not a quality ofreality. Immersion is like diving into cold water.
One of reality’s hallmarks is that its experienceis mundane, and any excitement it contains
comes from the thing experienced, not the act ofperception.
Paradoxically, the disconnect with reality become
most apparent in the most “realistic” simulations.
The more the viewer is supposed to be relieved ofthe effort of constructing a reality out of partial
information, the more obvious it is when it fails.

27This is why an artifact such as an aliased haloaround an otherwise well-rendered character is
so disturbing, or why the slightest anti-physicalmovement is so disruptive. Sidestepping thisproblem is the considerable work toward “cartoon”
renderings of virtual systems, which trade efforts at
photorealism for abstraction [Kline, 1999].
Abstraction is a powerful technique that chooses
to edit the qualities of a representation consciously
away from perfect concert with perceived reality.
Such an editing allows for parts of the scene that
do not contribute to authorial intent to be strippedaway or effaced, and other parts enhanced, offering
the viewer only the essence of the communication.
Abstraction, like photography, does some of thework of perception for you, but it goes one stepfurther, not aiming to mimic the retinal image, but
the ideal image. Of course abstraction can proceedbeyond increased legibility into opacity, but this has
an artistic agenda more difficult to unpack.
The virtual cartoon dog Dobie T. Coyote from Bruce
Blumberg’s Synthetic Characters Group at the
MIT Media Lab is a perfect example of abstractionused to enhance communication. His movements
and facial expressions are in a way hyperreal
since they are a distillation of the visual cues we
use to recognize a dog’s mood and behavior. It is
important to distinguish this kind of abstraction
from metaphor. The dog does not stand for a dog
simply because he is not “realistically” rendered.
There is no system of equivalence established towhich the dog as a symbol is appealing. We perceive
the cartoon dog as a specific dog and nothingelse. What abstraction does necessarily do isuniversalize, which has the effect of eroding certain
particulars in favor of others. However, in the caseof a virtual construct that has no referent in the realworld, the missing specificity pre-exists nowhere.

28

Figure 2.25. Screenshot from the AMP
II game engine. Almost realistic but
not quite. [http://www.4drulers.com/
amp.html]

Figure 2.26. Dobie T. Coyote from
Bruce Blumberg’s Synthetic Characters
Group. [http://web.media.mit.edu/~bruce/whatsnew.html#Anchor_new1]

Abstraction successfully avoids the vain attempt tobuild a virtual simulation with all of the detail andrichness of the real world.

Figure 2.27. Video conferencing facilities are available at the New Greenham
Park Hotel. [www.greenham-commontrust.co.uk/ images/video.jpg]

This same failure through slavish adherence tomaximizing sensory information is apparent
in the virtual reality of telepresence, in which a
non-present party is brought into “presence” by avirtualizing technology. In all of the telepresence
systems I have witnessed, the most obvious qualityof the remote parties is their non-presence. The
technology that is supposed to bring them closer
only serves to emphasize their distance from thegoings-on.
Having, experimented with webcams for personalconnection to help maintain a long distance
relationship, I can attest to their inadequacy. (Wewent back to telephone only.) Often a mentallyconstructed or abstracted reality is more compellingthan a sloppily constructed representation of afuller set of sensory information. Readers usuallyfind this the case with film adaptations of books
they love.
The inadequacies of “realism” in virtualenvironments make it worthwhile to look foralternative modes for dealing with the space inside
the machine.

292.4 The Problems With Interactivity
Where the problems of realism are problemsof space in the machine, the problems with“interactivity” are problems of the machine inspace.
There is an irony in the use of the words“active,” “interactive,” and “reactive” to describe
computational objects—both physical and virtual. It
is a common practice, as though nothing had thosequalities until the computer swooped down andstarted endowing ordinary objects with buttons andmicrophones. The truth is that non-computational
objects are far more active, interactive, and reactive
than any working computational version of thesame thing. The reason is that in order to consider
an object computationally, we must derive datafrom it, and that means outfitting it with sensors
in some way. As soon as we do that, we chop away
all of the interactions we have with that object thatare not meaningful to the specific sensor we havechosen. No matter how many sensors we add, weare taking a huge variety of interactive modalities
and reducing them to several. How could asimulation of a cup ever be as interactive as a cup?
Some argue that adding sensors to a physicalobject does not constrain its existing interactivity,
but augments it electronically. I believe thatis true as long as the object remains primarily
itself with respect to the user and does notundergo some metaphoric transformation into avirtual representation of itself or into a semantic
placeholder. That is difficult to achieve, andcannot be done as long as users must consult asecondary source to determine the quality of theirinteractions. For users to check a screen or evento listen to a tone to determine the pressure withwhich they are squeezing an object supersedes their30

own senses and reduces any squeezable object to apressure sensor. In order for a physical object to beaugmented rather than flattened by computation,
the computation must occur (or appear to occur)
inside the object and the consequences of thecomputation be registered by the object. The object
must also not become fragile or restricted in its
manipulability.
This challenges the claim of mouse-based Flashauthoring to be “interactive design.” It is perhaps asinteractive as a phone book, but it certainly isn’t asinteractive as an orange. In order for us to design
computational objects that achieve that level ofinteractivity we will have to concern ourselves withmore than the screen. The physical body of thecomputational object is vital to its interactivity.

31

3. Motivation
My motivation as a researcher stems from a desire
to address some of the deficiencies in the current
state of our interactions with machines, as detailed
above. But there is a personal motivation as well.
My family used to take trips to national parks.
These were some of my favorite vacations because
I liked to walk inside landscapes that were muchlarger than I was. I liked to be able to see things
distantly and then gradually to approach them andfind them to be even more richly detailed than I
had imagined them. I was a computer child too, so
I often thought about this in terms of resolution
and quantization—how the strongest flavors of
the real world were due to its infinite resolution.
Every pinecone had something interesting to say
under each one of its scales if you took the time to
examine it with eyes and fingers. No simulation I
had ever experienced had that power. They reachedonly as far as the attention of their creators. But
I dreamed of making that simulation. My fantasy
was to be able to take flight from where I stood andzoom in any direction to close in at high speed onanything that caught my interest. I would be able toexperience it in all of its detail.
The fantasy has faded, but what hasn’t left me isa love of the real. What I understand better now
are the limits of computation. I no longer dreamabout producing such a system inside the machine.
Instead I have turned my attention to an idea that
I think holds more promise: the integration of thereal and computed. Rather than try to simulate
the qualities of the world I love, why not let theworld stand and be present in all its complexity. I
have been trying to make systems that engage thephysical world rather than deny it.

32

Figure 3.1. I vividly remember Bryce
Canyon in Utah. [http://globetr.bei.tonline.de]

Of course, thanks to the house, a great
many of our memories are housed, andif the house is a bit elaborate, if it has acellar and a garret, nooks and corridors,
our memories have refuges that are
all the more clearly delineated. All
our lives we come back to them in our
daydreams.
—Gaston Bachelard[Bachelard, 1964; 8]

“We are continually living a solution ofproblems that reflection cannot hope tosolve.”— J. H. Van den Berg
[Van den Berg, 1955]

4.

Enter Spatial Computing
Spatial computing proposes hybrid real/virtual computation that erodes the barriers
between the physical and the ideal worlds. It is
computational design arising from a deep belief inphenomenology. This emphasis delimits it inside
the body of HCI practice, the majority of which
concerns itself exclusively with what it can measure.
Spatial computing is more interested in qualities
of experience. Wherever possible the machine inspace and space in the machine should be allowed
to bleed into each other. Sometimes this means
bringing space into the computer, sometime thismeans injecting computation into objects. Mostly
it means designing systems that push through thetraditional boundaries of screen and keyboard
without getting hung up there and melting into
interface or meek simulation.
In order for our machines to become fuller partners
in our work and play, they are going to need to join
us in our physical world. They are going to haveto operate on the same objects we do, and we aregoing to need to operate on them using our physicalintuitions:Because we have considered the relation between humanand machine as instrumental and prosthetic (subordinating
and conforming machines to the requirements of human
instrumentality), and even more because we have createdthem in the image of an ideally isolated individual, we
have denied our computers the use of the shifters (here,
now, you, we…) that might transform their servitude into
partnership [Cubitt, 1998; 35].

If we are not already, we are bound to become
human beings embedded inside our connected
machines. We will be the processors working within
the giant spatial networks that surround us. How
will we use space, place, and objects to direct thatcomputation?

33

5. Methodology
My goal at the Aesthetics + Computation Grouphas been to attack the boundaries between physicaland virtual spaces with small incursions from all
sides. Therefore my explorations have been manyand various in media and scope. Some have been
more about place, some more about objects. Each
one has led me further toward understanding
spatial computing. I imagine each of the projects
I developed as a component that could be
integrated into future systems that more powerfullycomplicate the real and virtual than any of themtaken singly.
In the course of my study my primary method has
been to make things first, and ask questions later.
This process privileges intuition over scientificinquiry because it does not produce artifacts
designed to test hypotheses. It is an engineering
methodology driven not by a functional brief but
instead only by the demand that the products havebearing on a broad set of concerns.
I have taken pains as I produced these projects toallow them to change as I made them to take theirbest course. It becomes clear only in the processof making what a thing’s most valuable form will
be. This freedom to allow ideas to change as they
became real has made my work better. Nothingleads to more tortured and awkward instantiationsof ideas than rigidity of purpose.
It was not always clear to me as I worked what theconnections between my projects were, and it has
required a period of introspection, a reprieve frombuilding, to hear what they have to say. The theory
has arisen from the artifacts, not the other way
around, and that is the only reason I have faith in it.
As William Carlos Williams said, “No ideas but inthings.”34

6. Precedents
Spatial computing is not a new discipline. It is
located well within the theoretical boundaries ofexisting fields and programs of research such as
HCI, Ubiquitous Computing, Invisible Computing,and Tangible Interfaces. Each of these is committed
in its own way to bringing together human andmachine space. All of the experiments I undertook
could easily belong to several of these. The
distinctions I mean to emphasize between themand spatial computing are small but importantdifferences of focus and in some cases philosophy.
HCI is the umbrella over all of these fields. Theinteraction between human and machine can beconstrued in many ways.
Ubiquitous and Invisible Computing
In 1988 at the Xerox Parc research lab, Mark
Weiser first articulated the idea of Ubiquitous
Computing. It was a call for technology to recede
into the background of our lives where we would
not even notice its existence:
Our preliminary approach: Activate the world. Provide
hundreds of wireless computing devices per person per
office, of all scales (from 1” displays to wall sized).
This has required new work in operating systems, userinterfaces, networks, wireless, displays, and many otherareas. We call our work “ubiquitous computing”. This isdifferent from PDA’s, dynabooks, or information at yourfingertips. It is invisible, everywhere computing that does
not live on a personal device of any sort, but is in the
woodwork everywhere [Weiser, 1988].

Weiser understood Ubiquitous Computing tobe the opposite of virtual reality. It admits no
possibility for the machine to represent space backto a user. It is the machine dissolved into space.
This is closely related to the invisible computer,as espoused by Donald Norman [Norman, 1998].
Norman has trouble both with graphical user35

interface (because it doesn’t scale gracefully to thelevel of complexity of modern software) and virtual
realities (because they confuse visual information
with spatial experience):
These well-meaning but naive technologists confuse what
people see when they move about the world with what
they experience. If we look only at the visual information
to the eye, there is no difference between moving the headto the left while the world stays stationary and moving theworld to the right while the head stays stationary. The eye“sees” a moving image in both cases, but our experiencesof the two cases are very different [Norman, 1998; 101].

Norman urges a move away from the multifunction
PC to the single-purposed “information appliance,”
whose operation is so obvious that it does not
require instructions. He advocates the purging of
metaphor from interface.
I primarily agree with Norman and Weiser’s
critique of existing interface practice. Spatial
computing also does not employ graphical userinterface (GUI) because it is metaphorical rather
than direct. And spatial computing also criticizes
pure virtual environments as they are commonly
implemented. But it certainly does not advocate
invisibility of the machine in every circumstance.
And neither does it shun spatial representationby computers. Spatial computing demands onlythat we be careful with visual representations ofspace, and rather than harbor vain hopes for theirreplacement of experiential reality, link them inmeaningful ways to existing spaces and objects.
Perhaps the most important difference between
spatial computing and Ubiquitous/Invisible
Computing is that spatial computing is notpolemical. I acknowledge and accept the obvious
benefits of GUIs. There is no way that physicalreality can offer the kind of chameleon flexibility
and complexity that instant metaphor can. I do nottherefore advocate the replacement of the PC as it
36

exists with spatial computing. Spatial computing
is inappropriate for tasks such as word-processing,which have no obvious physical connection to begin
with. It is well-suited to domains such as design,
which use the computer as virtual canvas or clay.

Figure 6.1. Brygg Ullmer’s metaDESK
uses a variety of physical tools and metaphors to allow users to interact withgeographical data.

Figure 6.2. Brygg Ullmer’s mediaBlocks lets users store and manipulate
media clips as if they were stored inwooden blocks.

Tangible Interfaces
Some recent and ongoing research at the Lab also
shares much with spatial computing. In particular,
Hiroshi Ishii’s Tangible Media Group has aninterest in physical manipulation of objects asa medium for computational control. The workof Brygg Ullmer such as his metaDESK [Ullmer,
1998], and mediaBlocks [Ullmer, 1997] provide avariety of ways to use physical objects and spaces to
explore and manipulate digital information. One ofthe primary differences between what Ullmer andthe rest of Ishii’s group have done and what I am
have been doing is that their work focuses directly
on interface. They are willing to use physicalobjects as icons “phicons.” These are objects
without previous valence to the user, often abstract
blocks or disks. Their manipulation does provide
control over a system, but it isn’t fundamentallydifferent from software interface except that it isbound by physical laws. They call these systems“TUIs” for Tangible User Interfaces. I thinktangibility is important, but it is not my primaryconcern. Tangibility is a necessary by-product ofcomputational engagement with real objects in real
spaces. I would not want to miss it, but I do thinkthat reducing physical object to interface controls
unnecessarily saps them of their own identity andautonomy. As Ullmer points out, they are symbolic,standing for something for something other thanthemselves [Ullmer, 2001].
Unlike Tangible Media, which deals with physicalobjects as interface, my aim is to obscure anddistribute interface so that it becomes impossible to37

locate its beginning. Interface itself is unavoidable.
It happens at the meeting of any two different
media. But in our interactions with physical objects
we are seldom aware of interface as such. Ourattention extends beyond the interface to the objectof our intention. I hope to allow for that push pastinterface in spatial computing.
A group at the Lab that has done significant work
toward embedding computation in existing objects
is Joe Paradiso’s Responsive Environments group.
They have placed sensors and computers in objects
such as shoes for dance and gait analysis without
making them fragile or limiting their use [Paradiso,2000]. They are also interested in sensor networks,
which effectively spread the locus of interface sowidely that it may become invisible. Matt Laibowitz
is currently defining a “Phenomenological Modelfor Distributed Systems” (based on the description
language SensorML), which deals explicitlywith issues of active computational perception[Laibowitz, 2003]. These projects go a long way
toward integrating the machine into human space.

Figure 6.3. Expressive footwear fromthe Responsive Environments group.

In recent talks, HCI promoter and researcher Bill
Buxton, has expressed concern over the difficultyof transducing objects. We have very few ways to
get them into and out of our machines. This is aconcern central to spatial computing.
There are also important precedents very close tohome. The Visible Language Workshop was thegroup at the MIT Media Lab that later became the
Aesthetics + Computation Group, of which I am amember. They did much of the pioneering graphics
work on integrating perceptual depth cues otherthan linear perspective into computer graphics.
In particular some of their research dealt withlayering, blur, and transparency [Colby, 1992].

38

Figure 6.4. The Visible Language
Workshop explored layering, translucency, and blur as visual tools.

On the humanist side of this research, Anthony
Dunne and Fiona Raby have been looking atways people react to objects with technologicalappendages [Dunne, 2001]. For instance theyembedded a GPS receiver in a table and had
people keep it in their homes for periods of time.
They found people became attached to the object
and its operation and were concerned when it
lost its signal. Some were compelled to take thetable outside where it could tell where it was.
The attachment people make to active objects isof central importance to spatial computing. The
qualities of design that establish that pseudoempathic relationship are part of what I hoped toengage.
Each of the individual projects I describe in thisthesis had its own specific precedents, and those I
will detail in their own sections.

39

7. Roadmap of Explorations
The six projects I describe in this thesis could beorganized on several different axes. They could beordered by their bias toward real or virtual space,or the amount they deal with objects versus theamount they deal with space. Instead I will present
them as a chronology because it will give thereader some idea of the circumstances that lead totheir conception and the forces that shaped theirdevelopment.

7.1 Installation
I arrived at the Aesthetics + Computation group
after two years studying Architecture at MIT. I was
ready to think about computation and space, andeager to explore the resources the group had tooffer. Among these was a set of inductive positionand orientation sensors called a “Flock of Birds”[http://www.ascension-tech.com/products/
flockofbirds.php], a surplus flat-panel LCD displaythat I could be allowed to dismember, and aminiature video camera. I quickly sketched out anidea for a system called Installation that would
allow users to create and modify virtual sculpturesthat were visible only through a viewing screen.
The viewing screen could be moved freely in space
to see the virtual constructs from any angle. This
involved the use of two of the inductive sensors(one to use as a 3D stylus, and one to track theposition and orientation of a viewing screen); onegutted flat panel; and the camera mounted on theback of the screen. The system took shape quickly
and ended up surprisingly close to my original
intention. In the end the system allowed users to
sketch free-form blobs with the stylus and theninstall them permanently at any depth into thespace of the room as seen through the view screen.
When the user moved the view screen, the objects
40

Figure 7.1. The Installation setup.

responded as if they were actually in the room.
I later wrote an external client for the system,
which I ran on several machines around the room.
Whenever a user threw an object close enoughto one of the clients, it would disappear from theviewing screen and appear on the screen of theclient. This gave the strong impression that one had
actually flung a virtual object through real space.

7.2 Internaut

Figure 7.2. Internaut.

After Installation, I returned to an old idea that I
had wanted to realize for some time—a mapping of
the structures of web sites into three-dimensional
spaces that could be navigated with a first-person
game engine. I guessed that there would be
qualities of three-dimensional space that would
give some added richness to the experience ofnavigating the web. After finding a suitable open
source game engine, Quake II, from ID Software,
I modified it to use maps that I generated fromthe structure and content of web sites. I called thesystem Internaut. The resulting virtual spaces
proved interesting in some regards but nearly
impossible to navigate. Users of the system thoughtof many ways to improve the legibility of the spaces
generated, but I think the fundamental flaw was
the naïve injection of space into a medium thatis fundamentally space-denying. Analysis of thisproject led me to understand the importance ofretaining reference to real space.

7.3 Stomping Ground
Shortly after this I got the opportunity to workwith the Responsive Environments group on arichly spatial installation at the MIT Museum. An
old project of theirs, the Magic Carpet, a carpet
as musical instrument, was to be permanently
installed in the MIT Museum, and they wanted to
add a visual component to it. The carpet had a grid41

of piezoelectric sensor wires underneath it to sense
footsteps and two Doppler radars to sense upper
body movement. Users could control the music it
made by where and how hard they stepped on the
carpet and the overall speed and direction of theirbody movements. The system had been used inperformance by dancers and had had a thoroughtour of the world. It was my job to take the same
sensor information that Kai-yuh Hsiao had madeinto music and make it visual. The resulting
system, now renamed Stomping Ground, used
rear-projection to present people on the carpet
with greater than life size images of their own legs
and feet with blobs rising out of the floor wherever
they stepped. In the resulting piece, the space of the
carpet was legibly translated into a virtual space inwhich people mingled with virtual forms.

Figure 7.3. Stomping ground.

7.4 Hotpants/LittleVision
After these experiments in screen-based virtuality,
my work took a turn toward the hand-held
object. I was part of a team that helped teach anundergraduate class in microcontroller design. Ouradvisor, John Maeda, had us create a development
environment from the ground up, which we
called Nylon. A common problem in elementary
hardware design classes is a frustrating bottleneckin actuation. No matter how interesting or exciting
student designs are, they are limited in their rangeof actions: maybe spinning a motor or lighting a
few LEDs. We decided to alleviate this problem bybuilding for them a palm-size output device that
had significant expressive range. We called thecircuit Hotpants. It was a grid of 10 by 14 red LEDs
each of which could be on, off, or half brightness.
We wrote a display language that a microcontroller
onboard the display interpreted so that students
could send primitive graphics commands tothe displays to do things like draw points, lines,
rectangles, and circles.
42

Figure 7.4. The Nylon microcontroller
teaching platform.

Figure 7.5. A proud workshop participant and his Hotpants.

For the purposes of the class, the device served asa display. But because of its size and shape, it wasmore than a screen. It was a physical entity to behandled and manipulated. Because each pixel was
visible, it wasn’t possible to forget the physicalityof the device and become seduced by the image
surface. The image was always teetering on theedge of legibility, requiring the viewer to position
themselves in space at just the right distance tomake it properly resolve. That said, some projects
from the class did not deal thoroughly with theobject qualities of the display, and instead tacked
it on as though its operation as a display somehow
excused it of its body.
But the embodiment of the display was of intenseinterest to me, and after the class I began to explore
it as an independent object. It had its own processor
and I supposed it could be used to store and playback small video sequences. I wrote software thatallowed image sequences to be compressed andburned directly into the display. This use of thedisplay we called LittleVision. Justin Manor wrote
video software that allowed us to shoot movies witha webcam and downsample them to the resolutionof the device. We ran several workshops in which
participants filmed tiny movies using their bodies
and large foamcore props. They got surprisingly
good results. The most engaging characteristics
of LittleVision were its size and weight, just large
and heavy enough to feel good in the hand. It was amorsel of video, an object to which a person could
become attached. Its thingness, its substance in theworld was its most important quality.
Figure 7.6. The Word Toss handhelds
demonstrating pointable computing.
Smoke provided by Justin Manor. (Note:breaks in the beam here are where thesmoke is thin. We don’t modulate it at
the speed of light.)

7.5 Pointable Computing
As I began to use LittleVisions, I started to think
about the possibilities and implications of theircommunicating with each other, which led me to an43

analysis of the spatial qualities of different modesof wireless information transfer. It struck me thatas the world moves away from wires and infrared communication in favor of radio-frequency(RF) technologies such as 802.11 and BlueTooth,
we are losing the specificity of address that aspatially directed connection offers. It is always
possible to tell what a wired device is attachedto—just follow the wire. And infra-red devices likeremotes are aimable within a fairly narrow cone
as is obvious when using a television remote. But
RF communications extend almost sphericallyfrom their source, making directional intention
impossible. We have to resort to selecting theobjects of our intentions from a list of names or
identifiers. My idea was to emphasize directionality
and specificity of spatial communication over
all other qualities, and therefore for my carrierof communication, I chose a laser beam, theapotheosis of directionality. I built a system forcommunication between devices that operates
much like an infra-red transceiver, but since it
was laser-bound, it was longer-range and totally
pointable. This pointability and the feedback theaimer got as a red spot on the object of control were
obvious examples of the benefit of maintaining aspatial relationship with computational objects.

7.6 EyeBox
My last experiment, EyeBox, went further in the
direction of integrating existing physical objects
into computation than any of the previous projects.
I made a simple 3D scanner out of a collection of
inexpensive webcams. I used a technique called“visual hull” reconstruction, which bounds thevolume of an object based on the intersection ofgeneralized cones produced from silhouettes ofthe object taken at multiple angles around it. The
technique is described more fully below. It is notcapable of reproducing every topography, but it
44

Figure 7.7. EyeBox is a mini-fridge
turned 3D scanner.

take surprisingly little sensing to produce a very
good representation of many everyday objects.
As interesting as EyeBox was as a 3D scanner,
it was at least as interesting as a model of a new
spatial interaction with a computer. The screen
in EyeBox was mounted on the door of the fridge,
and the system was operated by opening up
the computer and putting an object inside. The
repurposing of the space inside the machine asan active space, not just a cavity containing theguts of the machine engaged people’s spatial
understanding. It made intuitive sense to themthat they should be able to open the machine andput things inside. It was a very pleasurable andcomplete interaction.

45

Figure 7.8. The six projects organized along five different axes. They are positioned vertically on the scales of Space/
Object, Physical/Virtual, Feedback/Blind, System/Component, and Visible/Invisible. They are sized in the chart according to the amount that the project directly addresses that axis.

Space

PhysicalFeedback

SystemVisibleObjectVirtualBlind

Component

Invisible

Pointable Computing

EyeBoxLegend:Installation

46

Internaut

Stomping Ground

LittleVision

8. Installation

8.1 Introduction
Figure 8.1. Installation allowed users
to create virtual forms and install them
permanently into real space.

My first project was Installation, a system for thecreation of virtual forms and their permanent
installation into real space. Installation consistedof a viewing window and stylus. A tiny camera on
the back of the viewing window showed a live feedof the room behind the screen. The stylus and theviewing window were tracked in three dimensional
position and orientation to calibrate virtual
coordinates with real viewing position. Virtual
objects created in the system responded as though
they were physically in the space of the room. Once47

objects were placed in the environment, they stayed
there in perpetuity, pulsing and growing over time.

8.2 System Description
Installation was an exploration in what istraditionally called “augmented reality,” to indicatethat rather than trying to replace an experienced
reality with a virtual substitute, we are adding to
an existing reality with virtual constructs. Thiscertainly qualifies as spatial computing.
Installation presented itself as a cloth-draped
chest-height table with a very light flat-screen
panel resting on it, which had been liberated fromits housing and placed in a translucent plexiglass
frame with handles that allowed it to be held andmoved in two hands. In the panel, users could see
the room behind the screen in a live video feed. This
feed was coming from a tiny camera mounted in
the middle of the back of the screen. The screen did
not quite appear to be transparent, but somehow
it was an obvious leap for users to allow it to standin place of their eyes. Also resting on the table was
the end of a long black cord with a half-inch red
cube and a single button at its tip—the stylus. Whenusers picked up the stylus they noticed a pencil-likeobject that appeared onscreen and closely tracked
the stylus in the spatial representation. There
was no difficulty in understanding this mapping;it was a literal translation of real space to virtual
space, and users spent no time disoriented by it oradjusting to it.
When users brought the stylus in front of thescreen, a white haze settled over the video feed of
the room as if it had become suddenly foggy. The
fog cleared up if they moved the stylus behind thescreen. The foggy and clear states represented
the two operational states of the system, object

48

Figure 8.2. The Installation setup incontext.

Figure 8.3. The back of the systemshowing the camera.

Figure 8.4. The stylus.

Figure 8.5. Object creation mode. The
form tracks the user’s gesture.

Figure 8.6. In object placement mode,
the user can throw the object into thespace of the room.

creation, and object placement. In object creationmode, with the stylus in front of the window,
when users pressed the button, a blobby substanceappeared to be squirted out from the end of thepencil-like cursor. If users stopped moving, theblobby form continued to inflate. If users moved
the stylus quickly, the form was a thin track of thegesture in space, but if they moved slowly, the blob
inflated in place, making a thicker form. In thisway, users had direct gestural control over virtual
forms created in the system. It was easy to makepretzel-like knots or letters this way. Installation
was not intended as a drafting tool, but a simple
gestural sketcher for organic blobby forms. Users
could add many separate blobs to a single form bystopping and starting their drawings.
Once a form had been created, if users moved thestylus behind the screen, the pencil-cursor was
shown emitting a ray of laser-like red light. Thiswas object placement mode. The orientation of thestylus was tracked, so they could point the beamin any direction they pleased, even back toward
themselves. The object they created in creation
mode appeared attached to the laser beam a few
inches away from the tip of the pointer. Wherever
the user pointed the beam, the object followed.
When they pressed the button on the stylus, theobject shot further down the beam. A grid appeared
which helped to show users how far they had
cast the object into the scene. Otherwise it would
have been very difficult to tell how far away it was
since the object was of purely invented form, andits relative size held no information. When users
had positioned an object in the space of the room
where they wanted it, they could bring the stylus
back to the front of the screen, and the blob wasleft floating in space wherever they had put it. They
could then use the stylus to create other forms tofurther populate the space of the room.

49

When a user picked up the viewing window, thevideo feed moved in a predictable way because
the camera moved. The virtual forms represented
onscreen moved in exactly the way they would ifthey were truly positioned in space where they wereplaced. This allowed the user to move the viewing
window to look at the objects they had made fromany angle, even to cut through them by pushing
the window through space they occupied. Throughthe viewscreen, the objects as seen through thewindow were fully fledged members of the spaceof the room. They floated wherever they had been
put. In order to add some life to the system I gave
the forms the ability to change shape and grow overtime. If they were left too long, they grew out ofcontrol, filling the space of the room.

Figure 8.7. Moving the viewscreen
around causes the forms to react as
if they were exactly where they wereplaced in the room.

The system had no representation of the actual
geometry of the room. Therefore the only occlusionthat occurred to the objects came from other objects
in the system. If a user wanted to place an object
a mile away, they could, and at no point would it
disappear behind the far wall of the room. This
detracted somewhat from the completeness of theillusion. One of the very nice qualities of the system,
however, was that it was entirely self-calibrated.
That meant that it would work just as well in any
space. I did, in fact, show it in a remote location,and it required no special calibration. That movablequality could be important to potential applicationsof the system, so it would not do to have it interact
with a single pre-constructed 3D model of the scene
in front of the screen. However, gathering real-time
range data and integrating it into the system would
be an interesting future effort.
I added a networked client feature to the system, bywhich objects could be “thrown” to other machines
in the room—including the printer. To set up a

50

Figure 8.8. A client screen (outlined inblue tape) as seen through the viewscreen.

client, I installed the client software, which inits default mode, simply displayed a blank white
screen. I then entered a special mode on the mastersystem (the flat panel), in which I placed a sphere
into the room directly surrounding each client. I
taped blue tape around the border of the monitor
of each client, so that a user of the system could
identify them. Whenever a user was in placementmode, and they threw an object close enough to aclient, it would disappear from the viewing window,
and immediately show up on the client’s screen,rotating slowly in space. I set up the printer as aclient too, and when an object was sent there, it
disappeared from the viewing window and got
printed out. In this way, users actually had thesense that they were making objects and throwing
them around the room.

8.3 Technical details

Figure 8.9. System diagram.

Installation was a fairly simple piece ofengineering. It had six primary components, thePC, the client machines, the sensing system, thedisplay, the stylus, and the camera. The PC and theclients were totally ordinary Windows machines.
The PC talked to the client machines over wired
Ethernet. The camera was a small NTSC CMOS
camera that went right to a capture board in the
PC. The display was a flat-panel LCD monitor
with all its housing and shielding removed. Oncesuch an operation is done, a flat panel monitor
is a very light, wonderful thing (if perhaphs
not 100% FCC compliant). It had a laser-cut
plexiglass frame surrounding it that had handles
for its manipulation. This frame went throughtwo iterations, making it smaller and lighter.
The single button on the stylus, and the several
control buttons on the back of the display were
implemented as stolen key switches from a hacked-51

up keyboard—probably the easiest way to get abunch of momentary switches into a PC.
Sensing System
The sensing system was a “Flock of Birds” from
Ascension Technologies, an off-the-shelf inductive
position and orientation sensing system. Thissystem itself consisted of three separate types ofunit—the signal-processing boxes, which talked
to the PC via a serial connection, the base station,and the sensing coils. The base station was placedout of sight under the blue cloth. It was about aslarge and heavy as a brick. It emitted a magnetic
field along three axes that reversed itself at a
certain frequency. The two sensing coils, one for the
display, and one for the stylus were just coils of wire
wrapped in three different axes. By sensing the flux
caused by the field in each of the three directions
for each of the three axes of magnetic field, thesignal processing box is able to reconstruct thelocation and orientation of the sensor.
Software
All of the software was written in C++ using
OpenGL for graphics. Software development fell
into three categories. The first software layer
processed and integrated data from the sensors,
buttons and camera. The second layer acted tocalibrate the virtual space to the real space toestablish an appropriate projection for the viewing
window. The third layer was for creating the forms
themselves. I developed a method using spheres
connected with Catmull-Rom splines, which
provided a fast way to model and render complex
organic-looking forms.

Figure 8.10. Flock of Birds diagram.

Figure 8.11. The blobby forms were
spherical nodes connected with CatmullRom splines.

8.4 Precedents
ARToolkit
Installation shares features with many augmented
reality systems. Some, like AR Toolkit [Billinghurst,
52

Figure 8.12. The ARToolkit is used tocomposite a virtual plane into a video
image. [http://www.equator.ecs.soton.ac.uk/projects/arproject/fokker-ar.jpg]

2002], are purely vision-based. They spot known
patterns in the world which a user prints out aheadof time. They infer the location and orientationof the pattern by vision algorithms, and thencomposite previously-defined objects into the scene
at the same point. These systems typically act toannotate prepared scenes with prepared overlays.
They do not easily allow for creation of new forms
or positioning them in arbitrary places in space.

Figure 8.13. Rekimoto’s “Magnifying
Glass” approach uses a handheld screen
to superimpose information. [Rekimoto,
1995] [http://www.csl.sony.co.jp/
person/rekimoto/navi.html]

Figure 8.14. The Diorama system[Karahalios, 1998]

Overlay systems
Many augmented reality systems are used todisplay information about the world directly onto
it as a kind of floating wall text [Karahalios, 1998],[Feiner, 1997], [Rekimoto, 1995]. Like Installation,these systems calibrate virtual coordinates to real
spaces, but they are quite different in their focus
and intent. Augmented reality systems call upon
the virtual to annotate the real. Iconic tags orsymbols appear overlaid onto scenes to indicate
for instance, if there is mail in your mailbox. Thereis little attention to the forms or space in thevirtual, or their interactions with the real, and as aconsequence the virtual layer is entirely dominatedby the real, appearing as little more than an
intelligent heads-up display.
By contrast, Installation places more attention on
the virtual than the real. If there is a subordinate
world in Installation, it is the real world, whichappears as a reactive underlay for a richer
virtual environment. Perhaps Installation is lessaugmented reality than augmented virtuality.

Figure 8.15. The MARS project at Columbia University implements a mobile
augmented reality overlay system. [http://www1.cs.columbia.edu/graphics/
projects/mars/]“Eye in hand” systems
George Fitzmaurice seems to have been among
the first to describe and develop systems withhandheld screens tracked in space. He called
these “eye in hand” systems [Fitzmaurice, 1993].
(Interestingly, he used the very same tracking

53device I did ten years earlier. It is shocking how
little the field of 3D tracking has progressed.) It issurprising, considering that they do in fact map theeye to the hand, how intuitive the “eye-in-hand”
model is. This is seen to be a primary advantageof the technique [Tsang, 2002]. Since 1993, there
have been several notable systems for augmented
reality using handheld screens. One, the Virtual
Car, by Art + Com, used an overhead armature to
track the viewpoint of a screen used to display ahighly detailed model of a virtual Mercedes [Art +Com, 1997]. The Boom Chameleon, a similarly caroriented device also uses a hinged rig to track thescreen [Tsang, 2002]. This device traces its lineage
directly back to Fitzmaurice’s original concept.
There even appears to be a related product on themarket, WindowsVR from Absolut Technologies
in Brazil. Surprisingly, few of the other 3D eye-inhand systems uses a live camera feed. An exception
is a system from NTT, which uses the ARToolkit forvision-based registration [Matsuoka, 2002]. The
video feed was one of the most important featuresof Installation, and the easiest to implement. It
is possible that they eschewed it out concern thatreference to the real world would make small errors
in calibration noticeable. My research indicates thatpeople are tolerant, even ignorant, of a great deal of
misregistration as long as it is of the right kind.

Figure 8.16. A rendering of Art +Com’s Virtual Car system. [Art + Com,
1997]

Figure 8.17. The Boom Chameleon.
[Tsang, 2002]

This list of precedents, most of which I wasshamefully unaware of as I produced Installation,
indicates that this work has a rich history and also
an active present.

8.5 Evaluation and CritiqueInstallation removed the layer of spatial metaphor
inherent in most graphical computing by dealing
directly in the space of a room. An object created
two feet in front of the user was two feet in front of
54

Figure 8.18. The WindowsVR rig has
joysticks to register translation. [Absolut, 2002]

the user. They were free to step around it to operateon it from the side. This kind of readjustmentof viewing and working angle is exactly the kindof maneuver that we do continuously without
ever thinking about it in the real world, butwhich we must master some interface to achieve
in computational design. As Tsang points out,
manipulation of viewpoint in “eye-in-hand” systems
requires essentially no new learning. Furthermore,in traditional three-dimensional modeling,
operations that change the position of objects
viewed through the screen, implicitly change ourphysical position relative to the scene. But since weknow that we have not moved, we must imagine
that the entire virtual world displayed in frontof us has reoriented without the slightest hint ofinertia or other true physical effect. It makes theinteraction feel cheap and unreal, and separates us
from our work.
This problem with the traditional computational
representation of space became obvious onwatching people interact with Installation. Theyexperienced delight that the objects they createdbehaved the way their intuition demanded theyshould. There was an immediacy to the interaction,
which people had ceased to expect from machines.
It is ironic, perhaps sad, that the operations
that seemed magical to users of Installation are
the most mundane features of our real physicallives. That lifting a viewing window and looking
at a scene from a different angle was cause forwonderment, bespeaks the distressing inadequacyof typical human-machine interaction.

Figure 8.19. An immersive CAVE
simulation. Is this more convincing?
[http://resumbrae.com/info/mcn01/
session3/]

In the corner opposite augmented reality,
privileging the virtual to the complete exclusionof the real are immersive virtual environments.
What Installation called into question about thesesystems is whether it is necessary to jettison all55

of the richness and intricacy of the real world tocreate a convincing virtual experience. The easewith which Installation aroused a response fromits users indicated that there is a sumptuous
experiential quality to be gained by embedding avirtual world within a real one.
Forgiveness and relativity
Some human qualities that proved quite consistentover the course of my projects first became
apparent with Installation. First, it was reassuring
to discover how forgiving of certain discrepancies
the human sensory system is. This might be
expected given the tremendous extent to which ournotions of a consistent reality are constructed fromfragmentary sensory evidence and expectation. But
it was a surprise to me. The linear algebra I wasdoing to reconstruct the scene as users moved theviewing window was only so good. It corresponded
very roughly with what an actual window would
see. Yet the illusion was fairly convincing. That
had a lot to do with relativity of sensing. We havealmost no absolute references for sensing anything.
We gauge things entirely relatively to what else
we are experiencing at the moment. This can bedemonstrated in countless ways. There are color
experiments that show that we perceive color values
almost exclusively by value relative to the visual
field surrounding a point. This is well-known toany photographer or videographer who has to takewhite-balance into account. We cannot perceive
small global shifts in color temperature unless they
happen quickly enough that we can compare them
to a fresh memory.
I was fortunate also not to be overlaying virtual
objects onto real objects, in which Ronald Azuma
states discrepancies of 1/60th of a degree may be
noticeable. Instead there was a strong separation
between the physical and the real objects, and I

56

Figure 8.20. Three paintings of Salisbury Cathedral by John Constable. They
all use a different color palate to render
the scene, but they are all convincing.

did not endeavor to tie them tightly to each other.
Azuma in his survey of existing augmented reality
applications notes that these discrepancies areseverely limiting for certain applications like
medical imaging [Azuma, 1997].
Feedback
The believability of spatial connectedness wasquite strong. Although the screen did not behave
exactly as it would physically, it was impossible tosay exactly how it was off, and it hardly seemed tomatter since the misalignments were predictable,consistent, and could be counteracted by physicalfeedback. Azuma refers to a phenomenon calledvisual capture, in which any contradictory sensory
information tends to be overridden by the visual.
This effect was certainly noticeable in Installation.
Although the physical movement of the screen
may not have exactly matched the screen’s
representation, the visual took precedence, and thediscrepancy went mostly unnoticed.

Figure 8.21. A translation takes thegesture from in front to directly behindthe screen.

The importance of feedback can hardly beoverstated. As Norbert Weiner wrote, many
control problems disappear in the presence of ahuman operator with sufficient feedback [Weiner,
1948]. For instance, how hard should one pusha door to open it? The answer is “hard enough.”
We don’t know how hard we are going to haveto push a door, so we adjust our own exertion
based on instantaneous feedback we feel about
whether the door is yielding. Everything is relative
to momentary circumstance and experience. The
feedback loops inherent in Installation were very
tight. The control of the 3D cursor onscreen bymeans of the stylus was one instance. The cursor
was easy to control because it followed the hand
directly and it provided onscreen visual feedback
immediately. In fact, in object creation mode, there
was an inherent spatial translation in effect that57

took the gesture being made from in front of thescreen to behind it. Almost no user of the systemeven noticed it. An absolute translation became
unnoticeable in the face of tight feedback and good
relative correlation.
How little it takes
Another observation that became apparent
accidentally during the operation of the system(when the camera stopped working) was how
much I was getting from how little. All the camera
provided was a live video feed of the room to beplastered behind the virtual objects. It was notcalibrated or manipulated in any fashion. But
the moment it was removed, the system became
entirely flat. Even though users could still use thescreen to view the virtual forms from different
angles, the primary experience of their existingin the room was utterly gone. It was a shock, andworth remembering how powerful a simple live
image can be to create context.
Difficulty of depth
The challenge of conveying depth on a twodimensional medium is ancient. Installation
added to that discussion the capability to movethe display surface through the scene. But manyof the traditional problems of conveying depth
remained. J. J. Gibson identified 13 different cues
we use to perceive depth [Gibson, 1979]. Not very
many of them made it intact into Installation.
Stereo vision, a favorite of many augmented-reality
implementations, was gone. In the absence of any
physical referent for the shapes, it was impossible
to use their relative size in the scene as a depthcue. Almost the only things remaining to use
for depth-cueing were occlusion (of the objects
with themselves only), surface shading (but noshadows), and relative speed of movement in thevisual field. It was this last that proved the most58

Figure 8.22. Without the background,blobs are just blobs.

Figure 8.23. Georges Braque’s Fruitdish uses many perceptual cues to give arich illusion of depth without relying onlinear perspective.

useful, and the illusion of depth was best when
there were multiple objects in the scene at different
depths and the user was actively moving theviewing window.

Figure 8.24. These letters were legible
from the front. I wonder what they said .

It was interesting also to note how difficult it
was for users to draw in an unconstrained 3D
environment. They were used to having thestructure of a flat surface to press against when
making an image. It was difficult for them tocontrol the depth of their drawing. Often if they
were drawing letters, for instance, they would beusing as feedback only the single 3D view that thestationary viewscreen gave them. So they would
close their shapes only to the point of visible closurein a single 2D projection. When they then moved
the screen, they would see that their letters went
way out of plane and did not topologically close
at all. Most letters people drew were not legible
from angles different from the viewing angle atwhich they were drawn. To the extent that thiswas a failure of the system to translate the spatialintention of the user, I think it should be addressed.
What it represents is a failure of feedback. With
enough spatial information, users could certainly
close their forms. What it would require is a systemthat allowed for users to change their viewpoint
easily as they drew so they could actively perceive
their forms. This would probably best be attachedto the eye so that head movement could be used in
its natural way to disambiguate 3D projection.
Simplicity
One of Installation’s best innovations was a lack ofany visible onscreen interface elements except fora cursor. This helped the system to disappear. In
particular there were no graphical elements thatcalled attention to the plane of the viewscreen as
anything other than a window onto a 3D space. Any
buttons, sliders, or text would have set up a virtual59plane that would have been impossible to ignore.
It would have distracted from the sense of puretransparency that Installation aspired to. Mappings
were clear and reactive enough that the systems
driving them could be forgotten.
The throwing of objects to client screens was a good
example. When a user sent an object to a client,
the information to reconstitute the blob on theclient screen actually travelled over an Ethernet
network. It may have taken a trip to another floorof the building and then back before appearing on
the client, but it was invisible and irrelevant to theinteraction. As far as experience was concerned,
the object flew through the air and landed on thatscreen.
The importance of this transparency was madeobvious by its unfortunate lack in one case. One
client, the printer, sat in exactly the wrong place
to be aimed at by the system (way in front of thescreen, behind the user). Therefore rather than havepeople throw their objects to the physical printer,
I printed out a piece of paper with a picture of a
printer on it and taped it to the wall in front of thesystem as a physical icon for the printer. Whenpeople threw their objects to this icon, they printed
out on the printer behind them. This separationof the icon from the actual device shattered theillusion of the object’s spatial travel, and it exposed
the network plumbing underneath it all. Anywherethat metaphor becomes visible, it exposes its
separation from the reality for which it stands.
It became an important design criterion to avoid
metaphor and apparent interface wherever possible.

8.6 Future Work
A Platform for collaboration
The ideas explored in Installation become
particularly powerful when we imagine several
60

Figure 8.25. Giving forms a sharedspatial context allows them to be theobjects of collaborative effort.

Figure 8.26. Microsoft Bob suggested
the home as a metaphor for information organization. But it took place in a
fictional iconic space.

Figure 8.27. [Dourish, 2000] studied
storage and retrieval from a spatialmodel like this. It doesn’t have much tosay about our experience of real space.

windows at once looking onto the same evolving
environment. Then it becomes a model for
luxurious collaborative computation. This model
is applicable to any kind of communal formmaking, whether it’s physical form or abstract
information, meaning the ideas could equallyfind use in architectural design or large-systems
engineering. The fundamental idea is that oncea work object is placed into space it has a sharedcontext for simultaneous manipulation. Thisfacility is demonstrated by Tsang’s system, which
he explicitly proposed as a prototype for thecollaborative 3D design markup and critique[Tsang, 2002].
Storage and retrieval
It is easy to imagine the ideas in Installation being
used for storage and retrieval of information. What
could be more natural than to look for something
you placed in a physical location? A hierarchy offolders offers very little to the eye to act as retrieval
cues. Under most conditions, we cannot even be
sure that the position of an item will be constanton our screen. We spend time and energy orienting
ourselves to the ad-hoc spaces that the machine
tosses at us as fast as we can handle them. Instead
why not let the machine orient itself to our own
naturally inhabited space?
There have been attempts to apply a physicalmetaphor to information storage, but few of them
have used a real space as the containing envelope.
Most of the spaces have tended to be iconic or pureraw regions of linear perspective. I believe neither
one has the potential for association that a wellcorrelated real space has.

Figure 8.28. Web Forager from Xerox
Parc organized web data in a virtual
library [Card, 1996].

Installation explores the mixing of real and virtual
spaces, and in so doing, begins to fulfill the promiseof models for computation that respond to our basic
human facilities and intuitions.
619.

Internaut

9.1 Introduction
After Installation, I turned to a slightly more
abstract spatial problem. I wrote Internaut, asystem for mapping Internet structures into threedimensional virtual environments and exploringthem in a first-person game engine. As such, it did
not meet the requirements for spatial computing
as outlined above, but was, in fact, a project whose
deficiencies were instrumental to my constructionof that definition. The analysis of its flaws led
directly to my understanding of the importanceof spatial computing as opposed to purely virtualenvironments.

62

Figure 9.1. A web space made into avirtual space by Internaut. A map of thearea is shown in the upper right.

9.2 Technical Description

Figure 9.2. A map begins from a web
pages and trolls the links on that page.

Figure 9.3. A map grows. The root
node is shown in red.

Figure 9.4. In this detail we see thatthe page “cover” links at least to pages“demo,” “intro,” and “ideas.” These
are connected by springs (black lines),
which will punch doorways in the wallsof the rooms (blue lines).

The Internet constitutes an enormous electronicarchitecture that defines places without regard tophysical structure. We navigate these spaces with
web browsers, moving from place to place with aclick on a link. Internaut proposed that a physicalarchitecture could be derived from the shape of the
network and navigated with a first-person 3D gameengine. This was a several-step process, which
involved first making spatialized maps from web
sites and then processing them into a form in which
they could be virtually explored.
The maps were generated starting from a given
seed web page by a fairly simple procedure thatguaranteed several criteria in the three-dimensional
map that I deemed important for them to bemeaningful. First, every page from the same site
as the seed that was accessible by any path of links
should be represented. Second, any two pages thatlinked together should be immediately accessible
to each other. There are numerous ways to design a
process to do this, but the one I implemented relied
on a simple physics simulation running in Java.
The first page was represented by a node in a 2Dgraph with a point location. All links on this page
to pages at the same site were traversed in order,
and these sites were added to the graph as nodes
with springs connected to the root node. These
simple simulated springs pulled nodes together
with a force proportional to their length plus a
constant factor for their rest length. It should be
no surprise, that these new nodes, which wereadded to the graph at random locations settled
into a ring around the root site. A user was allowed
to click and pull on any node in the graph at any
time. All springs stretched to accommodate suchmanipulation, and snapped back into a relaxed63configuration when released. Each new page wasthen processed in the same way as the root nodein the order in which it was added. The resulting
network of nodes connected with springs wasa stretchy gyrating mass that was constantly
attempting to relax into the lowest energy state
consistent with its topology of connections.
The nodes were then separated from each otherwith walls that were the divisions of a Voronoi
diagram. A Voronoi diagram associates each nodewith the area surrounding it that is closer to it thanto any other node. This is always a lattice of convex
polygons surrounding each node, guaranteeingthat each node gets some share of physical space.
The springs connecting the nodes intersected these
Voronoi-generated walls at many points. Anywhere
they intersected, a doorway was drilled in thewall, insuring that any link became a navigable
path from one cell to another. This structure
successfully located pages in a 2D map close topages to which they were linked. Obviously there
are linking conditions possible in web sites that are
not possible to represent in a 2D map with strict
adjacency, but the method guarantees that thesewill be pulled together more strongly the further
they are separated, so it does a good job of creating
spatial representations of web structures.
The next task was to go from a map in this Javaapplication to a map usable in a 3D game engine.
I chose a modified form of the Quake II engine
from ID Software because it is now a mature
open source project. I generated a map file forthis engine with the largest image on any page
tiled onto its walls as repeating wallpaper. Thissurface image was the only distinguishing featureof any room. I undertook extensive changes to theengine to demilitarize it, removing the guns andgangs of monsters bent on killing the explorer, and64

Figure 9.5. The map is then processedin a Quake map editor.

Figure 9.6. I then had to demilitarize
the game.

Figure 9.7. After removing the gun andadding a mapping feature.

adding a mapping feature which displayed a mapof the entire site onscreen at all times. I retained
the engine’s capability to run in a networked
mode in which multiple players could explore
environment together, seeing each other, and evencommunicating via typed messages.

Figure 9.8. Soap bubbles makevoronoi patterns [Boys, 1959]. Referenced from [www.snibbe.com/scott/bf/bubbles.htm].

I showed the project repeatedly, letting users selectthe starting web site and then allowing them tonavigate the resulting three-dimensional map. As
I watched them try to orient themselves to thisremapping of Internet space, I became aware ofseveral things that would inform my subsequentwork.

9.3 Precedents

Figure 9.9. Scott Snibbe’s Boundary
Functions [http://www.snibbe.com/
scott/bf/]

Figure 9.10. Jared Schiffman’s honey.
[Shiffman, 2000]

For the self-organizing map component of the
project, I had many good precedents. This sortof problem has interested scientific and artistic
communities for a long time. Voronoi diagrams
have broad application to many problems inanalytic geometry and self-organizing systems. For
instance they can be used to position nodes in selforganizing neural networks [Suanders, 2001]. And
they arise naturally in many situations in whichsurface energies are being minimized as in soap
bubbles. They appeal to computational artists anddesigners for their organic appearance and easeof production. Jared Shiffman used them for theirorganic visual quality in Honey, an exercise incellular form [Shiffman, 2000]. Scott Snibbe usedthem for their partitioning ability in Boundary
Functions, in which participants stepping on aplatform are automatically separated from eachother by boundaries projected from above [Snibbe,
1998].
Simulated springs are even more commonly usedin computational design. They lend movements a65

squishy, organic feel. Benjamin Fry used springs
to organize web spaces in Anemone, which tracks
web traffic as a continually evolving network of
nodes representing web pages [Fry, 2000]. Judith
Donath has used springs to create a self-organizing
visualization of human social networks [Donath,1995].
Ideas of organic form and self-organization havebecome popular in theoretical architecture in recent
years. Greg Lynn uses such forms as “blobjects” inhis designs.

Figure 9.11. Ben Fry’s Anemone [Fry,2000].

Mappings of non-spatial networks into virtual
spaces are not new either. In 1996 Apple briefly
promoted a 3D flythrough technology called
Hotsauce for web page meta-information [Apple,
1996]. AT&T Research produced a system called
CoSpace, which used an additional layer of VRML
on top of existing web pages to represent web
spaces [Selfridge, 1999].
Other networked virtual environments were
designed spatially from the beginning. Certainly
networked first-person shooter games like Quake
III Arena have been successful. It is easy to convene
teenage boys in a virtual space with the lure of their
being able to shoot each other with impunity. We
are currently experiencing a small explosion ofnonviolent networked virtual environments thatare not meant to represent existing web spaces,
but to establish parallel virtual Internet spaces thatare constructed and inhabited by a broad public.
Examples include the Sims Online [Electronic Arts,2003], Second Life [Linden Labs, 2003], and There[There, 2003]. Several systems like these alreadyexist, but do not find wide use.
A strange feature of the places that users constructin these virtual environments is that they mimic

66

Figure 9.12. A study model of Greg
Lynn’s.

Figure 9.13. Apple’s Hotsauce metacontent 3D web flythrough plug-in.
[Apple, 1996]

structures in the real world. They put “roofs” ontheir “houses,” for instance. Why? There isn’t
any rain, or in fact weather of any kind to defend
against. And enclosure represents no protection
from outsiders. It must be a desire for familiaritythat drives people to make designs that takeno advantage of the liberation that they mightexperience in these worlds without physicallimitation.

9.4 Evaluation and Critique
Figure 9.14. CoSpace, a 3D web
browsing system from AT&T research.
[Selfridge, 1999]

Figure 9.15. The Sims Online. [Electronic Arts, 2003]

Users found wandering around the spaces
generated by Internaut engaging, but also foundthem confusing and difficult to navigate. Even withthe help of a map, they had difficulty finding their
way around or remembering where they had been.
I think there were several probable contributing
factors.
First there was the elimination of all text. I expect
people would have difficulty navigating any web
sites that had all the text removed and left only asingle image to denote each page. Links would nolonger describe their destinations in words, but betiny thumbnail images of the image on the page
they linked to. Navigating spaces like this would, I
expect be somewhat bewildering too.
But even in the absence of text, there was adifficulty in navigating the structure due to its
unfamiliar and inhospitable physical structure.
There is a reason that we do not construct
our building plans as Voronoi diagrams. The
spaces that these generate tend toward spatially
undifferentiated rotundas of doorways that make it
impossible to identify a dominant spatial axis. Even
when there is one, it is not shared by any adjacentcells. Under such conditions, it is often impossible

67

even to identify the portal through which one
entered a space.
We are careful in architectural plans to define
circulation space. We do not expect rooms tofunction both as destinations and corridors for
movement at once. The Voronoi plans make nosuch circulation. There are no clear means ofpassage between spaces that do not directly abut.
To get from one end of the space to the other it
is necessary to turn at every room, potentiallyeven away from the final destination. There is noorganizing logic that makes the space serve an
intention other than aimless wandering.
Use of an organizing geometry other than
Voronoi could potentially help this. There areexperiments in grammatical architectures that
could help point the way to saner structures[Brown, 1997]. That is one possibility for future
research. These geometries might allow for theuse of more information from the web sites
than simple topology. It should be possible, forinstance, to identify the primary entrances to the
web site. These should represent entrances to thevirtual space as well. (In the existing geometry
they are most likely to be buried at the center andsurrounded by a ring of ancillary pages.) It is likelythat some links from a page are more dominant
than others—larger text or higher on the page.
These should be represented by larger openings or
grander access.
Another possibility is that part of what makes
the Internet successful is that it is fundamentally
non-spatial. Certain conditions of spatiality do not
apply to it. For instance there is no such thing as aone-way connection in space. There are doors thatlock from one side, but adjacency is commutative.
Not so in a non-spatial network. One page may

68

Figure 9.16. Internaut tended to offer
the user a bewildering array of self-similar doorways.

Figure 9.17. Rule-based design from
Gero [4.290 Production Systems, Fall
2002].

link to another that has no idea of the existence ofthe referrer. This network of one-way streets has
the tendency to channel users toward sites that arecommonly linked to [Barabási, 2002; 57]. These
have a higher chance of being useful than the sitesthat are seldom referenced. There is also a trail ofbread crumbs that web-surfing leaves that a user
can always use to backtrack via the “Back” button.
No such facility exists in real space, although it
could be simulated by having movement leave a
trace in a virtual environment.
The most damning concern may be that thefundamental property of Internet space is thecollapse of distance. Distances are measured in
the number of clicks the path takes, and a long one
may be three. This implosion of space is necessary
to what makes the Internet a useful complement
to the real world. An advantage of shopping online
is that every store is equidistant at a distance of
one click, or the typing of its address. In orderto spatialize this condition, it would require abewildering portal—a spherical mall with thousandsof openings that would be a thrilling sight, but
hardly useful. It must not be necessary to cross any
space to have access to another. Once the intended
destination is identified, the need to “walk” there
only represents wasted time. Access must be as fast
as the delivery of information will allow. So perhaps
the idea of a spatial Internet is fundamentally
flawed. Cyberspace as Jean Baudrillard puts it is
Where all trips have already taken place; where thevaguest desire for dispersion, evasion and movement areconcentrated in a fixed point, in an immobility that hasceased to be one of non-movement and has become thatof a potential ubiquity, of an absolute mobility, which
voids its own space by crossing it ceaselessly and without
effort [Baudrillard, 1988, p. 32] .

In a study of the necessity of legibility for virtual
spaces, Ruth Dalton concluded that global
69

intelligibility is not important in systems suchas the web where that structure is not used fornavigation. Web space developed without any needfor an intelligible global structure, and to try to
impose one is likely a fool’s errand.

9.5 Future Work
Lots of the issues raised in the first part of mycritique could be addressed with sufficient
further work. We could try to generate rule-basedarchitectures that are more legible and easier tonavigate. While I think the program of virtual
representation of Internet structures has something
to teach us, particularly in using visualization to
uncover topologies and flows of the network, I do
not think its spatialization for browsing purposes isgenerally useful.
People suggest that it would be a good shopping
interface, in which a user could walk around andsee merchandise disposed around a space while
talking to others virtually browsing with them.
That is a possibility, and I think it would initiallybe exciting to some, but I don’t think its long-term
effectiveness would be any greater than nicely
displaying merchandise on a web page. The Sims
Online may well succeed, but I believe that thatwill have more to do with its nature as a game thanas a networked space. Remember that the nononline version of the Sims was wildly popular too. I
have come to believe that there is more interesting
territory to explore in the realm of spatial
computing, in which the spaces involved are real
spaces that the user already has attachment to andexperience with.

70

10. Stomping GroundFigure 10.1. A kid engrossed in Stomping Ground.

10.1 Introduction
Stomping Ground is a permanent installation at the
MIT Museum consisting of a musical carpet and a
projection of live video with superimposed blobs. It
is a collaboration between Joe Paradiso director of
the Responsive Environments group at the MediaLab, who made the carpet and the radars, Kai-yuh
Hsiao of the Cognitive Machines group, who wrote
the music, and myself, who designed the space andprogrammed the visual interaction.

10.2 System Description

Figure 10.2. Rewiring the carpet with
piezoelectric wires. [Photo by Stephanie
Hunt].

The carpet tracks the location and intensity offootfalls with a grid of sensors. Doppler radars
mounted on the sides of the projection wall trackthe overall direction and intensity of upper-body
motion. This information is used to create amusical composition that has two modes: one has
a richly layered ambient sound, and the other is71

aggressively percussive. The same data is fed to thegraphics system, which produces blobs that growupwards from the locations of footsteps. The blobs
are superimposed on a live video image showing thelegs and feet of people on the carpet (whole bodies
of very small people). The video and the forms in
it are warped by a virtual fluid simulation, which isstirred by stomping and upper-body activity.

10.3 Precedents
Figure 10.3. Kids exploring the carpet.

Prior to my involvement, the carpet had been
exhibited as part of exhibits on musical instruments
and hosted dance performances. As should be thecase in the extension any good work, the past work
served as my foremost precedent. I studied footageof these events, the sound and code of the musicmaking, and the technology behind the operation of
the carpet [Paradiso, 1997].
One of the music’s modes has a watery backgroundsounds, which led me to give the graphics anundersea feel. I used an intuitive 2D fluid-flow
model by Jeffrey Ventrella to warp the projection
based on flow induced by “forces” from the radars[Ventrella, 1997].
The blobby forms I adapted from Installation,connecting their nodes with springs, and subjecting
them to reverse gravity, which pulls them up fromthe base of the display and out of the picture.

10.4 Evaluation and Critique
It was an interesting challenge to come into a
project that already had such a thorough life
independent of visualization. I wanted both to fit
into the framework as it existed—the expressive
qualities of the music, the two modes—but I wanted
also to make my portion of the project my own. I
72

Figure 10.4. I implemented a fluid flow
model from [Ventrella, 1997] to warp
the video image.

wanted the visual component in the end not to be
separable from the whole experience.
Invisibility
Stomping Ground represents an intermediatestep in the integration of physical and virtualenvironments. The real space of the carpet isrepresented on the screen while virtual artifacts
swirl around on top. It is an augmented anddistorted mirroring. Unlike the direct and obvious
form-making control users have with Installation,in Stomping Ground, the link between behavior
and form produced is less obvious. More was being
decided by the system, making the system itself
more present as an agent. As much as it was a goalof Installation’s to make the system invisible, it wasa goal of the Stomping Ground’s to become a focus
of attention. It was the exhibit as much as the forms
and sounds made by it were. In that way it blurred
the line between instrument and artwork.

7311. Hotpants/LittleVision

11.1 Introduction
Hotpants was a handheld display device originallydesigned for use with the NYLON microcontroller
system [nylon.media.mit.edu], which we
produced to teach basic microcontroller design toundergraduates. Then as I became interested in
the display’s potential for autonomous operation, I
untethered it from NYLON, renamed it LittleVision,and began to use it as a standalone device for therecording and showing of short video segments.

11.2 System Description
Hotpants/LittleVision consists of a very simple
circuit which uses a PIC microcontroller todrive four shift registers and two current source
74

Figure 11.1. A bunch of LittleVisions
running tiny movies.

Figure 11.2. The Hotpants circuit layout. For a usable version, see Appendix
C.

chips, which in turn drive a matrix of 10 X 14 red
LEDs. These LEDs can be set to display at fullbrightness, half, or off. The board exposes a set ofprogramming pins, which are used to connect theboard to a PC for downloading of new movies. The
board stores about 300 frames, depending on how
well they compress, and plays them back at 12 persecond, for a total of 25 seconds of video. After
this period (or shorter if the movie contains fewer
frames), the movie loops. I have recently developed
a second board, a camera board, which can be used
to record movies directly to the LittleVision without
the use of a PC.
The circuit and its components are quite
inexpensive, and were designed with that criterion
in mind. There are much nicer display elements
available than these red LED arrays, but they are
all more costly. We have run several workshops inwhich participants film movies of themselves orother props and then burn them to the devices andtake them home. In one two-day workshop, we hadparticipants build their boards the first day andmake movies the second day.

Figure 11.3. A standalone camera
board turns LittleVision into a self-contained tiny video camera.

11.3 Technical Details
Hardware
The whole circuit was controlled by a PIC 16F876
microcontroller running at 20 MHz. It had 22
usable I/O pins. We were using it to drive four 5 X
7 LED arrays. The LED elements in the arrays werereferenced by row and column, so we did not havesimultaneous unique access to each one. Basicallywhat we had to do was turn on one column at atime and light each row that needed lighting inthat column. Then we quickly switched to thenext column, and so on. That meant that eachcolumn was only lit for a fraction of its possible
time. This was sad, as it cut down on brightness,but unavoidable. We did, however, play one nice75

trick, which was to treat the four arrays as two tallcolumns rather than one large array. That way we
could control each LED while keeping the columns
lit 1/5 of the time rather than 1/10, effectively
doubling the brightness. (This may make more
sense on inspection of the PIC code that drives it.
[Appendix C])
Unfortunately, that meant that we had to control
two columns of 14 LEDs independently. With 10
columns and 28 effective rows, we were saddledwith a burden of 38 outputs, which the PIC couldn’t
provide by itself, so we used shift registers. Shiftregisters turn serial outputs parallel by piping
clocked values to their output pins on a specificsignal. We hooked up 4 shift registers in series, andended up with 32 extra outputs controlled by 3 pinson the PIC (data, clock, and output enable).
Finally we had a potential problem with constantbrightness. We wanted all of the LEDs to beequally bright, but the PIC had a limited ability tosink or source current, which meant that when it
was lighting 14 LEDs at once, they would be toodim, and when it was lighting one, it would be toobright. So we ran the PIC column outputs through aDarlington current source chip to give it muscle. On
four AA batteries, the circuit ran with intermittent
use for about six months.
76

Figure 11.4. Annotated images of thecircuit innards.

Software
There were several different incarnations ofsoftware for Hotpants because it was used in a
number of different contexts. All of the software for
Hotpants had two components, one on a PC andone on the board. A system by Megan Galbraith
allowed a user to write programs in the Nylon
language and send them to Hotpants. I wrote thesetup that lets a user take movies with a webcam
and send them to the board. The software on thePC side and the firmware on the PIC is different foreach application. We burned a bootloader onto the
PIC ahead of time to make it easier to download
different programs to the board to change its
functionality.
The basic operation of the firmware on the PIC was
to change the values in the display buffer over time.
That became an animation. The actual refresh of
the screen column by column was done by timed
interrupt, so it remained consistent no matter whatelse was going on on the PIC.

Figure 11.5. Justin filming a tiny
movie.

We got three pixel levels (ON, HALF-ON, OFF) byusing two alternated screen buffers. A pixel thatwas half brightness was on in one buffer and off in
the other. That way it got half duty cycle. (Actually
it only got 1/3 duty cycle because we displayed
the second buffer two times out of three. That was
because it made the contrast between all-on andhalf-on better.)

11.4 Precedents
Precedents for Hotpants are somewhat hard tofind. It seems that existing technologies are always
either more or less than Hotpants. Handheld
displays that do more than Hotpants/LittleVision
are everywhere. These are on PDAs and the backsof digital cameras. There are beginning to be backlit

77

LCD picture frames sold, which are somewhat
similar in spirit to Hotpants, but deliver more
image fidelity than object-relationship. Products
less than Hotpants are the LED array components
themselves, which come in a huge variety of sizes
and configurations but have no built-in control
circuitry to drive them.
Pixilated LED displays are everywhere as banners,
and even architectural surfaces. People are startingto have video displays as small as watches. But
all of these try for an imagistic resolution. An
exception is Jim Campbell, an artist whose workwith LED arrays explores pixilation, motion, blur,and form. His pieces led me to realize that putting
a blurring filter over a highly pixilated display
makes the image easier to decipher. His pieces also
demonstrate how much recognition we get frommotion.

Figure 11.6. A digital picture framefrom Ceiva. [http://www.ceiva.com/]

11.5 Evaluation and Critique
Hotpants/LittleVision brought my attention to therealm of the handheld object, a scale which allows
users to form totally different kinds of attachments
than room-sized environments. LittleVisionessentially compressed room-scale activity andplaced it in the hand as a small electronic brick witha pleasant heft. Participants had a connection withthe scenes they were filming, and then immediately
thereafter held them in their palms. It was a very
different experience than it would have been to seethem on a television screen, or even on the LCD
panel of a handheld video camera. This difference
had a lot to do with a level of abstraction that thelimited resolution enforced.
10 X 14 is not very many picture elements. Complex
scenes are not recognizable. This forced an actof imagination onto the experience of viewing
a LittleVision that, like the cartoon rendering
78

Figure 11.7. The Nasdaq exchange in
New York has a full color LED wall.

Figure 11.8. From Motion and Rest
#5, Jim Campbell, 2002. [http://
www.jimcampbell.tv/]

discussed above, removed the distracting quality ofnear-perfection. The viewer could slip in and outof seeing figure, ground, or even individual pixels.
This slippage was also tied tightly to the distanceat which the object was viewed, which made people
experiment with it, bringing it close to their faces or
holding it as far away as possible. As in Campbell’swork, scenes that were impossible to understandwould sometimes snap into focus when they started
to move. Interestingly in Installation it was also
motion that brought out the sharpest qualities ofdepth. Human visual recognition owes a great deal
to motion [Nakayama, 1985].

Figure 11.9. Can you tell what thismovie is about? (Hint: It swims in theocean and has big sharp teeth.)

The screens on PDAs are designed not to confuse
the issue of pixel versus image. They display
images as faithfully as they are able at high enoughresolution that they are instantly recognizable.
Their displays are primarily surfaces of interface,which take up as much of one side as possible.
The interface draws the user’s attention to a flatspace of text and buttons, which totally overpowers
the substance of the object itself. They are always
fighting their physical existence, trying to becomethinner and lighter. They are rectangular tooverlap the palm in one dimension. This makes it
impossible to fold one’s thumb down across the topof them—the natural desire for holding palm-sized
objects. They are held like a stick, not a rock. Thereis something that destroys intimacy having to wield
an object that you would cup if you could.
The object-qualities of LittleVision are its mostimportant. It is appealing because its size, shape,
and weight make sense to the eye and hand. Any
smaller, and it would fall below a satisfying palmful.
Any larger and it would be unwieldy. LittleVisionsatisfies my goals for spatial computing by its being
a computational object that compresses space into asmall body while not having its object nature overpowered by its computational process.
7912. Pointable Computing

12.1 Introduction
One way to understand remote communication
is as a battle with the separating qualities ofspace. AT&T’s old slogan “Reach out and touch
someone,” made that explicit. The phone was to be
an electronic prosthesis for contact. But it has not
only been long distances that we have put effort
into nullifying. The “remote” in remote control
typically connotes no more than 15 feet. This kindof spatial collapse attempts to bring things just outof the sphere of reach into contact with the fingers.
It functions as an extension of touch, and most80

Figure 12.1. Word Toss handhelds
sending information over a visible laser.

remote controls resemble the kinds of interface wewould expect to encounter on an appliance itself.
This is not an interaction about communication,
however. It is strictly about control, and it operates
unidirectionally.
Remote control has moved a step further in recent
years to encompass remote data access. This has
pushed the technology beyond the capacity of
infra-red communication and into radio-frequency
territory with 802.11 and BlueTooth. The spatial
idea behind these technologies is different
from the directed-connection spatial model oftelecommunication and remote control. Instead,these technologies are proposed to replace wires.
Wires are simply physical connectors designed
to carry signals. They do exactly what their shape
implies. It has been possible until recently to tellwhat a machine is connected to by tracing its wires.
Now the wires are going away, and it is totally
unclear what connections are being made frommachine to machine. A useful assumption may bethat everything is connected to everything. Thereis no disconnect to privilege any one particularconnection over another.
And that is a problem. Now that we have essentiallyconquered spatiality with communication
technology, we are left floating in an
undifferentiated spacelessness. True we may haveeliminated the need to crawl around to the backof our machines to plug in devices, but we have
replaced that inconvenience with a new burden ofreference. We must assign everything with which
we want to communicate a unique identifier sothat we can select it from a list of things in range ofcommunication. We have essentially become likeour machines, who have no notion of directionality
or focus, and therefore must refer to things by ID.
This is not a humanizing direction of development.

81

What I proposed in Pointable Computing wasa solution to this crisis of non-space in wirelesscommunication.

12.2 System Description
Pointable Computing was simply a handheld
system for remote communication over visible
lasers. It was the absolute epitome of directed
communication. Until I learned to spread the beamslightly, it was so sharply directed that it was hard
to use at all. The purpose of the project was toexplore the possibilities and experiential qualities
of highly-directed communication and contrast it
with undirected technologies.

Figure 12.2. One of the two handheld
devices.

Technical description
The system consisted of two handheld devices
equipped with laser-diodes and phototransistors forsending and receiving of signals. I spread the beamslightly with a lens system to make it eye-safe andeasier to control for distant targets. Each handheld
had a display board (a repurposed Hotpants
display), a single button and a control wheel. I also
made a standalone wall-mounted receiver withthree Hotpants displays. Each of these systems was
driven by a PIC microcontroller.
Word Toss
The proof-of-concept application I designed forthe devices I called Word Toss. Each handheld
showed two words stacked vertically, a transitive
verb on top and a noun on the bottom. In one of thedevices, rolling its wheel changed the verb, and in
the other device, it changed the noun. Each device’s
laser was on by default. When the devices werealigned, their lasers hit the other’s receiver, and apixel in the top right of the receiving device would
light to indicate that it had acquired a signal. When82

Figure 12.3. The innards of the handheld. Clockwise from top: laser, send
button, receiver, Hotpants display, batteries, more batteries, and thumbwheel.

either device’s button was pressed, its laser was
pulse-modulated to send a message to the otherdevice. The message in Word Toss was simply theverb or noun selected with the wheel. The otherdevice received the message and changed its word
to match the word sent. It was also possible to use
the handhelds to send words to the wall-mounted
device, which displayed them. I was successful insending messages from more than 30 feet away.

12.3 Precedents

Figure 12.4. It was also possible to use
the handhelds to send messages to thiswall-mounted unit.

Figure 12.5. Laser tag systems haveused the total directedness of visible lasers for entertainment. [http://www.lazerrunner.com/primer.html]

Sending information on visible lasers has been
done for a long time. The field is called “Free Space
Optics” to denote that it is an optical transmissionthat requires no medium of conveyance otherthan air. Systems have been developed that cancommunicate up to 2 miles and at speeds of up to622 Mbits/sec. These devices do not use visible
lasers, and are certainly not handheld. But they
do demonstrate a kind of spatial pointing thatis similar in spirit if not in scale to Pointable
Computing’s.
Certainly existing infra-red communication, whichtends to have a conical spread, was a datum. The
most common specification IrDA allows many
devices to communicate with each other. The IrDA
specification demonstrates a vision very much likethe point-and-communicate model of Pointable
Computing [IrDA, 2000]. The deficiencies ofinfra-red LEDs are that their illumination quickly
spreads too much to be detected, making it suitable
only for very short range applications. By contrast
low power visible lasers can extend hundreds offeet without dissipating too much to be useful. The
limiting factor at long range with handheld lasers
is the unsteadiness of the hand, which distancemultiplies. The other liability of infra-red is that83

it is invisible, so the user is never quite sure is aconnection is being made or is reliable.
Pointable Computing’s second lineage is the wholefamily of computational pointing input devices.
Figure 12.6.

Pointing Devices Organized by Domain of Operation������������

�����������

�������

����� ����������

��������� �������

��������

����� ��������� �������

����������� ���������
����� ������

��������������
������

Metaphorical devices map spatial gestures and actions to a metaphorical machine space.
Documentary devices recover spatial qualities of the world and report them to the machine.
Literal devices are active operators that report an actual spatial or bodily condition. Literal devices
are differ from metaphorical devices in that they directly connect agent, environment, and machine.
Metaphorical devices are concerned only with the input environments locally relative to themselves.

8412.4 Use Scenarios
I developed several use scenarios to illustratepossible applications of pointable computing.
Universal remote
The most obvious use of Pointable Computing
would be to make a universal remote. Pointing
the device at any enabled object would turn thehandheld into a control for that object.
Figure 12.7. Universal remote sketch.

Figure 12.8. Active tagging sketch.

Active Tagging
Imagine yourself walking down an aisle of products.
You see one you would like more information
about or two you would like to compare. You point
your handheld device at them and they transmit
information about themselves back to you. Why isthis different from giving each product a passive
tag and letting an active reader look up information
in a database? The answer is about autonomy anddecentralization. If the information is being actively
sent by the object scanned, it does not need to beregistered with any central authority. It meansthat no central server of product information isrequired, and anyone can create an active tag
for anything without registering some unique
identifier. Note also that in this scenario we see
the likely condition that a non-directed wirelesscommunication like BlueTooth would be useful in
conjunction with a Pointable. The two technologies
complement each other beautifully.
Getting and Putting
In a vein similar to Brygg Ullmer’s mediaBlocks
project, it would make sense to use Pointable
Computing to suck media content from one sourceand deliver it to another [Ullmer, 1998]. Hereagain it would not be necessary to display muchon the handheld device, and one button might
be sufficient. An advantage in media editing that85

the Pointable has over a block is that there isno need to touch the source. That means that it
would be possible to sit in front of a large bank ofmonitors and control and edit to and from each one
without moving. It might even make sense to use a
Pointable interface to interact with several ongoingprocesses displayed on the same screen.
Instant Wiring
In this simple application, the Pointable is used
simply to connect together or separate wirelessdevices. If, for instance, a user had a set of wirelessheadphones which could be playing sound fromany one of a number of sources, there is no reason
they couldn’t simply point at the headphones andthen point at the source to which they wanted it
connected.

Figure 12.8. Getting and putting
sketch. The pointable sucks up information from the computer on the left andspits it out onto the computer on theright.

Sun Microsystems likes to say, “The network is
the computer.” This is a fairly easy formulation
to agree with considering how many of ourdaily computational interactions are distributed
among multiple machines. Any form of electronic
communication necessarily involves a network.
The shrinking and embedding of computation into
everyday objects implies that informal networks
are being created in the physical fabric of ourhomes and offices. If we assume that the networkof wireless devices around ourselves is essentiallya computer, we must admit that we spend our days
physically located inside our computers. Being
located inside the machine is a new condition forthe human user, and it allows the possibility of
directing computation from within. A pointing
agent, a kind of internal traffic router, is one
potential role for the embedded human being.
Reactive surfaces
Reactive surfaces are building surfaces, exterior
or interior, covered with changeable materials

86

Figure 12.10. Instant wiring sketch.
The user connects the computer toheadphones by pointing at it and thenat them.

coupled to arrays of pointable sensors. They makeuse of new materials that have changeable physicalproperties such as LCD panels, electrochromic
glass, OLEDs, or electroluminescents. It would
be possible with a pointable device to write atemporary message on a desk or wall or define
a transparent aperture in an otherwise shaded
window wall. Such an aperture might follow thepath of the sun during the day.

12.5 Evaluation and Critique
Figure 12.11. Reactive surfaces sketch.
A user makes a patch of transparency inan opaque surface.

Pointable Computing takes as its starting point
an emerging reality in which everyday electronic
devices communicate wirelessly. These devices
already have identities tied to their functions, bethey headphones, storage devices, or building
controls. They are not crying out for an additional
layer of interface. How can we address the newcapacity of things to talk to each other without
further mediating our relationships with them? Weneed the remote equivalent of touch, an interaction
focused on its object and containing its own
confirmation. Pointable Computing offers that by
way of a visible marker, a bright spot of light. The
user does not need to consult a screen to determine
if they are properly aligned. It is apparent. The
receiver may indicate that it has acquired the beam,
but that indication will always be secondary to thevisual confirmation that the object is illuminated.
The system did feel substantively different fromexisting modes of wireless communication. And
its primary difference was its spatial specificity. It
felt much like using a laser pointer, which has aremarkable quality of simultaneous immediacy anddistance. This I believe is due to its antiphysicalquality of tremendous length with infinite
straightness and lightness. It is like an ideal rod.
Also like a physical pointer, Pointable Computing87

is usable because it offers feedback. As can bedemonstrated by a game of “pin-the-tail-onthe-donkey” we cannot point very well without
continuing to reference what we are pointing at. A
laser spot is the perfect feedback for pointing—ask asniper with a laser scope.
As Norbert Weiner pointed out, any system
containing a human being is a feedback system[Weiner, 1948]. As users, people automaticallyadjust their behavior based on the overall
performance of the system. What makes the
Pointable Computing a robust communication
system is that the feedback loop containing thehuman being is direct and familiar. The human eye
has an area of acuity of 1–2°, implying that narrow,beam-like focus is the norm, not the exception forhuman perception. The rest of the visual field issampled by eye movements and then constructed in
the brain. Tight visual focus is the way we solve theproblem of reference without naming in a spatialenvironment. The feedback loop that enables theact of looking entails our observing the world andcorrecting our body attitude to minimize error of
focus. It happens so quickly and effectively that
we do not even notice it. The same feedback loop
can be applied to a point of focus controlled by thehands. It is not quite as immediate as the eyes, but
it is close. And, as it turns out, it doesn’t suffer fromthe kinds of involuntary movements that plagueeye-tracking systems. This is the same mapping as
the “eye-in-hand” virtual window model.
Pointing is an extension of the human capacity tofocus attention. It establishes a spatial axis relative
to an agent, unambiguously identifying anything inline-of-sight without a need to name it. This brings
our interactions with electronic devices closer toour interactions with physical objects, which wename only when we have to.

88Pointable Computing successfully takes
computation away from the screen and into thespace between things. Its use of simple, inexpensive
components, and its surreptitious hijacking of thehuman machine as a very fine controller makeit more appealing than many other options likemotion-tracking, inductive position sensing, orcomputer vision for establishing simple spatial
relations to a user. It requires no calibration, it
operates robustly under almost any conditions,and it weighs next to nothing. I expect to see
more systems employing laser-directed spatial
interaction.

8913. EyeBox

13.1 Introduction
My final project at ACG turned my attention verymuch toward physical objects. It focused on finding
a good way to get them into the computer. As many
people, including Bill Buxton have noted, even as
our machines get tremendously more powerfulinternally, our abilities to get things other thanprinted material in and out of them have notprogressed very far. The engines of computation
have digested very little of our world. In order forour machines to become fuller partners in ourwork and play, they are going to have to join us in90

Figure 13.1. The EyeBox 3D scannerwith a small windup robot inside.

our physical world. That means we are going tohave to introduce them to the objects that form thesubstance of our lives.

Figure 13.2. Working with a scanned
form (a vice) on the computer.

In EyeBox, I made a computer at which a user
could sit and work, and then they could open it
up and place an object inside. The object wasscanned in 3D and its form became available
for digital manipulation. However important it
was as an inexpensive 3D scanner, I think it wasmore important as an example of a simple spatial
interaction with a computer that made sense to
everyone who tried it. How should we get an object
into the computer? Open it up and put it in. Thatis how we get anything into anything in the realworld. It broke the barrier of the screen by making
it a door that hinged open to reveal a usable spacebehind it.

13.2 System Description

Figure 13.3. The screen was embeddedin the door of a mini-fridge.

EyeBox was made out of mini-fridge, three
webcams, two fluorescent lights, a microwave
turntable, and a flat panel display. Any dark-coloredobject nine inches on a side or less could be placed
into the box, and in approximately twenty seconds,
the machine rotated the object once around andproduced a full volumetric reconstruction of it
from the visual hull of 24 silhouette images (eightfrom each camera taken during the rotation). The
user began by opening up the fridge. They would
place an object on the turntable inside, which had
hash marks around its edge. They would close thefridge, and the turntable would begin to spin. The
user would see the camera images from the threecameras displayed onscreen as the object rotated.
After a full rotation, the screen would change toa 3D projection showing the 24 silhouette images
in their positions around the platform, and an
iteratively refining 3D reconstruction of the object

91

on the platform. Over the course of the next few
minutes, the representation of the volume of theobject would get progressively finer until it reacheda resolution of 512 X 512 X 512 voxels. Then it was
filtered to smooth the voxels, giving it a smoothershape.

13.3 Motivation
A goal in the project was to keep costs low. Verynice 3D laser digitizers are available for $8,000.
EyeBox is not as accurate as these, but it cost$100 to build (minus the flat panel, which isnecessary to the spatial interaction but auxiliary to
the scanning). There is an obvious place for suchinexpensive devices in industries such as rapid
fabrication, design, and entertainment.
Less obvious, but perhaps more important in thelong term is the need for computers to be able
to recover geometries from the world simply to
be more useful in problems that are meaningful
to human beings. Computers are wonderfuldevices for cataloging objects. It would begreat to be able to catalog objects as full threedimensional reconstructions of themselves. These
representations could be sent to others and printed
out either locally or remotely, yielding respectively
a 3D copier, and a form teleporter. Museums might
be interested in this to catalog artifacts or to exhibit
pieces in a way that users could place them in acabinet to find out more about them. It could be
used to let people leave impressions of objects inplaces where they would not leave the actual object.

13.4 Technical Description
EyeBox used a technique called visual hullreconstruction to recover volumes from thesilhouettes of objects. Methods of visual hull92

processing fall loosely into three categories: imagebased [Matusik, 2000], polyhedral [Matusik 2001],
and volume carving [Szeliski, 1993]. All of thesetechniques rely on the same basic principle—that asilhouette relative to a calibrated camera produces
a generalized cone of volume in which the object
must be located. These cones from several cameras
can be intersected to produce a representationof the volume at which they are all looking. It
takes surprisingly few cameras to get a fairly good
approximation of most common shapes.

Figure 13.4. The silhouette of the vicerelative to the camera on top represents
a cone of volume in which the vice mustlie. The intersection of this cone fromeach camera is the visual hull.

Figure 13.5. Figure showing how acup’s rim always hides its inside insilhouette.

Techniques for reconstructing form from silhouette
data are all capable of producing its “visual hull”
relative to the views taken. Abstractly, the visualhull of an object is the best reconstruction thatcan be made of it assuming views from every
angle. The visual hull, as discussed in Petitjean, is
a subset of an object’s convex hull and a supersetof its actual volume envelope [Petitjean, 1998].
Specifically, a visual hull technique cannot everrecover a full topographical concavity, such as theinside of a bowl. Such an indentation will be filled
in by the visual hull. This is because the techniquereconstructs volumes from their silhouettes, andno matter what angle one views an object from, acomplete concavity will be obscured by its rim insilhouette.
Image-based
Image-based techniques are the fastest because
they do not reconstruct three-dimensional form
at all. Instead they synthesize new views fromany angle by selectively sampling from the source
images directly. Since there is no volumetricrepresentation produced, they are not suitable
for true volumetric reconstruction problems. It is
possible to imagine, however, reformulating many
volumetric problems as image-based problems.
For instance, volumetric object-matching may

93be construed as an image search for the bestreconstruction to match a given image of an
unknown object. The challenge would be making it
fast enough to search all possible orientations of all
possible matching objects.
Polyhedral
Polyhedral techniques produce a surface
representation of the object (easily converted
into a volumetric representation if required) bygeometrically intersecting polygonalized versionsof the cones. This is relatively quick, and provides
an unaliased representation without the need for
iterative refinement. Extensions to this techniqueare able deal with the hulls as splines to let them
curve as in Sullivan and Ponce [Sullivan, 1998].
Polyhedral techniques allow for easy texturemapping of the original images back onto thereconstructed surfaces, giving another level ofdetail. I implemented this technique in several
different ways, but each time I ran into the sameproblem: it is highly sensitive to calibration andnumerical error. It is imperative that the geometric
operations used to construct the volumes be
numerically robust and have adjustable geometrictolerances. Methods for general volumetricintersection (constructive solid geometry) that havethese necessary characteristics are challenging to
implement and difficult to find as free software
libraries. So although in theory this may be the bestclass of methods, it is very difficult to get it to work
reliably on real-world data.
Volume carving
This is the simplest technique to implement andalso the slowest. It projects voxels from world space
onto each of the camera views. If a voxel projection
falls fully outside any of the silhouettes, it can bediscarded. This produces an explicit volumetricrepresentation at the cost of voxel aliasing and lots

94

Figure 13.6. Image-based visual hulls
from [Matusik, 2000].

Figure 13.7. A polyhedral reconstruction of a vice produced by the first version of EyeBox.

Figure 13.8. Gumby as volume-carved
by a single camera from the top.

of computation. I implemented it because I wanted
a volumetric representation for matching purposes
and it was the easiest to produce. Because of its
aliasing it is also somewhat more tolerant of error
in camera calibration than the polyhedral method.
This proved to be a significant advantage in theturntable driven scanner.

Figure 13.9. An octree node may be
subdivided into eight subnodes.

Figure 13.10. An object after smoothing.

Octree subdivision
Having chosen the volume carving method, I sped
it up by representing the volume as an octree. Thatis a recursively refined volumetric tree startingwith a root node representing the entire volume
to be scanned. When a projected node was foundto be cut by the silhouette from any camera, it was
divided into eight subnodes. This way whenevera large node was found to be outside of any of the
projections, it needed never to be subdivided orotherwise considered again. This sped processing
up dramatically. Another speed advance was toiteratively refine the octree representation by
one level at a time, running it on each camera ateach level. That way more large octree nodes were
rejected earlier, and did not slow it down. Octree
nodes that were wholly inside each silhouette weremarked too, so that on each iteration, the only
nodes that had to be processed were nodes that in
the previous level intersected silhouette boundaries
in some camera. This was tantamount to finding
the substantial structures early and then iteratively
refining their surfaces. It also meant that the user
saw the form improving over time and was free tostop the process whenever it got to a level of detailthey were happy with. I smoothed the surface byapplying a Gaussian filter to the voxel data and thenfinding an isocontour.

13.5 Precedents
Figure 13.11. A swept-laser 3D digitizer
from Minolta.

There are existing 3D digitizers which work moreaccurately than EyeBox. One such family measures95

parallax in a laser beam that it sweeps over thesurface of an object. These return point clouds
which can then be surfaced. They are, however, twoorders of magnitude more expensive than EyeBox.
The technique of reconstructing volume fromsilhouette data is not new. It is well worked out
and documented in a variety of sources. But it issurprisingly rarely implemented. Perhaps thatis because of a fear of incompleteness—if it can’t
promise to accurately digitize an arbitrary form,
what good can it be? I propose that a rough scancan be quite useful. Typical setups for the process
involve a single well-calibrated camera viewing an
object on a turntable as in Kuzu and Rodehorst[Kuzu, 2002]. The turntable is turned by hand or
motorized to provide an arbitrarily large numberof silhouette images to be acquired from a single
camera.
Fixed multiple camera setups exist, notably
Matusik, Buehler, and McMillan’s [Matusik, 2001],
which is capable of scanning people in a roomin real time. This setup requires a computer per
camera and one more as a central processor, so
it doesn’t qualify as a low-cost solution, but theirresults are stunning. It is also not designed forscanning palm-sized objects.

Figure 13.12. Four 1/10th life-size people from 54 Personen 1:
10, 1998-2001, by Karin Sander. [http://www.karinsander.de]96

Figure 13.13. The iterative refinementof Gumby. The user can stop the process
at any time.

There are also cultural precedents related to 3D
scanning. Karin Sander took full body scans of 54
people and then mechanically reproduced themat 1/10th life size. She then painted them fromphotographs and displayed them on pedestals.

13.6 Design and Operation
EyeBox as a mini-fridge is a second generation of
the system. The first was less refined.
Figure 13.14. The first version washoused in a foamcore box.

Figure 13.15. The six cameras were
disposed as shown.

Version 1
The first version, a foamcore cube 18 inches ona side with six cameras at fixed locations andno turntable, was quite successful. The camera
positions in the original version had to becarefully chosen to deliver the most amount of
non-redundant information. Therefore they were
not one-to-a-side, as might be supposed. Views
separated by close to 180° are primarily redundantfor silhouettes.
The first step in the construction was thedismemberment of the webcams. Then I built an
18” X 18” X 18” cube out of foamcore and put aplexiglass shelf in it 7” from the bottom. I cut holes
in the sides and top for the cameras and attached
two small fluorescent lights to the inside. I thencalibrated the cameras by Tsai’s method embedded
in a custom application I wrote for the system [Tsai,
1987]. Then I was ready to write the reconstruction
software.
The first step was to acquire a silhouette image
from each camera, which was very easy because of
the well-controlled imaging environment. For eachcamera, I simply subtracted an image of the empty
box and then thresholded the results.

Figure 13.16. With the lights on andthe original calibration cube inside .

97

The reconstruction proceeded as detailed in theoctree method outlined above.
Problems
There were some problems with the reconstructedobjects. Many of them had to do with the whitebackground. Light colored objects did not scan wellat all. Specularities on objects are always white andtended to be seen as background, drilling holes inobjects. In a future version of the system, I would
use a blue background to make segmentation
simpler. Reflections off the plexiglass were
troublesome. Finally, the box was rather large for
an effective scanning volume of 6” X 6” X 6”. That
could have been improved with wider angle lenses,
but the wider the field of view, the lower the qualityof the reconstruction. There were also errors ofvolume just due to spaces not visible to any camera.
This could have been helped with more cameras.

Figure 13.17. Gumby reconstructed.
The images on the left show the silhouetting process.

The second version of the system set out to solve
some of these problems. It used a rotating platterto effectively multiply the viewpoints from threecameras into 24. The rotating platform also helped
shrink the necessary size of the system. Sincecameras were only looking at the object fromone side, it was the only side that needed visual
clearance. It imaged against a rounded backgroundto get rid of dark corners in the empty volume.

Version 2
Version 2 was housed in a mini-fridge. I chose amini-fridge because it fairly closely matched thedimensions I determined were optimal, and I
could not resist the feeling of the seal made by afridge door. I gutted the fridge and drilled a holein the back to run cables out. I decided to orient
it standing up rather than lying down so as not
to evoke a coffin. Instead it is very clearly a mini-

98

Figure 13.18. Version 2 under construction.

Figure 13.19. The three cameras show
real-time feeds on the display.

Figure 13.20. System diagram.

fridge, and its hybridity is part of its strong appeal.
I used a water-jet cutter to cut out a large opening
in the door and mounted an Apple Cinema Display
in it. I salvaged an AC gearhead motor from a old
microwave turntable and mounted it inside thefridge with a shaft and a plexiglass turntable on
it. I glued three webcams to the interior of thefridge looking slightly off-center at the turntable. I
turned them off-center to maximize the probability
that they would perceive the edges of objects—
the source of all of my information. I was notconcerned that they might not be able to see bothedges at once because I rotated every object a full
360°. I disassembled two small fluorescent lights
and mounted them inside the cabinet pointing
directly back onto the curved white back surface.
My hope was that this would completely backlightthe subject and get rid of all the problems with
specularity. In fact it ended up still giving a strong
side light. I mounted a reed switch on the door
hinge to control the platter motor. When the door
closed, the platter spun.
My setup avoided having to carefully control the
speed or position of the turntable by placing black
marks at its edges in 45° increments. The total lightvalue from a small patch of the camera looking
from the top was used to determine when theturntable was in position to use a single video framefrom each camera as a still image from one angle.
Two of the marks were not black—one was red,and one was cyan. These were present to indicate
the starting position (considered zero degrees),
and the direction the platform was spinning. It
was necessary to determine the direction in realtime because the turntable motor was a cheap AC
motor lifted from a microwave, and it was therefore
impossible to know which direction it would turnwhen power was applied.

99

I calibrated the cameras by the same procedure as
the first version. Because I had not constructed thewhole system to engineering tolerances, I calibrated
each of the 24 views by hand rather than calibrating
three and performing rotations on them.
Results
All of the changes proved to be advantageous,and my results were somewhat better with the
new system. The biggest disappointment was
how little it improved. The fantastic advantage ofthe technique is that it takes so little information
to give very good results. After the first several
cameras, adding more gives diminishing returns.
It may be that 24 views is more than is necessary,and rotating the object may therefore be as well.
With the current cost of webcams at about $15,
maybe I should just settle for 12 in a stationary
setup. Not rotating has several advantages—easier,more consistent calibration, no moving parts, faster
operation. The primary advantage, though, to not
rotating the object is the improved magical qualityof producing a transformable 3D reconstruction
from an object that is totally stationary.

13.7 Analysis and Critique
EyeBox as a tool is more interesting as part of aworkflow that passes objects between physicality
and virtuality with ease. What it needs in orderto do this is a matching rapid output device. The
dominant model for architectural sketching is withlayers of trace paper placed down over the top of
an existing sketch. The sketch evolves as the layers
pile up. A means to rapidly reproduce form could
introduce this technique to three dimensionalmodeling.
Several people have pointed out that EyeBox would
do very well coupled with Installation. Real objects

100

Figure 13.21. A scanned object. (Alump of modelling clay that looked a lot
like this.)could be virtually duplicated and placed in otherparts of the space. Or an object that is needed as areference could be duplicated and placed virtuallyin the room before the actual object is removed. I
look forward to bringing to two together. It should
not prove too difficult.

Figure 13.22. A physical sculpture ofa skull that has been warped by computational transformation and then rebuiltby hand in bone. [Robert Lazzarini,“skulls,” 2000]

A question that EyeBox naturally raises is whatis the relationship of a formal duplicate to theoriginal. It is a simulacrum, but functional only inso much as the function of the original derives fromits form. And as a purely digital representation it
can serve no physical purpose without transduction
back into the real world. The power of the digitalmodel is its extreme malleability. We can operateon it in ways we could not imagine working on
physical objects. We have used the idealizedmachine space to break physical law. Hybrid real/virtual form-making promises to push future designin new directions.

101

14. Summary Conclusions
In the course of building and analyzing six
separate experiments in spatial computing, a
few fundamental design principles have become
apparent. These are not highly theoretical orcomprehensive but do offer a set of guidelines for
implementations of computational systems thatengage real space. I have not done enough to put
forward a comprehensive theory. My hope is thatthat will never be possible, allowing me to work
productively toward it for the rest of my career. The
variousness of the experiments has been essential
to the inductive process. The qualities evident
from experiments so widely disparate in scale andapproach are likely to retain some validity over theentire field.
Rather than restate everything from the analyses of
the individual experiments, here I will organize theresults briefly by theme and then propose a visionof a future application for the field.

14.1 Editing
Certain decisions must always be made in how thereal world will enter into the system and vice versa.
These are essentially questions of editing.
Suggestion vs. realism
The power of the live video feed in Installation
demonstrated that often a suggestion of the real isenough (particularly if the system is not immersive,
so that the simultaneous experience of the real is
not denied). The camera was not much to add, andit certainly did not fool the eye. But it establishedcontext of the interaction. The mind did the rest of
the work.
This is closely related to the kind of abstraction
at work in LittleVision, which presented a highly102pixilated, low-bandwidth representation of a visual
scene. It was just enough to indicate what wasgoing on. The work of the mind to bring the scene
into focus created an attachment to the interaction.
The understanding was a collaboration between theobject and the observer.
The message is this: Do not try to fool the senses!
It will not work. The senses are canny and aware ofmuch more than we will ever be able to simulate.
The closer we try to approximate reality, the more
noticeable our failure will be. Instead, we mustsuggest what we want the experience to be and trust
the mind of the user to do the rest.
Literalness vs. metaphor
Donald Norman in The Invisible Computer inveighsagainst metaphor, arguing that it always breaks
down somewhere [Norman, 1998]. Since there isno true equivalence there must be difference thatthe user cannot know when to expect. This may betrue, but I think a worse crime of metaphor is its
obscuring the identity of something real. However,there is often no real referent for the metaphors ofcomputer interface, making them essentially freefloating and instantly changeable. Hayles calls them“flickering signifiers” [Hayles, 1999; 31]. There is afantastic convenience to working inside a metaphor
factory that runs on nothing. There is no need forthe cumbersome limitations and specificities of thereal. This cannot be denied, and I do not think that
all computer interface should give up metaphor. I
believe that there is currently no better alternative
for the control of most existing software. However,it turns the screen into an impenetrable barrier,and it has no place in spatial computing, which has
as its fundamental objective the union of the realand computed. Objects should be themselves andshould not have to be referenced by an icon or a
name.

103

In graphical interfaces for all of these projects
combined, I believe I used a total of three buttons
and zero icons. I consider the buttons a mistake.
Design vs. engineering
The physical objects involved must be theproducts of design, which goes beyond a processof engineering. They should be approachable andpleasing. They should not deny their physicality bytrying to disappear, but use their forms for all of
their potential value.
There are many factors at work in whether an
object will have resonance with an audience. I
identify two.
First, if it is to be held, it must have a form that is
pleasing in size, weight, and texture. LittleVisiondemonstrated this.
Second, an object needs its own identity andautonomy. It must have no wires trailing off
it unless the wires are specific to its identity.
Besides making an object a crippled dependent onsomething external, wires restrict its manipulation.
A helpful technique, not exclusive of the others,
is to use objects with existing resonance and repurpose them. The mini-fridge cabinet of EyeBox
gave it an appeal to many that no custom cabinet
could have.

14.2 Qualities of interactions
One of the fields closely related to spatial
computing calls itself interaction design. The field
does not limit itself to interaction with computers,
but extends to other products of design such asarchitecture. The notion that interaction itself can

104be object of intentional design is an importantone, and my experiments indicate several essential
characteristics of interaction in spatial computing.
Full vs. limited
The ways a system appears to be usable are often
called its “affordances” [Gibson, 1979]. The
affordances of a successful spatial computation
system must be implemented so fully that there are
no invisible barriers to its operation that disturb theillusion it is working to create.
My experiments achieved various levels of fullness.
The affordances of Installation were the handling of
the screen and the stylus. These were both heavily
cabled, and therefore fell short of the fulfillment of
free manipulation. EyeBox’s shortcomings were in
the objects it did not handle well. The fullness of
the interaction was disturbed and questions of thesystem’s competence were unavoidable. LittleVision
was perhaps the fullest project by these standards.
It was entirely self-contained, and it did everythinga user might expect it to.
Feedback vs. one-way control
Feedback is essential to human control. The levels
and kinds of feedback offered by spatial systems
dramatically influence their usability.
Immediacy of feedback is the single most importantquality of interaction. We are able to control ouroperations in the world only relative to feedback we
receive about how they are proceeding. If a system
limits such feedback it becomes difficult to control.
This was apparent in almost every experiment.
Predictablity vs. inconsistency
The second ingredient in accommodating humancontrol is not frustrating expectation. A user’s
desire to control a system should require as
little conscious effort to achieve as possible.
105

This demands total consistency in operation andgratification of expected behavior. These are thefoundations that allow us to plan action based on
intention. This does not preclude surprise as anelement, but too much surprise in the control of asystem is just uncontrolability.
However, as demonstrated by Stomping Ground,perfect controllability is not always the goal. The
system itself seemed to have something of a willof its own. Users knew they were effecting theoutcome, but it was not quite clear how literally
or directly. For a piece that had a strong artistic
identity of its own, this turned out to be quite
appropriate. It engaged users in a dialog withthe system through which they were trying tounderstand their relationship to the sound andgraphics.

14.3 Shortcuts
Spatial computing is starting to seem like a lot ofwork. But there are welcome shortcuts that help
make such systems possible to produce without
years of effort and outrageous expense.
Relativity of perception
The relativity of sensory experience is something
to leverage. Since we have no absolute benchmarks
it is often not necessary to calibrate to a hardexternal standard, which can be extremely
difficult. This excludes certain applications thatdemand calibrated precision, such as surgicalenhancement. But many fields (entertainment,
design, visualization, for instance) remain open toa system that delivers the experience of real spatial
interaction without being absolutely true.

106

Opacity
Some systems should become transparent—
essentially unnoticeable to their users. Some should
remain solid and visible. There is no hard rule,
contrary to some opinions [Norman, 1998], that say
all successful systems become transparent. Much
depends on the intended focus of user attention. In
many cases the system itself is part of what should
be experienced. The extent to which a systemshould assert its presence must be considered andcontrolled closely by its designer.
Some systems exist to disappear. That was true of
Installation, for instance. The screen was to be seen
through. By contrast in EyeBox, the screen had an
important physical identity as a door to a cabinet.
It did not attempt to disappear. LittleVision always
remained present. Its presence as an object and its
unwillingness to render a photographic image kept
it from disappearing. Stomping Ground was verymuch a solid presence. It was meant to be seen.
Many virtual environments and systems foraugmented reality see transparency as their goal.
This is may be a design consideration for the
specific project, but it is not intrinsic to the field.
Visibility is part of a system’s not apologizing for
its existence. Since no system yet has become soseamlessly integrated into our sensory apparatus
that it truly disappears, it behooves systems to deal
responsibly with their visibility rather than pretend
to be invisible and fail.

10714.4 The Future of Spatial Computing
Technologies
We are beginning to see issues of space andinterface enter the mainstream computer market
as wireless networking and the tablet PC become
popular. These technologies respectively encourage
mobility and treatment of the screen as a literalspace. It remains, however, a planar barrier.
Display
OLED and other display technologies are goingto make digital displays larger, thinner, lighter,brighter, and more flexible. We are going to seethem augment many fixed-appearance surfaces
such as desks, walls, ceilings, and billboards. Thereis an opportunity for these surfaces to become morethan fixed surfaces, to fold, open, wrap, flip, stretch,bend, hinge, or twist. I hope these potentials are
realized.
Another class of display that is becoming available
is the 3D display. The Spatial Imaging group at theMedia Lab has a new 3D display that is capable ofshowing viewers full-color hologram-like images
from digital files. The illusion of depth is very
good. Displays like this one, particularly if they
can be transparent where there is no object, havethe potential to blend the real and virtual quite
powerfully.

Tracking
Tracking objects in space must become morereliable and affordable before we see it usedcommonly. GPS works well outdoors, but it isuseless inside or for precise location. Improvements
will come from either advances in computer vision

108

Figure 14.1. Some laptops now allow
their screens to be flipped and folded
down to make the machine into a tablet
controlled with a stylus. This is opens up
a whole variety of spatial interactions.

Figure 14.2. An image of a vice as seenon the new 3D display from the Spatial
Imaging Group at the MIT Media Lab.
(The vice was scanned with EyeBox.)algorithms or new physical technologies for spatial
sensing.
Scanning
Scanning objects in 3D will become faster, cheaper,and easier. It is unlikely that 3D scanners will be
popular devices for the home unless there is acompelling application other than design. Most
people do not own flatbed scanners today.
Printing
3D printing will become tremendously valuable
as the process speeds up and the machines andmaterials drop in price. There is no reason thatmany objects that are primarily formal cannot besold as digital representations and then printed
locally.

Applications
As the technologies improve we will begin to ask
more of our interactions with our machines. We
will demand to be treated as more than ten fingers
and a wrist.
Rather than try to augur the entire future of
the discipline, let me paint a picture of a single
application: a workplace with a spatially-sensitive
network “ZapNet” in place.
Every employee is issued a laser-pointer-like device
called a ZapStick. This is an enhanced pointabledevice capable of sending data over its beam andreceiving it from a wireless LAN. Every display
device in the office, be it a projection surface,laptop or desktop computer screen is outfitted withsensors to detect and read the ZapStick beam.
Figure 14.3. The ZapStick.

Any information being displayed can be sucked up
into a ZapStick or squirted out onto a display by109

one (provided the user is authorized to have it).
Meeting notes can be shot out to participants asthey arrive. Notes and files can be shot between
machines during the meeting. Further, participants
can scribble on the projection with their pointers
during the meeting, and that writing will be saved
and incorporated as an annotation layer on the file.
Large stores of information could be associatedwith physical objects around the office. In
addition to a networked server with a specificname, an artifact such as a building model in anarchitectural office could be the locus of all thedigital information pertaining to the project. The
ZapSticks could be used to pull information fromthis site or send information to it. Groups of thesemodels could be gathered together and brought tomeetings or conversations where they are germane.
Of course all of these digital interactions andsharing of information is possible without physicalanalogs, but it is what a technology makes easierthat it promotes. In this case, the augmentation of adata network with a physical network promotes theflow of digital information on top of existing social
interactions. We tie the space of data flow closer tothe space of conversation.
This is one small idea for spatial computation. It is
all too easy to imagine others. The real learning will
come in building them.

11015. References[Absolut, 2002]

WindowsVR, www.abs-tech.com/Produtos/3D_VR/VRHardware/hmds1/Virtual_Research/win_vr.html, 2002.

[Apple, 1996]

Apple Computer’s Hotsauce, http://www.inxight.com/news/
apple_initiative.html.

[Art+Com, 1997]

Art+Com, Virtual Car for Daimler-Benz AG, 1997,
www.artcom.de.

[Arthur, 1993]

Arthur, K. W., Booth, K. S., and Ware, C., Evaluating 3D task
performance for fish tank virtual worlds, ACM Transactions
on Information Systems, vol. 11, no. 3, July 1993, pp. 239-265.

[Azuma, 1997]

Azuma, R., T., A survey of Augmented Reality, in Presence:
Teleoperators and Virtual Environments, 6, 4, 1997, p. 355385.

[Bachelard, 1964]

Bachelard, G., The Poetics of Space, Jolas, M., trans., The
Orion Press, 1964.

[Barabási, 2002]

Barabási, A., Linked: The New Science of Networks, Perseus
Publishing, 2002.

[Baudrillard, 1988]

Baudrillard, J., The Ecstacy of Communication, New York:
Semiotext(e), 1988.

[Billinghurst, 2002]

Billinghurst, M. and Kato, H., Collaborative Augmented
Reality. Communications of the ACM, 45(7), 2002, pp. 64-70.

[Boys, 1959]

Boys, C.V., Soap Bubbles, Their Colors and Forces which Mold
Them. Dover Publications, 1959.

[Brooks, 2001]

Brooks, R., The Relationship Between Matter and Life, in
Nature 409, pp. 409-411; 2001.

[Brown, 1997]

Brown, K. N., Grammatical design, IEEE Expert: Intelligent
Systems and their Applications 12, 1997, pp. 27-33.

[Buxton, 1997]

Buxton, W., Living in Augmented Reality: Ubiquitous Mediaand Reactive Environments. In Finn, K., Sellen, A., andWilber, S. (Eds.). Video Mediated Communication. Hillsdale,
N.J.: Erlbaum, 1996; pp. 363-384.

[Card, 1996]

Card, S. K., Robertson G. G., York W., The WebBook and the
Web Forager: An information workspace for the World Wide
Web. Proc CHI ‘96 ACM Conference on Human Factors in
Computing Systems. New York: ACM Press, 1996; pp. 111-116.

[Carroll, 2001]

In Human-Computer Interaction in the New Millennium,
John M. Carroll, ed.; Addison-Wesley, August 2001, pp. 579601.

[Colby, 1992]

Colby, G. and Scholl, L., Transparency and Blur as Selective
Cue for Complex Information. Proceedings of SPIE’92.

111

[Conroy, 2002]

Dalton, R. C., Is Spatial Intelligibility Critical to the Designof Large-scale Virtual Environments? International
Journal of Design Computing, vol. 4, 2002, http://
www.arch.usyd.edu.au/kcdc/journal/vol4/dalton.

[Cubitt, 1998]

Cubitt, S., Digital Aesthetics, SAGE Publications, Ltd.,
London, 1998.

[Deering, 1992]

Deering, M., High resolution virtual reality, in Computer
Graphics, 26, 2, 1992, pp. 195-202.

[Donath, 1995]

Donath, J., Visual Who: Animating the affinities and activities
of an electronic community, ACM Multimedia 95 - Electronic
Proceedings, November 5-9, 1995, San Francisco, California.

[Dorsey, 1996]

Dorsey , J. and Hanrahan, P., Modeling and Rendering ofMetallic Patinas. Proc. of SIGGRAPH ‘96. In Computer
Graphics Proceedings, Annual Conference Series, 1996, ACM
SIGGRAPH, pp. 387-396.

[Dourish, 2000]

Dourish, P., Edwards, W.K., et. al., Extending Document
Management Systems with User-Specific Active Properties.
ACM Transactions on Information Systems, 2000. 18(2), pp.
140-170.

[Dunne, 2001]

Dunne, A. and Raby, F., Design Noir, the Secret Life ofElectronic Objects, August / Birkhäuser, Berlin, 2001.

[Electronic Arts, 2003]

Electronic Arts, Sims Online, 2003, http://www.eagames.com/
official/thesimsonline/home/index.jsp.

[Feiner, 1997]

Feiner, S., MacIntyre, B., Höllerer, T., and Webster, A., A
Touring Machine: Prototyping 3D Mobile Augmented Reality
Systems for Exploring the Urban Environment. In Proc. ISWC
’97 (Int. Symp. on Wearable Computers), pp. 74–81, 1997.

[Fitzmaurice, 1993]

Fitzmaurice, G. W., Situated information spaces and spatiallyaware palmtop computers, Communications of the ACM,
Special issue on Augmented Reality and UbiComp, July 1993,
36(7), p.38-49.

[Fry, 1999]

Ben Fry, Anemone, 1999, http://acg.media.mit.edu/people/
fry/anemone.

[Gibson, 1979]

Gibson, J.J., The Ecological Approach to Visual Perception.
Houghton Mifflin,Boston, 1979.

[Gombrich, 1969]

Gombrich, E. H., Art and Illusion, a study in the Psychologyof Pictorial Representation, Princeton University Press,
Princeton, NJ, 1969.

[Hayles, 1999]

Hayles, N. K., How We Became Posthuman, The University of
Chicago Press, Chicago & London, 1999.

[IrDa, 2000]

IrDA, Infrared Data Association, Point and ShootProfile, March 20, 2000, http://www.irda.org/standards/
specifications.asp.

[Ishii, 1997]

Ishii, H. and Ullmer, B., Tangible Bits: Towards Seamless
Interfaces between People, Bits and Atoms, in Proc. of112

Conference on Human Factors in Computing Systems (CHI‘97), (Atlanta, March 1997), ACM Press, pp. 234-241.
[Karahalios, 1998]

Karahalios, K., Diorama. in Conference abstracts andapplications: SIGGRAPH ‘98, ACM Press, New York, p. 297.

[Kline, 1999]

C. Kline and B. Blumberg, The Art and Science of Synthetic
Character Design. Convention of the Society for the Study ofArtificial Intelligence and the Simulation of Behavior (AISB),
Symposium on AI and Creativity in Entertainment and Visual
Art, Proceedings, Edinburgh, Scotland, April, 1999.

[Kuzu , 2002]

Kuzu, Y., and Rodehorst, V., Volumetric Modeling Using Shape
From Silhouette, 2002, www.fpk.tu-berlin.de/forschung/sonder/pub/DT4_kuzu.pdf.

[Laibowitz, 2003]

Laibowitz, M., Paradiso, J., Phenomenological Model forDistributed Systems, http://www.media.mit.edu/resenv/
phenom, 2003.

[Le Corbusier, 1923]

Le Corbusier, Towards a New Architecture, 1923, DoverPubns; (February 1986).

[Linden Labs, 2003]

Linden Labs, Second Life, 2003, http://lindenlab.com.

[Loos, 1908]

Loos, A., Ornament and Crime, 1908, in Ornament andCrime: Selected Essays, Mitchell, M. trans., Ariadne Press,
1998.

[Manovich, 1996]

Manovich, L., The Aesthetics of Virtual Worlds: Report from
Los Angeles, CTHEORY, www.manovich.net, 1996.

[Manovich, 2002]

Manovich, L., Generation Flash, www.manovich.net, 2002.

[Matsuoka, 2002]

Matsuoka, H., Onozawa, A., Sato, H., Nojima, H., Regenerationof Real Objects in the Real World, Conference Abstracts andApplications of SIGGRAPH 2002, p.77,p.243.

[Matusik, 2001]

Matusik, W., Buehler, C., and McMillan, L., Polyhedral Visual
Hulls for Real-Time Rendering, in Proc. Twelfth Eurographics
Workshop on Rendering, 2001, pp. 115-125.

[Matusik, 2000]

Matusik, W., Buehler, C., Raskar, R., Gortler, S., and McMillan,
L., Image-based Visual Hulls, in Proc. SIGGRAPH, 2000, pp.
369-374.

[Milgram, 1999]

Milgram, P., Colquhoun, H., A Taxonomy of Real and Virtual
World Display Integration, in Mixed Reality, (Ohta, Y.,
Tamura, H., eds.), Ohmsa Ltd., Tokyo, 1999.

[Nakayama, 1985]

Nakayama, K., Biological image motion processing: A review.
Vision Research, 25, 1985 , 625-660.

[Norman, 1998]

Norman, D., The Invisible Computer, MIT Press, Cambridge,
MA, 1998.

[Paquette, 2001]

Paquette, E., Poulin, P., Drettakis, G., Surface Aging by
Impacts, Proceedings of Graphics Interface 2001 , June 2001.

113

[Paradiso, 1997]

Paradiso, J., Abler, C., Hsiao, K. Y., Reynolds, M., The Magic
Carpet: Physical Sensing for Immersive Environments, in Proc.
of the CHI ‘97 Conference on Human Factors in Computing
Systems, Extended Abstracts, ACM Press, NY, pp. 277-278(1997).

[Paradiso, 1999]

Paradiso, J., The Brain Opera Technology: New Instruments
and Gestural Sensors for Musical Interaction and
Performance, Journal of New Music Research, 28(2), 1999, pp.
130-149.

[Paradiso, 2000]

Paradiso, J., Hsiao, K., Benbasat, A., Teegarden, Z., Design and
Implementation of Expressive Footwear, IBM Systems Journal,
Volume 39, Nos. 3 & 4, October 2000, pp. 511-529.

[Pausch, 1997]

Pausch, R., Proffitt, D., and Williams, G., Quantifying
immersion in virtual reality, SIGGRAPH’97.

[Pessoa, 2003]

Pessoa, L., de Weed, P., eds., Filling-in: From Perceptual
Completion to Cortical Reorganization, Oxford University
Press, 2003.

[Petitjean, 1998]

Petitjean, S., A Computational Geometric Approach to Visual
Hulls, Int. J. of Computer. Geometry and Appl., vol. 8, no.4, pp.
407-436, 1998.

[Piekarski , 2002]

Piekarski, W., and Thomas, B. H., Unifying Augmented Reality
and Virtual Reality User Interfaces , Technical report, January
2002, University of South Australia.

[Rekimoto, 1995]

Rekimoto, J., The Magnifying Glass Approach to Augmented
Reality Systems, International Conference on Artificial Reality
and Tele-Existence ‘95 / Conference on Virtual Reality Software
and Technology (ICAT/VRST ‘95).

[Robertson, 1997]

Robertson, G., Czerwinski, M., and van Dantzich, M.,
Immersion in Desktop Virtual Reality, UIST’97.

[Saunders, 2001]

Saunders, R., Simplified ART, http://www.arch.usyd.edu.au/~rob/java/applets/neuro/SimplifiedARTDemo.html, August
2001.

[Selfridge, 1999]

Selfridge, P. and Kirk, T., Cospace: Combining Web-Browsing and Dynamically Generated 3D Multiuser Environments,
SIGART 10, 1, 1999, pp. 24-32.

[Shiffman, 2000]

Jared Shiffman, Honey, 2000, http://acg.media.mit.edu/
people/jarfish/honey.

[Slater, 2002]

Slater, M., Presence and the Sixth Sense, in PRESENCE:
Teleoperators and Virtual Environments, MIT Press, 11(4) ,2002, pp. 435-439.

[Sullivan, 1918]

Sullivan L., The Tall Office Building Artistically Considered,in: Athey I., ed. Kindergarten Chats (revised 1918) and Other
Writings. New York 1947: 202-13.

[Sullivan, 1998]

Sullivan, S., and Ponce, J., Automatic Model Construction, Pose
Estimation, and Object Recognition from Photographs Using
Triangular Splines, IEEE Transactions on Pattern Analysis and114

Machine Intelligence, 20(10):10911096, 1998.
[Szeliski, 1993]

Szeliski, R., Rapid Octree
Construction from Image
Sequences, CVGIP: Image
Understanding, 58, 1 (July 1993),
pp. 23-32.

[There, 2003]

There, Inc., There, 2003, http://
www.there.com.

[Tsai, 1987]

Tsai, R. Y., A versatile camera
calibration technique for highaccuracy 3D machine vision
metrology using off-the-shelf TV
cameras and lenses, IEEE Trans.,
1987, RA-3, pp. 323-344.

[Tsang, 2002]

Tsang, M., Fitzmaurice, G.,
Kurtenbach, G., Khan, A. andBuxton, W., Boom Chameleon:
Simultaneous capture of 3D
viewpoint, voice and gestureannotations on a spatiallyaware display. Alias|Wavefront.
Submitted for publication, 2002.
[http://www.billbuxton.com/
boomChameleon.pdf][Ullmer, 1997]

Ullmer B., Ishii H., The metaDESK:
Models and Prototypes for
Tangible User Interfaces. Proc. of
UIST’97, pp.223-232.

[Ullmer, 1998]

Ullmer, B., et al., mediaBlocks:
Physical Containers, Transports,
and Controls for Online Media,
in Proceedings of SIGGRAPH ‘98,
ACM Press, 1998, pp. 379-386.

[Ullmer, 2000]

Ishii, H. and Ullmer, B., Emerging
Frameworks for Tangible User
Interfaces, in IBM Systems Journal
Volume 39 Nos. 3&4, 2000; pp.
915-931.

[Van den Berg, 1955]

Van den Berg, J. H., The
Phenomenological Approach in
Psychology, Charles C. Thomas,
publisher. Springfield, Illinois, 1955,
p. 61. Quoted in [Bachelard, 1964;
xvii.][Ventrella, 1997]

Ventrella, J., Arrows in the Fluid
Mind, A 2D Fluid Dynamics
Model for Animation Based on
Intuitive Physics, 1997 , http://
www.ventrella.com/Ideas/Fluid/
fluid.html.

115

[Venturi, 2001]

Venturi, R., Robert Venturi’s Disorderly Ode, Metropolis,
Sept. 2001.

[Weiner, 1948]

Weiner, N., Cybernetics: Or Control & Communication in the
Animal and the Machine, MIT Press, Cambridge, MA, 1948.

[Weiser, 1988]

http://www.ubiq.com/hypertext/weiser/UbiHome.html[Witmer, 1998]

Witmer, B.G., Singer, M.J., Measuring Presence in Virtual
Environments: A Presence Questionnaire, Presence, 7 (3),1998, pp. 225-240.

[Yarbus, 1967]

Yarbus, A. L., Eye Movements and Vision. Plenum Press, New
York, 1967.

[Zhai, 1996]

Zhai, S., Buxton, W., and Milgram, P., The partial-occlusion
effect: Utilizing semitransparency in 3D human-computerinteraction, in ACM Transactions on Computer-Human
Interaction, 3(3), 254-284.

116

Appendix A:
Other Experiments
The six projects featured in the body of the thesis
are those that best describe and comment on spatial
computing. In the time of my thesis work, however,I completed a number of other projects with close
ties to my core ideas. I describe some of them here.

Tear Me
Tear Me is a line of six thermal print heads
suspended from the ceiling. When people walk
below it, they are sensed with infra-red proximity
sensors, and the printers near them print downat them. Each printer repeats one word over andover, and together they form a sentence: “let”“me” “know” “if I’m” “reaching” “you.” Viewers
are encouraged to tug on the paper, and to tear off
excess as it gets long and drop it on the floor at theirfeet.

WeaverFigure A.1. Two photos of Tear Me as
installed in a gallery. The proximity sensor is at the top edge of the board here.

Figure A.2. Weaver explored the formsof woven and braided strands.

Weaver is a dynamic formal exploration of woven
and braided patterns as a generator of architecturalform. It grew out of a project I began withProfessor Peter Testa when I was in the Architecture
department prior to my work at the Media Lab. He
was exploring woven carbon fiber as a structural
material for building, and I did some work creatinggrammars for encoding weaves and writing
software to render the patterns.
Two years later, while at the Media Lab, I had theopportunity to work with Testa again to update mywork, making it a dynamic, real-time, somewhat
interactive system. Users rolled a trackball toorient the growing shape in space or click one ofits buttons to freeze the form for a few seconds for
closer inspection. The form itself was an evolving
weave that shots around the screen, intersectingwith itself, exploding toward the user, and creating
patterns of seemingly tremendous complexity out of117

a few simple geometrical rules. The shapes depend
both on the description of the weave pattern andthe topology of the surface on which the weave isapplied.

Active Statics
I worked with professors emeritus Ed Allen and
Waclaw Zalewski to create a set of eight highlyinteractive demonstrations for teaching elementary
structural design to architecture or engineering
students. For maximum transparency and creative
potential, these demonstrations were based ongraphic statics, a body of techniques used bysuch masters as Antonio Gaudi, Gustave Eiffel,
and Robert Maillart to create their structural
masterpieces.
The external forces on each structure are plotted toa scale of length to force on a load line. Working
from the load line, the forces in the members of
the structure are determined by scaling the lengthsof lines constructed parallel to the members. The
diagram of forces that results from this process iscalled the force polygon. In these demonstrations,the force polygon is constructed automaticallyand revised instantly and continually as changes
are made in the form or loading of the structure.
The intensities of member forces are indicatedby the lengths of the line segments in the force
polygon, the numerical values in a table, and by
the thicknesses of the members themselves. Allcomponents of each demonstration are color-coded:Blue indicates tension, red is compression, yellow
is zero force. External loads are black and reactions
are green.
What makes these demos important is their
feedback. Students are able to see analyticalfeedback as a direct consequence of formal
manipulation. The directness of this feedback loop
is unprecedented, and it is a tremendous aid to thedevelopment of intuition. Previously, the analysis
stage had to happen subsequent to design, and thetwo processes were separate. There was a practical
118

Figure A.3. Three of the eight interactive teaching demos. Any yellow point
can be manipulated. The analysis updates in real time.

limit to the number of times any designer would
modify a design and re-run an analysis. Only
with the analysis baked directly into the processof design can the number of iterations become
essentially infinite.
The demos are currently in use for courses atseveral universities. They are publicly available atacg.media.mit.edu/people/simong/statics/data/.

MomentMoment is a pure parametric design twodimensional design environment. It implements
a visual data-flow language for the prototypingof dynamic graphics. The process of designingin Moment is one of connecting terminals withwires, spinning them together into bundles whennecessary.
Moment is the last incarnation of a project thathas had many aliases and iterations, first as aseries of two constraint-based languages called
RunningThings and Paragon, next as a visual dataflow language, Paramour, and now finally, revised,
retooled, and expanded as Moment.
Figure A.4. The visual data-flow language Moment.

Moment is available for download at
acg.media.mit.edu/people/simong/moment/.

High Definition

Figure A.5. High Definition exploredthe space of dictionary definitions.

High Definition is a piece that like Internaut maps
an undimensioned space to a virtual space. In
this case the space of origin is the dictionary. The
piece begins with a single dictionary definition ofword chosen in advance. The definition trails theword, flapping and waving in a circle in 3D spaceon the screen. Every so often one of the words ofthe definition quivers and explodes into its own
definition, starting a new circle. This processcontinues recursively, creating a boiling tangle ofwords and definitions. The user can explore thisdefinition space at will.
119Appendix B:
SBalls CodeSBalls are useful for making organiclooking blobby tube shapes like the oneson the right. They are implemented in
C++ and require OpenGL and GLUT.

SBalls.h#ifndef _SBALLS_H_#define _SBALLS_H_#define BALL_SLICES 30#define FATNESS_MULTIPLIER 1#define BALL_STACKS 10#define CAP_STACKS 20#define TOTAL_STACKS (((nNodes-1)*BALL_STACKS) + 2 * CAP_STACKS +1)#define VERTEX(slice, stack) (fullVertices+(((slice)*TOTAL_STACKS+(stack))*3))#define VERTEX_INDEX_GL(slice, stack) ((slice)*TOTAL_STACKS+(stack))#define NORMAL(slice, stack) (normals+(((slice)*TOTAL_STACKS+(stack))*3))#define QUAD_SLICE(slice) (quads+((sli ce)*TOTAL_STACKS*2))#define NODE_VERTEX(slice, node) (nodeVertices+(((slice)*nNodes+(node))*3))
enum {BALLS_AND_WIRES = 0, SURFACES, BALLS_AND_SURFACES, WIRES};// A node is a sphere in the
struct SBallNode {// Settablefloat
pos[3];// Starting node location
float
dPos[3];// Move this much on each update
float
radius;// Starting radius
float
dRadius;// Amount of radius change per update};// Internal
SBallNode *next;
SBallNode *previous;
int
rotOffset; // Used to control twisting of outside surface

class SBalls {public:// Public properties
int displayMode;
float
color[4];float
offset[3];booluseTexture;// How to draw: BALLS_AND_WIRES, SURFACES, BALLS_AND_SURFACES, or WIRES// Base color, RBGA// Offset of blob// Texture-map onto blob or not// Construct/desctruct
SBalls();~SBalls();// Add nodes
void appendNode(float pos[3], float radius);
void appendNode(float posX, float posY, float posZ, float radius);
void appendNode(float posX, float posY, float posZ, float radius, float dX, float dY, float dZ, float dR);
void getExtents(float *xMin, float *xMax, float *yMin, float *yMax, float *zMin, float *zMax);
void draw();

120// Issue the GL calls to draw the SBalls object// If you’re using a texture, you need to set it up before calling this// with glTexImage2D or some such call// Updates
void update();void silentUpdate();void calculate();// Update node positions and sizes and recalculate every surface point (slow)// Just update node positions and sizes (fast)// Recalculate every surface point (slow)

int nNodes;// The ends of the nodelist
SBallNode *startNode;
SBallNode *endNode;private:float*nodeVertices;
unsigned int *quads;
float*fullVertices;float*normals;
float*texCoords;// Data structure as straight vertices: SLICES * NODES * 3// Data structure as quad strips SLICES * ((NODES * STACKS) + 1) * 2// Data structure as straight vertices: SLICES * ((STACKS * NODES) + 1) * 3// Data structure SLICES * NODES * STACKS * 3;// Data structure SLICES * NODES * STACKS * 2;

SBallNode* previousIfExists(SBallNode *cur);
SBallNode* nextIfExists(SBallNode *cur);// Utility functions
float distSq(float vec1[3], float vec2[3]);void vecCross(float vec1[3], float vec2[3], float result[3]);void vecCopy(float in[3], float out[3]);void vecMinus(float a[3], float b[3]);void vecPlus(float a[3], float b[3]);void vecTimes(float a[3], float b);
void vecAverage(float a[3], float b[3]);void vecAverage(float fromA[3], float toA[3], float fromB[3], float toB[3], float average[3]);float vecMag(float vec[3]);void vecNormalize(float vec[3]);void vecRotate(float vec[3],double theta, float axis[3]);void catmullromPoint(int upToNode, int slice, float t, float curvePoint[3]);float catmullromBasis(int i, float t);void fillQuadStructure();void fillVertexData();void makeNormals();};

float tempVec[3];float averageVec[3];float averageVecOrigin[3];float tempVec2[3];float tempVec3[3];float circleAxis[3];float tempVec4[3];float tempVec5[3];float inFromPrevious[3];float inFromNext[3];#endif121

SBalls.cpp/*----------------------------------------------------------------*//* SBalls forms a texture-mapped blobby surface as a series of/* spheres of different radius located along a curve and connected by/* Catmull-Rom splines.
/*/* Documentation in header.
/*/* 10/07/01 Simon Greenwold/* Modified 5/17/03/*----------------------------------------------------------------*/#include “SBalls.h”#include <GL/gl.h>#include <GL/glut.h>#include <float.h>#include <math.h>/* Some <math.h> files do not define M_PI... */#ifndef M_PI
#define M_PI 3.14159265358979323846#endif#ifndef MIN#define MIN(x,y) (((x) < (y)) ? (x) : (y))#endif#ifndef MAX#define MAX(x,y) (((x) > (y)) ? (x) : (y))#endifSBalls::SBalls() {startNode = NULL;
endNode = NULL;
nNodes = 0;
displayMode = BALLS_AND_WIRES;
color[0] = 0.0;
color[1] = 1.0;
color[2] = 0.0;
color[3] = 1.0;
offset[0] = 0.0;
offset[1] = 0.0;
offset[2] = 0.0;
useTexture = FALSE;}
SBalls::~SBalls() {SBallNode *cur = startNode;
SBallNode *next;
while (cur != NULL) {next = cur->next;
delete(cur);
cur = next;}
if (nNodes > 1) {delete nodeVertices;delete quads;
delete normals;delete fullVertices;delete texCoords;}}
void SBalls::getExtents(float *xMin, float *xMax, float *yMin, float *yMax, float *zMin, float *zMax) {*xMin = FLT_MAX;*xMax = FLT_MIN;*yMin = FLT_MAX;*yMax = FLT_MIN;*zMin = FLT_MAX;*zMax = FLT_MIN;
float x, y, z, r;for (SBallNode* nodePtr = startNode; nodePtr != NULL; nodePtr = nodePtr->next) {r = nodePtr->radius;x = nodePtr->pos[0];if (x - r < *xMin)*xMin = x - r;
if (x + r > *xMax)*xMax = x + r;
y = nodePtr->pos[1];if (y - r < *yMin)*yMin = y - r;
if (y + r > *yMax)

122

*yMax = y + r;}}z = nodePtr->pos[2];if (z - r < *zMin)*zMin = z - r;
if (z + r > *zMax)*zMax = z + r;

void SBalls::appendNode(float pos[3], float radius) {appendNode(pos[0], pos[1], pos[2], radius);}
void SBalls::appendNode(float posX, float posY, float posZ, float radius) {appendNode(posX, posY, posZ, radius, 0, 0, 0, 0);}
void SBalls::appendNode(float posX, float posY, float posZ, float radius, float dX, float dY, float dZ, float dR) {nNodes++;SBallNode *newNode = new SBallNode;
newNode->pos[0] = posX;
newNode->pos[1] = posY;
newNode->pos[2] = posZ;
newNode->radius = radius;
newNode->rotOffset = 0;
newNode->dPos[0] = dX;
newNode->dPos[1] = dY;
newNode->dPos[2] = dZ;
newNode->dRadius = dR;
if (!startNode)startNode = newNode;
newNode->next = NULL;
newNode->previous = endNode;if (endNode)endNode->next = newNode;endNode = newNode;if (nNodes > 1) {delete nodeVertices;delete quads;
delete normals;delete fullVertices;delete texCoords;}
nodeVertices = new float[BALL_SLICES * nNodes * 3];
quads = new GLuint [BALL_SLICES * TOTAL_STACKS * 2];
fullVertices = new float [BALL_SLICES * TOTAL_STACKS * 3];normals = new float [BALL_SLICES * TOTAL_STACKS * 3];
texCoords = new float [BALL_SLICES * TOTAL_STACKS * 2];}

fillQuadStructure();calculate();void SBalls::calculate() {// Draw average vectors
SBallNode *cur = startNode;if (!cur->next)
return;float rotAngle = 360.0 / BALL_SLICES;
int numNode = 0;
int rotOffset = 0;
while (cur) {if (cur == startNode) { // First nodevecCopy(cur->pos, circleAxis);
vecMinus(circleAxis, cur->next->pos);vecCopy(circleAxis, tempVec2);
tempVec2[1] += 30;
tempVec2[2] += circleAxis[2] + 40;
vecCross(circleAxis, tempVec2, averageVecOrigin);
vecTimes(averageVecOrigin, FATNESS_MULTIPLIER * cur->radius / vecMag(averageVecOrigin));}else if (cur == endNode) {// last nodevecCopy(cur->previous->pos, circleAxis);
vecMinus(circleAxis, cur->pos);vecCopy(circleAxis, tempVec2);
tempVec2[1] += 30;
tempVec2[2] += circleAxis[2] + 40;
vecCross(circleAxis, tempVec2, averageVecOrigin);123

vecTimes(averageVecOrigin, FATNESS_MULTIPLIER * cur->radius / vecMag(averageVecOrigin));}else {// a node in the middle
vecCopy(cur->pos, inFromPrevious);
vecCopy(cur->next->pos, inFromNext);
vecMinus(inFromPrevious, cur->previous->pos);
vecMinus(inFromNext, cur->pos);vecAverage(cur->previous->pos, cur->pos, cur->next->pos, cur->pos, averageVecOrigin);}

vecCross(inFromPrevious, inFromNext, tempVec2);
vecCross(averageVecOrigin, tempVec2, circleAxis);
vecTimes(averageVecOrigin, FATNESS_MULTIPLIER * cur->radius / vecMag(averageVecOrigin));// Get positions
float tempPts[BALL_SLICES][3];rotOffset += cur->rotOffset;for (int slice = 0; slice < BALL_SLICES; slice++ ) {vecRotate(averageVecOrigin, rotAngle, circleAxis);
vecCopy(averageVecOrigin, tempVec5);
vecPlus(tempVec5, cur->pos);vecCopy(tempVec5, tempPts[slice]);}// Determine rotational offsets to prevent twisting
float minCost = FLT_MAX;
SBallNode* prev;
float cost;if (cur != startNode) {// Offset of start is always zeroprev = cur->previous;for (int rot = 0; rot < BALL_SLICES; rot++ ) {cost = 0.0f;
for (int i = 0; i < BALL_SLICES; i++ ) {cost += distSq(tempPts[(i + rot) % BALL_SLICES], NODE_VERTEX(i,numNode - 1));}if (cost < minCost) {minCost = cost;cur->rotOffset = rot;}}
printf(“Min cost %d: %f, %d\n”, numNode, minCost, cur->rotOffset);}for (int i = 0; i < BALL_SLICES; i++) {vecCopy(tempPts[(i + cur->rotOffset) % BALL_SLICES], NODE_VERTEX(i, numNode));}//cur = cur->next;
numNode++;}}if (nNodes > 1) {fillVertexData();
makeNormals();}void SBalls::fillQuadStructure() {int nextSlice;
GLuint* index;
for (int slice = 0; slice < BALL_SLICES; slice++) {nextSlice = (slice + 1) % BALL_SLICES;
index = QUAD_SLICE(slice);for (int stack = 0; stack < TOTAL_STACKS; stack++) {index[0] = VERTEX_INDEX_GL(slice, stack);
index[1] = VERTEX_INDEX_GL(nextSlice, stack);
texCoords[VERTEX_INDEX_GL(slice, stack) * 2] = ((float)stack) / (TOTAL_STACKS-1) * (640.0f / 1024.0f);
texCoords[VERTEX_INDEX_GL(slice, stack) * 2 + 1] = ((float)slice) / (BALL_SLICES-1) * (480.0f / 512.0f);

}

}

}

index += 2;

void SBalls::fillVertexData() {float t = 0.0f;
float* index = fullVertices;
float curvePoint[3];float stackInc = 1.0 / BALL_STACKS;
for (int slice = 0; slice < BALL_SLICES; slice++) {if (nNodes > 1) {for (int capStack = 0; capStack < CAP_STACKS; capStack++) {124

tempVec4[0] = NODE_VERTEX(slice, 0)[0] - startNode->pos[0];tempVec4[1] = NODE_VERTEX(slice, 0)[1] - startNode->pos[1];tempVec4[2] = NODE_VERTEX(slice, 0)[2] - startNode->pos[2];tempVec5[0] = startNode->next->pos[0] - startNode->pos[0];tempVec5[1] = startNode->next->pos[1] - startNode->pos[1];tempVec5[2] = startNode->next->pos[2] - startNode->pos[2];}}vecCross(tempVec5, tempVec4, tempVec3);
vecRotate(tempVec4, (90.0 / CAP_STACKS) * (CAP_STACKS - capStack), tempVec3);
VERTEX(slice, capStack)[0] = startNode->pos[0] + tempVec4[0];VERTEX(slice, capStack)[1] = startNode->pos[1] + tempVec4[1];VERTEX(slice, capStack)[2] = startNode->pos[2] + tempVec4[2];for (int node = 1; node < nNodes; node++) {t = 0.0f;
for (int stack = 0; stack < BALL_STACKS; stack++) {catmullromPoint(node, slice, t, curvePoint);
VERTEX(slice, ((node - 1) * BALL_STACKS) + stack + CAP_STACKS)[0] = curvePoint[0];VERTEX(slice, ((node - 1) * BALL_STACKS) + stack + CAP_STACKS)[1] = curvePoint[1];
VERTEX(slice, ((node - 1) * BALL_STACKS) + stack + CAP_STACKS)[2] = curvePoint[2];//printf(“Catmull-rom: %f, %f, %f\n”, index[0], index[1], index[2]);t += stackInc;
index += 3;}}
VERTEX(slice, (TOTAL_STACKS - 1))[0] = NODE_VERTEX(slice, nNodes - 1)[0];VERTEX(slice, (TOTAL_STACKS - 1))[1] = NODE_VERTEX(slice, nNodes - 1)[1];
VERTEX(slice, (TOTAL_STACKS - 1))[2] = NODE_VERTEX(slice, nNodes - 1)[2];if (nNodes > 1) {int i = 0;
for (int capStack = (TOTAL_STACKS - 1) - CAP_STACKS; capStack < TOTAL_STACKS; capStack++) {tempVec4[0] = NODE_VERTEX(slice, (nNodes - 1))[0] - endNode->pos[0];tempVec4[1] = NODE_VERTEX(slice, (nNodes - 1))[1] - endNode->pos[1];tempVec4[2] = NODE_VERTEX(slice, (nNodes - 1))[2] - endNode->pos[2];tempVec5[0] = endNode->previous->pos[0] - endNode->pos[0];tempVec5[1] = endNode->previous->pos[1] - endNode->pos[1];tempVec5[2] = endNode->previous->pos[2] - endNode->pos[2];

}

}

}

}

vecCross(tempVec4, tempVec5, tempVec3);
vecRotate(tempVec4, (-90.0 / CAP_STACKS) * i, tempVec3);
VERTEX(slice, capStack)[0] = endNode->pos[0] + tempVec4[0];VERTEX(slice, capStack)[1] = endNode->pos[1] + tempVec4[1];VERTEX(slice, capStack)[2] = endNode->pos[2] + tempVec4[2];i++;void SBalls::makeNormals() {int nextSlice;
float around[3];float down[3];for (int slice = 0; slice < BALL_SLICES; slice++) {nextSlice = (slice + 1) % BALL_SLICES;
for (int stack = 0; stack < TOTAL_STACKS - 1; stack++) {vecCopy(VERTEX(nextSlice, stack), around);vecMinus(around, VERTEX(slice, stack));
vecCopy(VERTEX(slice, stack + 1), down);
vecMinus(down, VERTEX(slice, stack));
vecCross(down, around, NORMAL(slice, stack));//vecNormalize(NORMAL(slice, stack));}}}
vecCopy(NORMAL(slice, TOTAL_STACKS - 2), NORMAL(slice, TOTAL_STACKS - 1));

void SBalls::silentUpdate() {for (SBallNode* node = startNode; node; node = node->next) {vecPlus(node->pos, node->dPos);
node->radius += node->dRadius;if (node->radius <= 0.0)
node->radius = .05;}}
void SBalls::update() {silentUpdate();calculate();125

}void SBalls::draw() {SBallNode *cur = startNode;glPushMatrix();
glTranslatef(offset[0], offset[1], offset[2]);glLineWidth(1.2);
glDisable(GL_TEXTURE_2D);// Draw balls
if (displayMode == BALLS_AND_WIRES || displayMode == BALLS_AND_SURFACES) {glColor4f(1.0, 0.0, 0.0, 0.5);
cur = startNode;while (cur) {glPushMatrix();
glTranslatef(cur->pos[0], cur->pos[1], cur->pos[2]);glutWireSphere(cur->radius, 20, 20);}}

cur = cur->next;
glPopMatrix();

if (nNodes <= 1) {glPopMatrix();
return;}// Draw wires
if (displayMode == BALLS_AND_WIRES || displayMode == WIRES) {//draw the fast way
glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(3, GL_FLOAT, 0, fullVertices);
glDisableClientState(GL_NORMAL_ARRAY);
glDisableClientState(GL_TEXTURE_COORD_ARRAY);
glDisable(GL_TEXTURE_2D);}

glColor4f(color[0], color[1], color[2], 0.5); //color[3]);for (int slice = 0; slice < BALL_SLICES; slice++) {glDrawArrays(GL_LINE_STRIP, VERTEX_INDEX_GL(slice, 0), TOTAL_STACKS);}// Draw surfaces
if (displayMode == BALLS_AND_SURFACES || displayMode == SURFACES) {glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(3, GL_FLOAT, 0, fullVertices);
glEnableClientState(GL_NORMAL_ARRAY);
glNormalPointer(GL_FLOAT, 0, normals);if (useTexture) {glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glTexCoordPointer(2, GL_FLOAT, 0, texCoords);
glEnable(GL_TEXTURE_2D);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);}
glColor4f(color[0], color[1], color[2], color[3]);for (int slice = 0; slice < BALL_SLICES; slice++) {glDrawElements(GL_QUAD_STRIP, TOTAL_STACKS * 2, GL_UNSIGNED_INT, QUAD_SLICE(slice));}
glDisable(GL_TEXTURE_2D);}
glPopMatrix();}//evaluate a point on the B spline// i may vary from 2 to POINTS - 2// beyond that interpolation breaks down
void SBalls::catmullromPoint(int upToNode, int slice, float t, float curvePoint[3]) {/*float basis[4];for (int j = -2; j<=1; j++){basis[j + 2] = catmullromBasis(j,t);}126

for (int i = 0; i < 3; i++) {curvePoint[i] = basis[0] * previousIfExists(previousIfExists(upToNode))->circlePoints[slice][i] +basis[1] * previousIfExists(upToNode)->circlePoints[slice][i] +basis[2] * upToNode->circlePoints[slice][i] +basis[3] * nextIfExists(upToNode)->circlePoints[slice][i];}*/float basis;
int index;
curvePoint[0] = 0.0f;
curvePoint[1] = 0.0f;
curvePoint[2] = 0.0f;
for (int j = -2; j<=1; j++){basis = catmullromBasis(j,t);
index = MIN(MAX(upToNode + j, 0), nNodes - 1);
float*
point = NODE_VERTEX(slice, index);//}

if (slice == 2)//printf(“Retrieving, slice, index, [0]: %i, %i, %f\n”, slice, index, NODE_VERTEX(slice,index)[0]);//printf(“In Catmull-rom: %f, %i, %f\n”, basis, index, point[0]);//printf(“ Still in Catmull-rom: %i, %i, %i\n”, slice, nNodes, index);}

curvePoint[0] += basis * point[0];curvePoint[1] += basis * point[1];curvePoint[2] += basis * point[2];// Catmull-Rom spline is just like a B spline, only with a different basis
float SBalls::catmullromBasis(int i, float t) {switch (i) {case -2:
return ((-t+2)*t-1)*t/2;
case -1:
return (((3*t-5)*t)*t+2)/2;
case 0:
return ((-3*t+4)*t+1)*t/2;
case 1:
return ((t-1)*t*t)/2;}
return 0; //we only get here if an invalid i is specified}
SBallNode* SBalls::previousIfExists(SBallNode *cur) {if (cur->previous)
return cur->previous;
return cur;}
SBallNode* SBalls::nextIfExists(SBallNode *cur) {if (cur->next)
return cur->next;
return cur;}float SBalls::distSq(float vec1[3], float vec2[3]) {return (vec1[0] - vec2[0]) * (vec1[0] - vec2[0]) +(vec1[1] - vec2[1]) * (vec1[1] - vec2[1]) +(vec1[2] - vec2[2]) * (vec1[2] - vec2[2]);}void SBalls::vecCross(float vec1[3], float vec2[3], float result[3]) {result[0] = (vec1[1] * vec2[2]) - (vec2[1] * vec1[2]);result[1] = (vec1[2] * vec2[0]) - (vec2[2] * vec1[0]);result[2] = (vec1[0] * vec2[1]) - (vec2[0] * vec1[1]);}void SBalls::vecCopy(float in[3], float out[3]) {out[0] = in[0];out[1] = in[1];out[2] = in[2];}void SBalls::vecMinus(float a[3], float b[3]) {a[0] = a[0] - b[0];a[1] = a[1] - b[1];a[2] = a[2] - b[2];}void SBalls::vecPlus(float a[3], float b[3]) {a[0] = a[0] + b[0];a[1] = a[1] + b[1];a[2] = a[2] + b[2];}void SBalls::vecTimes(float a[3], float b) {// a - b// a + b// a * b127}a[0] = a[0] * b;a[1] = a[1] * b;a[2] = a[2] * b;void SBalls::vecAverage(float a[3], float b[3]) {vecPlus(a, b);
vecTimes(a, 0.5);}
void SBalls::vecAverage(float fromA[3], float toA[3], float fromB[3], float toB[3], float average[3]) {vecCopy(toA, average);
vecMinus(average, fromA);
vecCopy(toB, tempVec);
vecMinus(tempVec, fromB);
vecAverage(average, tempVec);}
float SBalls::vecMag(float vec[3]) {return sqrt(vec[0] * vec[0] + vec[1] * vec[1] + vec[2] * vec[2]);}void SBalls::vecNormalize(float vec[3]) {vecTimes(vec, 1.0 / vecMag(vec));}/*

Rotate a point p by angle theta around an arbitrary axis r
Return the rotated point.
Positive angles are anticlockwise looking down the axis
towards the origin.

*/
void SBalls::vecRotate(float vec[3],double theta, float axis[3]){float q[3] = {0.0,0.0,0.0};
double costheta, sintheta;
vecNormalize(axis);
costheta = cos(theta * M_PI / 180.0);
sintheta = sin(theta * M_PI / 180.0);q[0] += (costheta + (1 - costheta) * axis[0] * axis[0]) * vec[0];q[0] += ((1 - costheta) * axis[0] * axis[1] - axis[2] * sintheta) * vec[1];q[0] += ((1 - costheta) * axis[0] * axis[2] + axis[1] * sintheta) * vec[2];q[1] += ((1 - costheta) * axis[0] * axis[1] + axis[2] * sintheta) * vec[0];q[1] += (costheta + (1 - costheta) * axis[1] * axis[1]) * vec[1];q[1] += ((1 - costheta) * axis[1] * axis[2] - axis[0] * sintheta) * vec[2];q[2] += ((1 - costheta) * axis[0] * axis[2] - axis[1] * sintheta) * vec[0];q[2] += ((1 - costheta) * axis[1] * axis[2] + axis[0] * sintheta) * vec[1];q[2] += (costheta + (1 - costheta) * axis[2] * axis[2]) * vec[2];}vec[0] = q[0];vec[1] = q[1];vec[2] = q[2];128Appendix C:
Hotpants Technical Documents
These documents describe the construction andoperation of the Hotpants circuit. All of these
documents and additional information is available
at http://acg.media.mit.edu/people/simong/
hotpants/tech.

Component List
Component
Shift register
Oscillator
LED Displays
Microcontroller
Capacitor
Current source
SwitchHeaderBattery CaseAA Batteries

Distributor
Digikey
Digikey
JamecoDigikey
Digikey
Digikey
Digikey
Digikey
Digikey
Digikey

Part Number
TC74HC595AP-ND
X909-ND
118906
PIC16F876-20/SP-ND
P965-ND
TD62783AP-ND
EG1906-ND
A19343-ND
BC4AAW-ND
P107-ND

Quantity per board
4
1
4
1
1
2
1
1
1
4
129

LittleVision firmware
This listing is the code that is burned into themicrocontroller onboard the Hotpants to enable it
to play movies. The movie data is loaded later witha bootloader (optional) starting at address 0x0396.
Movie data is encoded with a change-marking
compression. Assume the following states:
OFF = 0
HALF = 1
FULL = 2
Board front (actual size)
C:\data\NYLON\HotPants6.pcb (Top layer)

Board back (actual size)
C:\data\NYLON\HotPants6.pcb (Bottom layer)

We start each frame in state OFF. We advancefrom pixel to pixel, starting from the top left, andproceeding across each row of 14 down the display.
We maintain a current global state. As we traverse
the LEDs, we change the state of each LED to theglobal state. For each new LED, we read one bitof data. If it is a zero, we keep the current global
state the same and simply set this LED to that. If it
is a one, we look at the next bit. If that bit is zero,we add one to the current state (wrapping aroundto 0 after 2). If that bit is one, we add two to theglobal state. There are 14 bits per word in the PIC
memory.
This code compiles using the PICC compiler fromCCS [http://www.ccsinfo.com].

#case#include <16F876.H>//#device *=16// Configure PIC to use: HS clock, no Watchdog Timer,// no code protection, enable Power Up Timer//#fuses HS,NOWDT,NOPROTECT,NOPUT, NOBROWNOUT, NOLVP#define DATA_START 0x0396#define FRAMES_ADDR DATA_START#define MOVIE_START (DATA_START+1)// USE BOOTLOADER#ORG 0x1F00,0x1FFF //for the 8k 16F876/7
void loader() { }Board silkscreen (actual size)
C:\data\NYLON\HotPants6.pcb (Silkscreen)130// DATA RESERVE#ORG DATA_START,0x07FF
void data0() {}#ORG 0x800,0x0FFF
void data1() {}#ORG 0x1000,0x17FF

void data2() {}#ORG 0x1800,0x1EFF
void data3() {}}// Tell compiler clock is 20MHz. This is required for delay_ms()// and for all serial I/O (such as printf(...). These functions// use software delay loops, so the compiler needs to know the// processor speed.
//#use DELAY(clock=20000000)// Hardware definitions#defineOE#defineDS#defineCKPIN_C1
PIN_C0PIN_C2#define#define#define#define#definePIN_B7
PIN_B6
PIN_B3
PIN_B4PIN_B5

C4
C3
C2
C1
C0

// Hard parameters#define W 10
#define H 14
#define SIG_DELAY 4 // Anything lower doesn't seem to work
#define REFRESH_PERIOD 7167 // Empirical. You can mess with it.

}

buf1[x] |= (1L<<(long)y);void getMovieWord() {movieWordLow = make8(movieWordPtr, 0);
movieWordHigh = make8(movieWordPtr, 1);
#ASM
movf
movieWordLow, W
movwf
EEADR
movf
movieWordHigh, W
movwf
EEADRH
bsf
EECON1, EEPGD
bsf
EECON1, RD
nop
nop
movf
EEDATA, W
movwf
movieDataLow
movf
EEDATH, W
movwf
movieDataHigh#ENDASM}

curMovieData = make16(movieDataHigh, movieDataLow);// Colors#define OFF0#define HALF 1#define ON 2

void initMovie() {nowState = 0;
curFrame = 0;
movieBitPtr = 0;
movieWordPtr = MOVIE_START;
getMovieWord();}unsigned long movieWordPtr;
int movieWordLow;
int movieWordHigh;unsigned int getBits(int nBits) {int i;
unsigned int result = 0;#byte STATUS=0x0003#byte EEDATA=0x010C#byte EEADR=0x010D#byte EEDATH=0x010E#byte EEADRH=0x010F#byte EECON1=0x018C#define EEPGD 0x0007#define RD 0x0000

for (i = 0; i < nBits; i++) {result <<= 1;
if (curMovieData & (1L << (13L (long)movieBitPtr)))
result++;movieBitPtr++;if (movieBitPtr > 13) {movieBitPtr = 0;
movieWordPtr++;
getMovieWord();}}return result;}//Movie data
#define FRAME_DELAY 15// Determines the speed of the movie

unsigned int delayCounter;
int movieBitPtr;
unsigned long curMovieData;
unsigned int movieDataLow;
unsigned int movieDataHigh;unsigned int nowState;unsigned long curFrame;
unsigned long frames;

void nextFrame() {int x, y;
int nextBit;
for (y = 0; y < H; y++) {for (x = 0; x < W; x++) {nextBit = getBits(1);if (nextBit) {nextBit = getBits(1);if (!nextBit) {nowState = (nowState + 1) % 3;}
else {
nowState = (nowState + 2) % 3;}}
point(x, y, nowState);}}// Buffer storage
int frameCol;
short int inBufRefresh;
int bufNum;
unsigned long buf0[10];unsigned long buf1[10];void clearScreen() {int i;for (i = 0; i < 10; i++) {buf0[i] = 0L;
buf1[i] = 0L;}}
void point(int x, int y, int color) {// int x, y;if (color == OFF) {buf0[x] &= ~(1L<<(long)y);buf1[x] &= ~(1L<<(long)y);}else if (color == HALF) {buf0[x] |= (1L<<(long)y);buf1[x] &= ~(1L<<(long)y);}else {
buf0[x] |= (1L<<(long)y);}

curFrame++;if (curFrame == frames) {initMovie();}void runMovie() {while (1) {nextFrame();delay_ms(FRAME_DELAY);}}
void clockSR() {output_low(CK);

131

}

delay_us(SIG_DELAY);
output_high(CK);
delay_us(SIG_DELAY);

output_low(DS);else
output_high(DS);delay_us(SIG_DELAY);

void setupCols() {if (frameCol == 0) {output_high(C0);}else {
output_low(C0);}clockSR();mask <<= 1;}clockSR();// TOP RIGHT
mask = 0b0000000000000001;
for (i = 0; i < 7; i++) {if (frameCol == 1) {output_high(C1);}else {
output_low(C1);}

if ((buf[frameCol] & mask) != 0) // Light this row
output_low(DS);else
output_high(DS);delay_us(SIG_DELAY);

if (frameCol == 2) {output_high(C2);}else {
output_low(C2);}clockSR();mask <<= 1;}clockSR();if (frameCol == 3) {output_high(C3);}else {
output_low(C3);}}void initTimer() {frameCol = 0;
delayCounter = 0;
CCP_1 = REFRESH_PERIOD;
enable_interrupts(INT_CCP1);
enable_interrupts(GLOBAL);
setup_ccp1(CCP_COMPARE_RESET_TIMER);
setup_timer_1(T1_INTERNAL);}

if (frameCol == 4) {output_high(C4);}else {
output_low(C4);}}output_low(OE); // oevoid drawFrameCol(unsigned long* buf) {int i;
long mask;
output_high(OE);
delay_us(SIG_DELAY);// oe// TOP LEFT
mask = 0b0000000010000000;
for (i = 0; i < 7; i++) {if ((buf[frameCol] & mask) != 0) // Light this row
output_low(DS);else
output_high(DS);delay_us(SIG_DELAY);
clockSR();mask <<= 1;}clockSR();// BOTTOM LEFT
mask = 0b0000000010000000;
for (i = 0; i < 7; i++) {if ((buf[frameCol + 5] & mask) != 0) // Light thisrow
output_low(DS);else
output_high(DS);delay_us(SIG_DELAY);
clockSR();mask <<= 1;}clockSR();// BOTTOM RIGHT
mask = 0b0000000000000001;
for (i = 0; i < 7; i++) {if ((buf[frameCol + 5] & mask) != 0) // Light
this row

132

clockSR();setupCols();

inline void readNumFrames() {movieWordPtr = DATA_START;
getMovieWord();frames = curMovieData;}
void main() {clearScreen();readNumFrames();
initMovie();
initTimer();runMovie();}#INT_CCP1
void DRAW_COL() {if (frameCol == 4) {if (bufNum == 3)
bufNum = 0;
else
bufNum++;frameCol = 0;}
else
frameCol++;if (bufNum == 0)
drawFrameCol(buf0);
else
drawFrameCol(buf1);}