Pixels towards Pixies: Post-Multimedia interactions with Air-Based Media
Yoichi Ochiai11Digital Nature Group, University of Tsukuba, Ibaraki, Japan.
http://digitalnature.slis.tsukuba.ac.jp, email: wizard@slis.tsukuba.ac.jp

Figure 1. (a) Interaction concept of Computational Ether [1], (b) Laser Plasma Graphics induced by Femtosecond laser [2],(c) Acoustic Field [3] rendered and visualized by dry ice (d) tactile feedback rendered by both acoustic and light fields [4]actuate objects around users to inform them. These fields of
Abstract
physical quantities rendered around users work as
Towards post multimedia interactions, we envision real world“Computational Ether” that we could describe it along theoriented interactions with Computational Generated Fields on
metaphor of “Ether” in classical physics. This Computational
Air; we called this interaction concept as “Computational Ether”.
Ether suspends objects floated and also works as aerial media to
Along this concept we have conducted studies to propose arender the graphical and haptic expressions. The method of
method for realizing new physical interactions that expands
handling field quantities by CGH is well known and matured,malleability of non-digital material towards programmablehowever this concepts framework enables us to clarify how tomatter in the real world [1]. Our method utilizes light field [2] andalter the physical environment by appropriate field design and to
acoustic field [3] these are calculated, generated, and controlled
consider application areas.
by computers. By applying such computational fields of quantities
and physical phenomena around the holographic field, weCalm Technologies: We believe that field-oriented
transformed ordinary air into interactive visual media [2], haptic
programmable matter is one means of realizing Calm Technology
media [4], and actuators [3]. To implement these interactive[6], i.e., a technology that informs us however does not demand
media, we employed Computer Generated Hologram andour focus or attention. Mark Weiser suggested the vision of Calmrendered with ultrasound generated by acoustic phased array andTechnology in 1996. Some two decades on, our daily lives arelaser induced plasma generated by femtosecond laser sources
surrounded by computers [7]. As Weiser pointed out, calmness iswith Spatial Light Modulators. In this paper, we introduce thea fundamental challenge for all technological design for the next
results of our research group such as case studies; aerial haptic
50 years; we are now facing issues related to calmness.
interaction, aerial touch displays, 3D manipulation of objects,
As he pointed out, a vision designed with an appropriate
new material expression displays, and so on. Finally, we discusscombination of scientific aspects, technology, design, andthe advantages and limitations of our method. We believe these
implementation can drive us toward a solution for calmness, i.e.,
technologies stimulate display researches from 2D pixels on theawareness of ubiquitous computers [7]. We believe this problem
surface towards 3D pixies flying in the air.
can be solved if ubiquitous or pervasive computers can alter the
Author Keywords
real world to be “programmable” by hidden and separated field
Human Computer Interaction; Computer Graphics; Femtosecond
generators that activate invisible medium. To implement them,Laser; Haptic Interface; Tangible Interaction; CGH
we noticed computational generated field on air that is renderedby separated generators. In this case, the technology itself (digital
1. Introduction: Computational Ether
resources or machines) is hidden, and the actuation targets are
Towards Real World Interactions: In 1965, Ivan Sutherland
mixed with other non-digital objects. These are notstated that the ultimate display is a room that can control the
distinguishable until they are actuated or activated, and therefore
existence of matter [5]. We envision that digital resources should
do not require our attention unnecessarily. Then we considered
be accessible as freely as non-digital resources (e.g., physicalthe air is one of most suitable medium to utilize for calmness. We
objects) at the goal of IoT technologies. We also believe that nonhave utilized invisible infrared laser and inaudible ultrasound todigital material should be as malleable as digital material ingenerate holograms for calmness interactions that connect
digital resources (e.g., data in computers). To realize the concept
computational resources and activated matters without attention.
of the room that Ivan Sutherland suggested, invisible medium thatconnects the matters and environmental computational resources
2. Related Work: Programmable Matteris necessary for the human computer interaction. Thus we
Visions: Programmable matters - controlling objects in the realconsider to control the “air” as computational medium.
world is now popular topic in the computer graphics, display, andIn conventional studies we have utilized CGH, wave fronts (lighthuman-computer interaction communities. Various ideas andand acoustic waves), and air to display information to users or toFigure 2. Concept of Computational Ether (Field Distribution) (a) construction of energy distribution,(b) animation (c)holographic distribution (d) simulated results (e) rendered result [3]Figure 3. Principle on common ground: the rendering process of holographic images by SLM and phased array.
visions to organize and realize this concept have been proposed –e.g., programmable matter [8], radical atoms [9], and actuationinterfaces [10]. These proposals focus on controlling real objects
through a computer and generating physically programmablematerial. These concepts will play very important roles in futureinteractive environments because they expand the range ofcomputer applications from “painted bits” to the real world [11].
Two methods are currently available to control objects in the realworld. In one, objects actuate themselves, whereas they are
actuated by their surroundings in the other. The latter method is
divided into two types of actuation: contact and noncontact. We
address the noncontact approach by utilizing invisible resources
and activate air medium (concept is shown in Figure 2).
Volumetric displays and screens: The case studies that weraised this articles [2] [3] [4] are generally categorized three
dimensional volumetric displays. Studies directed toward
controlling the spatial position of an active screen and display arealso being actively pursued in display researches. There are twokinds of the studies; one aims to achieve multi-perspective 3D
image and the other aims to realize deformation of planner
screens for haptic and/or other purposes. Multi-perspective 3D
dis-play is a popular topic in computational display areas. [12]constructs 3D images with a rotated mirror and projection. On theother hand, there are researches that focus on the dynamic
deformable screen and display. For example, the deformableactuated screen “Project FEELEX” [13] constructs 3D forms on
the screen surface using an actuator array set under the screen.
Inform [14] has proposed an interactive deformable screen, calledthat handles and/or interacts with other objects. A noncontactlyactuated deformable screen [15] employs an ultrasonic-phasedarray to deform a colloidal screen. While there is a 3D solution[16] that uses a plasma 3D volumetric display. Our researches are
based on these conventional studies and we aimed not only toshow 3D image but to mix 3D interaction with user’s
environment.

3. Principle on Common Ground
Computer Generated Hologram: We basically utilize
computer generated hologram to generate the computational
acoustic and light fields however note that. Because of thelimitation on laser intensity endurance of SLM, we combine

SLM, galbano mirror, and varifocal lens to generate images inlaser applications and because of the limitation in resolution ofphased array, we combined holographic control and spatial
control to generate computational acoustic field.
The spatial phase control of light and acoustic field enables thefocusing position to be controlled along both the lateral (X andY) and axial (Z) directions. The complex amplitude (CA) of thereconstruction from the computer-generated hologram (CGH) Ur
is given by the Fourier transform of the designed CGH pattern Uh:
𝑈" 𝜈$ , 𝜈& =𝑈( 𝑥, 𝑦 exp −𝑖2𝜋 𝑥𝜈$ + 𝑦𝜈&= 𝑎" 𝜈$ , 𝜈& exp [𝑖𝜑" (𝜈$ , 𝜈& )]𝑈( 𝑥, 𝑦 = 𝑎( 𝑥, 𝑦 exp [𝑖𝜑( (𝑥, 𝑦)]

𝑑𝑥𝑑𝑦
(1)
(2)

where ah and φh are the amplitude and phase of the hologram
plane displayed on the SLM or Phased Array, respectively. In theexperiment, ah is constant because the light irradiation on the
CGH is considered to be plane wave with a uniform intensity
distribution and also it is constant on the acoustic field because
amplitude of all transducers on ultrasonic phased array is samevalue. φh is obtained by hologram calculation algorithm, whereasar and φr are the amplitude and phase of the reconstruction plane,
respectively.
The spatial intensity distribution of the reconstruction is actuallyobserved as |Ur|2 = ar2. To control the focusing position along thelateral (X and Y) direction, the CGH is designed based on asuperposition of CAs of blazed gratings with a variety of azimuth
angles. If the reconstruction has N-multiple focusing spots, the
CGH includes N-blazed gratings. To control the focusing position
along the axial (Z) direction, a phase Fresnel lens pattern φp(x, y)= k (x2+y2)/2f with a focal length f is simply added to φh, where k= 2π/λ is a wave number.
In this case, the spatial resolution of the SLM and the interval
length of transducers on Phased Array determines the minimum
focal length of each computational field respectively. In
conventional study [17], ultrasonic phased array was controlled
to render a single point in X and Y by given (1) and Z position isdecided by this phase Fresnel lens pattern on ultrasonic phasedarray.

Figure 4. System and Applications (a) Pixie Dust [3] (b) Fairy Lights in Femtoseconds [2] (c) Cross-Field Haptics [4]Haptic Images: Haptic images on cross field project [4] aregiven by a combination of an SLM image and galvano mirror andradiation pressure of ultrasonic focal points. Haptic image Hi is
the summation of the time series of the focal points, that is,
𝐻< =𝑈" (𝜈$ , 𝜈& )×𝑖> ×𝑡> ＋

𝑓A (𝑥, 𝑦, 𝑧) ×𝑝×𝑡D(3)

where Ur represents the laser focal points given by (1), tl is time
duration, il is laser intensity, fp is the radiation pressure ofultrasonic focal points [19], p is the acoustic pressure, and ta is thetime duration.

4. Case Studies
In this section we introduce several case studies. Common groundof these studies are employing “air” to generate stimulation
towards user by implementing Computational Ether concept
shown in Figure 4.
Graphics by Computational Acoustic Fields: This is the
project named Pixie Dust [3] and systems are shown in Figure4(a-1). We have utilized ultrasonic phased array and generate
acoustic fields in three dimensionally positions. In conventional
research on acoustic levitation, small objects are trapped in the
acoustic beams of standing waves. We expanded this method bychanging the distribution of the acoustic-potential field (APF).
Acoustic-Potential Field is generated by four ultrasonic phasedarrays [19]. Using this technique, we could generate the graphics
using levitated small objects (Figure 4(a-2,3)). Our approach
made available many expressions, such as the expression bymaterials and non-digital appearance. These kinds of expressions
are used in many applications, and we aim to combine them withdigital controllability. In the current system, multiple particles arelevitated together at 4.25-mm intervals. The spatial resolution of
the position is 0.5 mm. Particles move at up to 72 cm/s. The
allowable density of the material can be up to 7 g/cm3. For thisstudy, we use three options of APF: 2D grid, high-speed
movement, and combination with motion capture.
These are used to realize floating screen or mid-air rastergraphics, mid-air vector graphics, and interaction with levitatedobjects. And also it can be combined with haptic feedback [19]with aerial tangible images by levitated materials.
Graphics by Computational Light Fields: This is the
project named Fairy Lights in Femtoseconds [2] and systems areshown in Figure 4(b-1). We have utilized femtoseconds laser
source and spatial light modulator to generate laser induced
plasma by laser light fields in three dimensionally positions. Thisis a display method of rendering aerial and volumetric graphics
using femtosecond lasers. A high-intensity laser excites physicalmatter to emit light at an arbitrary three-dimensional position.
Popular applications can thus be explored, especially because
plasma induced by a femtosecond laser is less harmful than thatgenerated by a nanosecond laser. There are two methods ofrendering graphics with a femtosecond laser in air: producing
holograms using spatial light modulation technology andscanning of a laser beam by a galvano mirror.
The holograms and workspace of the system proposed here
occupy a volume of up to 1 cm3; however, this size is scalable
depending on the optical devices and their setup. We tested twolaser sources: an adjustable (30–100fs) laser that projects up to
1,000 pulses/s at an energy of up to 7mJ/pulse and a 269fs laserthat projects up to 200,000 pulses/s at an energy of up to
50µJ/pulse. We confirmed that the spatiotemporal resolution of
volumetric displays implemented using these laser sources is
4,000 and 200,000 dots/s, respectively (Figure 4(b-2,3)).
In this project we proposed design methods of “tangible 3D
images” by utilizing femtosecond laser induced plasma. The
spatial resolution of voxels is 100µm interval and participants inthis study states that displayed object can not be distinguishedfrom physical solid materials.
Haptic and Cross-Field Applications: This is the project
named Cross-field aerial haptics [4] and systems are shown in
Figure 4(c-1). In this project we presented a new method ofrendering aerial haptic images that uses femtosecond-laser lightfields and ultrasonic acoustic fields. In conventional research, asingle physical quantity has been used to render aerial haptic
images. In contrast, this method combined multiple fields (lightand acoustic fields) at the same time.
While these fields have no direct interference, combining them

provides benefits such as multi-resolution haptic images and asynergistic effect on haptic perception. We conducted user studies
with laser haptics and ultrasonic haptics separately and tested
their superposition. The results showed that the acoustic field
affects the tactile perception of the laser haptics. We exploredaugmented reality/virtual reality (AR/VR) applications (Figure4(c-2,3)) such as providing haptic feedback of the combination ofthese two methods. This project is aimed at combining these two
project from the viewpoint of haptic feedback and how to
combine them as unified system.

5. General Discussions and Future Work
Spatiotemporal Resolution on each system are determined by its
hologram generators. The key factor of spatiotemporal resolution
is refresh rate and endurance of spatial light modulators for ourlaser systems. These two factors decide the limitation of parallel
access to aerial voxels. To reduce the other optical components
such as galvano mirror and varifocal lens, we have to increase therefresh rate of SLM in our systems. For our acoustic systems the
key factor is size of transducer arrays. Transducer size itself
decided by the wave length of ultrasound. To achieve highresolution of hologram image, we have to increase the size oftransducer arrays.
Our concept of Computational Ether covers noncontact anddisplay technologies that conceal the generators from human sightand increase the programmability of the actuation of matter. We
believe this is a worthy contribution the realization of Calm
Technologies. We believe field-oriented programmable matter isa useful concept in the development of Calm Technologies fromthe aspect of interface selectivity. We employed non-digitalmaterials to bring about actuation [4]. These are examples inwhich non-digital materials turn into programmable matter. Oneof the drawbacks of noncontact display and actuation is that theefficiency of energy transmission is lower than that ofconventional contact-based methods (e.g., led wired from electric
circuit, motor). In spite of this, the absence of bulky mechanicalsystems in the workspace is a large advantage of the noncontact
actuation technologies from the viewpoint of calmness.

6. Conclusions
In this article, we described real world oriented interactions withcomputational generated field of quantities on air towards post
multimedia interactions and we introduced interaction concept as“Computational Ether”. Along this concept we have conductedcomputer generated hologram research to propose a method forrealizing new physical interactions that expands their malleability
towards programmable matter in the real world. As case studies
we raised several implementations by applying suchcomputational fields of quantities and physical phenomena
around the holographic field. We introduced the results of ourresearch group such as case studies; aerial haptic interaction,
aerial touch displays, 3D manipulation of objects, new material
expression displays, and so on. The resulting system can beapplied to display technologies such as Computer Graphics,
Entertainment Computing, and Human Interfaces. Finally, we
discuss the advantages and limitations of our method. weemployed Computer Generated Holograms rendered withultrasound generated by acoustic phased array and laser induced
plasma generated by femtosecond laser sources, Spatial Light
Modulators, and optical components. We believe thesetechnologies stimulate display researches from 2D pixels on thesurface towards 3D pixies like programmable matter.

7. References[1] Ochiai,Y. Graphics by Computational Acoustic Fields,
Doctoral Dissertation, The University of Tokyo, 2015.
[2] Ochiai, Y., Kumagai, K., Hoshi, T, Rekimoto, J.,
Hasegawa, S., and Hayasaki, Y. Fairy Lights in
Femtoseconds: Aerial and Volumetric Graphics Renderedby Focused Femtosecond Laser Combined with
Computational Holographic Fields. ACM Trans. Graph.35,
2, Article 17 (February 2016), 14 pages, 2016.
[3] Ochiai, Y., Hoshi, T., and Rekimoto, J. Pixie dust: graphics
generated by levitated and animated objects incomputational acoustic-potential field. ACM Trans. Graph.
33, 4, Article 85 (July 2014), 13 pages, 2014.
[4] Ochiai, Y., Kumagai, K., Hoshi, T, Rekimoto, J.,
Hasegawa, S., and Hayasaki, Y. Cross-Field Aerial
Haptics: Rendering Haptic Feedback in Air with Light andAcoustic Fields, In Proc of CHI 2016 , CHI'16, May 07-12,
2016, San Jose, CA, USA, 2016.
[5] Sutherland, I. E. The ultimate display. Proc. IFIP Congress(1965), 506-508. 1965.
[6] Weiser, M. and Brown, J.S. The coming age of calm
technology. Beyond Calculation (1997), 75-85. 1997.
[7] Weiser, M. The Computer for the 21st Century. Scientific
American 265, 3 (1991), 66-75. 1991.
[8] Goldstein, S.C., Campbell, J.D., and Mowry, T.C.
Programmable matter. Computer 38, 6 (2005), 99-101.
2005.
[9] Ishii, H., Lakatos, D., Bonanni, L., and Labrune, J.B.
Radical atoms: Beyond tangible bits, toward transformablematerials. Interactions 19, 1 (2012), 38-51. 2012.
[10] Poupyrev, I., Nashida, T., and Okabe, M. Actuation andtangible user interfaces: The Vaucanson duck, robots, andshape displays. Proc. TEI, ACM (2007), 205-212. 2007[11] Ishii, H. and Ullmar, B. Tangible bits: Towards seamless
interfaces between people, bits and atoms. Proc. CHI, ACM(1997), 234-241. 1997[12] Jones, A., Mcdowall, I., Yamada, H., Bolas, M., Debevec,
P. Rendering for an interactive 360&deg; light field
display. ACM Trans. Graph. 26, 3 (July). 2007.
[13] Iwata, H., Yano, H, Nakaizumi, F., and Kawamura, R.
Project FEELEX: Adding haptic surface to graphics. Proc.
SIGGRAPH, ACM (2001), 469-476. 2001.
[14] Follmer, S., Leithinger, D., Olwal, A., Hogge, A., and Ishii,
H. inFORM: Dynamic physical affordances and constraints
through shape and object actuation, Proc. UIST, ACM(2013), 417-426. 2013.
[15] Ochiai, Y., Oyama, A., Hoshi, T., and Rekimoto, J.
Poppable display: A display which enables people tointeract with popping, breaking, and tearing, Proc. GCCE,IEEE (2013), 124-128. 2013.
[16] Kimura,H., Uchiyama, T., and Yoshikawa, H. 2006. Laser
produced 3D display in the air. In ACM SIGGRAPH 2006
Emerging Technologies, ACM, New York, NY, USA,SIGGRAPH ’06. 2006.
[17] Hoshi, T., Takahashi, M., Iwamoto, T., and Shinoda, H.
Noncontact tactile display based on radiation pressure ofairborne ultrasound. IEEE Transactions on Haptics 3, 3,
155-165, 2010