OUT OF THIN AIR

A team at the University of Sussex, UK, has created a virtual butterfly that can hover in space.

Researchers are creating 3D images that float in space. By Elizabeth GibneyW

ith the tap of a keyboard, Ryuji
Hirayama brings a listless foambead to life. The white speckjumps up and hovers with perfect stillness in space. Anothertap, and the dot transforms
into a luminous butterfly-likeshape, which flaps its wings as
it circles inside a black box. Diego Martinez
Plasencia, Hirayama’s colleague at the University of Sussex in Brighton, UK, reaches
into the box to show that there are no strings.
The effect seems to be pure magic. “I showed it
first to my daughters. They were like — ‘wow’,”
Martinez Plasencia says, his eyes widening inchild-like delight.
Behind the mid-air metamorphosis is a.
dev
rese
rst
hgirllA.
detimiLerutaNregnirpS
9102©

272 | Nature | Vol 575 | 14 November 2019

relatively simple set-up. Two slender arraysof 256 tiny speakers above and below the beadmove it by generating ultrasound waves. Theobject darts so fast that all the eye sees is a continuously evolving 3D image a few centimetresacross, drawn in the air as if by a high-speed
Etch a Sketch. The same ultrasound speakers
that create the image can also generate audio
and tactile sensations. Reach towards the butterfly, and your finger might feel a flutter. In
another case, a smiley face appears, accompanied by the strains of Queen’s ‘We Will RockYou’. Remarkably, most of the components
used to create these effects are off-the-shelf.
“It’s an elegant and exciting platform,”says Daniel Smalley, a physicist at Brigham
Young University in Provo, Utah, who last year

unveiled a similar technique, using lasers tosteer around a fleck of cellulose to produce
images1. Until now, few physicists thought itwould be possible to use sound to move a beadfast enough to create such a display, he says.
In August, Tatsuki Fushimi, a physicist at theUniversity of Bristol, UK, and his collaborators
became the first to show that it was feasible.
But their bead took longer to trace out shapes,
meaning that only images smaller than 1 centimetre across could appear as a single, continuous object2. The Sussex team’s work is “a pieceof engineering that makes us believe things
we didn’t think were possible,” says Smalley.
The acoustic device, described in Nature(see page 320)3, is the latest example of a
3D-image-generation technology known as

EIMONTAS JANKAUSKIS/UNIV. SUSSEXFeaturevolumetric display, which differs in fundamental ways from technologies such as holograms, virtual reality and stereoscopes. Thosemore-familiar approaches use tricks of thelight to create the illusion of depth, and canbe life-sized and photorealistic. But holograms
can be seen only from certain angles, virtual
reality and stereoscopes require headgear,
and all these techniques can cause eye strain.
Free-space volumetric displays, by contrast,
use lasers, electric fields, fog projections andother approaches to create truly 3D images
that viewers can see from any vantage point.
In that way, they’re the closest any display
technology has come to Princess Leia’s SOS
message in the 1977 film Star Wars.
Research into volumetric displays is evenolder than that film. And the approach has a
crucial advantage over holograms, because
it requires much less computational power.
But despite decades of effort, free-space displays are still limited to small, crude drawings,
and they have struggled to get off the groundcommercially, says Smalley. Still, he is hopeful that work bringing together different andmore-practical technologies, including acoustic levitation, will help volumetric display tofind its killer app. It might be used in detailedinteractive mock ups for medical trainees,perhaps, or to give people the ability to chatwith distant relatives rendered in 3D. And theSussex team’s acoustic method wouldn’t necessarily require a long phase of development
to make its way out of the laboratory, says
Smalley. “I would make a bet on this technology becoming commercial before many of theother technologies we work on.”Lights and soundVolumetric displays that are already on themarket usually work by making 2D images
3D. The Voxon VX1, for example, projects
photons onto a screen that vibrates rapidly up
and down. Timed right, this creates a 3D image
without the need for special eyewear. But thedisplay’s complex mechanical parts mean it islocked behind glass and has so far found only
niche uses, such as in museum displays.
In 2006, Hidei Kimura made one of thefirst attempts to draw images directly onto
3D space4. Kimura, chief executive of Burton, afirm based in Kawasaki City, Japan, and his academic collaborators developed a techniquethat used a laser to knock electrons off air
molecules, causing them to glow. By moving
the laser’s focal point at high speed, they could
create luminescent dots of plasma that build
to make a coarse image. “With nothing, wecan create 3D images directly in the air,” says
Kimura, who envisages using the techniqueto broadcast emergency information into thesky, or to project 3D replays above the pitch ata sporting event.
The plasma technique creates relatively
stable images, but it faces some major

limitations: it is low resolution (one laser blast
equals one point in the image) and the laser is
so intense that it could cause burns, says Yoichi
Ochiai, a computer scientist and artist at theUniversity of Tsukuba in Japan.
In 2016, Ochiai’s team adapted the plasma
technique, using a lower-energy, shorter-pulse
laser to make images that are safe to touch5. Ata few millimetres across, the images are muchsmaller than the Kimura team’s. But by using
lasers that pulse at a higher frequency andmodulators to shape the beam into multiple
focuses, the team can ramp up the resolution
to 10–200 times higher than that used in Kimura’s work. This allows them to create more
complex images, such as pinhead-sized fairies.

working to improve their understanding of
how the bead responds to the forces acting on
it, which would allow them to move it faster,to draw more-complex images by levitating
multiple beads at once and to integrate sightand touch more closely. In the current set-up,
the tactile sensation and image don’t occur
in exactly the same place, because the fields
needed to create them can interfere with eachother. Ochiai’s group has already found a way
to bring together touch and sight by using
fields that do not interfere: an acoustic field fortactile feedback and a laser to draw tiny images
in plasma. The group has used the approachto draw braille dots in the air7.

“A piece of engineering thatmakes us believe things wedidn't think were possible.”

Inevitably, any 3D display gets compared toStar Wars holograms. But Barry Blundell, aphysicist specializing in 3D technologies at theUniversity of Derby, UK, cautions against trying to use volumetric technology to make rich,photorealistic displays. “No one would lookat a sculpture and compare it to a painting,”
he says. He adds that efforts to compete withholograms have often led to commercial deadends, and that the displays are best suited toapplications that would be impossible in othermedia but don’t require high detail, such as
interactive displays capable of showing complex 3D movements.
Interactivity could be powerful, Smalley
says. Surgeons in training might use such displays to practise threading a catheter throughthe vessels of the heart, for instance. With one
million moving particles, he adds, “you canhave a disembodied face — do face-to-facetelepresence”. Creating avatars of people in aspace could give a stronger sense of presence
than a photorealistic image seen through virtual reality, he says.
At the Sussex lab, a million-particle display
seems a long way off. Only time will tell if thegroup’s approach will pave the way to such
numbers. After showing off his sphere’s limited repertoire of tricks, Hirayama shuts off
power to the speakers. The flapping butterfly
vanishes, and the bead that created it drops
and bounces gently on the display’s base.
Hirayama picks it up and sets it in a box withhundreds of others, ready at any time to create
magic in thin air.

At Sussex, the acoustic 3D display started
with another stalwart of science fiction: thetractor beam made famous in the 1960s tele­vision series Star Trek. Since 2012, Sriram
Subramanian, who leads the team, has pioneered ways of crafting sound waves to create
points of high pressure that can trap and movesmall objects6. But it was only when Hirayama
came to the laboratory in 2018 that the teamfound a way to use sound to create images.
To appear as a solid picture, a bead needs tocreate each image frame in less than one-tenthof a second. Until now, acoustic levitation hastended to focus on holding objects as steady as
possible; motion happens relatively slowly ina stop–start fashion, from one stable point toanother. Hirayama’s innovation was to kick thebead before it comes to rest, calculating eachnew target point inside hardware designed
especially for the computation. This meantthat the team could change the field’s focus
40,000 times per second. The bead reaches
speeds as high as 8.75 metres per second,which “looks like teleportation” when a 2-millimetre-wide bead crosses a few centimetresof space, says Hirayama. As the bead moves,a rapidly changing LED bathes the display inlight to create colour.
The team was inspired by Smalley’s work
using lasers to move and illuminate a speckof cellulose plant fibre1. Using the same number of particles and data, Smalley says hisimages are only one-tenth the size of the Sussex team’s, but have ten times the resolution.
The Sussex technique does have a drawback: it needs speakers on two sides of thedisplay, which restricts a viewer’s ability tointeract with the display and limits its size.
But with hardware upgrades, Subramaniansays it could be possible to use a different kindof acoustic wave to create images with speakers on just one side. The researchers are alsoInteractive advantage

Elizabeth Gibney is a senior reporter with
Nature in London.
1. Smalley, P. E. et al. Nature 553, 486–490 (2018).
2. Fushimi, T., Marzo, A., Drinkwater, B. W. & Hill, T. L. Appl.
Phys. Lett. 115, 064601 (2019).
3. Hirayama, R., Martinez Plasencia, D., Masuda, N. &Subramanian, S. Nature 575, 320–323 (2019).
4. Kimura, H., Uchiyama, T. & Yoshikawa, H. Proc. SIGGRAPH‘06 https://doi.org/10.1145/1179133.1179154 (2006).
5. Ochiai, Y. et al. ACM Trans. Graphics 35, 17 (2016).
6. Marzo, A. et al. Nature Commun. 6, 8661 (2015).
7. Ochiai, Y., Hoshi, T., Hasegawa, S. & Hayasaki, Y. Proc.
2016 CHI Conf. Human Factors Comput. Syst. 3238–3427 (2016).

.
devresersthgirllA.
detimiLerutaNregnirpS
9102©

Nature | Vol 575 | 14 November 2019 | 273