Holographic Whisper: Rendering audible sound spots inthree-dimensional space by focusing ultrasonic waves
1

Yoichi Ochiai 1, 3 *University of Tsukuba
Ibaraki, Japanwizard@slis.tsukuba.ac.jp,
1heisuzuki@gmail.com

2

Takayuki Hoshi 2, 3 *The University of Tokyo
Tokyo, Japan
star@star.rcast.u-tokyo.ac.jp

ABSTRACT

We propose a novel method of spatial audio rendering
using ultrasound. An ultrasonic phased array generates one
or more focal points in air, and they act as point sources ofaudible sound when the ultrasound waves are modulated.
Our sound-point loudspeaker has two major advantages
over conventional ultrasound-based sound-beam (superdirectional) loudspeakers. The higher audience selectivity
means that our sound-point loudspeaker can deliver soundto the ears of the target person, whereas a sound-beam
loudspeaker delivers sound to not only the target person but
also other persons standing in the same direction. The otheradvantage is lower exposure to ultrasound; while an audible
sound beam travels along an ultrasonic beam in a soundbeam loudspeaker, audible sound can be heard along thedirection perpendicular to the ultrasonic beam in our soundpoint loudspeaker. This paper reports the principles of oursound-point loudspeaker, prototype construction, evaluation,
and applications.
Author Keywords

Spatial sound control, Ultrasound, Phased-array focusing,Self-demodulation effect, Point source, Aerial interaction
ACM Classification Keywords

H.5.1. Information interfaces and presentation (e.g., HCI):
Multimedia information systems: Audio input/output
INTRODUCTION

Audio information is one of the essential components ofmultimedia content and real-world interactions. In contrast
to light rays, sound waves have lower directivity and arereceived by listeners who were not targeted by the speaker.
Conventional loudspeakers are almost omnidirectional,
which is usually an undesirable feature in public places. A
technology to provide different audio information to* Joint first authors.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights forcomponents of this work owned by others than the author(s) must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee. Request permissions from
Permissions@acm.org.
CHI 2017, May 06 - 11, 2017, Denver, CO, USA
Copyright is held by the owner/author(s). Publication rights licensed to
ACM.
ACM 978-1-4503-4655-9/17/05…$15.00
DOI: http://dx.doi.org/10.1145/3025453.3025989

3

Ippei Suzuki 1

Pixie Dust Technologies, Inc.
Delaware, United States{ochyai, star}@pixiedusttech.com

different users is desired in ubiquitous computing (in otherwords, non-wearable computing). With such a technology,
personalized sound delivery becomes possible and soundenvironments can be designed more freely. Continuous
efforts have been made to this end, such as wave field
synthesis, directional audible-sound loudspeakers, andultrasound-based super-directional loudspeakers.
Wave field synthesis [1, 2] is a method to reproduce wave
fronts from virtual sound sources with a large number ofloudspeakers. The listening space has to be densely
surrounded by the loudspeakers because the principle is theKirchhoff-Helmholtz integral, and so this method is very
costly to implement. Besides, this method aims to
reproduce a sound environment and it is not suitable forgeneration of localized sound areas. Rather than thismethod, we discuss directional and super-directional
loudspeakers for our purpose.
Conventional directional loudspeakers are arrays of subunits (small audible-sound loudspeakers). All the sub-units
are driven by an identical signal and, as a result, planesound waves are radiated. The plane waves of audible
sound form a narrow sound beam [3]. Steering of the beamis also possible [4]. Although these loudspeakers produce
sound beams in a narrow angle, all the persons standing
within that angle receive the audio information. An
ultrasound-based super-directional loudspeaker [5-8]utilizes the plane waves of ultrasound as an end-fire arrayof audible-sound sources, based on the self-demodulation
effect of air, to generate a narrower sound beam. In thiscase, listeners are exposed to high-intensity ultrasound.
Another category of sound manipulation is the productionof sound spots in air. An array of loudspeakers generates
focal points of audible sound waves by tuning time delays,or phase delays, between the loudspeakers (i.e., phasedarray control) [9]. Another method uses multiple
ultrasound-based super-directional loudspeakers to generate
audible sound at the crossing points of ultrasonic beams[10]. These methods have spatial requirements: The former
requires a large device consisting of hundreds ofloudspeaker sub-units and the latter requires multiple
devices surrounding the listening space.
We introduce a new loudspeaker that generates discretesound producing spots in air with a single ultrasonic device,that we named “Holographic Whisper.” Our proposition isGeneration of focal points of audible sound waves has beenpreviously reported [9]. A 120-ch loudspeaker arraygenerated focal points of audio signals such as voices andmusic. A single loudspeaker sub-unit had dimensions of 66
× 107 mm, and the array had dimensions of 1,650 × 1,350
mm. The diameter of a focal point depends on soundfrequency, and it was approximately 400 mm for 1 kHz,
which was comparable to the size of a human head.
Because we aim to realize a similar effect without such alarge array, we use ultrasound instead of audible sound.
Figure 1. Position of this study among related work.
Holographic Whisper focuses on ultrasound-based movablesound points (red box). It can produce steerable sound beams(red dotted box) when the focal length divided by the device
size is larger than a threshold value.

to apply the phased-array control technique to an array ofultrasonic transducers. In other words, the transducers are
driven by different phase-shifted signals individually,
instead of being driven by an identical signal. The phasedifferences are adequately calculated for an intended spatial
distribution (e.g., a single or more focal points) of
ultrasound after interference. We intend to generate focal
points of ultrasound that act as sound point sources.
The primary contribution of this paper is the production of
aerial sound point sources with an ultrasonic phased arrayfor three-dimensional (3D) audio interaction. The
remainder of this article is organized as follows. First, we
introduce related work on loudspeakers and ultrasonicphased arrays. Second, we theoretically describe theprinciples and control methods of the ultrasonic phasedarray. Third, we show the implementation and experimental
results of the prototype of the proposed focusing
loudspeaker. Fourth, we give examples of applications.
Finally, we discuss the limitations and scalability. We
believe that this study fills a gap between the design spacesof 3D visual images and sound images, and contributes tothe progress of human-computer interaction (HCI) researchand leads to various new applications.

Ultrasound-based super-directional/spot loudspeakers

Ultrasound enables the creation of loudspeakers that havenarrower directivity than audible-sound directional
loudspeakers [5-8]. An ultrasound-based super-directional
loudspeaker consists of an array of ultrasonic transducers
that are driven by an identical signal. The principle consistsof two effects. One is the self-demodulation effect, which is
the radiation of audible sound from modulated highintensity ultrasonic waves in air. This means air filled withmodulated ultrasonic waves acts as a virtual sound source.
The other is the parametric array effect, which is theformation of an audible sound beam. The ultrasonic arrayradiates an ultrasonic plane wave (a beam of ultrasound).
This means that virtual sound sources are aligned along thisultrasonic beam in air. The phases of these virtual soundsources are different because of the propagation of theultrasonic plane wave. As the result of interference ofaudible sound waves, a narrow beam of audible sound isgenerated.
Studies on the steering of audible sound beams from superdirectional loudspeakers have been published [11, 12]. To
achieve this, the phase delays between the transducers arecontrolled based on the steering angle, θ [rad]: When thespacing between the transducers is d [m] in the case of a 1D
transducer array, the phase delay of the i-th transducer, Δφi[rad], is calculated as follows,∆ϕ i = i × kd sin θ ,(1)where k [rad/m] is the wavenumber of the ultrasound.

RELATED WORK

Our ultrasonic loudspeaker renders point sources of audible
sound using an ultrasonic phased array. Related
technologies are audible-sound directional/spot loudspeakers and ultrasound-based super-directional loudspeakers. Figure 1 shows the positon of our study among
these technologies. Phased-array control of ultrasonicwaves for other purposes is also an important topic. We
introduce these topics as follows.

Sound spots can be generated by using multiple superdirectional loudspeakers [10]. The ultrasonic beams haveslightly different frequencies. Audible sound is generated
within the crossing point of these ultrasonic beams as aresult of beat formation. In this configuration, the target
person is necessarily exposed to high-intensity ultrasound.
Because we aim to realize a similar effect with lower
exposure to ultrasound, we generate a focal point instead ofa beam (Fig. 2).

Audible-sound directional/spot loudspeakers

Phased-array control of ultrasonic waves

Directional loudspeakers, especially for public spaces,
already exist in the market [3]. Sub-units of a loudspeakerarray are driven by an identical audible sound signal, andform a plane wave of audible sound (audible-sound beam).
By controlling these sub-units individually, it is possible tosteer the beam [4].

Recently, ultrasound has been used to produce haptic
feedback in air, by using the principle of acoustic radiationpressure, which is a nonlinear effect where high-intensity
ultrasound presses the object surface. The phased-array
focusing technique is used to achieve high intensity at theamplitude modulation (AM) to modulate ultrasound, which
is implemented into digital control circuits as pulse-widthmodulation (PWM) of the driving rectangular waves. These
topics are explained as follows.
Self-demodulation effect

The temporal change of ultrasonic (primary) waves, pu [Pa],leads to the radiation of audible sound (secondary) waves,pa [Pa] [5]. This effect is called self-demodulation, and isexpressed by 2 1 ∂2 β ∂2 ∇ − 2 2  pa = − 4 2 pu2 ,c ∂t 
ρc ∂t
Figure 2. Super-directional loudspeaker and proposed soundpoint loudspeaker. The latter loudspeaker can send audible
sound to only the target person by focusing the ultrasound.

Figure 3. (Left) Spatial distribution of ultrasound on the focal
plane, which is known as the Airy pattern. (Right) Focal
length, device size, diameter of the focal point, and focal depth.

focal point [13, 14]. Generating multiple focal points
simultaneously is possible [15, 16]. The same effect is also
used to draw images on object surfaces [17, 18].
Acoustic levitation is another application of ultrasonicphased arrays. The principle of acoustic levitation is that
potential wells in an ultrasonic spatial distribution suspend
small objects. A line-like standing wave [19] is the simplest
configuration, which is formed by two opposite phasedarrays that generate focal points at the same position. By
widening the ultrasonic beam in one direction and using
four phased arrays, a grid-like standing wave is formed, tomake a projection screen floating in air [20]. Multiple
objects can be moved independently [21]. Generating
potential wells with a single phased array has also been
demonstrated [22, 23].
The trend that ultrasonic phased arrays are actively used in
the HCI field motivated us to apply it to audible-sound fieldmanipulation. Because audible sound radiation requires
lower intensity than force generation for noncontact haptic
feedback and acoustic levitation, we can distribute theacoustic power of ultrasound among more focal points.
PRINCIPLES

We utilize the self-demodulation effect to radiate audible
sound from ultrasound. In order to achieve an effective
ultrasonic intensity at a localized area (focal point), we
employ a phased-array focusing technique. We use(2)

where c [m/s] is the speed of sound in air, β is the nonlinear
parameter, and ρ [kg/m3] is the mass density of air. The left
side of this equation is the wave equation of the secondarywave, and the right side is the driving force arising from thesecond-order differential of the square of the primary wave.
This effect becomes obvious when pu is effectively highand its square value is not negligible. Here, we name thisthe audible-sound radiation threshold pTH [Pa]. Both of AM
and frequency modulation (FM) can be used to modulate
ultrasound.
In the case of super-directional loudspeakers, ultrasonicbeams are used and the self-demodulation effect is always
accompanied by the parametric array effect. As a result,beams of audible sound are generated. We intend to
separate the self-demodulation effect from the parametric
array effect. For this purpose, we use the phased-array
focusing technique to generate a focal point that is the only
place from where audible sound is radiated.
Phased-array focusing

In order to generate a focal point, the distance from the i-thtransducer to the focal point, li, is an important parameter incalculating the phase delay, differently from the beamsteering technique expressed by Eq.(1). This is expressed as∆ϕ i = k (l0 − li ) .

(3)

Although here we consider the i-th transducer in a 1D array,the extension to the (i, j)-th transducer in a 2D array isstraightforward. The position of the focal point can be
freely moved by changing the phase delays. Note that thecalculated phase delays become identical to those of aconventional super-directional loudspeaker when theposition of the focal point is set at an extremely long
distance. We can then generate and steer a sound beam,instead of generating a sound point.
The sound pressure distribution at the focal plane is the
Airy pattern (Fig. 3 (left)) that is well known in the field of
optics. The diameter of a focal point, wr [m], generated by acircular phased array is theoretically given by

wr = 2.44λ

R
,
D

(4)

wave propagation and observation angle, θ [rad], of the n-th
point source are nkΔw and -nkΔw cosθ, respectively. The
amplitude of audible sound received at the observation
point, p(θ) [Pa], is derived as follows:

A jnk∆w − jnk∆w cosθeen =0 NN −1p(θ ) = ∑Figure 4. Discrete model of the focal point to estimate thedirectivity of audible sound. The virtual point sources are
distributed along the direction of ultrasound propagation.

where λ [m] is the wavelength of ultrasound, R [m] is the
focal length, and D [m] is the diameter of the array. Thisequation indicates that a higher frequency results in asmaller focal point, and the focal length and array size havea trade-off relationship. The focal depth, wz [m], (Fig. 3(right)) is derived from the geometrical relationship (i.e.
wr : wz = D/2 : R), which is given by2

Rwz = 4.88λ   .
D

(5)

We tune the amplitude of ultrasound so that pu > pTH only
inside the focal point, to make a point source of audible
sound.
Directivity of audible soundThe focal depth dominantly determines the directivity ofradiated audible sound. We estimate it by assuming the
focal point as an array of point sources with a length of the
focal depth and neglecting the width of the focal point (Fig.
4). The number of point sources is N, i.e., wz = (N-1)Δw.
The amplitude of audible sound received at a faraway
observation point from each point source is assumed as A/N[Pa]. Note that, in reality, the amplitude of each point
source is not the same and gradually decreases with wave
propagation owing to air absorption and secondary-wave
radiation. The interval between the n-th and (n+1)-th point
sources is Δw [m]. The phase delays originating from the=A

sinc[k ( wz + ∆w)(1 − cos θ ) / 2]
sinc[k∆w(1 − cos θ ) / 2](6)where j is the imaginary unit. We obtain the following
equation as Δw → 0,p(θ ) → A sinckw z (1 − cos θ )= AΦ (θ ) .
2(7)That is, a sinc function expresses the directivity, Φ(θ), andthe focal depth is a parameter when the wavenumber ofaudible sound is given.
Figure 5 shows examples of the directivity, Φ(θ), of 1-kHzsound for various wz. The focal point acts as a sound point
source if the directivity is quasi-omnidirectional as we
intend, and the focal point generates one or more beams if
the directivity has zero points for some angles. It is useful
to see the directivity as a function of R/D in designing it.
For example, the amplitude at θ = π/2 rad becomes -3dB
when wz = 150 mm and it is no longer quasiomnidirectional. This situation occurs when R/D ≈ 1.9 (R ≈320 mm with D = 170 mm) for λ = 8.5 mm. Therefore, we
can use this phased array as a quasi-point-source
loudspeaker at the range of R/D < 1.9 for frequencies lessthan 1 kHz. Note that it becomes similar to a superdirectional speaker at the range of R/D > 1.9.
Pulse-width modulation

As mentioned above, ultrasonic waves are modulated with
audible sound. We implement PWM into a digital control
circuit to achieve AM of ultrasonic carrier waves. The
driving signals are 40-kHz rectangular waves having

wz = 50 mm

wz = 150 mm

wz = 500 mm

wz = 1500 mm

wz = 5000 mm(R/D ≈ 1.1)

(R/D ≈ 1.9)

(R/D ≈ 3.5)

(R/D ≈ 6.0)

(R/D ≈ 11.0)

Figure 5. Simulated directivities of focal points of various lengths. R/D is also shown, which is calculated assuming λ = 8.5 mm.
Ultrasonic waves propagate from left to right as shown in Fig. 4. It is quasi-omnidirectional up to R/D ≈ 1.9.

Figure 6. Circuit configuration of the developed prototype. It communicates with the control PC via a Wi-Fi network.

Figure 7. Phase delays to generate an ultrasonic focal point.

adequate phase delays. These signals are initially 3.3 V0-p,
and then amplified to V0 = 24 V0-p by push-pull amplifiers.
The direct current component is removed by high-passfilters before these signals drive the transducers. Because
ultrasonic transducers act as narrow band-pass filters, theredundant frequency components in the rectangular waves
are eliminated and only the 40-kHz component drives thetransducers. The amplitude of the 40-kHz sinusoidal wave,
V1 [V0-p], is not proportional to the pulse width, W [s], andis expressed as

V1 = V0 W
sin  2π 
π  T 2(8)

where T [s] is the period of the ultrasound (25 μs for 40kHz ultrasound).
In the PWM method shown above, audio data sampled at
44.1 kHz is not exactly reproduced because the pulse widthis updated at 40 kHz. This can be solved by sampling audio
data at 40 kHz or using transducers of 44.1-kHz resonant
frequency.
PROTOTYPE
Specifications

We developed a prototype of our audio focusing device(Fig. 6). It consists of 269 transducers (40 kHz resonant
frequency, 10 mm diameter, T4010B4, Nippon CeramicCo., Ltd.) arranged in a 170-mm-diameter circular area.
The maximum value of sound pressure at the center of the
focal point is approximately 2,400 Pa RMS (estimated), andthe diameter and depth of the focal point is calculated as wr= 24.4 mm and wz = 57.4, respectively, for a focal length R

SDS A

SDS B

Figure 8. Commercially-available super-directional
loudspeakers used in the experiments for comparison.

= 200 mm. This focal point is small enough to act as aquasi-point source of audible sound. The position of the
focal point is digitally controlled with a resolution of 1/32
times the wavelength (approximately 0.25 mm for 40-kHz
ultrasound) and can be refreshed at 1 kHz.
The prototype phased array consists of three circuit boards.
The first board contains the array of ultrasonic transducers,
the second is a driving board including an FPGA and pushpull amplifier ICs, and the third is a computer board(RasberryPi 2) for Wi-Fi communication and control. The
control application was developed in C++ and runs on ArchLinux.
Here, we explain how to output a single focal point. The
RasberryPi 2 communicates with external computers via a
Wi-Fi network to receive commands, and then sends thedata (the coordinates of the focal point and pulse width) tothe driving board. The driving board receives the data,
calculates the adequate phase delays for all the transducers
based on the distance to the focal point (Fig. 7), andgenerates the driving signals. The driving signals are sent to
the transducers via the amplifiers and HPFs. Note that we
can modify the phase-delay calculation algorithm to form
spatial distributions other than a single focal point. PWM
control is conducted digitally with a resolution of 624 steps.
Although this resonant-type PWM is nonlinear as
mentioned above, linear conversion from audio data toFigure 11. Experimental setup. A microphone is set in front ofloudspeakers to record sound signals.

Figure 9. Experimental setup. A microphone is fixed on an XY
stage to scan the sound pressure distribution above HW.

pulse width is used in the prototype within an
approximately linear range (from around 100 to 500 of 624
steps) for simplicity. Precise nonlinear conversion using asinusoidal function will improve the sound quality.
EXPERIMENTS

We conducted two experiments for evaluation of the spatial
distribution and audible sound characteristics of ourHolographic Whisper (HW) device. We compared our
prototype device with two commercially-available superdirectional speakers; SDS A (k2617 parametric speaker kit,
Tri-State) and SDS B (super-directional speaker kit, Switch
Science) shown in Figure 8. The WH generated a single
sound point at 150 mm above (along the z-axis) the
speakers. With this focal length, the focal depth is wz = 16mm and the focal point effectively acts as a point source.
All sound signals for evaluation were recorded in 24 bit and
48 kHz WAV format.
Spatial acoustic distribution

The spatial distribution of a 1 kHz sine wave was measured.
Figure 9 shows the experimental setup. The HW was
directed upward, and each speaker was driven by a 1-kHzsine signal. An x-y stage was used to move the microphone(ATM-350, Audio-Technica) over the speakers. The soundsignal was captured by an audio interface (QUAD
CAPTURE, UA-55, Roland). The capture intervals in the xand y- directions were 10 mm.
Figure 10 shows the amplitude distribution of 1-kHz soundradiated from the HW on the x-y planes at different
distances along the z-axis. The amplitude is a relative value.
It is shown that the HW could successfully generate a soundsource only on the focal plane, which is within a 30 × 30
mm area. The sound near the surface is observed, however
it does not affect the focal point. This comes from theelectric circuit and can be reduced in future devices.
Signal characteristics

We conducted experiments in order to show that the HW
can reproduce audio data. A song “Twinkle, twinkle, little
star” starting from two different keys, C3 (130.81 Hz) andC4 (261.63 Hz) was used. The HW and two SDSs were set
vertically to radiate ultrasound upwards. The audible soundwas captured at a distance of 750 mm. We used a
microphone (ECM-CG60, Sony) and an audio interface(DR-07MKII ， TASCAM). The focal length of the HW
was adequately long so that the HW radiated a plane wave
in this experiment. That is, all the speakers generated planewaves toward the microphone.
Figures 11 and 12 show the experimental setup and results,
respectively. The uppermost panel of Fig. 12 shows theoriginal temporal waveform and the upper middle, lower
middle, and bottom panels show the Fourier transforms of
the original signal, and recorded signals from SDS A, SDS
B, and WH, respectively. The 16 kHz and 8 kHz
components included in all the data are subharmonics of the
40 kHz carrier frequency.
Figure 13 shows the magnified images of Fig. 12. The HW
seemed to reproduce signals from 3 kHz to 8 kHz moreaccurately in comparison with SDS B, and approximately
as well as SDS A. At a minimum, we can conclude thesound quality of HW is no less than SDS A and B.

Figure 10. Measured spatial distribution of 1-kHz audible sound before, on, and after the focal point, R = 150 mm.

User study 1: Azimuth indication

Holographic Whisper can move the sound point source on
the x-y plane. This user study was conducted to confirm
whether the participants can perceive the position of the
focal point. Each participant was asked to freely walkaround the device, to hear sound, and to answer where theamplitude is the highest. Three different positions (thechairs A, B, and C, as shown in Fig. 14) were tested foreach participant. As a result, 42 (= 14 × 3) trials wereconducted in total. The coordinates (x, y, z) of the focal
point for A, B, and C are (0 mm, 300 mm, 500 mm), (300mm, 300 mm, 500 mm), and (300 mm, 0 mm, 500 mm),
respectively.
Figure 15 shows the merged results. There is just one error,
and these results indicate that the participants perceived theposition of the focal point with high accuracy.
User study 2: Height perception

Figure 12. From the top, original signal and its spectrum, andspectrums of measured signals from SDS A, SDS B, and HW.

Holographic Whisper can also move the sound point source
along the z-axis. This user study was conducted to examine
whether humans can perceive the height of the focal point.
Each participant was asked to freely move his/her head up
and down, to hear sound, and to answer the perceived
height of the sound point source. Three different heights(the device surface, the air space above the device, and theceiling, as shown in Fig. 14) were tested for eachparticipant. As a result, 42 (= 14 × 3) trials were conductedin total. Note that, in the case of focusing at the ceiling,
Holographic Whisper acted as a super-directional
loudspeaker (i.e., the device generated a sound beam rather
than a sound point).
Figure 15 shows the merged results. The diagonalcomponents (i.e., right answers) have the highest scores,
and these results indicate that the participants could
discriminate the three heights.
User study 3: Height perception (low amplitude)

Figure 13. Magnified images of a single note from Fig. 12.
There is no significant difference between these signals.
USER STUDIESThe perceived position of a sound image generated by
Holographic Whisper was investigated. Holographic
Whisper was facing upwards (Fig. 14). It generated a soundpoint source at various positions on the x-y plane and along
the z-axis. Fourteen people (4 females and 10 males, agefrom 18 to 36, no auditory difficulties) participated in thefollowing studies. Before each user study, they were
instructed regarding the ability of Holographic Whisper andexperienced all the trial conditions.

This user study was conducted to evaluate how correctly
humans can perceive the height of the focal point. The
amplitude of audible sound was lower than the other user
studies in order to highlight the focal point as a soundsource. Each participant was asked to freely move his/herhead up and down, to hear sound, and to answer theperceived height of the sound point source. Four different
heights (200, 400, 600, and 800 mm, as shown in Fig. 14)
were tested for each participant. As a result, 56 (= 14 × 4)
trials were conducted in total.
Figure 15 shows the merged results. Although the innercases (400 and 600 mm) tend to be answered correctly, theboth-end cases (200 and 800 mm) tend to be answered
wrongly.
APPLICATIONSHolographic Whisper enables new types of audio
applications. Because its sound focal points are invisible
and intangible, it is also applicable to augmented reality(AR) and virtual reality (VR) applications. It is especiallyFigure 14. Experimental setups for user studies of azimuth indication and height perception (long and short distances).

Figure 16: Audio application: Meeting table.
Audio applications
Sound sources floating over a table

In this application, Holographic Whisper is mounted in thecenter of a table facing upward (Fig. 16). A sound source isplaced at the eye-level of audiences. In the case of
teleconference, audiences cannot help but look at the soundsource (i.e., loudspeaker) when a person who is at a remoteplace talks. However, they can maintain eye contact witheach other using Holographic Whisper.

Figure 15. Experimental results: Confusion matrices.

The most significant feature of this setup is that audible
sound comes from an intermediate position between thetable surface and ceiling where a sound point source isgenerated. If we place an ordinary loudspeaker instead of
Holographic Whisper, audiences would perceive audible
sound was originating from the table surface. If we place asuper-directional loudspeaker, audiences would perceivethat audible sound was originating from the ceiling as aresult of reflection.
Multi-language simultaneous interpretation

suitable if the position and/or movement of the soundsource plays an important role. Here we show examples ofpossible applications.

Holographic Whisper can deliver different messages to theears of different persons, in the same setup (Fig. 16), withsmaller volume, if the positions of the ears are known ortracked by image processing. This ability is suitable forsuperimposed with indications or messages such as “do nottouch me.”A similar effect is produced by a conventional superdirectional loudspeaker with reflection on a real object.
However, the reflection angle is determined by the incident
angle and the object’s geometry. In the case of Holographic
Whisper, these conditions do not affect sound radiation.
Superposition on real environment

Figure 17. Audio application: Targeting walking person.

Holographic Whisper can enrich AR games such as Ingress[25] and Pokémon GO [26]. When players come to checkpoints, called portal/Pokestops, messages and sound effects
can be provided without bothering other people.
Superimposition on mid-air graphics

Recently, various technologies have been developed torender mid-air graphics, such as 3D displays and headmounted displays. Ordinary methods to provide 3D audio
are binaural headphones and 5.1-ch surround sound systems.
However, Holographic Whisper can superimpose soundsources on mid-air graphics, e.g., virtual birds singing as
they fly. Users feel as if these graphics emit sounds withoutwearing devices or using multiple loudspeakers.
Representation of invisible objects

Figure 18. AR application: Stuffed animals talking to a user.

simultaneous interpretation at international meetings. Eachparticipant can receive his/her mother tongue from atranslator without wearing an earphone.
Targeting an individual pedestrian

Holographic Whisper can move the position of a soundpoint source, and this feature could be useful for following
a target person for explanation in a museum andannouncements in a café if their positions are tracked byimage processing. Moreover, talking to a person who isbehaving suspiciously, using Holographic Whisper, may
contribute to the prevention of crimes. A sound point sourceis always generated near the ear even if the target person iswalking (Fig. 17).
A similar effect is produced by a conventional superdirectional loudspeaker with steering function. However,reflection from the walls and ceiling is not desired incrowded places, and so the installation must be designed
carefully. Holographic Whisper does not produce a soundbeam but a sound point, and so it has a higher degree of
freedom of installation than super-directional loudspeakers.
AR/VR applications
Superimposition on real objects

Holographic Whisper can also add audio to real objects. For
example, sound sources are placed in front of mouths ofpersons in displays or stuffed animals (Fig. 18). Then, users
feel as if these persons or animals emit sounds from theirmouths. It is also possible to for inanimate objects to be

Even if there is no visual information, people can feel the
movement of objects by their sound localization ability.
The moving sound sources generated by Holographic
Whisper can represent invisible ghosts, virtual birds, or tiny
insects flying around users.
DISCUSSION
Limitation
Ultrasound exposure

Intense ultrasound is required to radiate audible sound. It
can potentially damage human ears, and so carefulmanagement is important for safe use of ultrasonic devices.
The issue of ultrasound exposure has previously beenstudied and reported [24]. It is generally recommended to
avoid unnecessary ultrasound exposure by shortening
listening time, limiting ultrasound amplitude, and keep aproper distance from ultrasonic devices so that ultrasonicwaves are well absorbed in air before reaching the ears.
Holographic Whisper has the advantage of providing lessultrasound exposure to a listener in comparison with superdirectional loudspeakers. A super-directional loudspeakerradiates an audible-sound beam in the same direction as anultrasonic beam. This leads to ultrasound exposure when
delivering audio to audiences. However, Holographic
Whisper generates an audible-sound point source fromfocused ultrasound. The radiated audible sound is quasiomnidirectional, i.e., not only parallel to the propagation
direction of ultrasonic waves but also perpendicular to that.
Hence, audiences can receive audible sound whilst standingout of the area of intense ultrasound.
Here we estimate the ultrasonic sound pressure thataudiences receive. Suppose that we have 130 dB SPL

(typical value for self-demodulation effect [5]) at 1 cm fromthe center of the focal point. Because the sound pressure
distribution on the focal plane is the Airy pattern (the firstorder Bessel function of the first kind divided by the
distance) as shown in Fig. 3, the sound pressure decreases
to 96 dB SPL at 50 cm from the focal point. This is lower
than the exposure limit, 110 dB SPL, recommended in [27].

sources are possible by further sophisticated phase control.
The simplest method is generating two spot sources bydividing the phased array into two sections. A computergenerated hologram (CGH) is another method, which isusually used in the field of laser optics [29] and also
applicable to acoustics [30]. Note that all focal points
radiate the same sound because they are synchronized.

Effective range

There is a possibility to produce the directivity of radiatedaudible sound by arranging point sources in air. In otherwords, the ultrasonic phased array generates an array ofpoint sources of audible sound in air. The control method istuning the distance between the point sources, instead of thephase delays, so that the radiated sound waves interferewith each other. The directivity of arranged point sources
can also be altered by slightly changes a CGH of ultrasoundto rearrange the positions of point sources.

Holographic Whisper’s directivity gets narrower to be
similar to that of super-directional loudspeakers when the
focal length R is set at a long distance. The device size D isa parameter to compensate this effect because R/Ddetermines the directivity as shown in Fig. 5. The current
prototype (D = 170 mm) can generate a quasi-point source
up to R ≈ 320 mm (R/D ≈ 1.9). If the focal length up to 1 m
is desired, the device size should be larger than 526 mm.
This is a trade-off relationship between the effective range
and the device size.
Grating lobes

Phased arrays generally suffer from grating lobes(especially large side lobes), which are unnecessary beams
originating from the discreteness of transducer arrays. Thiseffect is explained as aliasing owing to the transducer
interval being longer than the wavelength. In the current
prototype, grating lobes are generated at approximately 40°
from the main lobe (d = 10 mm and λ = 8.5 mm).
These grating lobes radiate sound at a volume lower thanthe main focal point owing to the directivity of thetransducers and they are not noisy. We can also avoid thesegrating lobes to affect unrelated people by controlling theposition and the timing of the sound point source
adequately, based on vision sensing, under the condition
that people are moving and sparse.
In order to eliminate grating lobes, smaller diametertransducers such as [28] (d = 5.2 mm and λ = 8.5 mm) canbe used so that they can be arranged at decreased intervals.
Then, the angle of the grating lobes is increased, and finallygoes beyond the directivity of transducers.
Amplitude of audible soundThe amplitude of audible sound is determined by the
amplitude of ultrasound and modulation depth (fluctuationof amplitude for AM and frequency for FM) according toEq.(2). It is desirable that the amplitude of ultrasound issmall, and so a reasonable method is AM with ultrasonic
amplitude as small as possible with a maximum modulation
depth (i.e., from zero to maximum amplitude).
The amplitude of audible sound determines not only the
directivity of sound point as discussed after Eq.(7) but also
the audible range of sound point source. These parameters
are in trade-off relationships, and designed carefully
according to the purpose.
Arrangement of multiple sound sources

Holographic Whisper is not limited to a single sound point
source. Various arrangements of multiple sound point

Future work

The prototype of Holographic Whisper is currently at thestage of feasibility study. The future work includes thefollowing issues.
The PWM control of the driving signals should be
optimized. Equations (2) and (8) express the nonlinear
characteristics of the self-demodulation effect and theresonant phenomenon, respectively. However, linear
conversion from audio data to pulse width is used in the
prototype for simplicity, which causes a distortion of sound.
Any high-frequency background noise should be eliminatedfor comfortable user experiences. Electric engineers saythat this may come from the mechanical vibration of
surface-mounted ceramic capacitors and it is an issue incommercial products such as PC. We plan to replace thesecapacitors with low-noise ones.
The positioning and tracking of the user is important inmost of Holographic Whisper’s applications. Vision
sensing is suitable for our purpose because of its wide
sensing area, easy installation, and wide variety of available
post-processing methods. We intend to develop systems andservices using Holographic Whisper.
CONCLUSION

We proposed a sound-point loudspeaker to render soundpoint sources in air by using an ultrasonic phased array. We
explained the principles: Self-demodulation effect, phasedarray focusing, directivity of audible sound, and pulsewidth modulation. We confirmed a point source of audible
sound is successfully generated. We conducted theexperiments on sound quality and spatial distribution, and auser study on perceived position of a sound source in aroom. We listed the examples of possible applications. The
proposed sound-point loudspeaker has several advantages
in comparison with conventional super-directional
loudspeakers, such as audience selectivity and low
ultrasound exposure.

REFERENCES

1.

A.J. Berkhout. 1988. A holographic approach toacoustic control, Journal of the Audio Engineering
Society 36, 977-995.

2.

A.J. Berkhout, D.D. Vries, and P. Vogel. 1993.
Acoustic control by wave field synthesis, The Journalof the Acoustical Society of America 93, 2764-2778.

3.

BOSE. F1 flexible array loudspeaker system. fromhttp://www.bose.com/prc.jsp?url=/promotions/entry_pa
ges/f1/index_en.jsp

4.

JBL. IntelliDisc-DS90. fromhttp://www.jblpro.com/www/products/installedsound/intellidisc-ds90

5.

M. Yoneyama, J. Fujimoto, Y. Kawamo, and S.
Sasabe. 1983. The audio spotlight: An application of
nonlinear interaction of sound waves to a new type ofloudspeaker design. The Journal of the AcousticalSociety of America 73, 5, 1532-1536.

6.

K. Aoki, T. Kamakura, and Y. Kumamoto. 1991.
Parametric loudspeaker - Characteristics of acoustic
field and suitable modulation of carrier ultrasound,Electronics and Communications in Japan 74, 9, 7682.

7.

F.J. Pompei. 1999. The use of airborne ultrasonics forgenerating audible sound beams, Journal of the AudioEngineering Society 47, 9, 726-731.

8.

W.-S. Gan, J. Yang, T. Kamakura. 2012. A review ofparametric acoustic array in air. Applied Acoustics 73,
1211-1219.

9.

K. Shinagawa, Y. Amemiya, H. Takemura, S. Kagami,
and H. Mizoguchi. 2007. Three dimensional simulationand measurement of sound pressure distribution
generated by 120 ch plane loudspeaker array. In
Proceedings of IEEE International Conference on
Systems, Man and Cybernetics, 278-283.

10. T. Matsui, D. Ikefuji, M. Nakayama, and T. Nishiura.
2014. Multiple audio spots design based on separating
emission of carrier and sideband waves. In
Proceedings of International Congress and Exposition
on Noise Control Engineering, 6313-6317.
11. W.-S. Gan, J. Yang, K. S. Tan, and M.-H. Er. 2006. A
digital beamsteerer for difference frequency in aparametric array. IEEE Transactions on Audio, Speech,
and Language Processing 14, 3, 1018-1025.
12. C. Shi, Y. Kajikawa, and W.-S. Gan. 2014. An
overview of directivity control methods of theparametric array loudspeaker. APSIPA Transactions on
Signal and Information Processing 3, e20.
13. T. Hoshi, M. Takahashi, T. Iwamoto, and H. Shinoda.
2010. Noncontact tactile display based on radiationpressure of airborne ultrasound. IEEE Transactions on
Haptics 3, 3, 155-165.

14. K. Hasegawa and H. Shinoda. 2013. Aerial display ofvibrotactile sensation with high spatial-temporal
resolution using large-aperture airborne ultrasoundphased array, In Proceedings of IEEE World Haptics
Conference, 31-36.
15. T. Carter, S. A. Seah, B. Long, B. Drinkwater, and S.
Subramanian. 2013. Ultrahaptics: Multi-point mid-air
haptic feedback for touch surfaces. In Proceedings ofthe 26th Annual ACM Symposium on User Interface
Software and Technology, 505-514.
16. S. Inoue, Y. Makino, and H. Shinoda. 2015. Active
touch perception produced by airborne ultrasonichaptic hologram, In Proceedings of IEEE World
Haptics Conference, 362-367.
17. Y. Sugiura, K. Toda, T. Hoshi, Y. Kamiyama, T.
Igarashi, and M. Inami. 2014. Graffiti fur: Turning
your carpet into a computer display, In Proceedings ofAnnual ACM Symposium on User Interface Software
and Technology, 149-156.
18. A Marzo, R. McGeehan, J. McIntosh, S.A. Seah, and
S. Subramanian. 2015. Ghost touch: Turning surfaces
into interactive tangible canvases with focusedultrasound, In Proceedings of International Conferenceon Interactive Tabletops & Surfaces, 137-140.
19. T. Hoshi, Y. Ochiai, and J. Rekimoto. 2014. Threedimensional noncontact manipulation by opposite
ultrasonic phased arrays, Japanese Journal of Applied
Physics 53, 07KE07.
20. Y. Ochiai, T. Hoshi, and J. Rekimoto. 2014. Pixie dust:
Graphics generated by levitated and animated objects
in computational acoustic-potential field. ACM
Transactions on Graphics 33, 4, article no. 85.
21. T. Omirou, A. Marzo, S.A. Seah, and S. Subramanian.
2015. LeviPath: Modular acoustic levitation for 3D
path visualisations, In Proceedings of Annual ACM
Conference on Human Factors in Computing Systems,
309-312.
22. A. Marzo, S. A. Seah, B. W. Drinkwater, D. R. Sahoo,
B. Long, and S. Subramanian. 2015. Holographic
acoustic elements for manipulation of levitated objects.
Nature Communications 6, article no. 8661.
23. A. Marzo. 2016. GauntLev: A wearable to manipulate
free-floating objects, In Proceedings of Annual ACM
Conference on Human Factors in Computing Systems,
3277-3281.
24. C. Q. Howard, C. H. Hansen, and A. C. Zander. 2005.
A review of current ultrasound exposure limits.
Journal Occupational Health and Safety - Australia
and New Zealand 21, 3, 253-257.
25. Niantic Inc. Ingress. from https://www.ingress.com/
26. Niantic Inc. Pokémon GO. fromhttp://pokemongo.nianticlabs.com/

27. C.Q. Howard, C.H. Hansen, and A.C. Zander. 2005. A
review of current ultrasound exposure limits, The
Journal of Occupational Health and Safety of Australia
and New Zealand 21, 3, 253-257.
28. Murata Manufacturing Co., Ltd. MA40H1S-R. fromhttp://www.murata.com/englobal/products/productdetail.aspx?cate=cgsubUltrason
icSensors&partno=MA40H1S-R

29. J. Bengtsson. 1994. Kinoform design with an optimalrotationangle method. Applied Optics 33, 29, 68796884.
30. Y. Xie, C. Shen, W. Wang, J. Li, D. Suo, B.-I. Popa, Y.
Jing, and S.A. Cummer. 2016. Acoustic holographic
rendering with two-dimensional metamaterial-based
passive phased array, Scientific Reports 6, article no.
35437.