Effect of handy microphone movementin mixed reality visualization system of sound intensity
Yuta KATAOKA(1) , Wataru TERAOKA(2) , Yasuhiro OIKAWA(3) , Yusuke IKEDA(4)(1)

Waseda University, Japan, kataoka8894@akane.waseda.jp(2)

Waseda University, Japan, tera-wata1129@fuji.waseda.jp
(3)

(4)

Waseda University, Japan, yoikawa@waseda.jp

Tokyo Denki University, Japan, yusuke.ikeda@mail.dendai.ac.jp

Abstract
We have proposed a system of real-time measurement and visualization for three-dimensional (3D) sound field
by using mixed reality (MR) device. MR device enables effective visualization with depth perception using
the 3D computer graphics (3DCGs) superimposed on the real world. Our system can superimpose 3D soundintensity data on the measurement points in the see-through head mounted display (STHMD). It helps us not
only to understand the sound field but also to design it, to detect the source of noise, etc. Sound intensities
are measured by freely and slowly scanning the sound field with a handy 4-ch microphone array. However, themeasurement results of sound intensity would be affected by assuming a moving microphone array as fixed one.
In this paper, we investigate the effect of microphone movement on sound intensity. To clarify the maximum of
handy microphone speed to be assumed, microphone speeds were measured in measurement of sound field byusing our proposed MR visualization system. Then, simulation experiments in a plane wave and a spherical wave
were conducted to compare the sound intensities by fixed and moving microphone array at different speeds.
Keywords: Acoustic imaging, sound intensity, Doppler effect, Mixed reality(MR), Augmented reality(AR)1

INTRODUCTION

The social request for comfortable sound environment has been increasing. To improve sound environment, it isimportant to understand physical phenomena of sound field. In particular, sound field visualization technologies
help us to intuitively understand the sound field information, because the spatial behavior of sound propagation
is suited for visual perception which has higher spatial resolution. They have been utilized for estimating a noise
source, checking a sound energy flow, acoustically designing rooms, etc. In previous studies, many visualization
methods were proposed by using such as beamforming method [1, 2], acoustic holography [3, 4, 5] and opticalvisualization [6, 7, 8].
As one of sound field visualization technology, we have proposed three-dimensional (3D) visualization systems for sound propagation with a handy 4-ch microphone array by using augmented reality (AR)/mixed reality(MR) and simultaneous localization and mapping (SLAM) [9, 10, 11, 12]. AR/MR refer to the technologies tomerge the real world constructs with virtual constructs generated by computer [13]. It enables us to represent
the depth information in vision while adjusting to the movement of viewpoint. Thus, the measured informationis translated to a object of 3D computer graphics (3DCG) with spatial and sound information. Then, observers
with a head mounted display (HMD) can watch many measurement results while freely changing the viewpoint
in a space. In particular, the real-time measurement and visualization systems [9, 10, 12] enable us to place a3DCG object of sound intensity at each measurement point by slowly moving a handy 4-ch microphone arrayfor understanding the relationships between the measurement results and actual objects.
Recently, a system of real-time measurement for 3D sound intensity by using the optical see-through HMD(OSTHMD) with SLAM [12] achieves free movement of measurement positions in a broad area. By an assumption in our previous visualization methods, the velocity of moving 4-ch microphone array must be slow
enough to ignore the Doppler effect. To visualize a wide range of sound field in a shorter time, it is preferable

4438

Origin-marker
Creation of 3D cone object

3D soundIntensity object

Display

4-ch microphone arraywith Mic-marker
Audio interface
Calculation of 3D sound intensity
PC

Receiving sound intensity data
UDP

OSTHMD： Sound signal： Information of intensity： Positional information

Figure 1. System overview. When a measured position and an intensity are transmitted to the OSTHMD, a 3D
object is shown on the display of OSTHMD.
to measure the sound field moving the 4-ch microphone array at higher speed by removing Doppler effect.
For sound sources moving at high speed, many de-Dopplerization methods have been proposed with multiple
microphones [2, 4, 5]. In this system, the Doppler effect on sound intensity is caused by moving the microphonearray. In addition, average of sound intensities over analysis time is measured by a microphone array at a fixed
position. However, a moving microphone array integrates sound intensities over time and the path of moving
microphone.
In this paper, the effect of the Doppler shift on sound intensity when moving the microphone array, is quantitatively investigated by simulation. Next chapter provides an overview of the system used in this paper. Then,the moving velocity of the microphone was measured with our proposed system. Based on these results, 3D
sound intensity due to the movement of the microphone array is formulated. Finally, the numerical simulationis conducted to compare the effects that a plane and spherical wave received.

2

MEASUREMENT AND VISUALIZATION SYSTEM

2.1 System overview
In this paper, we investigate the influence of moving the microphone array in the real-time measurement andvisualization system [12]. Figure 1 shows the system overview. Microsoft HoloLens [14] is used as OSTHMD.
SLAM installed at HoloLens makes the spatial map of the measured place and understand the position of observer in the spatial map. Based on the spatial map and observer’s position, 3DCG representing 3D soundintensity is superimposed to be fixed to the corresponding position on the real space. When the AR marker
installed at the 4-ch microphone array is detected by a camera of OSTHMD, the microphone positional information is sent sequentially to PC which is connected with the microphone array. Similarly, the sound signals are
obtained sequentially, and are calculated to the sound intensity by cross-spectral method [15]. Sound intensity
Ir between two microphones could be expressed as follows:
Z f2
1
G12 (f )
df,
(1)
Ir =
2πρ∆r f1
fwhere ρ is the atmospheric density, ∆r is the distance between microphones, f1 and f2 are the lower andupper frequencies of the analyzed frequency range respectively, and G12 is the cross spectrum of two soundsignals. Figure 2 shows a 4-ch microphone array we used. Six sound intensities that represent as I01 , I02 ,4439

Figure 2. Image of the 4-ch microphone array. I01 , I02 , I03 , I12 , I13 and I23 represents the sound intensity
between two microphones. By combining these intensities, it is possible to calculate the 3D sound intensity atcenter of array O.
I03 , I12 , I13 and I23 are measured by the 4-ch microphone array in the direction of all six pairs of the four
microphones. We calculated 3D sound intensity from the six sound intensities. Each element of 3D soundintensity I~ = (Ix , Iy , Iz ) is
Ix

=

Iy

=

Iz

=

I01 − I02 − 2I12 − I13 + I23,
4
I01 + I02 + I03√,
6−I01 − I02 + 2I03 + 3I13 + 3I23
√
.
4 3

(2)
(3)
(4)

The sound intensity level is
Li = 10 log10

~
|I|
,
I0

(5)

where I0 = 10−12 W/m2 is a reference intensity. Finally, the 3DCG cone object is positioned at each measurement point by combining the transmitted 3D sound intensity data and its positional information in OSTHMD as~ and the color of the cone shows Li .
shown in Fig. 3. The direction of the cone shows that of I,
A
B

A

Louds
pe
a
ke
r

B

C

C

Figure 3. The state of measurement. The cone object of sound intensity is plotted by scanning the sound fieldwith the microphone array.

4440

Figure 4. The velocity of 4-ch microphone array when movement is at normal (Normal speed) and quick (Highspeed).
2.2 Velocity of 4-ch microphone array
To investigate the influence of the Doppler shift by the movement of microphone array, the velocity of microphone array in our visualization system was measured. The position of the microphone array is detected with a
AR marker by OSTHMD at 60 fps. Assuming that the movement path of the microphone array is line in 1/60
s frame interval, the velocity between two points is calculated from positional information. In experiment, thedistribution of velocity was confirmed by acquiring 12000 samples each for the normal and quick movement.
As shown in Fig. 4, the averages of velocity at the normal and quick movement are 1.12 m/s and 3.77 m/s,
respectively. The maximum velocity at the quick movement is 7.98 m/s.

Figure 5. Arrangement of microphones and sound fields of plane wave and spherical wave. 4-ch microphonearray is located on x-axis. It moves to the direction V .

4441

3

THEORY

3.1 The effect of moving microphone in a plane wave
The medium is assumed to be stationary. A plane wave travels in the positive direction of x-axis with the
speed of sound C. A 4-ch microphone array moved with velocity V . Then, the angle between the direction of
propagation and the direction of V is described by θ as shown in Fig. 5.
The Doppler frequency shift f 0 in the plane wave field is as follows,f0 =C − V cos θ
f,
C

(6)

where f is frequency of a sound source [16]. Using Eq. (6), the sound obtained by the moving microphone
Smoved is represented as,
2iπV cos θf t
0
C
Sfixed ,(7)
Smoved = e(ikr−2πf t) = e√
where i = −1, k is a wave number vector, r is a position of the microphone and Sfixed is the sound obtainthe fixed microphone [17]:~Sfixed = ei(k~r−2πf t) .
(8)
Thus, the sound affected by the Doppler shift can be described by the sound of the fixed microphone. By using
the cross-spectrum G obtained from these four sound waves Smoved , the sound intensity obtained by the moving
microphones can be calculated from the Eq. (1).
3.2 The effect in a spherical wave
When the microphone moves in sound field of a spherical wave, it is necessary to consider the reduction due tothe distance from the sound source. Assuming that the sound source is placed at the origin, the distance between
the source and the initial measurement point in the x-axis is r. The microphone array travels at velocity V .
The sound measured at a distance r is defined as [17]:Sfixed =ei(kr−2πf t)
.
4πr

(9)

The distance between the measurement point and the source is sequentially changed as r0 = |r + vt|. Then, thesound obtained by the moving microphone Smoved can be represented as follows,0reei(kr−2πf t)=
Smoved =4πr02iπ(r 0 −r)f t
C

r0

Sfixed .

(10)

Thus, it is necessary to multiply signal of each microphone with a coefficient which changes with the distance.
The 3D sound intensity obtained by the moving microphone array in the spherical wave can be calculated byusing the cross-spectrum method from the obtained sound wave Smoved as well as the plane wave condition.

4

NUMERICAL SIMULATIONIn this numerical simulation, the initial position of the source was located at (0 m, 0 m, 0 m), and the initialcenter position of the 4-ch microphone array r0 was located at (5 m , 0 m , 0 m). The sound signals were
measured with 44.10 kHz of sampling frequency and 4096 sample of analysis length. The frequency band used
for the cross-spectrum method was an octave band of 1 kHz.
4.1 Plane wave simulation
First of all, we confirmed that the formula of the moving microphone array could be proved by theory when
the source information was known. Figure 6 shows comparison of the sound intensities measured by moving

4442

Figure 6. Sound intensities of plane wave at different velocities of the 4-ch microphone array along x-axis.
The Right side was calculated by moving the microphone array, and the left side was calculated from the fixedmicrophone array by using an estimation method.
the microphone array in x-axis and the sound intensity compensated by a coefficient based on Eq.10. A plane
wave of a single frequency 707 Hz is used as the sound source. The difference of sound intensities was definedas 10 log 10(|Imoved |/|Ifixed |), where Imoved is the 3D sound intensity by moving microphone array and Ifixed isthe fixed 3D sound intensity. The results showed that the fixed microphone and the actual moving microphone
can accurately reproduce the direction and level difference regardless of the velocity.
From this result, it was found that when the velocity of microphone was positive, the frequency of the soundsource was shifted downward. Then, the sound intensity was out of the range of frequency band due to the
Doppler shift. Thus, about 10 dB of the difference of intensity level occurred compared to the fixed microphone.
Thus, it was considered that the Doppler shift affects the intensity calculation result at the start and end of thefrequency band [707 Hz 1414 Hz]. Then, by changing the frequency of the sound source near the start pointof the band pass, the difference in the intensity level caused by the moving speed of the microphone in x-axis
was confirmed. Figure 7 shows a result that the Doppler shift affected the sound intensity level by changing afrequency of sound source. [688 Hz 710 Hz] at 8 m/s of moving microphone array velocity, and [704 Hz 718
Hz] at 8 m/s were more than 10 dB of the difference in sound intensity. Therefore, we will need to take into
account that the effect on intensity if the sound source have a frequency band near the edge of the intensity
frequency band in a measurement. On the other hand, even when measuring at a normal speed of 2.5 m/s, it
was confirmed that the influence of 7 dB at the maximum was received.
4.2 Spherical wave simulation
Figure 6 showed that the plane wave does not affect the direction of intensity. When the sound source isa spherical wave, the direction of sound intensity is affected. Then, the influence on the 3D sound intensity
caused by the direction of the microphone movement is simulated in Fig. 8. When the microphone moves
perpendicular to the propagation direction of the sound wave, the influence of the sound intensity level issmall. However, the difference between the direction of intensities of the fixed and moving microphones had
a difference of about four degrees in high speed condition. In this paper, although the movement direction of
the microphone is limited to movement only along the axis in order to simplify the problem, rotations of the
microphone and movement specific to handy microphone array are occurred.

4443

Figure 7. Sound intensity level difference between fixed and moving microphone caused by change of frequency
of sound source. Specific frequencies affected the frequency band for calculating cross-spectrum method byDoppler shift.

5

CONCLUSIONSIn this paper, we investigated the effect caused by scanning the handy 4-ch microphone array of a real-timemeasurement and visualization system using OSTHMD. The velocity distribution of the handy 4-ch microphonearray moved at normal speed and high speed was measured in our system. By multiplying the sound signals
acquired from the fixed microphones by coefficients, When the source information is known, the 3D soundintensity obtained by the moving microphone array can be compensated. In numerical simulation, the differenceof 3D sound intensity level was over 10 dB due to only the Doppler shift in a plane wave. The directional effectof the 3D sound intensity reaches 4 degrees when the microphone array moves at high speed perpendicularly
to the propagation direction of spherical wave. Thus, it is necessary to remove the effect of 3D sound intensity
when measuring at high speed. Although the simulation was performed in the moving direction along the axis

Figure 8. Relationship between the direction of moving microphone array and difference of sound intensity ina spherical wave. Unlike the plane wave, the direction of intensity was affected by moving microphone.

4444

to simplify the conditions in this paper, the effect of 3D sound intensity with the other specific trajectory of
handy microphone will be considered.

REFERENCES[1] J. C. Chen, K. Yao and R. E. Hudson, “Source localization and beamforming,” IEEE Signal Process. Mag.,19, 30–39 (2002)[2] G. P. Howell, A. J. Bradley, M. A. McCormick, and J. D. Brown, “De-Dopplerization and acoustic imagingof aircraft flyover noise measurements,” J. Sound Vib. 105(1), 151–167 (1986).
[3] J. D. Maynard, E. G. Williams and Y. Lee, “Nearfield acoustic holography: I. Theory of generalized
holography and the development of NAH,” J. Acoust. Soc. Am., 78, 1395–1413 (1985).
[4] S.-H. Park and Y.-H. Kim, “Visualization of pass-by noise by means of moving frame acoustic holography,”J. Acoust. Soc. Am. 110(5), 2326–2339 (2001).
[5] D. Yang, Z. Wang, B. Li, Y. Luo, and X. Lian, “Quantitative measurement of pass-by noise radiated by
vehicles running at high speeds,” J. Sound Vib. 330(7), 1352–1364 (2011).
[6] K. Ishikawa, K. Yatabe, C. Nachanant, Y. Ikeda, T. Onuma, H. Niwa and M. Yoshii, “High-speed imaging
of sound using parallel phase-shifting interferometry,” Opt. Express 24, 12922–12932 (2016).
[7] Y. Oikawa, K. Yatabe, K. Ishikawa and Y. Ikeda, “Optical sound field measurement and imaging using
laser and high-speed camera,” 45th Int. Congr. Expo. Noise Control Eng., pp. 258–266, Aug. 2016.
[8] K. Ishikawa, K. Yatabe, Y. Ikeda, Y. Oikawa, T. Onuma, H. Niwa and M. Yoshii, “Interferometric imagingof acoustical phenomena using high-speed polarization camera and 4-step parallel phase-shifting technique,”Proc. SPIE, vol. 10328, 103280I, Nov.2017.
[9] Atsuto Inoue, Yusuke Ikeda, Kohei Yatabe, Yasuhiro Oikawa, “Visualization system for sound field using
see-through head mounted display,” Acoust. Sci. & Tech., Vol.40, No.1, pp.1–11, 2019.1.
[10] Atsuto Inoue, Kohei Yataba, Yasuhiro Oikawa, Yusuke Ikeda, “Visualization of 3D Sound Field using
See-Through Head Mounted Display,” Proc. SIGGRAPH 2017 Posters, Article No.34, Jul.2017.
[11] W. Teraoka, Y. Kataoka, Y. Oikawa and Y. Ikeda, “Display System for Distribution of Virtual Image
Sources by using Mixed Reality Technology,” Inter-noise 2018, Aug. 2018.
[12] Yuta Kataoka, Wataru Teraoka, Yasuhiro Oikawa, Yusuke Ikeda, “Real-time Measurement and Display
System of 3D Sound Intensity Map using Optical See-Through Head Mounted Display,” Proc. SIGGRAPH
Asia 2018 Posters, Article No.71, Dec.2018.
[13] P. Milgram and F. Kishino : “A Taxonomy of Mixed Reality Visual Display,” IEICE Trans. Inf. Sys.,
E77-D, 12, pp. 1321-1329 (Dec.1994)[14] Microsoft HoloLens https://www.microsoft.com/ja-jp/hololens 2019-05-29.
[15] J. Y. Chung, “Crossspectral method of measuring acoustic intensity without error caused by instrument
phase mismatch,” J. Acoust. Soc. Am., 64, 1613–1616 (1978).
[16] P.M. Morse and K. U. Ingard, Theoretical Acoustics(McGraw-Hill, New York, 1968), pp. 698-701.
[17] E. G. Williams, “Fourier Acoustics,” Academic Press, (1999)

4445