3-D sound image localization in reproduction of 22.2 multichannel audio
based on room impulse response generation with vector composition
Kaige ZHENG(1) , Misaki OTSUKA(2) , Takanobu NISHIURA(3)(1) Ritsumeikan University, Japan, is0446ex@ed.ritsumei.ac.jp(2) Ritsumeikan University, Japan, is0206pi@ed.ritsumei.ac.jp(3) Ritsumeikan University, Japan, nishiura@is.ritsumei.ac.jp

Abstract
To simplify content creation processes and improve the realism of 22.2 multichannel system, a method to localize sound images is derived in this paper, which is an improvement of the method based on vector baseamplitude panning. The sound image localization can be realized by a speciﬁc direction and distance. A soundﬁeld is simulated to make spatial impressions in 22.2 multichannel reproduction, which includes reﬂection andreverberation, and the simulation approach is based on room impulse response generation with vector composition. To control the sound pressure on the ears of the listener, the amplitude of the input signal is attenuatedin advance by the distance of sound image. Evaluation experiments were carried out both subjectively and objectively with binaural recording. The improvement in the reproduction of sound image’s distance is realized,
while the direction of the sound image stays the same as the method based on vector base amplitude panning.
Keywords: 22.2 multichannel, Sound localization

1

INTRODUCTION

With the development of 3-D sound ﬁeld reproduction technologies, home theater systems are able to create amore realistic experience. In Japan, NHK (Japan Broadcasting Corporation) has developed the 8K Super HiVision standard with 22.2 multichannel audio system [1] in recent years. The traditional way of creating create
surround audio contents is using a panning tool to mix the sound signal, but it is difﬁcult to do so in 22.2
multichannel. Besides, the position of a sound image, including distance and direction, is also hard to control
with panning. To solve these problems and make the creation process easier, a new method to create contents
in 22.2 multichannel sound ﬁelds has to be developed.
To keep the compatibility with old surround systems, the object-based rendering method Vector Base Amplitude
Panning (VBAP) [2] is used. With the position of sound images, the output signal can be generated automatically based on the conﬁguration of target loudspeaker system. This method focus on the direct sound of objects,
therefore the direction information of sound images is easily generated, but the distance can not be controlled
precisely.
In this paper, a method to localize sound images based on room impulse response (RIR) generation is derived.
This method is described in 3 sections: amplitude correction for input signal, RIR generation, and output signal
generation. The input signal’s amplitude is adjusted by the distance of the sound image, and the RIR of eachchannel is generated based on the characteristics of the target sound ﬁeld and the position of the sound image.
This part can reproduce reﬂections and reverberations that are key to human perception of the distance of thesound source. The output signal is generated by the convolution of RIR and the input signal that has beenadjusted in the ﬁrst part.

2

22.2 MULTICHANNEL AUDIOThe loudspeaker arrangement for a 22.2 multichannel sound system is shown in Fig. 1. The numbers in theﬁgure are the azimuth angle of loudspeakers. With the loudspeakers being placed in elevation directions, a 3-D

5252

0

60

300

90

270
120

180

240
30

60

0 330

300
270

‫ݎ‬
90

120

180

240

0

60

300

Front

: Loudspeaker: LFE(Low Frequency Effect): Listening point
Radius ‫ ݎ‬ൌ ͳǤͻ m

Left

Right

Back

Figure 1. The loudspeaker arrangement for 22.2 multichannel sound system.
sound ﬁeld can be reproduced by this system, and sound images can be located from all positions.

3

SOUND IMAGE LOCALIZATION BASED ON VECTOR BASE AMPLITUDE PANNING

In the VBAP based method presented in this section, 3 loudspeakers, which are the closest loudspeakers to theposition of the sound image, is used to localize the sound image, and the gain factors for these loudspeakers
are calculated from the position vector of the sound image.
When the loudspeakers are selected, the gain factors are calculated as

andp = (px py pz )T ,(1)

qi = (qi,x qi,y qi,z )T ,(2)

g = (g1 g2 g3 )T ,(3)

p = q1 g1 + q2 g2 + q3 g3 ,
⎛ ⎞ ⎛
⎞⎛ ⎞
px
q1,x q2,x q3,xg1
⎝ py ⎠ = ⎝q1,y q2,y q3,y ⎠ ⎝g2 ⎠ ,pz
q1,z q2,z q3,z
g3

(4)

⎛ ⎞ ⎛
g1
q1,x
⎝g2 ⎠ = ⎝q1,y
g3
q1,z

q2,x
q2,y
q2,z

⎞−1 ⎛ ⎞q3,xpx
q3,y ⎠ ⎝ py ⎠ ,
q3,z
pz

(5)

(6)

where p is the position vector for the sound image, which points to the center of gravity of the selected
loudspeakers, qi is the position vector for the i (∈ {1, 2, 3})th loudspeaker, and g is the gain factor vector.

5253

Step 2࢞ሺ࢔ሻɾ‫ܘ‬࢞Ԣሺ࢔ሻ

Amplitude correction
for input signal

Step 1

Step 4

RIRs database

Design of
output signals (22ch)

‫ڮ‬

Input
parameters

‫ܟ‬

‫ڮ‬

࢟࢏ ሺ࢔ሻ

Step 3࢞ ࢔ ɾ‫ܘ‬ɾࢀࢊ ɾࢉ‫ڮ‬Design of
22ch RIRs‫ܘ‬ɾࢀࢊ ɾࢉ‫ܐ‬Ԣ࢏ ሺ࢔ሻStep 5࢞ሺ࢔ሻɾ‫ܘ‬

Design of LFE
output signals (2ch)࢟‫ۺ‬۴۳ǡ࢐ ሺ࢔ሻ

22.2ch multichannel soundFigure 2. Overview of the proposed method.
The output signal for each channel is generated by the multiplication of the input signal and the gain factors.
This method is used to render the direct sound of the sound image, but due to the position of the sound image
is limited in the area where the selected loudspeakers are, it is hard for the listener to tell the distance of thesound image. To improve the presence of rendered contents, a new method, which should be able to control
the distance of the sound image, needs to be developed.

4

SOUND IMAGE LOCALIZATION BASED ON ROOM IMPULSE RESPONSE GENERATION WITH VECTOR COMPOSITION

The overview of the proposed method is shown in Fig. 2. 3 steps are included in this method: amplitude
correction for input signal, design of 22ch RIRs, and output signal generation. Human uses sound pressure andreﬂected sound to perceive the distance of the sound image, which is controlled by the ﬁrst and second stepof this method separately. Amplitude correction is performed based on the 3-D position vector of the soundimage, and 22ch RIRs are designed by an adaptive algorithm, which can simulate the desired sound ﬁeld for
reproduction. To reproduce the ILD (Interaural Level Difference) and the ITD (Interaural Time Difference) thatare important to human’s perception of direction, the RIRs are also processed with amplitude correction anddelay.
4.1 Input parameters
As shown in Fig. 2, the input parameters are: the input signal x(n), the time index n, the position vector of thetarget sound image p = (px py pz )T , the reverberation time of the desired sound ﬁeld Td , and the speed of soundc.
4.2 Amplitude correction for input signal
The amplitude correction for input signal is performed asx (n) = αx(n),βα = 10 20 ,

5254

(7)
(8)

Distance decay [dB]0

: Theoretical value: Approximation function

-5
-10
-15

-20
-25
-30

0

1

2

3

4

5
6
Distance [m]

7

8

9

10

Figure 3. The results of the approximation function for distance attenuation.
and
β=

⎧
⎪
⎪
⎪
⎨ −9.55096⎪⎪⎪⎩ 9.55096

0.47082i∈{1,2,··· ,22}min

if(dC < r)

di

min

0.47082dii∈{1,2,··· ,22},

(9)otherwise

where β is the distance attenuation in dB from the sound image to the ith loudspeaker, which is calculatedby an approximate function, di is the distance between the ith loudspeaker and the sound image, dC is the
distance between the center of the 22.2 multichannel system and the sound image, and r is the radius of the22.2 multichannel system. The sound pressure of the sound image is controlled by distance by performing thisstep, and the output signal is x (n). The distance attenuation from Eq. (9) is compared with theoretical value,
and the results are shown in Fig. 3.
4.3 Design of 22ch RIRs
The proposed method generates RIRs by applying subsequent reverberation to early reﬂection, which is assumed
to be uncorrelated. The early reﬂection, which is equal to estimated RIRs, is generated by an adaptive algorithm,
which is written as
w = [w0 , w1 , · · · , wN−1 ]T ,(10)hi (n) = [hi,0 (n), hi,1 (n), · · · , hi,N−1 (n)] ,(11)x (n) = [x (n), x (n − 1), · · · , x (n − N + 1)]T ,

(12)

T

T 

a(n) = w x (n),

(13)

22

a (n) = ∑ hi (n)T x (n) ,(14)e(n) = a (n) − a(n),

(15)

i=1

and

where w includes the measured RIRs of the reproduction sound ﬁeld. The RIRs are selected from an RIR
database. N(=80 ms) [3] is the length of hi (n), which are the estimated RIRs. hi (n) is adapted based on LMS(Least-mean-square) algorithm, where a (n) is the output signal, a(n) is the target signal, and e(n) is the error
at current sample n. The algorithm is repeated based on the correction amount e(n), which is written as5255

hi (n + 1) = hi (n) + 2μe(n)gi x (n),(16)

where μ is the step size, and gi is the normalized gain factor. The algorithm for calculating the normalized
gain factor gi is based on vector composition, which is written as
⎛ ⎞ ⎛
px
l1,x
⎝ py ⎠ = ⎝l1,y
pz
l1,z
⎛

and

l1,x
L = ⎝l1,y
l1,z

l2,x
l2,y
l2,z

l2,x
l2,y
l2,z

⎛

⎞
g1
· · · l22,x ⎜ g2 ⎟⎜ ⎟· · · l22,y ⎠ ⎜ . ⎟ ,⎝ .. ⎠· · · l22,z
g22

(17)

⎞
· · · l22,x· · · l22,y ⎠ .
· · · l22,z

(18)

⎞

It is assumed that the position vector of the target sound image p points to the center of gravity of the selected
loudspeakers. The gain factor gi can be calculated by Eq. (17), but the matrix L is a nonsingular matrix, andtherefore has no inverse. In this algorithm, the gain factor gi is calculated with L+ , which is the pseudo inverse
matrix of L. It is written as
⎞
g1
⎛
l1,x
⎜ g2 ⎟⎜ ⎟ ⎝⎜ .. ⎟ = l1,y
⎝ . ⎠
l1,z
g22
⎛

and

⎛

l1,x
L+ = ⎝l1,y
l1,z

l2,x
l2,y
l2,z

⎞+ ⎛ ⎞· · · l22,xpx· · · l22,y ⎠ ⎝ py ⎠ ,· · · l22,z
pz

(19)

l2,x
l2,y
l2,z

⎞+
· · · l22,x· · · l22,y ⎠ .
· · · l22,z(20)

The gain factors are inversed in channels that are symmetric with respect to the center of the 22.2 multichannel
system. To prevent cancellation in low-frequency band, a normalization process is performed asgi =gi + |gmin |,gmax + |gmin |(21)

where gmax and gmin are the maximum and minimum values of gi . The output RIRs h i (n) are calculated ash i (n) =hi (n)
,
r(n)

(22)

and
r(n) = [r(n), r(n − 1), · · · , r (n − (Td − N) + 1)]T ,(23)

which is the result of applying subsequent reverberation to hi (n). The subsequent reverberation is generated
from exponentially attenuated white noise. [4]4.4 Design of output signals
The output signals yi (n) and yLFE, j (n) are designed based on the RIRs which are calculated in the last section.
The output signals for LFE (Low Frequency Effect) channels in the 22.2 multichannel system are generated by
an LPF (Low-Pass Filter). The algorithm is written as

5256

yi (n) = h i (n)T x (n),yLFE, j (n) = bT x n −

and

(24)

dj
,
c

(25)

b = [b0 , b1 , · · · , bTd −1 ]T ,(26)x (n) = [x (n), x (n − 1), · · · , x (n − Td + 1)]T ,(27)

where x (n) is a Td length of signal, which is extracted from the corrected signal x (n) in Eq. (7). j (∈ {1, 2})
is the index for LFE loudspeakers, and b is the LPF for LFE signals. By reproducing these signals throughall the loudspeakers simultaneously, the localization of the sound image is realized, with the direction and the
distance well controlled.

5

OBJECTIVE EVALUATION EXPERIMENT

5.1 Experimental conditions
In this experiment, the objective evaluation of the reproduction accuracy of the sound image was carried out
by calculating the ILD (Interaural Level Difference) [5]. It shows the difference between the sound pressure oftwo ears, which are represented by sleft (n) and sright (n). It is written as
ILD = 20 log 10

s −1sleft (n)|| ∑Nn=0s −1| ∑Nn=0sright (n)|

,

(28)

where Ns is the length of binaural signals. Based on the ILD (=0 [dB]) when the virtual sound image exists infront of the listener, the ideal tendency is that the ILD rises when the sound image is on the left of the listenerand decreases when on the right.
To evaluate the reproduction accuracy of the distance of the sound image, the sound pressure ratio of thebinaural signals was calculated as
E = 10 log 10

N −1ss2e (n)∑n=0,
Ns −1 2se (n)∑n=0(29)

where se (n) is the binaural signal e (∈ {left, right}) when the sound image is localized at the reference point.
The sound pressure ratio E tends to decrease because the amount of distance attenuation increases as the distance of the sound image becomes farther than the reference point.
In this experiment, the conventional VBAP method, the proposed method, and the real environment (recorded
as binaural sound using Neumann KU100 microphone) were compared. To avoid the inﬂuence of the physicalloudspeakers, the signals are not recorded (except the real environment), but generated as
22

2

i=1

j=1

se (n) = ∑ ki,e T (n)yi (n) + ∑ k j,e T (n)yLFE, j (n),(30)

where ki.e (n) is the BRIR (Binairal Room Impulse Response) from the ith loudspeaker to the binaural microphone, which was measured in advance. Other conditions for this experiment is shown in Table 1.
The direction is in counterclockwise with the front of the listener being 0. The distance represents the distancefrom the center of the 22.2 multichannel system. The input signal was a white noise in 16 [bit] / 48 [kHz]. The
step size μ of the adaptive algorithm in the proposed method was 10−12 . The adaptive algorithm was repeated
400 times.

5257

Table 1. Experimental conditions for objective evaluation
Environment
Reverberation time
Direction
Distance
HeightOfﬁce roomT[60] = 650 [ms]0, 30, · · · , 180 [deg.]
1.0, 2.0, · · · , 5.0 [m]
1.4 [m]: Real environment (Binaural sound): Conventional method: Proposed method: Theoretical value: Real environment (Binaural sound): Conventional method: Proposed method

Figure 4. Left: Experimental results by direction with a distance of 1.0 [m]. Right: Experimental results by
distance with a direction of 30 [deg].
5.2 Results
The results are shown in Fig. 4. When the distance of the sound image was ﬁxed as 1.0 [m], both theconventional method and the proposed method can reproduce the same transition of ILD that approximates thereal environment. No signiﬁcant difference was identiﬁed between the real environment and the two methods
with a t-test performed, where the signiﬁcance level is 5%. When the direction of the sound image was ﬁxed
as 30 [deg], the distance of the reference point in Eq. (29) was ﬁxed as 1.0 [m], and the theoretical value wascalculated using Eq. (9), the energy ration tends to decrease as the distance of the sound image going farther,but due to the fact that the distance controlling is insufﬁcient in the conventional method, the results of theconventional method are relatively ﬂat, compared to the proposed method, which leads to the conclusion thatthe proposed method is able to control the distance of the sound image. Comparing to the real environment, theenergy ratio is greatly reduced with the proposed method, showing that the amplitude correction in the proposed
method needs to be optimized for the reproduction environment.

6

SUBJECTIVE EVALUATION EXPERIMENT

6.1 Experimental conditions
In this experiment, the subjective evaluation of the reproduction accuracy of the sound image was carried out,
which focused on the distance of the sound image. When the direction of the sound image was ﬁxed as 0[deg], 5 subjects were asked to tell the distance of the sound image. The subjects were also asked to listen toevaluation signals of 1.0 [m] and 5.0 [m] before the experiment began. Sony MDR-CD900ST headphone wasused for binaural reproduction. Other conditions for this experiment are shown in Table 2.
6.2 Results
The results are shown in Table 3, which represent the average correct answer rate of the 5 subjects. The
improvement on the reproduction of the distance of the sound image was obvious when the distance was set

5258

Table 2. Experimental conditions for subjective evaluation
Condition
Environment
Temperature
Humidity
Ambient noise level

Binaural soundSound proof room
18.7 [◦ C]
30.1 [%]
LA = 30.4 [dB]

Conventional and proposed method22.2ch multichannel reproduction room
20.3 [◦ C]
26.1 [%]
LA = 38.0 [dB]

Table 3. Experimental results for subjective evaluation
Presented distance [m]135Binaural sound [%]
60
34
32

Conventional method [%]54
28
2

Proposed method [%]
50
28
65

to 5 [m]. Combining the results from the objective evaluation, it is clear that the proposed method has better
control of the distance of the sound image in the range of 3 to 5 [m].

7

CONCLUSIONS

A new method to localize sound images based on RIR generation is proposed in this paper, and the improvement of this method is conﬁrmed comparing the conventional method, which is based on VBAP. This new
method can be used to simplify the content creation process for 22.2 multichannel systems. In the future,
the accuracy of sound image localization based this method can be improved by optimizing parameters of theadaptive algorithm for other reproduction environments.

ACKNOWLEDGEMENTS
This research was partly supported by Ritsumeikan Global Innovation Research Organization, JST COI and
JSPS KAKENHI Grant Numbers JP18K19829 and JP19H04142.

REFERENCES[1] K. Hamasaki, T. Nishiguchi, R. Okumura, Y. Nakayama and A. Ando, "A 22.2 Multichannel Sound System
for Ultra-High-Deﬁnition TV (UHDTV)," SMPTE Motion Imaging Journal, vol. 117, no. 3, pp. 40-49, 2008.
[2] V. Pulkki, "Virtual sound source positioning using vector base amplitude panning," Journal of the AudioEngineering Society, vol. 45, no. 6, pp. 456-466, 1997.
[3] B. G. Shinn-Cunningham, N. Kopco and T. J. Martin, "Localizing nearby sound sources in a classroom:
Binaural room impulse responses," The Journal of the Acoustical Society of America, vol. 117, no. 5, pp.
3100-3115, 2005.
[4] T. Yoshimura, Y. Wakabayashi, T. Fukumori, M. Nakayama and T. Nishiura, "Sound localization with
distance perception based on simulated room impulse response using 16ch head-enclosed loudspeakerarray," in RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing 2017
(NCSP’17), pp. 421-424, 2017.
[5] N.Roman, D. Wang and G. J. Brown, "Speech segregation based on sound localization," Journal of the
Acoustical Society of America, vol. 114, no. 4, pp. 2236–2252, 2003.

5259