PROCEEDINGS of the
23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

Evaluation of moving sound image localization for reproduction of 22.2 multichannel audio using up-mix algorithm
Hiromu SUZUKI1; Takanobu NISHIURA2
1

Graduate School of Information Science and Engineering, Ritsumeikan University, Japan
2

College of Information Science and Engineering, Ritsumeikan University, Japan

ABSTRACT22.2 multichannel audio had been developed for ultra-high definition televisions. It cansimultaneously reproduce a three-dimensional (3-D) sound field. However, it is difficult to easilyacquire multichannel signals because of equipment costs and so on. The up -mix algorithm forgenerating multichannel signals from the existing fewer channel signals has been proposed to solve
this problem. Whereas, the conventional up-mix algorithm has been proposed without the 3-D soundsource location. As a result, it cannot reproduce the moving sound image in vertical and backward
directions. In this paper, we propose a novel up-mix algorithm which extends two channel signals to22.2 multichannel signals while using the 3-D location of the sound source. Both objective andsubjective evaluations were carried out to compare with the accuracy of reproduced sound image
location. Results of the evaluations show that the proposed up-mix algorithm can reproduce 3-D
moving sound images more accurately than the conventional one.
Keywords: 22.2 multichannel audio, 3-D sound field, moving sound image

1. Introduction
New sound field reproduction systems are necessary along with the development of video
technology. 22.2 multichannel audio [1] is a new reproduction system in conformity with ultra-highdefinition televisions. In the conventional system such as 5.1 multichannel audio, all loudspeakers are
set on the horizontal plane. Therefore, it cannot reproduce a three -dimensional (3-D) moving soundimage. 22.2 multichannel audio sets the loudspeakers in three layers: upper, middle and lower. So, it
can reproduce 3-D moving sound image.
In 22.2 multichannel audio, the sounds which are recorded by microphones corresponding to eachdirection are reproduced from each loudspeaker. However, this production method is not suitablebecause of the high cost of recording. To solve this problem, we use the up-mix algorithm forgenerating multichannel signals from the existing fewer channel signals. Whereas, the conventional
up-mix algorithm has been proposed without the 3-D sound source location. As a result, it cannot
reproduce the moving sound image in vertical and backward directions. In this paper, we propose anovel up-mix algorithm which extends two channel signals to 22.2 multichannel signals while using
the 3-D location of the sound source. In addition, we try to reproduce 3-D moving sound images with22.2 multichannel audio using a novel up-mix algorithm.

2. 22.2 multichannel audio
2.1 Design
Fig.1 shows the design of 22.2 multichannel audio. This system can divide three layers: upper,
middle and lower. It consists of 9 loudspeakers in the upper layer, 10 loudspeakers in the middle layer,
3 loudspeakers and 2 woofers in the lower layer. These layers can reproduce a 3-D sound field. Fig.2
shows the label of loudspeakers and the installation interval in each layer. This design is based on theconventional study about the optimal installation interval of loudspeakers.
1
2

5260

is0312ps@ed.ritsumei.ac.jp
nishiura@is.ritsumei.ac.jp

Figure 1 - Design of 22.2 multichannel audio.

(a) Upper layer(b) Middle layer(c) Lower layer

Figure 2 - The label of loudspeakers and the installation interval.
2.2

Effects
The listener obtains the following effects [2] for sound field reproduction with the design shownin Section 2.1.
i.
ii.
iii.
iv.

The arrival of sound from all directions
Sound localization in any positions
Construction of high-quality 3-D sound field by reverberation
Matching of video images and sound images
These effects greatly contribute to high sound field reproduction.

2.3

Problem
Driving 22.2 multichannel audio needs as many as 24 channel signals. One of the ways to generatethese signals is a recording method using 22 spherical microphones. However, this method is costly
because it records sound sources using individually microphones. To solve this problem, we focus on
up-mix algorithm: generating multichannel signals from the existing fewer channel signals. In thispaper, we propose a novel up-mix algorithm which extends two channel signals to 22.2 multichannel
signals while using the 3-D location of the sound source. In addition, we challenge the constructionof a 3-D moving sound image using generated signals. 5261

3. Conventional up-mix algorithm based on reverberation separation
3.1 Overview
Multi-channel audio requires faithful reproduction of the direct sound and the reverberation. The
reproduced sound should be separated into the direct sound and the reverberation because thereverberation comes from all directions in any places. Therefore, we consider applying conventional
up-mix algorithm [3] based on the reverberation separation to 22.2 multichannel audio.
Fig.3 shows the overview of converting two channel signals to 22.2 channel signals using
conventional up-mix algorithm. The procedure is following.
i.

Reverberation separation
Two channel signals ‫ݏ‬ଵ ሺ‫ݐ‬ሻ and ‫ݏ‬ଶ ሺ‫ݐ‬ሻ are separated into direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ݐ‬ሻ andreverberation ‫ୖݔ‬ୣ୴ ሺ‫ݐ‬ሻ using reverberation separation based on Multi-Step Linear Prediction(MSLP) [4].

ii.

Frame division
In this paper, direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ݐ‬ሻ and reverberation ‫ୖݔ‬ୣ୴ ሺ‫ݐ‬ሻ divided into ݂ frame toreproduce 3-D moving sound image. Divided sounds ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ and ‫ୖݔ‬ୣ୴ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ arereproduced from the loudspeaker for each frame; ‫ ݐ‬ᇱ is the time index in ݂ frame.

iii.

2-D Amplitude panning (direct sound)
Direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ is mapped as 2-D sound image in the left or right using 2-D
amplitude panning. 2-D amplitude panning is achieved by controlling the amplitudes of twochannel signals based on added 2-D sound image localization. The coefficients ݃େ୭୬୴ǡଵ ሺ݂ሻ and݃େ୭୬୴ǡଶሺ݂ሻ to control two channel signals are calculated as follows:݃େ୭୬୴ǡଵ ሺ݂ሻ ൌ ߶ ൅  ߠ௙ǡටʹ൫ଶ ߶ ൅ ଶ ߠ௙ ൯݃େ୭୬୴ǡଶ ሺ݂ሻ ൌ ߶ െ  ߠ௙ǡටʹ൫ଶ ߶ ൅ ଶ ߠ௙ ൯

(1)

(2)

where ߶ is the angle between the listener and the loudspeaker, ߠ௙ is the angle between thelistener and the sound image at ݂ frame, ݅ ൌ ͳ denotes the left channel, ݅ ൌ ʹ denotes the rightchannel. Finally, the two channel signals ‫ݕ‬ୈ୧୰ୣୡ୲ǡଵ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ and ‫ݕ‬ୈ୧୰ୣୡ୲ǡଶ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ are calculated asfollows:
s.t. ݅ ൌ ͳǡʹ‫ݕ‬ୈ୧୰ୣୡ୲ǡ௜ ሺ‫ ݐ‬ᇱǡ ݂ሻ ൌ ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ݃େ୭୬୴ǡ௜ ሺ݂ሻ.
(3)

Figure 3 - Conventional up-mix algorithm.

5262

Furthermore, the center channel signal ‫ݕ‬ୈ୧୰ୣୡ୲ǡଷ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ applies the direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ
without panning:‫ݕ‬ୈ୧୰ୣୡ୲ǡ௜ ሺ‫ ݐ‬ᇱǡ ݂ሻ ൌ ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻǤ
s.t. ݅ ൌ ͵ᇱሺ‫ݐ‬The three generated signals ‫ݕ‬ୈ୧୰ୣୡ୲ǡ௜ ǡ ݂ሻ map specific loudspeakers shown in Fig.4.
iv.

(4)

Signal extension (reverberation)Reverberation ‫ୖݔ‬ୣ୴ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ comes from all directions in any spaces. Therefore, it should be reproduced
by remaining 19 loudspeakers. The generated 19 reverberations ‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ are shown as follows:‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ ൌ ‫ୖݔ‬ୣ୴ ሺ‫ ݐ‬ᇱ ǡ ݂ሻǡ

s.t. ݆ ൌ ͳǡ ǥ ǡͳͻ(5)where ݆ shows loudspeaker channel to reproduce the reverberation. The 19 generated signals‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ mainly map the rear loudspeakers in Fig.4.
v.

Low-pass filtering (bass)
The bass reproduced from the woofer is generated by the low-pass filter. The standard for 22.2
multichannel audio [3] defines to play less than 120 Hz as bass. The two basses ‫୐ݕ‬୊୉ǡ௞ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ apply
low-pass filter as follows:‫୐ݕ‬୊୉ǡ௞ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ ൌ ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ ‫୔୐߬ כ‬୊ ሺ‫ ݐ‬ᇱ ǡ ݂ሻǡ s.t. ݇ ൌ ͳǡʹ(6)ᇱ
where ݇ shows woofer channel, ߬୐୔୊ ሺ‫ ݐ‬ǡ ݂ሻ is the coefficient of the low-pass filter.

3.2 Problem
The conventional up-mix algorithm separates two channel signals into direct sound andreverberation; direct sound is mapped three front loudspeakers, reverberation is mapped remaining 19
loudspeakers. However, the following two problems occur when trying to apply the generated signals
to the 22.2 multichannel audio.
i.

The difficulty of reproducing the sound field moving up and down
22.2 multichannel audio needs to reproduce 3-D moving sound image in any spaces. Whereas,
the conventional up-mix algorithm has been proposed without the 3-D sound source location. We
should apply 3-D sound image localization method using upper and lower loudspeakers.

ii.

The need to generate a large number of reverberations
In the actual sound field, different reverberations come from all directions due to the reflectionof the wall and so on. Whereas, the conventional up-mix algorithm generates the same
reverberations to play 19 loudspeakers. We should adopt a method to generate multiple
reverberations corresponding to 22.2 multichannel audio.

In this paper, we propose a novel up-mix algorithm using 3-D location of the sound source.

Figure 4 - The mapping method using conventional up-mix algorithm.

5263

4. A novel up-mix algorithm using the 3-D location of the sound source
A novel up-mix algorithm can reproduce a free moving sound image using the 3-D location of thesound source. Fig.5 shows the overview of converting two channel signals to 22.2 channel signals
using a novel up-mix algorithm. The procedure is following.
i.

Reverberation separation
Two channel signals ‫ݏ‬ଵ ሺ‫ݐ‬ሻ and ‫ݏ‬ଶ ሺ‫ݐ‬ሻ are separated into direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ݐ‬ሻ andreverberation ‫ୖݔ‬ୣ୴ ሺ‫ݐ‬ሻ in the same manner as conventional up-mix algorithm.

ii.

Select loudspeaker
This algorithm determines the loudspeakers for reproducing direct sound or reverberation basedon added 3-D location of the sound source. The 3-D localization ൫‫ݎ‬ǡ ߠ௙ ǡ ߶௙ ൯ is acquired
automatically or manually from media such as video; ‫ ݎ‬is the distance from listening point toloudspeaker, ߠ௙ is the azimuth from listening point to sound source location at ݂ frame, ߶௙ isthe elevation from listening point to sound source location at ݂ frame. Direct sound selects threeloudspeakers close in distance to the sound source location at ݂ frame. The distance ݀௜ between
sound source location and loudspeaker is calculated as follows:݀௜ ൌ ටʹ‫ ݎ‬ଶ െ ʹ‫ ݎ‬ଶ ൫߶௜ᇱ െ ߶௙ ൯  ߠ௙  ߠ௜ᇱ ൅  ߠ௙  ߠ௜ᇱ ǡ(7)

where ߠ௜ᇱ is the azimuth from listening point to loudspeaker at ݂ frame, ߶௜ᇱ is the elevation fromlistening point to loudspeaker at ݂ frame. The exception is that the all selected loudspeakers
should not be in the same layer.
iii.

Frame division (direct sound)
The direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ݐ‬ሻ is divided into ݂ frame after loudspeaker selection in Step ⅱ.
Divided direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ is localized in 3-D sound field by Vector Base Amplitude
Panning (VBAP) [5] in Step ⅳ.

iv.

VBAP (direct sound)
VBAP is one of the 3-D amplitude panning method by vector synthesis, this method realizes
any 3-D sound localizations. Fig.6 shows that VBAP divides the reproduction space into triangle
consisting of three loudspeakers and pans amplitude based on the calculated weighting factors ܏ ൌ்൫݃୔୰୭୮ǡଵ ݃୔୰୭୮ǡଶ ݃୔୰୭୮ǡଷ ൯ . The weighting factors ܏ are calculated as follows:(8)܏ ൌ ‫ିۺ‬ଵ ‫ܘ‬ǡwhere ‫ ܘ‬is unit vector from the listening point to virtual sound image, ‫ ۺ‬ൌ ሺ࢒૚ ࢒૛ ࢒૜ ሻ is matrix ofunit vectors ࢒૚ ǡ ࢒૛ ǡ ࢒૜ from the listening point to each loudspeaker. The weighting factors݃୔୰୭୮ǡଵ ǡ ݃୔୰୭୮ǡଶ ǡ ݃୔୰୭୮ǡଷ for ݂ frame are obtained by Eq. (8) calculated for each frame.

Figure 5 - The mapping method using a novel up-mix algorithm.

5264

The panned sounds ‫ݕ‬ୈ୧୰ୣୡ୲ǡ௜ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ are the product of the calculated weighting factors݃୔୰୭୮ǡଵ ǡ ݃୔୰୭୮ǡଶ ǡ ݃୔୰୭୮ǡଷ and the direct sound ‫ݔ‬ୈ୧୰ୣୡ୲ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ as follows:‫ݕ‬ୈ୧୰ୣୡ୲ǡ௜ ൫‫ݐ‬Ԣ ǡ ݂൯ ൌ ‫  ݔ‬൫‫ݐ‬Ԣ ǡ ݂൯݃୔୰୭୮ǡ௜ ሺ݂ሻǤ       ǤǤ ݅ ൌ ͳǡ ǥ ǡ͵

(9)

v.

Block shift method (reverberation)
Block shift method [6] can artificially generate many impulse responses from an original
impulse response. Generated impulse responses are similar to the original one and low correlation
between them. We can generate multiple reverberations by convoluting the impulse responses
generated by block shift method and the original reverberation. The flow of reverberation
generation using the block shift method is shown as follows.
First, original impulse response is divided into multiple ݊ blocks ‫ݎ‬௡ ሺ‫ݐ‬ሻ by time windows‫ݓ‬ሺ‫ݐ‬ሻ.‫ݎ‬௡ ሺ‫ݐ‬ሻ is generated as follows:‫ݎ‬௡ ሺ‫ݐ‬ሻ ൌ ‫ݓ‬൫‫ ݐ‬െ ሺ݊ െ ͳሻ൯ܶௗ ‫ݎ‬஺ ሺ‫ݐ‬ሻǡ
(10)
where ܶௗ is the interval between two adjacent time windows, ‫ݎ‬஺ ሺ‫ݐ‬ሻ is reverberation part in theoriginal impulse response.
Second, each divided block is randomly shifted and replaced by a new block to generate a newreverberation part ‫ݎ‬஻ ሺ‫ݐ‬ሻ.‫ݎ‬஻ ሺ‫ݐ‬ሻ is generated as follows:‫כ‬‫ݎ‬஻ ሺ‫ݐ‬ሻ ൌ σே(11)௡ୀଵ ‫ݎ‬௡ାே‫ כ‬ሺ௡ሻ ሺ‫ ݐ‬൅ ܰ ሺ݊ሻܶௗ ሻǡ‫ כ‬ሺ݊ሻ
where ܰ is the total number of ݊ blocks, ܰ
is a random number with an uniformேே
distribution ranging from െ to .
ଶଶFinally, we can obtain new 19 channel impulse responses ݄஻ǡ௝ ሺ‫ݐ‬ሻ by combining the direct
sound part with the reverberation part ‫ݎ‬஻ ሺ‫ݐ‬ሻ . New 19 channel reverberations ‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ݐ‬ሻ arecalculated as follows:‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ݐ‬ሻ ൌ ‫ ݔ‬ሺ‫ݐ‬ሻ ‫݄ כ‬஻ǡ௝ ሺ‫ݐ‬ሻǤ       ǤǤ ݆ ൌ ͳǡ ǥ ǡͳͻ
(12)

vi.

Frame division (reverberation)
The generated reverberations ‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ݐ‬ሻ are divided into ݂ frame. Divided reverberations‫ୖݕ‬ୣ୴ǡ௝ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ map 19 loudspeakers except for three loudspeakers what reproduce direct sound.

vii.

Low-pass filtering (bass)
The basses ‫୐ݕ‬୊୉ǡ௞ ሺ‫ ݐ‬ᇱ ǡ ݂ሻ for two woofers are generated by the low-pass filter in the samemanner as conventional up-mix algorithm.
22.2 multichannel signals generated by the above steps can reproduce a 3-D moving sound image.

Figure 6 - Vector Base Amplitude Panning.

5265

5. Evaluation experiment
We carried out the objective evaluation experiment to confirm the effectiveness of the proposed
method.
5.1 Conditions of objective evaluation experiment
Table 1 shows the experimental conditions. Table 2 shows the experimental equipment. In thisexperiment, we calculated Interaural Level Difference (ILD) to evaluate the localization direction of
the 3-D moving sound image. This objective experiment calculates the ILD of three moving soundsources; measured sound source, evaluation sound source reproduced by conventional up-mix
algorithm, evaluation sound source reproduced by proposed up-mix algorithm. Furthermore, we
identify localization error by calculating the average of ILD errors ‫୐୍ܧ‬ୈ between measured soundsource and evaluation sound source. ‫୐୍ܧ‬ୈ is calculated as follows:
‫ܨ‬െͳ

ͳ
ʹ
‫୐୍ܧ‬ୈ ൌ ඩ ෍൫‫ ܫ‬ሺ݂ሻ െ ‫ ܫ‬ሺ݂ሻ൯ ǡ
‫ܨ‬

(13)

݂ൌͲ

݂‫ܯ‬൅‫ܯ‬െͳσԢหܴ൫‫ݐ‬Ԣ ǡ ݂൯ห‫ ݐ‬ൌ݂‫ܯ‬‫ܫ‬୉୴ୟ ሺ݂ሻ ൌ ቌ ݂‫ܯ‬൅‫ܯ‬െͳቍ ǡσԢห‫ܮ‬൫‫ݐ‬Ԣ ǡ ݂൯ห(14)‫ ݐ‬ൌ݂‫ܯ‬where ‫ ܨ‬is the total number of ݂ frames, ‫ ܯ‬is frame width, ‫ ݐ‬ᇱ is the time index in ݂ frame,‫ܫ‬୉୴ୟ ሺ݂ሻ is the ILD of each evaluation sound source, ‫ୖܫ‬ୣୟ୪ ሺ݂ሻ is the ILD of each measured soundsource, ‫ܮ‬ሺ‫ ݐ‬ᇱ ǡ ݂ሻ and ܴሺ‫ ݐ‬ᇱ ǡ ݂ሻ are received signals recorded by dummy head microphone.
We carried out the experiment using six 3-D movement patterns shown in Table 2.
Table 1 - Experimental conditions in objective evaluation.
Environment
Experiment room

Ambient noise level୅ ൌ 37.0 dB

Reverberation time଺଴ ൌ 300 ms

Sampling frequency

48 kHzSound source

White noise (3 s)ձ from front right to front leftղ from back left to back right3-D movement patterns

ճ from front up to front downմ from back down to back upյ from front left to back leftն from front right to back right

Table 2 - Experimental equipment in objective evaluation.
Dummy Head Microphone
NEUMANN, KU 100

5266

Loudspeaker

YAMAHA, VXS5

Woofer

YAMAHA, VXS10S

Microphone amplifier

THINKNET, MA-2016C

Loudspeaker amplifier

YAMAHA, XMV 8280

A/D, D/A converter

RME, MADIface USB

Figure 7 - The average of ILD errors.

5.2 Experimental results in objective evaluation
Fig.7 shows the experimental results. Horizontal axis represents evaluation sound source, and left
vertical axis represents the average of ILD errors ‫୐୍ܧ‬ୈ between measured sound source andevaluation sound source. From Fig.8, we confirmed that the average of ILD errors decreased in all
sound sources. This result suggests that the 22.2 multichannel signals generated by novel up-mix
algorithm can reproduce a 3-D moving sound image similar to a real sound source.

6. Conclusions
In this paper, we proposed a novel up-mix algorithm which extends two channel signals to 22.2
multichannel signals while using the 3-D location of the sound source. We confirmed the effectivenessof the proposed method from an evaluation experiment with interaural level distance. As future works,
we challenge the reproduction of approach sound based on the control of localization distance using
a novel up-mix algorithm.

ACKNOWLEDGEMENTS
This work was partly supported by JST COI and JSPS KAKENHI Grant Numbers JP18K19829 andJP19H04142.

REFERENCES
1. K. Hamasaki, T. Nishiguchi, R. Okumura, Y. Nakayama and A. Ando, "A 22.2 Multichannel Sound
System for Ultrahigh-Definition TV (UHDTV)," in SMPTE Motion Imaging Journal, vol. 117, no. 3,
pp. 40-49, April 2008.
2. Recommendation ITU-R, "Performance requirements for an advanced multichannel stereophonic soundsystem for use with or without accompanying picture," BS. 775, 2012.
3. K. Kinoshita, T. Nakatani and M. Miyoshi, "Blind upmix of stereo music signals using multi-step linear
prediction based reverberation extraction," IEEE International Conference on Acoustics, Speech andSignal Processing, Dallas, TX, USA, 2010, pp. 49-52.
4. K. Kinoshita, M. Delcroix, T. Nakatani and M. Miyoshi, "Suppression of late reverberation effect on
speech signal using long-term multiple-step linear prediction," IEEE Transactions on Audio, Speech,
and Language Processing, vol. 17, no. 4, pp. 534-545, May 2009.
5. V. Pulkki, "Uniform spreading of amplitude panned virtual sources, "Proceedings of the 1999 IEEE
Workshop on Applications of Signal Processing to Audio and Acoustics," New Paltz, NY, USA, 1999,
pp. 187-190.
6. C. Mori, T. Nishiguchi and K. Ono, "Measurement and generation method of many impulse responses
for 22.2 multichannel sound production," 2016 IEEE International Symposium on Broadband
Multimedia Systems and Broadcasting (BMSB), Nara, 2016, pp. 1-4.

5267