2010.9.25 IEEE IC-NIDC 2010@Beijing, China

Implementation of a high-definition
3D audio-visual display based on Higher-order
Ambisonics using a 157-loudspeaker arraycombined with a 3D projection display

T. Okamoto1,2), Z. L. Cui1), Y. Iwaya1,3) and Y. Suzuki1,3)
岡本拓磨

崔正烈

岩谷幸雄

鈴木陽一

1) Research Institute of Electrical Communication, Tohoku University, Japan
2) Graduate School of Engineering, Tohoku University, Japan
3) Graduate School of Information Sciences, Tohoku University, Japan東北大学
1) 電気通信研究所，2) 大学院工学研究科，3) 大学院情報科学研究科

1Introduction and research aim
Implementation of 3D audio visual display
Realizing communications with high sense-of-presence in digital content
not only actual environment but also virtual and mixed environment

Clarifying human audio-visual and various multimodal perceptions
including spatial information

Today’s topics
Implementation of 3D sound field reproduction system

WFS or BoSC

using 157-loudspeaker arraysound field reproduction based on Higher-Order Ambisonics (HOA)• highly sound field reproduction around a center of arraythan Wave Field Synthesis (WFS) and Boundary Surface Control (BoSC)
Realization of highly synchronous all 157-audio playback system

Implementation 3D audio-visual display system

Ambisonics

Combining HOA reproduction system with 3D projection display
Relationship between
System synchronous measurement between audio and visual signals reproduction accuracy
and frequency

2

Our previous 3D sound field reproduction
system based on HOA
Sound field reproduction system using 157-loudspeakerarray based on HOA
5th order decoding, which is highest order in the world, was realized
Previous system was implemented by 4 PCs
Audio playback asynchronous between each PC was up to 51 samples (1.1 ms)

http://puredata.info

T. Okamoto et al. Proc. Ambisonics’ 10 20103Implementation of highly synchronous
all 157-audio playback system
Completely synchronous all 157-audio playback systemusing 3 MADI system (audio sampling frequency: 48 kHz)All 157 audio signals are controlled by 1 PC
Introduction of Multichannel Audio Digital Interface (MADI) system
A MADI system can control 64 audio input/output with completely synchronous
Using CoreAudio, which is MacOS X
audio driver, for recognizing 3 MADI
system as one unified audio interface
Moreover, another audio driver, JackPilot, is introduced for distributing
audio in-out processes to cores of CPUs

Evaluation of interchannel
synchrony using TSP (Time Stretched Pulse)Jack Pilot off• 1 to 3 samples of asynchronous
between each MADIJack Pilot on• Completely synchronous all 157channels

4Proposed 3D audio-visual display system
Combining HOA system with 3D projection display
Audio system: Completely synchronous 157-loudspeaker array system
Visual system: Stereo shutter technique with acoustic transparent screen
For integrating the audio system and the visual system, the trigger signal wasintroduced

5

Arrangements
Arrangements of loudspeakers, projector and screen
The sweet spot of for sound field
reproduction by HOA decoding is
the center of the array
The viewer's position for proper
stereovision was set identical tothe acoustical sweet spot

6

System synchronous measurement
between audio and visual signals
System synchronous measurement using luminance signal
and TSP signals
ResultUnsigned maximum
asynchronous between thedrawing on the screen and 157audio stream was 51 samples(= 1.1 ms)
This value is much shorter thanthe detection threshold ofaudio-visual asynchrony byhuman observers

7Concluding remarks
Implementation of 3D audio-visual display
Realizing sound field reproduction system by 5th order Ambisonics using
157-loudspeaker array
Realizing completely synchronous 157 audio playback system introducing
3 MADI system and two audio drivers, CoreAudio and Jack Pilot
Implementation 3D audio-visual display system combining HOA systemwith 3D projection system
System synchronous between audio and visual signals was 1.1 ms and is much shorter
than the detection threshold of human perception

Realizing not only actual environment but also virtualenvironment 3D audio-visual information reproduction
Future works
Evaluation of the accuracy of reproduced sound field by proposed system
Experiments for clarifying human audio-visual and various multimodal
perceptions
Control audio and visual information including spatial information

8Concept of HOA
Simple example of orthogonal expansion
Taylor expansion
f (x) =∞
�f (n) (a)

n=0

n!

f � (a)f �� (a)f (n) (a)2(x − a) = f (a) +(x − a) +(x − a) + · · · +(x − a)n + · · ·
1!
2!
n!
n

Orthogonal expansion of sound field information based on
spherical harmonics in HOA
p(kr, θ, φ) =+∞�m=0im jm (kr)m
�n n
BmYm (θ, φ)n=−mnSound information is decomposed to expansion coefficients Bm
usingspherical harmonics Ymn (θ, φ) of each direction (θ, φ)Using more higher order m , more accurate sound field is reproduced

0 0
�

P(cosϕ)ifn=0,N√m m(2m + 1) (m − |n|)!
nn n
Ymn (θ, ϕ) =2Nm Pm (cos ϕ) cos nθ if n > 0, Nm =√4π(m + |n|)!
n −n
2Nm Pm (cos ϕ) sin nθ if n < 0,9Sound field recording and reproduction
based on HOA system
Aspects of HOA system
Using more higher order m , moreaccurate sound field is reproduced
2More than (m + 1) channels areneeded to encode or decode order mRecording system and reproduction
system are independent each othernBmHOAm

Ambisonics

01234-4-3-2-10

1

2

34n

10

Implementation of sound field recording
system based on HOA
T. Okamoto et al., Proc. SOIM-GCOE 2010 (in print)

HOA recording system using a 121 spherical microphone array
9th order decoding, which is highest order in the world, was realized
Completely synchronous 121-audio recording system using Digital Electric
Condenser Microphone and FPGA board

Realizing actual sound field recording andreproduction system

11