PROCEEDINGS of the
23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

Audio-visual mutual effect of spatial impression in architectural
space
Chiharu HIRAOKA1; Takane TERASHIMA2; Ayumi ISHIKAWA3; Yasunobu TOKUNAGA4;
Xue GONG5;
1,2,5
3
4

Mie University, Japan

National Institute of Technology, Gifu College, Japan

National Institute of Technology, Maizuru College, Japan

ABSTRACTThe purpose of this study is to clarify the influence of visual information on subjective impression forroom sound field. Authors have carried out subjective experiments on it using room impulse response
signal as auditory stimulus and interior panorama picture as visual stimulus, and found thatreverberance/clarity for room sound field is affected by information sensible from room volume,
dimensions, geometry or finishing etc. And then in this report, experiments about visual effect on spatial
impression (ASW and LEV) in auditorium were conducted to clarify the influence of room volume, size or
geometry on them. Auditory stimuli were made in anechoic chamber by binaural microphone recording
under the conditions that directions of early reflections and late reverberation were varied, and were
presented to subjects through headphone. Visual stimuli were made of panorama pictures captured aroundroom interior and were presented by Head Mounted Display (HMD). ASW and LEV were measured undersingle-mode or multi-mode presentation of visual /auditory stimuli, and mutual effect of visual and auditory
sense were examined in architectural space.
Keywords: spatial impression, ASW, LEV, audio-visual multimodal effect

1. INTRODUCTION
It is well known that auditory impression for room acoustics is influenced by visual information and wehave been studying on audio-visual mutual effect in architectural space, by interior and room acoustics. In
previous paper, subjective experiments about the visual effect on Reverberance with interior panorama
pictures and impulse response signal as audio and visual stimuli were carried out [1,2]. Then existence ofvisual effects on Reverberance and conditions of manifestation were clarified. And in this paper, next
subjective experiments were carried out to clarify the influence of visual information on "spatial
impression" experienced in music spaces. “Spatial impression” is one of the most important experience inmusic space like concert hall. Auditory phenomena in terms of spatial impression are classified into three
types: Auditory Source Width (ASW), Listener Envelopment (LEV), and the size of the room that forms
the sound field [3]. In this report, subjective evaluation experiments were performed in regard to ASW and
LEV using interior panorama images and virtual sound fields of architectural spaces, then the influence ofvisual information was analyzed.

2. SPACIOUSNESSASW is defined as "the width of a sound image fused temporally and spatially with the direct soundimage". ASW is influenced by sound pressure level, frequency and arrival direction of earlyreflections, then interaural cross-correlation coefficient (IACC) or early lateral energy fraction (LF)1chiharu@e.arch.mie-u.ac.jp
tera@arch.mie-u.ac.jp
3
ishikawa@gifu-nct.ac.jp
4
tokunaga@maizuru-ct.ac.jp
5gongxue@e.arch.mie-u.ac.jp
24562

is an objective index of ASW. It is considered that ASW becomes larger as IACC becomes smaller or
as LF becomes larger [4]. The equation of LF is defined as follows [5] .
/./1∫/.//2 %& (() *+, - .(
𝐿𝐹 =
(1)
/./1
∫/ %& (().((p(t): sound pressure at receiver position, t: time after direct sound arrival, f: azimuth of reflection incidence)

LEV is defined as " the degree of fullness of sound images around the listener, excluding a sound image
composing ASW". Morimoto et al. reported that LEV is influenced by arrival direction of late reflections
and their sound pressure level, then defined front/back energy ratio (FBR) as an index shown in thefollowing equation [6].
3FBR=10log 4
[dB]
(2)
35

(Ef : energy of reflections arriving from front, Eb : energy of reflections arriving from back)

3. OUTLINE OF EXPERIENTIn this report, experiments on ASW and LEV were conducted as different series, and further two steps (Iand II) are included respectively. At step I, stimuli were evaluated subjectively by paired comparison whileat step II, by rating scale method. We call these 2 series x 2 parts of experiments "Ex. ASWI", "Ex. ASWII","Ex.LEVI" and "Ex.LEVII".
3.1

Experimental stimulus

Figure 1. Interior of space A
Figure 2. Interior of space B
Table 1. Properties of spaces targeted in experiments
Space

Use

Volume［m8 ］

Seat Capacity

T30 [s]C80 [dB]A

university auditorium (multi use）

12000

1650

1.4

1.0

B

town hall (multi use)

5200

604

1.8

-1.1

Figure 1 and 2 show the interior images of architectural spaces targeted in these experiments and theirproperties are shown in Table 1. The panorama VR images of these spaces used as visual stimuli werecreated by composing 3 photographs captured from 3 directions of the real space through fisheye lens.
The method of creating auditory stimulus is as follows. The impulse response measured in real space wasdecomposed into 3 parts, direct, early reflected and reverberant sound. Then these 3 parts of sound werereproduced from different loudspeakers installed in the anechoic chamber. The location of loudspeakers
radiating early reflections or reverberant sound was altered so that IACC/LF/FBR of the sound field couldchange. Reproduced impulse response signals were binaurally recorded as new ones i.e. auditory stimuli
which are different in reflected sound structure and used in experiment.
Table 2. Name and IACC/LF value of auditory stimulus (Ex.ASW)
Azimuth
[°]

0

30

60

75

90

Space

Name

IACC

LF

Name

IACC

LF

Name

IACC

LF

Name

IACC

LF

Name

IACC

LF

A

a0

0.94

0

a30

0.60

0.38

a60

0.57

0.69

a75

0.55

0.73

a90

0.50

0.63

B

b0

0.93

0

b30

0.73

0.41

b60

0.66

0.69

b75

0.67

0.80

b90

0.68

0.78

4563

Table 3. Name and FBR value of auditory stimulus (Ex.LEV)
Azimuth
[°]

0

60

90

120

150

Space

Name

FBR

Name

FBR

Name

FBR

Name

FBR

Name

FBR

A

c0

67.2

c60

67.2

c90

63.2

c120

-1.8

c150

-1.8

B

d0

69.5

d60

69.5

d90

66.7

d120

0.4

d150

0.4

Figure 3. Loudspeaker arrangement increating auditory stimulus (Ex.ASW)

Figure 4. Loudspeaker arrangement increating auditory stimulus (Ex.LEV)For Ex.ASW, as the arrangement of loudspeaker is shown in Figure 3, the direct sound (arriving from 0 to
10 ms) and reverberation (after 80 ms) was reproduced from loudspeaker 1, the first early reflection sound(10 ~ 45 ms) from loudspeaker 2, and the second early reflection (45 ~ 80ms) from loudspeaker 3 basically.
Loudspeaker 1 was fixed in front of recorder, while loudspeaker 2 and 3 were placed laterally andsymmetrically in the same radius. The height of loudspeakers was adjusted to the position of the recorder'sear. The azimuth of loudspeaker 2 and 3 were set to one of 30°, 60°, 75° and 90°. In the case of earlyreflections from 0° direction, all the signals were reproduced through loudspeaker 1. The name and
IACC/LF values of auditory stimuli used in Ex.ASW were shown in Table 2.
The arrangement of loudspeakers in creating auditory stimulus for Ex.LEV shown in Figure 4. Direct
sound and early reflections were reproduced through 3 fixed frontal loudspeakers (direct sound: azimuth 0°,early reflections: azimuth 30°), while late reverberation was reproduced through loudspeakers moving fromside to back, and the direction azimuth of late reverberation arrival to the recorder was altered to one of
0°,60°, 90°, 120° and 150° (reverberation was reproduced through fixed frontal loudspeaker in case of 0°).
Actually, in experiment, auditory stimulus was provided to subject in the form of convoluted music withthe recorded impulse response. Dry source of music is "Water Music" composed by Handel [7].
3.2 Experimental method
The number of subjects were 10 or 11 by experiment. In both experiments, the subject wears ahead-mounted display (HMD) and headphones standing upright in the laboratory. Both experiments
included two steps, step I (paired comparison) and step II (rating scale method). In paired comparison, a
subject was asked to answer orally which is larger in ASW or LEV out of stimuli pair. In rating scale
method, a subject was asked to answer orally the number of 7 step scale category in ASW or LEV for the
stimulus. Stimuli were presented to subjects in 2 modes, auditory stimulus only (single-mode) andcombination of audio-visual stimuli (multi-mode).
The visual stimuli are panorama images of space A and B. The auditory stimuli used in Ex.ASWI were a0,a30, a90, b0, b30 and b90, while all the ax and bx were used in Ex.ASWII. In Ex.LEVI auditory stimuli of c0, c90,
c150, d0, d90 and d150 , while all the cx and dx were used in Ex.LEVII.

4564

Experiment

Table 4. Used stimuli and number of trials
Single-mode
Multi-mode
StimuliNum.
Stimulia0, a30, a90,
A-a0, A-a30, A-a90,
6b0, b30 b90
B-b0, B-b30, B-b90
a0, a30, a60, a75, a90,
A-a0, A-a30, A-a60, A-a75, A-a90,
10b0, b30, b60, b75, b90
B-b0, B-b30, B-b60, B-b75, B-b90
c0, c90, c150,
A-c0, A-c90, A-c150,
6d0, d90, d150
B-d0, B-d90, B-d150
c0, c60, c90, c120, c150,
A-c0, A-c60, A-c90, A-c120, A-c150,
10d0, d60, d90, d120, d150
B-d0, B-d60, B-d90, B-d12, B-d150

Num.

Ex.ASWI(paired comparison)

Ex.ASWII(rating scale method)

Ex.LEVI(paired comparison)

Ex.LEVII(rating scale method)

6
10
6
10

Total Num.
6+6C2 =66

10+10=20
6+6C2 =66

10+10=20

In multi-mode stimulation, these auditory and visual stimuli are randomly combined and simultaneously
presented to subjects. The numbers of trials are summarized in Table 4.
Table 5. Subjective evaluation item (Ex.ASWⅡ)

Table 6. Subjective evaluation item (Ex.LEVⅡ)

Apparent Source Width
Answer0

Meaning

Small

1

2

3

4

Envelopment
5<------------------->

6

Answer0

Large

Meaning

Small

1

2

3

4

5

<------------------->

6
Large

In rating scale method (step 2 of each experiment), the subjective evaluation items are shown in Table 5
and 6. In Ex.LEVII, subjects were asked to answer the direction of arrival of the reverberant soundadditionally.

4. EXPERIENTAL RESULT

Relative distance of ASW in multi-mode

4.1

Result of Ex.ASW

Figure 5. Relative distances of ASW in Ex.ASWIThe relative distances between the stimuli are
1.0
calculated from the evaluation values measured in
0.8
Ex.ASWI on the assumption of Thurstone’s case V.
The evaluation for all auditory stimuli in both
0.6
space and both modes are shown on a number line
0.4
as relative distances in Figure 5. In this figure,
ASW for b0 in single-mode is the smallest, and
0.2
ASW for b90 in multi-mode is the largest. In Figure
0.06, the points corresponding to stimuli are plotted-0.2
with single-mode value on the horizontal axis andmulti-mode value on the vertical axis. In this
-0.4
figure, all the points for space B are located above-0.6
the diagonal line, which means that ASW inA
B
multi-mode is higher than single-mode. On the-0.8
other hand, the points for space A are located-1.0
almost in parallel with the diagonal line, which
-1.0
-0.5
0.0
0.5
1.0
means that ASW in single-mode and multi-mode
Relative distance of ASW in single-mode
are evaluated to the same extent.
Figure 6. Relationship of relative distancesof single- and multi-mode
4565

1.0

Relative distance of ASW

Relative distance of ASW

1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0

A (multi)

A (single)

B (multi)

B (single)

0.4

0.6

IACC

0.8

0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
1.0 0.0

A (multi)
B (multi)
0.2

0.4

LF

A (single)
B (single)

0.6

0.8

1.0

Figure 7. Relationship of ASW and IACC
Figure 8. Relationship of ASW and LF
Figure 7 shows the relationship between the relative distance of ASW and IACC for all stimuli, andFigure 8 shows the relationship with LF. The relative distance of ASW decreases as IACC increases
almost linearly, while it increases slightly logarithmically with LF. The ASW for a30 was larger than a90
which is expected to be larger than a 30 from IACC or LF. The reason why the ASW of a 30 was higher
than other stimuli cannot specified.
6

A (multi)
A (single)

evaluation average of ASW

Average of ASW

6

80

100

5
4
3
2
1
0
-20

0

20

40

60

5
4
3
2
1

B (multi)
B (single)

0
-20

0

20

40

60

80

Azimuth of early reflection arrival [°]Azimuth of early reflection arrival [°]Figure 9. Evaluation average of ASW
and standard deviation (A)Figure 10. Evaluation average of
ASW and standard deviation (B)100

The results of Ex.ASWII are shown in Figure 9 and 10. Figure 9 shows the averages and standard
deviations of ASW for auditory stimuli in space A, and Figure 10 shows them in space B with the arrival
direction of the early reflections. Unlike the results of Ex.ASWI, the fluctuation of the average ASW bystimuli is small overall. In Ex.ASWI, it was easy for the subjects to judge because of relative comparison
between the stimuli, whereas in Ex.ASWII it might be a little hard for subjects to judge the absoluteevaluation for a stimulus without reference, then variance of ASW came to be large, and large standard
deviation in these figures.
Figure 9 shows that ASW for space A tends to draw a mountain shape which has maximum at 60° insingle-mode and 75° in multi-mode. There is statistical significance in the change of ASW with respect tothe azimuth of early reflection arrival by ANOVA. Except for 30° and 60°, the average ASW in multi-mode
exceeded those in single-mode, but there was no significant difference between the two average ASW at
any arrival direction.
In Figure 10, ASW for space B tends to increase as the azimuth of early reflection arrival in single-mode,
but in multi-mode, it is almost constant regardless of the arrival direction. The average ASW in multi-mode
were higher than that in single-mode for all arrival directions except 90°. However, there was no significant
difference between the two average values except at the azimuth 0°. Furthermore, in both modes, there wasno significance in the change of average ASW with the arrival direction by ANOVA.

4566

Difference of average ASW between modes

1.5
1.0
0.5
0.0
-0.5
-1.0

A

B

-1.5
0

20

40

60

80

Figure 11 shows the change of ASW
difference between in multi-mode andsingle-mode with arrival direction ofearly reflection. Most of the valuesplotted in the figure are located above
0, indicating that the evaluation in
multi-mode is larger than that insingle-mode on the whole. In addition,in terms of difference from 0,
statistical significance is detected atazimuth 0° and 30° for space A and at
0° for space B. And the ASW
difference for space B between modes
tends to decrease as the azimuth
increases.

Aziuth of reflection arrival[°]Figure 11. Difference of average ASW between multi- and single-mode

4.2 Results of Ex.LEV

1.0

1.0

0.8

0.8

0.6

0.6

0.4
0.2
0.0
-0.2
-0.4
-0.6

A(multi)B(multi)

-0.8
-1.0
0

30

60

90

relative distance of LEV

Relative distance of LEV

Figure 12. Relative distances of LEV in Ex.LEVI

0.4
0.2
0.0
-0.2
-0.4

-0.6
A(single)-0.8
B(single)-1.0
120
150 -20

Azimuth of reverberation arrival [°]Figure 13. Relationship between relative distanceof LEV and azimuth of reverberation arrival

A (multi)
B (multi)
0

20

40

A (single)
B (single)
60

80

FBR[dB]
Figure 14. Relationship between relative
distance of LEV and FBR

Similar to Ex.ASWI, the relative distances of LEV between the stimuli are calculated from the evaluation
values measured by the experiment Ex.LEVI on the assumption of Thurstone’s case V, and are shown in
Figure 12. In this figure, the LEV for d150 in multi-mode is maximum, and minimum for d0 in single-mode.
Also, there are significant differences between average LEV in multi-mode and single-mode at d0 and d150.
The distribution of LEV is larger for space B than space A.
The change of relative distance with azimuth of reverberation arrival is shown in Figure 13. And Figure
14 shows LEV change with FBR. LEV for space A is maximum at c90 in both multi- and single-mode,
whereas LEV for space B is maximum at d150, i.e. it becomes larger with the azimuth of reverberation
arrival in both modes. The cause of the former is considered that contrary to the setting of the experiment,4567

the listening level of the auditory stimulus c90 was larger than other stimuli. In Ex.LEVII, this point wascorrected and the listening level of auditory stimuli was confirmed to be equal again. For every auditory
stimulus, LEV in multi-mode is larger than single-mode for both space A and B.
Figure 15 shows the LEV change with azimuth ofreverberation arrival using the rating scale method
5in Ex.LEVII. In both spaces A and B, the LEV ismaximum when the azimuth of reverberation4
arrival is 60° in multi-mode; LEV for the auditory
stimulus in which reverberation comes from slight
front tends to increase in multi-mode. On the other3
hand, in single-mode, LEV for the auditory stimuli
with azimuth of 120° is the largest for space B;
2
LEV for the auditory stimuli in which reverberation
comes from side or a little back tends to increase.
A (multi)
A (single)
1
For space A, there is little fluctuation of LEV with
B (multi)
B (single)
azimuth.
0
Only a significant difference of LEV for space A
0
30
60
90
120
150
between multi- and single-mode was detected with
Azimuth of reverberation arrival[°]respect to the auditory stimulus c60 whose azimuth
Figure 15. Relationship between averageof reverberation arrival is 60°. We suppose that
LEV and azimuth of reverberation arrival
azimuth 60° could happen to be the condition atwhich audiovisual audio-visual integration occurred, and the expected LEV from the visual stimulus might
be close to the perceived LEV from the auditory stimulus. We also suppose that since the expected LEV by
the visual stimulus was larger than the auditory perceived LEV, the effect of bringing the latter closer to theformer could increase perceived LEV at the condition.
On the other hand, for the auditory stimulus d120 for space B, we suppose that there also audiovisual
integration could occur, and the spread of expected LEV from the visual stimulus might be smaller than theauditory perceived LEV, then the effect of bringing the latter closer to the former could decrease perceived
LEV.
6

6

5

5

4
3
2

A (multi,group1)
A (single,group1)
A (multi,group2)
A (single, group2)

1
0
0

30

60

90

120

evaluation of LEV

average LEV

Average LEV

6

4
3
2

B (multi,group1)
B (single,group1)
B (multi,group2)
B (single,group2)

1
0

150

0

30

60

90

120

150

Aziuth of reverberation arrival[°]Aziuth of reverberation arrival[°]Figure 17. Comparison of results for two
Figure 16. Comparison of results for twogroups (space B)
groups (space A)
Total results of Ex.LEVII seems to be unclear and the cause could be considered individuality of subjects.
Then cluster analysis was carried out, and 10 subjects were classified into two groups of eight (group1) andthree (group2). Figure 16 shows the comparison of LEV change among groups for scape A. For group1,
LEV changes largely according to the azimuth of reverberation arrival and have a peak value at d120, while
LEV change for group2 is flat, but only LEV at c60 is larger than others in multi-mode. In both groups, there
is no significant difference between modes except at c60 for group1. Figure 17 shows the comparison of
LEV change among groups for scape B. For group1, LEV changes largely according to the azimuth ofreverberation and have a peak value at d60 in multi-mode, at d90 in single-mode. LEV change for group2 isflat and there is no significant difference between modes for group2.

4568

0.9

In Ex.LEVII, the subjects were also asked toanswer the arrival direction of the perceived
reverberation for each stimulus. Figure 18 shows
0.7
the results of the selection rate of the arrival
direction of reverberation (front or back)
0.6
perceived by the subject. In both spaces, at 0° and
0.5
150° of the arrival directions of reverberation,
0.4
there were differences in selection rate between
multi- and single-mode. In multi-mode, auditory
0.3
stimuli whose reverberation comes from 0° are
0.2
perceived as coming from the front (correctly) byA(multi)A(single)most of subjects, whereas in single-mode they are
0.1
perceived by almost a half of subjects as coming
B (multi)
B (single)
0
from the rear (mistakenly). At 150° almost the
0
30
60
90
120
150
same was seen, and the direction of reverberation
Azimuth of reverberation arrival[°]arrival tends to be correctly perceived in
Figure 18. Selection rate of reverberation
multi-mode compared with single-mode.
arrival direction (Front)
The reason why almost a half of subjects
perceived mistakenly in single-mode is supposed as follows. Since the auditory stimulus was created bybinaural recording and presented to subjects through headphone, some subjects have different soundimages from others due to the difference in the shape of outer ear in single-mode, but in the multi-mode,the error of the sound image position could be revised correctly by visual information. In order to avoid
such problems of sound image localization, we plan to try to present auditory stimuli by loudspeakers infuture experiments.

Selection rate

0.8

5. SUMMARY・ In this paper, ASW was generally evaluated higher in multi-mode presentation than single-mode, andthe influence of visual information on auditory spatial impression was recognized.
・ Also, LEV has the tendency to be evaluated higher by the addition of visual information, and theinfluence of visual information on LEV was recognized.
・ By adding visual information, the judgment of the arrival direction of reflections and reverberation
could change and then spatial impression could change.
・ It is supposed that when the spatial impression expected from the visual stimuli is approximated tothat perceived from the auditory stimulus, audio-visual integration could occur, and the spatial
impression could shift closer to the visual impression.
・ In future experiments, it is necessary to review the method of subjective evaluation andcreating/presenting auditory stimuli.

REFERENCES
1. A. Ishikawa and T. Terashima: Study on the effect of sense of incongruity caused by combination withvisual information in subjective reverberance for indoor sound field in architectural space, Proc. of theFifth International Conference on Human-Environment System (ICHES2016 Nagoya), 20030, p.10
(2016)
2. A. Ishikawa, T. Terashima and Y. Tokunaga: Effect of Visual Stimulus on Subjective Impression ofIndoor Sound Fields with Various Reverberation Times, Proceedings of internoise2014, the 43th
International Congress on Noise Control Engineering, 133 (2014)
3. M.Morimoto, H. Fujimori and Z. Maekawa: Discrimination between auditory source width andenvelopment，Journal of the Acoustical Society of Japan (J), pp.449-457, Vol.46, No.6 (1990)
4. P. Damaske: Direction dependence of the spectrum and the correlation function of the signals received at
the ears, ACUSTICA, Vol.22, pp.191-204 (1969/70)
5. M. Barron and A.H. Marshall: Spatial impression due to early lateral reflections in concert halls：thederivation of a physical measure，Journal of Sound and Vibration, Vol.77, pp.211-232 (1981)
6. M. Morimoto, K. Iida and K. Sakagami: The role of reflections from behind the listener in spatial
impression, Applied Acoustics, Vol.62, pp109-124 (2001)
7. ANECHOIC ORCHESTRAL MUSIC RECORDING, 1993, DENON

4569