A21:
VIRTUAL ACOUSTICS

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0343

Perception and Emotion Evocation by Rock Art Soundscapes of
Altai (Russia)
Samantha LÓPEZ-MOCHALES1,2, Raquel APARICIO-TERRÉS1,2, Margarita DÍAZ-ANDREU3,4,5,Carles ESCERA1,2,4,6
1

Brainlab – Cognitive Neuroscience Research Group, Department of Clinical Psychology and Psychobiology,
Faculty of Psychology, University of Barcelona, Barcelona, Spain
2

Institute of Neurosciences, University of Barcelona, Barcelona, Spain

Departament d’Història i Arqueologia, Universitat de Barcelona, Barcelona, Spain

34

Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain

Institut d’Arqueologia de la Universitat de Barcelona (IAUB), Barcelona, Spain

56

Institut de Recerca Sant Joan de Déu (IRSJD), Esplugues de Llobregat, Spain

ABSTRACTThe archaeoacoustics of rock-art studies how past communities chose certain acoustic environments to set
activities involving the production of paintings and carvings. Although sound is ephemeral, acoustic
environments persevere over time. Acoustic and perceptual sciences can study the sensory potential ofsoundscapes through experimentation on modern-day participants.
This study sought to explore whether archeological rock-art sites have particular acoustics that would elicit
distinct emotional reactions. Sixty participants were presented with a series of natural sounds convolved with
the impulse responses from six positions in Altai (Russia), three of them were in front of rock-art panels, andthree of them not. Participants were interrogated about the feelings exalted by the sounds they heard, using
ten psychoacoustical and emotional scales.
All the descriptor ratings were significantly affected by the sound that was listened. Also, participants rated
all sounds as significantly more pleasant with the acoustics from not in front of the panels. The presence,closeness, alert, peace and calm evoked by sounds were also significantly influenced by the location.
Keywords: psychoacoustics, archaeoacoustics, emotion, rock art

1. INTRODUCTION
Archaeoacoustics of rock art inquiries about how and why past communities chose certain acoustic
environments for the production of paintings and carvings (Díaz-Andreu et al., 2015). Acoustics affect both
listeners’ emotional reactions and sound perception (Västfjäll. 2012). By studying the emotions elicited by
the acoustics of archaeological sites of interest in modern-day participants, we aim to better understand theauditory experience endured by ancient communities (Kolar, 2013). This is possible through auralization, thepresentation of an anechoic sound convoluted with the impulse response of a space of interest to listeners(Vorländer, 2008). Listeners’ individual emotional reactions can be measured during the listening (Västfjäll
et al., 2002; Pätynen & Lokki, 2016) to ultimately extract conclusions about the selection process of the mostsuitable places for rock art production. The aim of the research presented here was to investigate, from a
psychoarchaeoacoustics point of view (Valenzuela et al., 2020), the acoustic environments of rock art sites
from the Republic of Altai, Federal District of Siberia (Russia).
There is a tight relation between rock art, sound and the landscape in several cultures around the globe(Díaz-Andreu et al., 2021; Díaz-Andreu & Mattioli, 2021). In the area of Altai, people living in the area
believe that rocks are inhabited by mythical beings (Tátar, 1996). There are no written sources indicating an
explicit relation with rock art production, but the similarity of the economic system of the period where theethnographic information was taken, as well as the widespread geographical area where similar beliefs arefound, makes us to assume that many of these beliefs may have its roots in the past.

By reproducing thecharacteristic rock art soundscapes of the region of Altai and studying its effects on auditory perception andemotion, we may better understand the auditory experience lived by the communities who participated in thesocial and religious events that took place in the rock art sites.
In the present work we recreated the soundscapes of six locations along the region of Altai, three ofthem close to rock art panels (art+ condition), and three away from rock art panels (art- condition),convolving its impulse responses with 17 sounds characteristic of the natural landscapes of Siberia. We
presented the sounds in a spatial audio format, 3rd order Ambisonics and interrogated participants about theirreactions during the listening. We expected to observe differences between conditions in all thepsychoacoustical and emotional dimensions studied. This would lead us to confirm the hypothesis that theperceived acoustic characteristics of a place played a role in the consideration of that space as sacred and inits selection for rock art production.

2. METHODS
Sixty human volunteers participated on this experiment. The study was approved by the Bioethics
Committee of the University of Barcelona according to the Code of Ethics of the World Medical Association(Declaration of Helsinki). All participants gave written informed consent.
Seventeen different sounds of 20 seconds duration, from natural sources including animals (an ibex, alynx, a leopard, a sheep, a bear, a cricket, an eagle, a crane, a pack of grey wolves and a plover) and othernatural phenomena (the wind, the rain, a bonfire, a snowstorm, a river, a waterfall and a thunderstorm). Fiveadditional sounds were used for the first five trials of each block, for participants to practice on the task, butthese data were not further analyzed.
Each sound was convolved with six impulse responses (IRs) recorded in different natural landscapesof the Republic of Altai, using the MIMO IR technique (Farina & Chiesi, 2016; Farina, 2020). Three of thesewere recorded in front of rock art panels (art+ condition, figure 1) and three in front of walls without presenceof rock art (art- condition). The objective was to isolate the effects of the variable presence of rock art. The
six resulting versions of the stimulus set were classified in three bundles (one, two and three), each including
one art+ and one art- version.

Figure 1. Rock art panles of Kalbak Tash 1 (A), Kalbak Tash 2, panel 2 (B), and Torgun I5 (C), in front of which the impulse
responses employed as art+ condition of the present study were recorded.

Participants were randomly assigned to three groups of 20 participants each, and each group waspresented with one of the three subsets of stimuli.
Four acoustic parameters (contemplated in ISO 3382-1 standards, AENOR, 2010) were calculatedwith the ARTA acoustic software (version 1.9.4.1) across octave bands from 125 to 2000Hz, for each of thesix impulse responses: speech clarity (C50, measured in decibels), music clarity (C80, measured in decibels),
reverberation time (T20, measured in seconds) and late lateral sound level (Lj, measured in decibels). Table
1.

Table 1. Values for the acoustic parameters: speech clarity (C50, dB), music clarity (C80, dB), reverberation time (T20,
seconds) and late lateral sound level (Lj, dB), calculated from the six impulse responses employed in the study. Average of theoctave bands from 125 to 2000 Hz.

Group 1

Group 2

Group 3

C50 (dB)

C80 (dB)

T20 (s)Lj (dB)

Ctrl

15.14015.4700.029-25.464Exp

32.90043.8000.089-44.418

Ctrl

14.13018.8700.017-29.000

Exp

29.71036.6000.090-34.804

Ctrl

20.63031.4100.017-23.006

Exp

27.12034.8900.119-49.574

The experiment was carried out in the Immersive Psychoacoustics Laboratory (immpaLAB), set upby the Artsoundscapes project at the Faculty of Psychology of the University of Barcelona (Spain).
Participants started by filling in questionnaire about religiosity and spirituality [the Centrality of Religiosity

Scale (Huber and Huber, 2021)], a State-Trait Anxiety Inventory (STAI, Spielbelger et al., 1983). Then,
participants were presented with one of the three subsets of stimuli described previously, depending on thegroup they were randomly assigned. Stimuli were presented in random order, first in one condition (art+ orart-) and then in the other. Half participants listened to the art+ condition first, and the other half, the otherway around. After listening to each sound, participants were interrogated about how they perceived the soundand the emotions it elicited, using ten descriptors. These were present, spacious, enveloping, deep, close,
alert, tension, peace, calm and pleasure. All of these have been previously employed in different studies
involving auditory stimuli (Russel, 1980; Watson & Clark, 1988; Barron, 1993; Zentner et al., 2008; Trost etal., 2012).
A total of ten factorial ANCOVA analyses were carried out, one for each descriptor studied. Factors in
the model were the group (with three levels), the sound (with seventeen levels) and the art (with two levels,
art+ and art-). Three covariates were also considered: one score derived from the Centrality of Religiosity
Scale and two scores derived from the STAI questionnaire. All the p-values were corrected for multiple
testing using the Benjamini-Hochberg (FDR) method. In all cases, the significance level considered was α =0.05. All the analyses were carried out using the software R (version 3.6.2) and the ez package (Lawrence,2016).

3. RESULTS
The influence of the factor of interest (art) was observed over the ratings of pleasure. Also, aninteraction between the factors art and group was observed over the ratings of the descriptors present, close,
alert, peace, calm and tension. The ratings of tension were also modeled by the triple interaction of the factors
group, art and sound.
In the case of pleasure the ratings were higher when listening to the sounds convolved with the artimpulse response (M = 3.968, SD = 3.285) compared to the art+ (M = 3.729, SD = 3.278, figure 2).

Figure 2. Ratings of the descriptor 'pleasure' significantly influenced by the factor ‘art’.

For the descriptors presence, closeness, alert, peace and calm, post-hoc comparisons were carried out
to analyze the effect of the factor art on each group separately. The factor art had a significant influence over
the ratings that participants from the third group attributed to these descriptors. Participants from this grouprated the sounds convolved with the art+ impulse response as more present (M = 7.338, SD = 2.118) andcloser (M = 7.182, SD = 2.124), as well as evoking more alert (M = 5.706, SD = 3.052), less peace (M =3.585, SD = 3.116) and less calm (M = 3.728, SD = 3.240) in comparison to its art-, in which the sounds
were rated as less present (M = 6.511, SD = 2.065), less close (M = 6.171, SD = 2.193) and evoking less alert(M = 5.200, SD = 2.944), more peace (M = 4.193, SD = 3.185) and more calm (M = 4.289, SD = 3.183).
Figure 3.

Figure 3. Ratings of the descriptors ‘presence’, ‘close’, ‘alert’, ‘peace’ and ‘calm' significantly influenced by the factor ‘art’
on each group.

For the descriptor tension a post-hoc analysis was carried out to observe the influence of the factor art
on each group and sound separately. For the sound of plover a significant influence of the factor art was
found in the first and third groups. Participants from the first group rated the sound of the plover in the art+
condition -convolved with the impulse response recorded in front of the rock art panel of Kalbak Tash I- assignificantly less tension-evoking (M = 3.350, SD = 2.368) than in the art- condition (M = 5.650, SD = 2.834).
On the contrary, in the third group, participants rated the sound of the plover in the art+ condition -convolved
with the impulse response recorded in front of the panel of Torgun i5- as more tension-evoking (M = 6.050,
SD = 2.564) than in the art- condition (M = 4.150, SD = 1.725). Figure 4.

Figure 4. Ratings of the descriptor 'tension' when listening to the sound of 'plover' significantly influenced by thefactor ‘art’ in the third group.

Since various studies have shown a relation between emotional valence and energy distribution along
the frequency spectrum (Scherer, 1986; Banse & Scherer, 1996; Waaramaa et al., 2010; Patel et al., 2011),
we further analyzed the spectrum of the sound of the plover convolved with the impulse responses presented
to the first and third groups. Figure 5 shows a distribution of amplitude along the frequency spectrum of thissound convolved with the mentioned four impulse responses, with a peak frequency in all cases around 3000
Hz.

Figure 5. Relative amplitude of frequencies between 200 and 8000 Hz of the sound of ‘plover’ convolved with the four
impulse responses listened by participants from groups 1 (pink) and 3 (blue), in art- (in front of a wall away from rock art
panels, line with circles) and art+ (in front of a panel with rock art, dashes with triangles) conditions.

4. DISCUSSION
Previous studies showed that dynamic changes in musical stimuli (Pätynen & Lokki, 2016) anddynamic responsiveness of the acoustic space (Green & Kahle, 2018; Pätynen & Lokki, 2018) are key foremotional exaltation during the listening. Late lateral sound level (Lj) reaches higher values in the artimpulse responses employed in the present study, compared to the art+ ones. The higher Lj translates intohigher dynamic responsiveness and might explain the greater pleasure reported by participants when listening
to the sounds presented with art- acoustics.
The acoustics in the position where the rock art panel of Torgun I5 is located have a longer
reverberation than its corresponding art-. Therefore, the perceived distance from the sound source should begreater in front of the rock art panel (ARUP, 2020; Bronkhorst & Houtgast, 1999; Kolarik et al., 2015).
However, we observe the opposite. The reason might be that the reverberation is still too short to be causing
an increase in the perceived distance from the sound source, and in contrast, the increased sound amplificationin the art+ position could cause an increased perceived presence and closeness of the sounds. Since soundlevel is inversely related to calmness (MSSS, 2016) this also explains the higher ratings of alert and lower
ratings of peace and calm in that position.
Loudness (Granot & Eitan, 2015) and pitch (Granot & Eitan, 2015, Ilie & Thomson, 2015) are known
to be directly related with a sensation of tension. Therefore, the greater amplification of the high-pitched peakfrequency that we observed could be the cause for a stronger tension evocation when listening to the soundof the plover in the art- impulse response of the first group and the art+ impulse response of the third group.
Díaz-Andreu et al. (2022) found a greater sound amplification in all the sites analyzed in the lower Chuya
river area, with the exception of Kalbak Tash I, that behaved opposite to its neighboring rock art sites. Thesefindings match with the opposite results obtained of the tension reported by participants from the first andthird groups when listening to the sound of a plover, and lead us to think that Kalbak Tash I does not offer agood representation of the acoustic behavior of the rock art sites of the area. Also, that in this particular casethe sound strength parameter was not especially related to the selection of this spot for decoration (DíazAndreu et al., 2022).

5. CONCLUSIONS
After conducting the present study, our hypothesis was partially confirmed; an effect of the acoustic
environment was observed over several psychoacoustic and emotional dimensions. However, only one effectwas consistent across the different rock art sites. Therefore, further research is necessary in order to confirm
that the observed phenomena are repeated in other rock art sites and, therefore, that acoustics could haveplayed a role in the selection for rock art production.

ACKNOWLEDGEMENTS
This article is part of the ERC Artsoundscapes project (Grant Agreeement No. 787842) that hasreceived funding from the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme. PI: Margarita Díaz-Andreu.

REFERENCES
1. AENOR (Asociación Española de Normalización y Certificación, 2010). Medición de parámetros
acústicos en recintos (ISO 3382-1).
2. ARUP (2020). Reverberant Sound Level (From RT). Strutt Help.
https://strutt.arup.com/help/Building_Acoustics/Lw2Lp.htm
3. Banse, R., and Scherer, K. (1996). Acoustic profiles in vocall emotion expression. J. Pers. Soc. Psychol.
70, 614–636. DOI: 10.1037/0022-3514.70.3.614.
4. Barron, M. (1993). Auditorium Acoustics and Architectural Design. Spon Press. ISBN: 0-203-87422-6.
5. Bronkhorst AW, Houtgast T. Auditory distance perception in rooms. Nature. 1999;397:517–520. DOI:
10.1038/17374.
6. Diaz-andreu, M., Mattioli, T. (2015). Archaeoacoustics of rock art: quantitative approaches to theacoustics and soundscape of rock art. Proceedings of the 43rd Annual Conference on Computer
Applications and Quantitative Methods in Archaeology, 1 and 2. ISBN: 9781784913373.
7. Díaz-Andreu, M., Mattioli, T. (2021). Rock Art, Music and Acoustics: A Global Overview. B. David and
I. J. McNiven (Scholarly Research Reviews). The Oxford Handbook of the Archaeology andAnthropology of Rock Art (503-528). Oxford Handbooks Online. DOI:
10.1093/oxfordhb/9780190607357.013.8.
8. Díaz-Andreu, M., Mattioli, T. and Rainsbury, M. P. (2021). The Cultural Understanding of Sound inRock Art Landscapes: The Limits of Interpretation. In Gjerde, J. M. y Arntzen, M. S. (coords.):
Perspectives on Differences in Rock Art. Equinox. 244-265. London.
9. Díaz-Andreu, M., Jiménez-Pasalodos, R., Rozwadowski, A., Alvarez Morales, L., Miklashevich, E. andSantos da Rosa, N. (2022). The soundscapes of the Lower Chuya River area, Russian Altai.
Ethnographic sources, indigenous ontologies and the archaeoacoustics of rock art sites. Journal ofArchaeological Method and Theory.
10. Farina, A. (2020, June 12). How we take acoustical measurements in the Artsoundscapes project, andhow we got there. Artsoundscapes Blog. https://www.ub.edu/artsoundscapes/how-we-take-acousticalmeasurements-in-the-artsoundscapes-project/
11. Farina, A., Chiesi, L. (2016). Measuring Spatial MIMO Impulse Responses in Rooms Employing
Spherical Transducer Arrays, Conference Paper. Conference on Sound Field Control, 2016 July 18-20,
Guildford, UK.
12. Huber, S., and Huber, O. W. (2012). The Centrality of Religiosity Scale (CRS). Religions 2012, 710–
724. DOI: 10.3390/rel3030710.
13. Kolar, M. A. (2013). Tuned to the Senses: An Archaeoacoustic Perspective on Ancient Chavín. The
Appendix, 1(3).
14. Kolarik, A. J., Moore, B. C. J., Zahorik, P., Cirstea, S. and Pardhan, S. (2015). Auditory distanceperception in humans: a review of cues, development, neuronal bases, and effects of sensory loss.
Attention, Perception and Psychophysics, 78: 373-395. DOI:10.3758/s13414-015-1015-1.
15. Lawrence, M. A. (2016). Ez: Easy Analysis and Visualization of Factorial Experiments. R package
version 4.4-0. Vienna: R Core Team.
16. MSSS (Ministère de la Santé et des Services sociaux, 2016). Exemples de sources de bruit et deréactions humaines selon le niveau de bruit. Publications du ministère de la Santé et des Services
sociaus, 15-004-06W.
17. Patel, S., Banzigar, T., Scherer, K. R. (2011). Perception of the proximal voice features for emotional
speech. Journal of the Acoustical Society of America, 130, 2574 (2011). DOI: 10.1121/1.3655320.
18. Pätynen, J., and Lokki, T. (2016). Concert halls with strong and lateral sound increase the emotional
impact of orchestra music. J. Acoust. Soc. Am. 139, 1214–1224. DOI: 10.1121/1.4944038.
19. Russell, J. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6),1161–1178. DOI: 10.1037/h0077714.
20. Scherer, K. (1986). Vocal afect expression: A review and a model for future research. Psychological
Bulletin, 99(2), 143–165. DOI: 10.1037/0033-2909.99.2.143.
21. Spielberger, C.D., Gorsuch, R.L., Lushene, R., Vagg, P.R. and Jacobs, G.A. (1983) Manual for theState-Trait Anxiety Inventory. Consulting Psychologists Press, Palo Alto.
22. Tátar, M. M. Mythology as an Areal Problem in the Altai-Sayan Area: The Sacred Holes and Caves. In
Shamanism and Northern Ecology. Religion and Society 36., edited by Juha Pentikäinen. Berlin, New
York: Mouton de Gruyter, 1996. DOI: 10.1515/9783110811674.267.
23. Trost, W., Ethofer, T., Zentner, M., & Vuilleumier, P. (2012). Mapping aesthetic musical emotions in thebrain. Cerebral Cortex, 22(12), 2769–2783. DOI: 10.1093/cercor/bhr353.

24. Västfjäll, D., Larsson, P., and Kleiner, M. (2002). Emotion and auditory virtual environments: Affectbased judgments of music reproduced with virtual reverberation times. Cyberpsychol. Behav. 5, 19–32.
DOI: 10.1089/109493102753685854.
25. Västfjäll, D. (2012). Emotional Reactions to Sounds without Meaning. Psychology, 03(08), 606–609.
DOI: 10.4236/psych.2012.38091.
26. Vorländer, M. (2008). Auralization. Berlin: Springer. ISBN: 978-3-540-48830-9.
27. Waaramaa, T., Laukkanen, A. M., Airas, M. and Alku, P. (2010). Perception of emotional valences andactivity levels from vowel segments of continuous speech, J. Voice, vol. 24, no. 1, pp. 30–38, 2010.
DOI: 10.3109/14015439709075330.
28. Watson, D. and Clark, L. A. (1988). Development and Validation of Brief Measures of Positive and
Negative Affect: The PANAS Scales. Journal of Personality and Social Psychology, 54(6), 1063–1070.
DOI: 10.1037/0022-3514.54.6.1063.
29. Zentner, M., Grandjean, D., & Scherer, K. R. (2008). Emotions Evoked by the Sound of Music:
Characterization, Classification, and Measurement. Emotion, 8(4), 494–521. DOI: 10.1037/15283542.8.4.494.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0388

Immersive audio inside the Greek-Roman theatre of Tyndaris:
comparison between past, current and future conditions
Lorenzo LAVAGNA1; Louena SHTREPI2; Antonella BEVILACQUA3; Angelo FARINA4; Arianna
ASTOLFI5;
1
2

Università di Parma, Italia

Politecnico di Torino, Italia

ABSTRACTThe acoustics applied to the ancient monuments has been found to be of great interest, especially whenother disciplines interface with the sound perception of virtual reality. This paper deals with the digitalreconstruction of the acoustic conditions related to the Greek-Roman theatre of Tyndaris through its
historical architectural evolution from the 4th century BCE. The auralization of the selected audio
recording has been applied to four different scenarios, representing the theatre in its past original condition(PC), in the current conditions (CC), in a future condition (FC) that includes an acoustic shell on theorchestra area, and in a hypothetical condition with the original scene preserved, but the current degradedcavea (SC). The absorption coefficients of the digital model have been calibrated based on the measured
results of the current condition. These data have been used as input for the PC and FC models. The same
source-receivers set-up has been used in the three conditions. Furthermore, a listening test has been
submitted to 8 volunteers who have been involved in the assessment of the subjective acoustic parameters,that is, the Intensity, Reverberation, Clarity, Tonal Balance, Listener envelopment, Apparent source width,
Distance and the source localization in the virtual environment.
Keywords: Ancient theatres, subjective acoustic tests, Auditory VR

1. INTRODUCTION
During the last decade, a growing interest towards the use virtual reality to make our cultural
heritage more accessible has emerged. One of possible application of this tool is the evaluation of
the effect of architectural interventions on the acoustic performance of historical buildings during
the design process (1). The purpose of this work is to investigate the benefits of comparing amethodic evaluation conducted with the objective parameters, given by ISO 3382 -1:2009, with thesubjective data gathered through listening tests with spatialized audio in virtual reality in the casestudy of different conditions of an ancient open-air theatre.
The acoustic simulation tools are nowadays easily available to a wide variety of audience,composed also of non-experts who approach the most developed technology given its contained
costs. This situation occurs also in specific contexts, spanning from the acoustic space design to thereconstructions of non-existing buildings (archaeo-acoustics), including the virtual world of thegaming industry (2). However, if the latest generation of technology creates new research
opportunities for the experts in acoustics, at the same time it can potentially raise several concernson the accuracy and reliability of the acoustic simulations. On this bas is, the issues can be faced
especially when there is a lack of a fully understanding of critical acoustic problems pertaining toboth simulation algorithms and modelling techniques (3).
1 lorenzolavagna@gmail.com
2 louena.shtrepi@polito.it
3 antonella.bevilacqua@unipr.it
4 angelo.farina@unipr.it
5 arianna.astolfi@polito.it

Postma and Katz have described the potentials of auralization in virtual reality applied tohistorical reconstruction in (4) and (5).
Moreover, acoustic simulations for particular spaces such as ancient theatres present several
challenges (6). It has been shown that the objective acoustic parameters (ISO 3382-1:2009) do not
fully describe the acoustic characteristics (7), thus a more “listening” - related approach could be
used to reach accurate acoustic models of existing conditions as well as to investigate acoustic
effects for any adaptive reuse (8).
This paper proposes the integration of Ambisonic auralization and its applications in an adaptive
reuse project related to the ancient theatre of Tyndaris. The theatre has been considered in its pastoriginal condition (PC), in the current conditions (CC), in a future condition (FC) that includes anacoustic shell on the orchestra area, and in a hypothetical condition with the original scene
preserved (SC). A description of the shape and the geometrical characteristics, including material
choices, for the proposed acoustic shell is also developed. Furthermore, the methodologies toachieve reliable results in open-air theatres have been developed and explored along with a carefulcalibration procedure of the three specific conditions.

2. MATERIALS AND METHODOLOGY
2.1 Case study overview
The ancient theatre of Tyndaris is located on the seaside of Sicily, specifically on a cliff facing
the natural landscape of the Tyrrhenian Sea at a height of 180 m. It is surrounded by the ruins of theancient city of Tyndaris that was founded in 396 BCE by G reeks. The birth of this Greek colony was
specifically due for retaining the exiles of the nearby city of Messenia (modern M essina).
The shape of this theatre has considerably changed throughout the centuries. During the 4th
century BCE a scenic building was added to close-up the orchestra, and during the late imperial ageof the Roman empire substantial modifications have been made to adapt the theatre to host theperformance of gladiators’ spectacles and fights against wild beasts.
After important natural events, like landslides and earthquakes, the Greek theatre was buried forseveral centuries until the archaeological excavations occurred in 1838 and continued between 1960
and 1998.
Nowadays the theatre hosts also artistic performance, but its existing a coustic state iscompromised by many factors: the lack of the original scenic building, responsible of usefulreflections supporting the actor's voice, the absence of a large part of the steps of the cavea and thegeneral deterioration of the materials, impacting the reflectivity, and the increasing noise levels of
the surroundings due to traffic.
2.2 Calibration process
The measured results herein used for the calibration process are related to the acoustic survey
carried out in September 2015 by the Applied Acoustics Research Group of the Department ofEnergy of the Politecnico di Torino, extensively described in (9).

Figure 1 – a) Source and receiver positions in Ramsete.
b) Axonometric view of the calibration model showing the mapping of the materials in the current situation.

The chosen software for the simulations is Ramsete, a widely employed geometrical acoustics (GA)software based on pyramid tracing, an evolution of the conical beam tracing (10).
The same number and location of source and receivers have been used during the measurements
and in Ramsete, in order to proceed with the calibration process of the abso rbing and scattering
coefficients of the materials placed in the theatre, which was done by trial and error.
This paper makes use of the results used in previous research studies, that have validated an
acoustic model reflecting the existing conditions of the theatre by tuning the spectra of the measured
values of the main acoustic parameters with those obtained by the digital simulatio n (11). All thematerials in the model have been mapped as shown in Figure 1b.
Table 1 summarises the absorption coefficients applied to the materials used for the acoustic
calibrations. The sound absorption coefficients are affected by a high level of uncertainty asreported by Vorländer (12) especially due to the consistent aging effect of the materials.
Table 1 – Sound absorption coefficients.
63

125

250

500

1000

2000

4000

8000

OPENINGS

1

1

1

1

1

1

1

1

BRICK

0.2

0.2

0.2

0.1

0.05

0.05

0.05

0.05

WOOD

0.4

0.4

0.3

0.2

0.04

0.02

0.02

0.02

STONE - OPUS QUADRATUM

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

STONE - OPUS SILICIUM

0.36

0.36

0.39

0.43

0.43

0.47

0.49

0.47

STONE 50% EARTH 50%

0.31

0.25

0.28

0.29

0.2

0.15

0.15

0.15

EARTH

0.25

0.35

0.42

0.43

0.3

0.21

0.25

0.25

EARTH 80% GRASS 20%

0.39

0.35

0.38

0.39

0.25

0.2

0.2

0.3

BEATEN EARTH

0.25

0.22

0.35

0.35

0.22

0.1

0.15

0.2

GRASS

0.2

0.25

0.36

0.37

0.38

0.28

0.19

0.18

The calibration procedure is based on the principles proposed in (4). The following description
highlights the details of the geometrical acoustic (GA) model constructio n. In particular:• The impulse responses (IRs) have been used like a reference point for the ca libration
process. In this case, measured values are gathered from the 2015 survey.
• A 3D model has been realised by using AutoCAD 2022, allowing a reconstruction of thetheatre reporting the most relevant geometrical features and details (13). Specifically, anumber of 2261 faces have been modeled in the DXF format file, then exported to be usedin Ramsete.
• A material mapping has been performed based on a photogrammetry of the site and wide angle photos. Reflecting the conditions existing during the measurements, the cavea was inpart covered by wooden benches and plastic chairs; the orchestra instead was covered by awooden platform.
• The reverberation time (T20) and strength (G) are the main acoustic parameters that havebeen considered sufficient for the calibration process (14). The tuning of the values,
developed like an iterative process, has been completed when the difference between
measured and simulated values is within 5% and 1 dB, respectively (15).
2.3 Construction of PC and FC models
From the CC model calibration, shapes of the architectural components and materials weremodified to create the virtual reconstruction of the past conditions (PC) resembling the Hellenistic
configuration of the theatre, as shown in Figure 2.

Figure 2 – Axonometric view of models used for the simulation of the Hellenistic configuration.
The solution chosen for the project of adaptive reuse has already been proposed in 2016 by
Giovanni Bouvet (16). It consists of a scenic structure that takes advantage of the Canac’s Law toprovide the early reflections useful for increasing the loudness of the sound signal. This futurescenario was named FC.
The innovation of the reuse design project is based on a new acoustic shell ch aracterised by acurved shape. This reflecting surface was generated by an evolutionary solver (i.e. Galapagos), a
plugin for Grasshopper deploying computational morphogenesis, using a simple Image Source
Method (ISM) algorithm, partially based on François Canac's studies.

Figure 3 – Structure of the acoustic shell
The shape of this geometry has been developed considering the recommendations of the Syracuse
Charter (17), asserting that temporary structures can integrate through gaps in order to optimise theacoustic performance: the modular approach of the singular components allows for a lightweightremovable structure that does not impose any permanent impact on the archaeolo gical site.
One additional fourth modelling scenario (SC) was created by transporting the scenic building
from PC to CC to assess the impact of the scenic building on the sound field without thereverberation generated by fully intact cavea of PC.

2.4 Ambisonic auralizations
From each of the four simulations, and for the selected source and listener positions, one IR was
exported to create the auralizations for the listening test. The IRs were exported from the receiverpoint R5 in B-format with an ambisonic order of 3 (3OA – 16 channels). They were then used forthe convolution of two anechoic recordings, one speech sample and one music sample, using themultichannel convolution matrix VST plugin X-MCFX running inside the sound processing program
Plogue Bidule.
The resulting tracks, each with 16 channels, were then muxed with a 360° video made byrendering an equirectangular image of a 3d model of the corresponding scene, centred around thereceiver point R5 in Blender. The mux was realized with FB360 Encoder, a free software developed
by Facebook that is able to use a 3OA track to create a .mkv file with 10 audio channels that can beread by a Meta Quest 2 in standalone mode.
2.5 Listening test
The resulting immersive scenes have been compared in a subjective listening test, structured as
an AB comparison (18) presented inside the virtual reality environment using a Head Mounted
Display (HMD), specifically a Meta Quest 2, and open headphones (Sennheiser HD 650).
Twenty-four auralization pairs were created, 8 of which were presented in a random order to eachparticipant. Each pair was followed by a set of 3 questions, presented inside the immersive video:
1. How different are the two signals on a scale of 0 to 10?
2. What kind of difference did you perceive? (Multiple choice answer: Intensity, Reverberation,
Clarity, Tonal Balance, Listener envelopment, Apparent source width, Distance)
3. Which signal did you prefer between first and second?
The listening methodology has been used also in other studies and has been extensively described in(5). Eight participants, self-evaluated normal hearing subjects aged between 22 and 32, were given
written instructions before the test, including the definitions of the perceptual attributes required toanswer the second questions. An extensive description of the same attributes can be found in (5);they were also given two training pairs before starting the test and they were allowed to replay eachcomparison as much as they needed. Their progress was constantly followed, taking advantage of
the wireless screen mirroring capability offered by the Meta Quest 2.
At the end of the test, participants were asked to remove the HMD to answer a last question about
the position of the sound source. Participants were provided a tablet showing the planimetry of thetheatre where they could draw freely with a stylus to indicate the position and dimensions of thesource.
All of the listening test sessions have been conducted inside the anechoic chamber of the Politecnico
di Torino in order to avoid any external disturbing noise. Closed headphones would have beenequally efficient at preventing distraction, but they would have impaired communication between
the tester and the subject.

Figure 4 – a) Set-up fore listening test in the anechoic room at Politecnico di Torino.
b) Rendering of the 3D model of the theatre in the current condition with the addition of an acoustic shellcorresponding to FC.

3. RESULTS
3.1 Objective parameters comparison
Figures 5, 6 and 7 show a comparison of CC, PC and FC based on the main objective parametersconsidered during the simulations.
The virtual reintegration of the missing parts of the cavea, together with the reconstruction of thefrons scaenae in PC, generated a noticeable increase of the T 20 and G and drastic decrease of C 50 and
C 80 . The effectiveness of the reflective shell in FC is proved by an average increment in G of 1.8 dB
and a better Clarity of speech, +2.4 dB on average for C 50, compared to CC; yet this improvement
could be unperceivable for a common spectator.

Figure 5 – Comparison of the values of Gmid for CC, PC and FC.

Figure 6 – Comparison of the values of T20 for CC, PC and FC.

Figure 7 – Comparison of the values of C50 for CC, PC and FC.

3.2 Subjective comparison
The results of the listening test show a clear preference of FC over CC, especially whencomparing speech auralizations; in this case FC was preferred 75% of the times, suggesting that the
impact of the acoustic shell on the sound field could be noticeable for the audience. The more
reverberant scenery of PC was still preferred over the other two, even for speech, although theparticipant reported a noticeable decrease of clarity.
Seven of eight participants were able to clearly identify the position of the sound source at thecenter of the orchestra when asked at the end of the session.
Being allowed to replay the audio pair as much as they needed was very useful for theparticipants since most of them expressed an initial sense of disorientation. Many subjects also
reported they were not sure if they were basing their preferences on visual or auditory input.

4. CONCLUSIONS
Aim of this preliminary study was to investigate the benefits of comparing a methodic evaluation
conducted with the objective parameters, given by ISO 3382-1:2009, with the subjective data
gathered through listening tests with spatialized audio in virtual reality in the case study of ancient
open-air theatre
The answers given by the subjects of the listening test result useful to evaluate the accuracy of
the geometrical-acoustic models. Moreover, they confirm the previous assumptions based on theanalysis of objective parameters. However, given the limited number of participants (= 8) the
reliability of the results must be further investigated with more listeners that would allow to apply
more advanced statistical methods. Some of the critical aspects of the test were related to therepeatability of the results when the participants were asked to evaluate the same auralization pair
twice, and to the influence of the visual stimulus on the evaluations. Future research will focus on
reducing these uncertainties in order to be able to take advantage of subjective listening tests toevaluate possible projects of adaptive reuse and to create accurate reconstructions of pastconfigurations of the historically valuable buildings of our cultural heritage.

5. ACKNOWLEDGEMENT
The funding for this study has been provided by the European Union’s Joint Programming
Initiative on Cultural Heritage project PHE (The Past Has Ears, phe.pasthasears.eu).

REFERENCES
1. Llorca Bofí J. The generative, analytic and instructional capacities of sound in architecture:
fundamentals, tools and evaluation of a design methodology. 2018;
2. Moloney J, Harvey L. Visualization and “auralization” of architectural design in a game engine based
collaborative virtual environment. In 2004. p. 827–32.
3. Martellotta F. Caveats and pitfalls in acoustic simulation of non-existing buildings. In: Proceedings ofthe 2nd Symposium on The Acoustics of Ancient Theatres. Verona; 2022.
4. Postma BNJ, Katz BFG. Creation and calibration method of acoustical models for historic virtual reality
auralizations. Virtual Real. 2015 Nov 1;19(3):161–80.
5. Postma BNJ, Katz BFG. Perceptive and objective evaluation of calibrated room acoustic simulation
auralizations. J Acoust Soc Am. 2016;140(6):4326–37.
6. Shtrepi L, Lavagna L, Bevilacqua A, Farina A, Astolfi A. Challenges in calibration of acoustical models
for historic virtual reality auralizations. In: Proceedings of the 2nd Symposium on The Acoustics ofAncient Theatres. Verona; 2022.
7. Rindel JH. Meaningful acoustical parameters for open-air theatres. In: Proceedings of the 2nd
Symposium on The Acoustics of Ancient Theatres. Verona; 2022.
8. Lavagna L, Shtrepi L, Farina A, Bevilacqua A, Astolfi A. Acoustic design optimization through the useof auralization: how does it sound like? In: Proceedings of the 2nd Symposium on The Acoustics ofAncient Theatres. Verona; 2022.
9. Astolfi A, Bo E, Aletta F, Shtrepi L. Measurements of Acoustical Parameters in the Ancient Open-Air
Theatre of Tyndaris (Sicily, Italy). Appl Sci. 2020 Aug;10:5680.
10. Farina A. RAMSETE-a new Pyramid Tracer for medium and large scale acoustic problems. 1995 Jan;
11. Ambisonics as a tool for architectural preservation. The virtual soundscape of the ancient theatre of
Tindari
Webthesis[Internet].
[cited2022
May
7].
Availablefrom:https://webthesis.biblio.polito.it/21074/
12. Vorländer M. Prediction tools in acoustics-Can we trust the PC ? In 2010.
13. Siltanen S, Lokki T, Savioja L, Christensen C. Geometry Reduction in Room Acoustics Modeling. Acta
Acust United Acust. 2008 May 1;94:410–8.
14. Bo E, Shtrepi L, Pelegrín Garcia D, Barbato G, Aletta F, Astolfi A. The Accuracy of Predicted
Acoustical Parameters in Ancient Open-Air Theatres: A Case Study in Syracusae. Appl Sci [Internet].
2018;8(8). Available from: https://www.mdpi.com/2076-3417/8/8/1393
15. ISO 3382-1:2009 “Measurement of room acoustic parameters - part 1: performance spaces.” [Internet].
ISO.
2009.
Availablefrom:https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/04/09/40979.html
16. Bouvet GA, Shtrepi L, Bo E, Echenagucia TM, Astolfi A. Computational design: acoustic shells forancient theatres. In: Forum Acusticum [Internet]. Lyon, France; 2020 [cited 2021 Nov 21]. p. 1581–5.
Available from: https://hal.archives-ouvertes.fr/hal-03235296
17. Carta di Siracusa “per la conservazione, fruizione e gestione delle architetture teatrali antiche.” 2014.
18. Boley J, Lester M. Statistical Analysis of ABX Results Using Signal Detection Theory. In AudioEngineering Society; 2009 [cited 2022 Aug 21]. Available from: https://www.aes.org/elib/online/browse.cfm?elib=15022

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0143

Ecological Momentary Assessment and virtual acoustic
environments for hearing research – differences and similarities
Inga HOLUBE1; Petra VON GABLENZ1; Jule POHLHAUSEN1; Joerg BITZER1
1

Institute of Hearing Technology and Audiology, Jade University of Applied Sciences, Oldenburg, Germany

ABSTRACT
Ecological Momentary Assessment (EMA) is a method which includes repeated ad-hoc surveys in real-life
as well as the collection of objective environmental data. In this way, EMA provides snapshots of individual
listening experiences affected by hearing loss, living environment, and life-style. The method is considered
to be more ecologically valid then clinical test methods or virtual acoustic environments for assessing hearing
abilities. It provides a large amount of data referring to real-world conditions, tasks, behavior, and emotions.
In comparison with lab experiments, however, EMA comes with uncontrolled parameter settings, many
confounders, questionable compliance, and low reliability. In EMA, comparisons between participants or test
settings (e.g., with and without hearing devices) are hindered by the lack of reproducibility since real-lifesituations cannot be precisely reproduced at different points in time. In this contribution, clinical settings,
virtual test setups in the laboratory, and real-life experiments in EMA studies are compared in terms of theirdifferences and similarities. Each of the approaches has its advantages and disadvantages, and it is thesedifferences that make them complementary rather than competitors.
Keywords: Ecological Momentary Assessment, clinical settings, virtual acoustics

1. INTRODUCTION
In hearing research, EMA is a method to repeatedly assess momentary listening experiences ineveryday life over a longer period, e.g., several days or weeks (1, 2). The subjective assessments canbe supplemented by recordings of objective acoustic parameters. Nowadays, modern technologies areoften used for this purpose, such as the open-source EMA system olMEGA consisting of two headworn microphones, a transmission box, and a smartphone (4, 5) presented in another contribution tothis conference (3).
Intuitively, EMA allows for data collection with higher ecological valid ity (6) than measurement
methods in the laboratory. However, the advantage of higher ecological validity is bought by, forexample, less controlled experiments and many confounders. Realistic replications of everyday
situations in the laboratory (7) or virtual experimental setups (8) could act as a link between controlled
and standardized clinical measurement conditions and everyday life. The analysis of everyday
situations can contribute to the selection of prototype listening situations ( 9, 10) that allow
controllable laboratory experiments relevant to everyday life. In this contribution, EMA is contrasted
with clinical settings and virtual experimental setups (see Tables 1 and 2).

2. Comparisons
2.1 Aspects of study design
Hearing tests with pure tones, speech tests, and questionnaires are used in clinical settings fordiagnosis purposes and for the validation of technical hearing devices (hearing aids, implants). The
pure-tone and speech tests can be considered as controlled experiments. Data are collected underpredefined conditions with a fixed and standardized set of parameters. Due to the narrow time period
available in the regular clinical routines, the measurement time for each test is limited to a few minutes.
From the participant’s perspective, travel and waiting time must be added to the measurement timeitself to estimate the burden of the clinical examination. During the measurement, maximum
1

inga.holube@jade-hs.decompliance must be assumed, which generally can be observed and ensured by the investigator.
Controlled and high compliance experiments are also performed in virtual test setups. In contrast
to clinical settings, there is usually more time available in research for conducting measurements in
virtual test setups. The common measurement period in the laboratory is up to 2 h and can be extended
by additional appointments. Travel time must also be added to the participant’s burden. The available
measurement time allows data collection under specifically selected, predefined conditions withsystematic variation of parameters. Particularly in virtual setups, a wide range of subjective andobjective methods is used to examine hearing and hearing-related perceptions. An overview ofmeasurement methods for listening effort, for example, is given by McGarrigle et al. (11) and Klink
et al. (12, 13).
EMA studies, on the other hand, are basically uncontrolled experiments with many confounders.
The measurement equipment, i.e., at least a smartphone, has to be worn by the participants in their
daily life mostly for several days or weeks. Thus, the observation time is much longer in EMA
applications than in laboratory experiments. Several surveys are completed digitally per day andadditional objective data can be collected. With this approach, EMA can allow for a representative
design (14), i.e., aiming at a representative image of everyday life by the repeated assessment ofrandomly selected listening situations. There are several sources for selection bias. Assumingly themost important, participants cannot always respond immediately to a prompt. For example, the use ofa smartphone would be impolite in conversational situations or safety-critical in road traffic situations.
In addition, the risk of careless responses (15) increases with increasing measurement duration. As
far as compliance is concerned, essentially only retrospective control is possible in EMA bycalculating the ratio of the number of surveys and the number of prompts. A lot of studies have shownthat compliance varies widely between participants (2).
Table 1 – Comparison of clinical setting, virtual test setup and EMA by design aspects
Design aspect

Clinical setting

Virtual test setup

EMA

Experiment type

Controlled

Controlled

Uncontrolled, with many
confounders

Conditions

Predefined with
fixed standardized
set of parameters

Specifically selected andpredefined withsystematic variation ofparameters

Representative design except
for improper conditions;parameter variations dependon the individual everyday life

Appointment

Onetime and short,in the clinic

One to several and max.
2 h each, in the lab

Longer period of observations
in everyday life

BurdenTravel, waiting, andmeasurement time

Travel and measurement
time

Carry equipment for several
days or weeks

Methods

Pure-tone detection,
speech recognition,
retrospective
questionnaires

Divers subjective andobjective methods

Several surveys per day;additional objective data

ComplianceObservable andassessable;
goal: 100%

Observable andassessable;
goal: 100%

Retrospectively calculated;careless responses possible

2.2 Communication task
In everyday life, the most apparent goal of communication is the exchange of information. For thispurpose, the content of speech must be understood and processed. Generally, communication
situations require not only listening, but also one’s own speaking. On average, the proportion of time
per day spent speaking on one’s own is about 10% (16). In EMA studies, this natural interpersonal
contact is not substantially influenced by data collection, therefore an almost natural behavior can beassumed.
In contrast, the tasks in clinical hearing testing is characterized by only listening and repeating thepresented speech. No real communication takes place during the measurements, i.e., it involves
unnatural behavior without interpersonal contact and meaningful responses. Although the task invirtual test setups often also includes the repetition of presented speech stimuli, other tasks can be
integrated. An example is the DiapixUK task (17), in which two people must converse in order tosolve a joint task. However, there is always the risk that the interpersonal contacts are too artificiallyestablished with the consequence that the behavior may turn out unnatural.
Table 2 – Comparison of clinical setting, virtual test setup and EMA by various test features
Test features

Clinical setting

Virtual test setup

EMA

Communication task

Only listening andrepeating speech
items

Depends, but mostly
only listening

Content of speech has to beunderstood and processed

Personal contact

None

Artificial

NaturalBehavior

Unnatural

Possible unnatural(Mostly) natural

Reproducibility

Exact

Exact

LimitedSound pressure level

Speech and/or noise at
60 or 65 dB SPL

Selected and calibrated
appropriate for thesimulated condition

Set by the momentary
situation

Signal-to-noise ratio

Speech recognition
threshold mostly at
negative SNR

Not specified

Mostly positive

Attention

One target speakerwith one noise source

Diverse conditions

Divided and selected; one
to several sound sources

Participants

Patients

Volunteers in theirleisure time

Volunteers in their leisure
time

Motivation

Medical goal:
diagnostics orrehabilitation

Tasks not necessarily
meaningful andrelevant; (almost)constant motivation
during measurement

Meaningful and relevanttasks; motivation varies
over time; includes stress,
excitement, personal
involvement, emotions,fatigue over time2.3 Reproducibility
In clinical settings and virtual test setups, measurement conditions can be accurately reproduced
at any time. This is usually not possible for EMA studies. Even if the participant visits the same
listening situation at different times, e.g., a restaurant or watching TV at home, one cannot assume
the same acoustic conditions for different occasions. Additionally, the number and variety of listening
situations may be different for the same participant in different study periods. From (18, 19), among
others, it was shown that the agreement of the type of sound classes between two study periods was
only at correlations coefficients of r=0.64 (18) and r=0.52 (19), respectively.
2.4 Sound pressure levels
In clinical settings, speech recognition is typically determined at sound pressure levels (SPL) for
speech of 60 or 65 dB, which are considered average levels in communication situations (20). Theselevels are also used for speech-in-noise testing. Virtual test setups aim to replicate everyday situationsin the laboratory. Therefore, levels which correspond to the selected everyday situation are choosen.
Pohlhausen et al. (16) used the EMA system olMEGA to analyze SPL in daily life of 13 elderly
participants. Type of sound sources not considered (speech or noises), a median SPL of 50.3 dBA was
obtained, which was reduced to 49.0 dBA when the data segments that are very likely to contain theparticipants’ own voices were excluded. Because the participants were often in quiet environments at
home, the results are consistent with those of Pearsons et al. (21), who observed speech levels of
55 dB at background noise levels below 45 dB. However, median SPL of about 54 dBA for backgroundnoise and 64 dBA for speech were measured in home environments by Wu et al. (10). In crowded
listening situations, SPL for speech reach a median of about 70 dBA.
2.5 Signal-to-noise ratios
Since in everyday situations the speech level increases with the background noise level, a negative
signal-to-noise ratio (SNR) is rarely observed (10, 22, 23). However, in clinical settings, speech
recognition in noise is often determined at negative SNR. This is because the speech recognition
threshold (SRT) for speech recognition scores of 50% can be determined quickly and with highreproducibility using adaptive procedures. For matrix tests (24), the SRT reference value for young
normal-hearing persons is about -7 dB SNR. However, persons with hearing loss can also show anegative SRT (25).
2.6 Attention
In clinical settings, speech recognition is determined for a target speaker, which is usuallypresented via a loudspeaker positioned in the front of the test person (0° direction). The test is
performed in quiet or with one noise source, i.e., attention can be directed to the target speaker. A
wide range of conditions can be implemented in virtual test setups. An example is the Concurrent
Oldenburg sentence test (CCOLSA, 26), where three target speakers are presented and an alternation
between divided and selective attention is required. In everyday situation, any combination ofdifferent sound sources and of selective and divided attention can occur. As an example, Figure 1
shows the percentage of the number of sound sources in an EMA study (27). In most situations, theparticipants reported only one sound source. However, in about a quarter of the situations, more than
one sound source was present. Wu et al. (10) found that speech targets come from the front in the mostcommon listening situations and report a high proportion of diffuse noise situations when it is not
quiet.
2.7 Motivation
In clinical settings, participants are patients who usually have a relevant medical interest in
diagnosis or rehabilitation. In contrast, studies in virtual test setups or with EMA involve volunteers
participating in their leisure time. The tasks in virtual test setups are not necessarily meaningful andrelevant to the participants. As the measurements are performed under the super vision of aninvestigator, it can be assumed that motivation is approximately constant during the measurements.
Nevertheless, it is difficult to create the same level of stress in laboratory situations that can occur inpersonally volatile situations in everyday life, especially for participants who frequently participate
in laboratory studies (28).
EMA studies are also conducted with volunteers and their motivation to respond to smartphone
prompts may vary over time, but the data collected refers to real-life situations which include

meaningful and relevant tasks. Daily life is accompanied by different levels of stress as well as
excitement, personal involvement, emotion, and fatigue. Moreover, the importance that is individuallyattached to certain situations, may have an impact on the assessment. As an example from an EMA
dataset (27), Figure 2 shows the relationship between listening effort, loudness, and importance ofhearing well. Listening effort increases not only with loudness, but also in important listening
situations compared to less important listening situations.

Figure 1 – Percentage of the number of sound sources in an EMA study.

Figure 2 – Listening effort in dependence of loudness (soft, medium, and loud) and importance(unimportant, partly important, and important)

3. CONCLUSIONSIn this contribution, clinical setting, virtual test setups, and the EMA method were compared withrespect to distinctive criteria and exemplified. The methods pursue different goals and should not beseen as competitors, but as complements. For the future it would be desirable if elements of the EMA
method and virtual test setups were adopted in everyday clinical practice to increase the ecologicalvalidity of clinical settings.

REFERENCES
1. Stone AA, Shiffman S, Atienza AA. The science of real-time data capture – self-reports in health research.
2007. Oxford University Press.
2. Holube I, von Gablenz P, Bitzer J. Ecological momentary assessment in hearing research: Current state,challenges, and future directions. Ear and Hearing. 2020; 41(Suppl. 1): 79S-90S.

3. Holube I, von Gablenz P, Bitzer J. Evaluating hearing devices in real life using Ecological Momentary
Assessment. Proceedings of the 24th Inernational Congress on Acoustics, October 24 to 28, 2022,
Gyeongju, Korea.
4. Kowalk U, Kissner S, von Gablenz P, Holube I, Bitzer J. An improved privacy-aware system for objective
and subjective ecological momentary assessment. In: Santurette S, Dau T, C.-Dalsgaard J, Tranebjaerg
L, Andersen T, Poulsen T, editors. Proceedings of the International Symposium on Auditory andAudiological Research. Vol. 6: Adaptive Processes in Hearing. The Danavox Jubilee Foundation; 2017.
p. 25-40B.
5. Kowalk U, Franz S, Groenewold H, Holube I, von Gablenz P, Bitzer J. olMEGA: An open source android
solution for ecological momentary assessment. GMS Z Audiol (Audiol Acoust). 2020; 2:Doc08: 1-9.
6. Keidser G, Naylor G, Brungart D, Caduff A, Campos J, Carlile S, Carpenter M, Grimm G, Hohmann V,
Holube I, Launer S, Lunner T, Mehra R, Rapport F, Slaney M, Smeds K. The quest for ecological validity
in hearing science: What it is, why it matters, and how to advance it. Ear and Hearing. 2020; 41(Suppl.
1): 5S-19S.
7. Smeds K, Larsson J, Dahlquist M, Wolters F, Herrlin P. Live evaluation of auditory preference, alaboratory test for evaluating auditory preference. J Am Acad Audiol 2021; 32:487-500.
8. Hohmann V, Paluch R, Krueger M, Meis M, Grimm G. The virtual reality lab: Realization and applicationof virtual sound environments. Ear Hear 2020; 41 (Suppl. 1): 31S-38S.
9. Wolters F, Smeds K, Schmidt E, Christensen EK, Norup C. Common Sound Scenarios: A context-driven
categorization of everyday sound environments for application in hearing-device research. J Am Acad
Audiol 2016; 27:527-540.
10. Wu Y-H, Stangl E, Chipara O, Hasan SS, Welhaven A, Oleson J. Characteristics of real-world signal tonoise ratios and speech listening situations of older adults with mild to moderate hearing loss. Ear andHearing 2018; 39:293-304.
11. McGarrigle R, Munro KJ, Dawes P, Stewart AJ, Moore DR, Barry JB, Amitay S. Listening effort andfatigue: What exactly are we measuring? A British Society of Audiology Cognition in Hearing Special
Interest Group ‘white paper’. International Journal of Audiology 2014; 53(7), 433-440.
12. Klinik KB, Schulte M, Meis M. Measuring listening effort in the field of audiology – A literature review
of methods, part 1. Zeitschrift für Audiologie 2012; 51(2):60-67.
13. Klinik KB, Schulte M, Meis M. Measuring listening effort in the field of audiology – A literature review
of methods, part 2. Zeitschrift für Audiologie 212; 51(3), 96-105.
14. Dhami MK, Hertwig R, Hoffrage U. The role of representative design in an ecological approach tocognition. Psychol Bull 2004; 130:959-988.
15. Welling J, Fischer R-L, Schinkel-Bielefeld N. Is it possible to identify careless responses with post-hoc
analysis in EMA studies? In: Adjunct Proceedings of the 29 th ACM Conference on User Modeling,
Adaptation and Personalization (UMAP ’21 Adjunct), June 21-25, 2021, Utrecht, Netherlands. ACM,
New York, NY, USA.
16. Pohlhausen J, Holube I, Bitzer J. Near-ear sound pressure level distribution in everyday life considering
the user’s own voice and privacy. Acta Acustica (in review).
17. Baker R, Hazan V. DiapixUK task materials for the elicitation of multiple spontaneous speech dialogs.
Behav Res 2022; 43:761-770.
18. von Gablenz P, Kowalk U, Bitzer J, Meis M, Holube I. Individual hearing aid benefit: Ecologicalmomentary assessment of hearing abilities. In: Kressner A, Regev J, C.-Dalsgaard J, Tranebjaerg L,
Santurette S, Dau T, editors. Proceedings of the International Symposium on Auditory and AudiologicalResearch. Vol. 7: Auditory Learning in Biological and Artificial Systems. The Danavox Jubilee
Foundation; 2020. p. 213-220.
19. von Gablenz P, Kowalk U, Bitzer J, Meis M, Holube I. Individual hearing aid benefit in real life evaluatedusing ecological momentary assessment. Trends in Hearing. 2021; 25:1-18.
20. Benson RW, Hirsh IJ. Some variables in audio spectrometry. J Acoust Soc Am 1953; 25(3):499-505.
21. Pearsons KS, Bennett RL, Fidell SA. Speech levels in various noise envrionments. Office of Health andEcological Effects, Office of Research and Development, US EPA. 1977.
22. Smeds K, Wolters F, Rung M. Estimation of signal-to-noise ratios in realistic sound scenarios. J Am Acad
Audiol 2015; 26(2):183-196.
23. Weisser A, Buchholz JM. Conversational speech levels and signal-to-noise ratios in realistic acoustic
conditions. J Acoust Soc Am 2019; 145(1):349-360.
24. Kollmeier B, Warzybok A, Hochmuth S, Zokoll MA, Uslar V, Brand T, Wagener KC. The multilingual
matrix test: Principles, applications, and comparison across languages: A review. International Journal

of Audiology 2015; 54:sup2, 3-16.
25. Krueger M, Schulte M, Zokoll MA, Wagener KC, Meis M, Brand T, Holube I. Relation between listening
effort and speech intelligibility in noise. American Journal of Audiology 2017; 26:378-392.
26. Heeren J, Nuesse T, Latzel M, Holube I, Hohmann V, Wagener KC, Schulte M. The Concurrent OLSA
test: A method for speech recognition in multi-talker situations at fixed SNR. Trends in Hearing 2022;
26:1-12.
27. Holube I, von Gablenz P, Kissner S, Bitzer J. Ecological Momentary Assessment reveals the relation
between listening effort, demand, and motivation. (submitted)
28. Holube I, Haeder K, Imbery C, Weber R. Subjective listening effort and electrodermal activity inlistening situations with reverberation and noise. Trends in Hearing 2016; 20:1-15.

ABS-0227

Spatial audio-visual congruency effects in virtual reality environments
Cosima A. ERMERT(1) ; Jonathan EHRET(2) ; Torsten W. KUHLEN(2) ; Chinthusa MOHANATHASAN(3) ; Sabine J.
SCHLITTMEIER(3) ; and Janina FELS(1)(1) Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany,{cosima.ermert, janina.fels}@akustik.rwth-aachen.de(2) Visual Computing Institute, RWTH Aachen University, Germany, {ehret, kuhlen}@vr.rwth-aachen.de(3) Work and Engineering Psychology, Institute of Psychology, RWTH Aachen University, Germany,{chinthusa.mohanathasan, sabine.schlittmeier}@psych.rwth-aachen.de

ABSTRACT
Virtual reality (VR) is becoming increasingly important in psychoacoustic research as it allows for controllable
and reproducible experimental environments. For the perceived plausibility of virtual environments, the congruency of acoustic and visual stimuli is of high relevance. However, auralization of virtual sound sources canintroduce localization inaccuracies that induce a spatial mismatch between the visual and audio signals, e.g., byusing simplified spatial sound. This, in turn, could increase participants’ cognitive load, which could be a disadvantage for listening research in audiovisual VR environments compared to conventional laboratory settings.
With the aid of the well-established serial recall paradigm, this work investigated to what extent such spatialcongruency effects affect verbal short-term memory for heard speech depending on the angle of mismatch between acoustic and visual stimuli. Furthermore, we tested if the participant’s performance varied when the task
was done in a standard setup with a 2D computer screen presentation or in a virtual 3D scene presented on ahead-mounted display (HMD). In our experiment (n = 23) we found significant differences in the reaction time
between the screen and the virtual reality (VR) environment as well as in dependence of audio-visual spatialcongruency.

Keywords: Short-Term Memory, Serial Recall, Audio-Visual, Spatial Congruency, Virtual Reality

1

INTRODUCTION

In everyday human interaction, verbal communication is of high importance. Although verbal information is
primarily received in the auditory domain, corresponding visual cues like lip movements can affect auditory
perception and help language understanding [1]. This is especially important in challenging listening situations
with background noise or distracting talkers. The process of linking information from multiple sensory modalities (e.g., visual, auditory) is referred to as multisensory integration. More precisely, the integration of visualand auditory stimuli is called audio-visual integration. Multimodal presentation of stimuli has been shown tobe beneficial for, e.g., short-term memory performance [2]. When integrating audio-visual information, humans
can tolerate a certain degree of incongruency, e.g., in a video call when a poor internet connection causes a
temporal difference between the audio and video channel. Research by Slutsky et al. [3] showed that audiovisual stimuli with a discrepancy larger than 150 ms are no longer perceived as a unit. Chen and Vroomen [4]report a temporal just noticeable difference (JND) of ±200 ms. Similar effects can be observed in the spatial
domain: the spatial ventriloquism effect describes the tendency to perceive spatially separated audio and video
stimuli as a unit, given the separation is small enough. Generally, such discrepancies should not exceed 5◦ [3].
However, in other studies, visual and auditory stimuli were perceived as one unit for angle discrepancies aslarge as ϕ = 30◦ [5].
In recent years, VR is applied increasingly in research as it allows for controllable and reproducible close-to-real-life experimental environments. So far, mostly traditional setups with computer screens have been usedin cognitive psychology research (cf. [6]). Since such artificial scenarios differ from those in everyday life, it isalso possible that different cognitive processes are challenged in reality [7].
3D visual environments are oftentimes accompanied by a virtual acoustic environment to provide coherent spatial sound and thus enhance immersion. However, individualization of spatial cues, e.g., by measuring
an individual head-related transfer function (HRTF), is a time-consuming and expensive procedure. Thus, especially for commercial applications and for research groups with no access to elaborate spatial reproduction
methods, simplified spatial sound is used which could result in a localization blur [8] and, in turn, in perceived audio-visual incongruency. As short-term memory performance benefits from audio-visual integration,such uncertainties could increase the difficulty of remembering heard speech.
This study aims to investigate if short-term memory for heard speech is influenced (1) by audio-visual angle
discrepancies and (2) by the visual presentation environment, i.e., computer screen or VR scene on an headmounted display (HMD). For this purpose, a listening experiment was implemented. To measure short-term
memory performance for heard material, the auditory verbal serial recall (aVSR) paradigm was used [9]. Here,the participants hear a sequence of verbal items and are asked to recall them in the presented order after ashort retention interval. The listening experiment was performed both in a VR environment and a traditional
laboratory setup with a computer screen. Furthermore, audio-visual integration was challenged by introducing
angle discrepancies.

2

LISTENING EXPERIMENT DESIGN

2.1 Visual environment
With the technical developments in recent years, VR environments are increasingly applied in cognitive psychology research. In contrast to traditional experimental setups with computer screens, VR enables the simulationof close-to-real-life scenarios. One goal of this work was to investigate whether short-term memory for heardmaterial is influenced by different modes of visual presentation, i.e., traditional computer setups or VR. For
that purpose, two visual conditions were introduced to the aVSR: the visual stimuli were either presented on acomputer screen or an HMD.
As a plausible visual cue and to simulate a realistic conversational situation, a female virtual human was
generated with MetaHuman Creator. To determine the voice speaking the auditory stimuli which matches thevirtual human face best, a prestudy was carried out using the recordings from Oberem and Fels [10]. This wasdone to reduce the influence of a content mismatch in the audio-visual stimuli on the participants’ performance.
The prestudy was generated using SoSci Survey 3.3.11 [11]. The 57 participants (18 male, 36 female, 3 divers)
aged between 22 and 65 years (M = 35.0 years, SD = 8.6 years) rated the voice female voice ’fb’ from thedatabase as most pleasant and fitting. To obtain a lively appearance, an idle animation and lip-syncing via
Oculus Lipsync were added to the virtual human for the main experiment.
In the 2D computer screen environment (hereafter referred to as Screen Environment), the face of the virtualhuman was presented on a Fujitsu B24T-7 screen with a 24" diagonal and a resolution of 1920 × 1080 pixels
placed on a desk in front of the participant. The 3D environment (hereafter referred to as VR Environment) was
visualized using an HTC Vive Pro Eye HMD. Here, the virtual human was shown to be seated on a chair infront of the listener in such a way that the height and distance at which the face was positioned stayed constantin between environments.
2.2 Audio-visual angle discrepancies
The visual target was positioned at a fixed position in front of the listener at an azimuth angle of ϕ = 0◦ . Fourabsolute azimuth angles were chosen for the audio target position. The selection of audio-visual mismatch angles should include cases small enough so that the visual and auditory stimuli are still expected to be perceived
as a unit and cases large enough so that an audio-visual mismatch is obvious. Research on the ventriloquism
effect showed that a visual and auditory stimulus can be perceived as a unit for angle discrepancies as large as
ϕ = 30◦ [5]. To cover this interval, |∆ϕ| = 15◦ and |∆ϕ| = 30◦ were selected. As an obviously mismatching casean angle deviation of |∆ϕ| = 60◦ was chosen, following the work by Fintor et al. [12], where a conversation

Figure 1. One trial in the implementation of the aVSR in Unreal Engine consists of four steps: countdown,
audio-visual presentation of stimuli, retention period, and recall period.
between two talkers at ϕ = ±60◦ is investigated. Lastly, as a baseline condition, |∆ϕ| = 0◦ was included, where
no audio-visual angle discrepancy occurs. As in [12], the visual and all auditory sources were positioned at a
distance of 2.5 m from the participants’ head center.
2.3 Paradigm
The well-established auditory verbal serial recall paradigm was used in this study to investigate short-term
memory performance. The study was implemented using Unreal Engine 4.27 based on the aVSR code by
Schlittmeier et al. [6]. One experimental trial is displayed in Figure 1. Each experimental trial started with acountdown, where three rectangles decreasing in size were presented visually, with an interval of 0.5 s between
appearances. Subsequently, an arbitrary digit sequence was chosen from a predefined set. Each digit sequence
consisted of 8 elements, namely the digits from 1 to 9, with each digit only appearing once in a sequence.
Obvious sequences like 1-2-3-4-5-6-7-8 were excluded from the predefined set. The serial positions of thedigits were approximately balanced over all sequences. Every number sequence was only presented once toeach participant.
The digits were presented auditorily at a rate of one digit per second. Simultaneously, the virtual human withmatching lip-synchronization was shown as the visual stimulus. After a retention period of 3 s, the presented
digits were arranged on the screen in a randomized order in a matrix. The participants were then asked toreconstruct the remembered presentation order by clicking the respective digits. There was no time limit for thenumber reconstruction task. No corrections were possible, as the digits disappeared upon clicking. Afterwards,
the next trial began.
2.4 Auralization
The target signal consisted of recordings of the digits 1 to 9 in German of the female voice ’fb’ by Oberem
and Fels [10]. No recording of the digit 5 was included in the publication but was available as a raw recording
and post-processed accordingly. To avoid a visual fixation on the potential source positions, it was decided
against loudspeakers when choosing the audio reproduction method. Instead, the source sources were realized
as virtual sources using Virtual Acoustics (VA) [13], version 2020a. The signals were auralized in a freefield environment with no room acoustics. To ensure plausible spatial cues, a generic HRTF of the Institutefor Hearing Technology and Acoustics (IHTA) artificial head [14] with a 5◦ × 5◦ resolution was used, which
provides sufficiently precise localization in dynamic scenes for most listeners as shown by Oberem et al. [15].
The target signal was played back via Sennheiser HD650 headphones in both visual environments. Headphoneequalization was performed by measuring an individual headphone-related transfer function (HpTF) for eachparticipant with the method proposed by Masiero and Fels [16]. The target signal was calibrated to 60 dB(A)at the ear canal entrance with the IHTA artificial head (Note: the IHTA head is not equipped with an ear canalsimulator).
To enable head movements while creating the perception of a fixed sound source position, head tracking was

employed. The position of the sound receiver’s head center was continuously updated in VA. Different tracking
systems were chosen for the two visual environments. In the HMD environment, the tracking system provided
by HTC was used to both update the visual position of the virtual pawn in Unreal and, via the VA plugin
for Unreal, the position of the sound receiver in the VA environment. In the Screen Environment, this tracking
system could not be used, as the participants did not wear the HMD. Thus, a tracking body was attached tothe headphones which could be tracked with OptiTrack infrared cameras in Motive 2.1.1. Positional data wasstreamed to VA with the OptiTrack Unreal Engine 4 plugin 2.27 and the VA plugin for Unreal.
2.5 Participants
In the scope of this study, 23 adults (7 female, 17 male, 1 divers) aged between 19 and 36 years (M = 26.16
years, SD = 4.28 years) were recruited amongst students and assistants at the IHTA and via the institute’s
listening experiment online booking system. Participants had to be native German speakers (self-reported) toparticipate. To ensure normal hearing below 25 dBHL, an audiometry up to 14 kHz was performed prior to
the listening experiment. Furthermore, participants had to pass a Snellen Test (20/30) [17] to examine theireyesight. Participants with corrected-to-normal vision were admitted if their visual aid could be worn under the
HMD without discomfort. All participants received compensation in the form of sweets and a 10 e voucher fora local bookstore.
2.6 Room
The experiment was carried out in the virtual reality laboratory (VRLab) of the IHTA at the RWTH Aachen
University in Germany (l × w × h = 5 m × 8 m × 2.8 m). The walls are covered with curtains, which enclose aquadratic ground plane of 5 m × 5 m for the experimental area [18]. To ensure congruency between the Screen
Environment, where the visual target is only visible on the PC screen, and the VR Environment, where theparticipant is fully immersed in a virtual room, a 3D model of VRLab was generated in Unreal Engine 4.27.
While in the Screen Environment the model served as the background for the virtual human, it surrounded theparticipants as a virtual environment in the VR Environment.
2.7 Procedure
The participants completed all trials for one visual environment (Screen or VR) before moving on to the next
environment. The start environment was balanced over participants. In each environment, written instructions
for the task were displayed on white background. After that, all participants went through a training block
consisting of eight trials (two per audio-visual angle discrepancy condition) to familiarize themselves with therespective controller and the task itself. Participants could ask questions and take a break afterwards. In themain block, each audio-visual angle discrepancy condition was tested 12 times, resulting in a total of 48 main
trials per environment. The trials were split up into three blocks of 16 trials each (four per audio-visual angle
discrepancy condition) to avoid fatigue effects. The order of the angle discrepancy conditions was randomized
within each block. After each main block and before switching the visual environment, the participants had thechance to take a break of no fixed length. The duration of the whole experiment was ca. 60 min long.

3

RESULTS AND DISCUSSION

The results are evaluated with regard to two dependent variables: the reaction time and the performance accuracy in terms of the proportion of correctly recalled digits per trial. The reaction time is defined as the time
passed between two button clicks (or the time between the start of the feedback period and the first button
click for serial position one) and is measured in seconds. The performance accuracy is computed as the sumof digits recalled at the correct serial position in one trial divided by the total number of digits per trial, here
eight, and thus ranged between 0 (no digits recalled correctly) and 1 (all eight digits recalled correctly). All
statistic evaluations were performed in IBM SPSS Statistics, version 28.0.1.1.
A 2 × 4 repeated measures analysis of variance (ANOVA) was conducted for both the reaction time andperformance accuracy, with the visual environment (Screen, VR) and the audio-visual angle discrepancy |∆ϕ|(0◦ , 15◦ , 30◦ , 60◦ ) as the two within-subjects factors. Prior, data cleaning was performed by excluding trials

2.5

∗∗

0.8

2.0

0.6

1.5

0.4

1.0

0.2

0.5
Screen Environment

Reaction time [s]Performance accuracy

1.0

VR Environment

Figure 2. Distribution of performance accuracy and reaction time between the Screen, the VR Environment, andboth environments (– median, ◦ mean). The reaction time between environments is significantly different.
that exceeded the range of M ± 2 · SD per participant and factor combination.
Performance accuracy values were normally distributed except for |∆ϕ| = 60◦ and |∆ϕ| = 30◦ in the Screen
Environment, as assessed by the Shapiro-Wilk test with α = .026 and α = .019, respectively. For the reaction
time, normality violations were found only for |∆ϕ| = 60◦ in the VR Environment (α = .009). According toBerkovits et al. [19], the ANOVA is robust against these violations if the requirement for sphericity is met.
This was confirmed with the Mauchly test. Thus, the ANOVA results could be interpreted.
3.1 Effect of visual environment
In Figure 2, the distribution of the performance accuracy and reaction time data are displayed for the two visualenvironments Screen and VR. No significant differences in performance accuracy could be found between theenvironments, F(1, 22) = 0.139, p = .713. This leads to the assumption that the performance accuracy of theparticipants was not influenced by the visual environment they had to perform the task in. For the reaction

Reaction time difference [s]

0.5
0.4
0.3
0.2
0.1
0
-0.1

1

2

3

4

5

6

7

8

Serial position
Figure 3. Difference in reaction time between the VR and the Screen Environment in seconds over the serial
position ± the standard error.

time a significant difference between the environments could be found, F(1, 22) = 9.018, p = .007, with alarger reaction time in the VR Environment (M = 1.029 s, SD = 0.286 s) compared to the Screen Environment(M = 1.135 s, SD = 0.404 s). This result can potentially be explained by the handling of the controllers. Whilein the Screen Environment a mouse was used, the digit selection VR Environment was done with a VR controller,
which requires more physical and time-consuming movement by the participants.
Especially at the beginning of the feedback phase before selecting the first digit, the VR controller has to be
moved from the resting position to the number pad, while the mouse cursor naturally stays within the area ofthe computer screen. Thus, the largest difference in reaction times between the environments is to be expected
at the first serial position. As visualized in Figure 3, this is the case. Paired t-tests of the reaction time between
the VR and Screen Environment for each serial position showed significantly lower reaction times in the Screen
Environment for the first serial position, t(22) = 3.579, p = .002. No other serial position did yield significant
differences in reaction time.
3.2 Effect of audio-visual angle discrepancies
The course of the performance accuracy over the audio-visual angle discrepancies |∆ϕ| is displayed in Figure 4.
The angles were chosen to cover both audio-visual spatial congruent cases (|∆ϕ| = 0◦ ), cases in which audiovisual spatial integration has been shown in literature (|∆ϕ| = 0◦ , 30◦ ), and cases where a spatial separation
between the audio and visual stimulus is obvious (|∆ϕ| = 60◦ ). From the choice of the angles, a generaldecrease in performance was expected for angles where audio-visual discrepancies become obvious. However,
the results of the ANOVA did not show a significant influence of the angle discrepancies on the performanceaccuracy, F(3, 66) = 0.869, p = .462. There are multiple possible explanations for this finding. Either no audiovisual integration was necessary to successfully complete the task, as all the information was contained in theauditory stream while the visual stream (lip movements) only offered repetitive information. Or, audio-visual
integration did happen but the chosen angle discrepancies did not influence (or only to a degree not visible
with n = 23) the results. It is also possible that the face and the lip-movement of the virtual human, which
was generated artificially, were not close enough to reality and thus irritate the participants. Although notsignificant, the graphs show an increase in performance accuracy for larger angular differences, indicating better
performance for larger separation between the audio and video stream. These tendencies should be investigatedfurther in future research.
For the reaction time, a significant influence of the audio-visual angle discrepancies could be found, F(3, 66) =Performance accuracy

0.85

0.80

0.75
0◦

15◦

30◦

60◦

Angle discrepancy |∆ϕ|Screen Environment

Both Environments

VR Environment

Figure 4. Proportion of correct recalled digits per trial ± standard error for the Screen, the VR Environment,
and both environments over the audiovisual angle discrepancies |∆ϕ|.

Reaction time [s]

1.2

1.1

1.0

0.9

0◦

15◦

30◦

60◦

Angle discrepancy |∆ϕ|Screen Environment

Both Environments

VR Environment

Figure 5. Reaction time between digits ± standard error for the Screen and the VR Environment over theaudiovisual angle discrepancies |∆ϕ|.
2.833, p = .045 (cf. Figure 5). However, Bonferroni post-hoc tests did not identify two angles leading to significantly different reaction times. On a descriptive level, the reaction time increases from |∆ϕ| = 0◦ to 30◦ anddecreases again towards a minimum at |∆ϕ| = 60◦ . This leads to the assumption that less reflection time wasneeded in the case of stronger stream segregation. More research is necessary to investigate this tendency.
Finally, the interaction between the visual environments and audio-visual angle discrepancies was investigated. No interaction could be found for the reaction time (F(3, 66) = 0.688, p = .562) or the performanceaccuracy (F(3, 66) = 0.700, p = .555).

4

CONCLUSION

The results of the listening experiment revealed that the performance accuracy did not differ significantly between the Screen and the VR Environment. The reaction time differed significantly, presumably due to thedifferent physical effort required to operate the different controllers (computer mouse, VR controller). No significant influence of the audio-visual angle discrepancies on the performance accuracy was found. The reaction
time was influenced significantly by the angle discrepancy in general. However, post-hoc tests did not reveal apair of angles that showed a significant difference. Overall, the findings are qualified by several limitations thatsuggest new directions for future research.

ACKNOWLEDGMENTS
This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation): SPP2236- Projectnumber 444724862; Listening to, and remembering conversations between two talkers: Cognitive research using embodied conversational agents in audiovisual virtual environments. The authors would like toexpress their very great appreciation to Alissa A. Wenzel for assistance in the implementation and conductionof the experiment. Furthermore, the authors would like to thank Karin Loh, Lukas Vollmer, Pascal Palenda,
Carolin Reimers, Manuj Yadav, and Oliver Renaldi for their help and fruitful discussions.

REFERENCES[1] Gonzalez-Franco M, Maselli A, Florencio D, Smolyanskiy N, Zhang Z. Concurrent talking in immersive
virtual reality: on the dominance of visual speech cues. Scientific Reports. 2017;7(1).

[2] Quak M, London RE, Talsma D. A Multisensory Perspective of Working Memory. Frontiers in Human
Neuroscience. 2015;9.
[3] Slutsky DA, Recanzone GH. Temporal and spatial dependency of the ventriloquism effect. NeuroReport.
2001;12(1):7–10.
[4] Chen L, Vroomen J. Intersensory binding across space and time: A tutorial review. Attention, Perception,
and Psychophysics. 2013;75(5):790–811.
[5] Thurlow WR, Jack CE. Certain Determinants of the “Ventriloquism Effect.” Perceptual and Motor Skills.
1973 Jun;36:1171–84.
[6] Schlittmeier SJ, Mohanathasan C, Fintor E. Paradigm for Measuring Verbal Short-Term Memory Capacity
for Auditorily or Visually Presented Items: Verbal Serial Recall Task. RWTH Publications. 2021:RWTH2021-09604.
[7] Fels J, Ermert CA, Ehret J, Mohanathasan C, Bönsch A, Kuhlen TW, Schlittmeier SJ. Listening to, andremembering conversations between two talkers: Cognitive research using embodied conversational agentsin audiovisual virtual environments. In: 47. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik
e.V. (DEGA); 2021. p. 1328-31.
[8] Murphy D, Neff F. Spatial Sound for Computer Games and Virtual Reality. In: Game Sound Technology
and Player Interaction: Concepts and Developments. IGI Global; 2011. p. 287-312.
[9] Hurlstone MJ. Serial Recall. Oxford University Press; 2021 (in press).
[10] Oberem J, Fels J. Speech Material for a Paradigm on the Intentional Switching of Auditory Selective
Attention; 2020:RWTH-2020-02105.
[11] Leiner DJ. SoSci Survey [Computer software]. 2019. Available from https://www.soscisurvey.de[12] Fintor E, Aspöck L, Fels J, Schlittmeier SJ. The Role of Spatial Separation of Two Talkers’ Auditory Stimuli in the Listener’s Memory of Running Speech: Listening Effort in a Non-Noisy Conversational Setting.
International Journal of Audiology. 2022 May;61(5):371-9.
[13] Institute for Hearing Technology and Acoustics (IHTA), RWTH Aachen University. Virtual Acoustics– A Real-Time Auralization Framework for Scientific Research [Computer Software]. Avaliable fromhttp://www.virtualacoustics.org/[14] Schmitz A. Ein digitales Kunstkopfsystem. In: 21. Jahrestagung für Akustik. Deutsche Gesellschaft für
Akustik e.V. (DEGA); 1995. p. 771-4.
[15] Oberem J, Setzer D, Koch I, Richter JG, Seibold JC, Fels J. Experiments on localization accuracy with
non-individual and individual HRTFs comparing static and dynamic reproduction methods. In: 44. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2018. p. 702-5.
[16] Masiero B, Fels J. Perceptually Robust Headphone Equalization for Binaural Reproduction. 2011.
[17] Snellen H. Probebuchstaben zur Bestimmung der Sehschärfe. H. Peters; 1873.
[18] Kohnen M, Vorländer M. Virtual Scene Adaption for Compensation of the Reproduction Room. In: NoiseControl for a Better Environment: Proceedings of INTERNOISE 2019: The 48th International Congress
and Exhibition on Noise Control Engineering. Madrid, Spain: Sociedad Española de Acústica; 2019. p.
2995-3992.
[19] Berkovits I, Hancock GR, Nevitt J. Bootstrap Resampling Approaches for Repeated Measure Designs:
Relative Robustness to Sphericity and Normality Violations. Educational and Psychological Measurement.
2000;60(6):877-92.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0311

Comparison of speech intelligibility in a real and virtual living room
Julia SCHÜTZE1; Stephan D. EWERT1; Christoph KIRSCH1; Birger KOLLMEIER1
1

Medical Physics and Cluster of Excellence Hearing 4 All, Carl von Ossietzky Universität Oldenburg, Germany

ABSTRACT
A mismatch between the outcomes of “classical” audiological tests and the perceived benefits of aided
hearing impaired patients leads to an interest in more ecologically valid testing methods, better reflecting thepatients’ performance in real-world situations.
In daily-life, a highly relevant indoor (home) environment is the living room in which listening and speechcommunication typically involve different target or interferer sources on a sofa or chair, a TV set, or in an
adjacent room connected with an open door.
This study compares speech intelligibility in a controllable laboratory environment closely resembling an
average German living room with an adjacent kitchen and in the according virtual representations. Speech
recognition thresholds of normal hearing listeners in the real-world living room lab and in three different
acoustic reproductions were compared using measured and simulated room impulse responses: binaural
presentation via headphones, a small-scale 4-channel loudspeaker array, and a 3-dimensional 86-channel
loudspeaker array in an anechoic environment. Target and interferer positions were permuted over four
different positions in the living room lab, including an acoustically challenging position in the adjacent
kitchen room without line of sight. Implications are discussed in the context of the reproductions and potential
applications with hearing impaired listeners.
Keywords: virtual acoustics, ecological validity, speech intelligibility

1. INTRODUCTION
Moving challenging daily-life acoustic situations into the lab is a step in hearing research towards
ecologically valid testing (1). Typically, a mismatch between proposed benefit of hearing aids andexperienced benefit by the hearing aid wearer in real-life situations is observed (e.g., 2). A typicalacoustic communication environment at home for hearing-impaired and normal hearing people is theliving room (3, 4). Virtual acoustic environments offer the potential of more ecologically valid testing,enabling controlled, lab-based reproduction of existing real-world environments and other realistic
complex acoustic environments.
Speech intelligibility has been measured in several virtual environments: in a virtual undergroundstation with measured and modeled impulse responses auralized via 60 loudspeakers (5), in an officemeeting scenario reproduced using higher-order ambisonics (HOA) recordings, rendered on a 64
channel spherical loudspeaker array (6), and in a simulated reverberant cafeteria in a 3-D loudspeakerarray with 41 equalized loudspeakers (7). Virtual sound environments auralized via a spherical arrayof 29 loudspeakers using Ambisonics or nearest loudspeaker method have been validated by acomparison to the real environment (8). A comparison of speech intelligibility in a real room comparedwith speech intelligibility in virtual reproduction, has been performed for a reverberant reference room
(9).
In the current study, we focus on a direct comparison of a real living room environment, establishedas “living room lab” in the university building and simulated acoustic reproductions. Once thesimulation is verified, it offers the advantage that arbitrary conditions can be generated forpsychoacoustical testing as well as for large datasets, e.g., for machine learning. Speech recognition
thresholds (SRTs) were measured with loudspeakers as sources in the real room - the ground truth –and were compared to SRTs measured with simulated acoustic reproductions: Besides binaural
reproductions using headphones, a minimum, small-scale 4-channel horizontal loudspeaker array in alistening booth was used as well as a reproduction with a spherical 86-channel loudspeaker array in1julia.schuetze@uni-oldenburg.de

an anechoic chamber. The small-scale loudspeaker array in the listening booth resembles a
reproduction system typically available at a commercial hearing aid distributor or in a clinical context.
The large scale system offers more optimal 3-dimensional reproduction without undesired acoustic
effects of the reproduction room itself. Both loudspeaker reproduction systems enable the listener towear hearing aids during the measurement and allow for natural head movements.

2. METHODS
2.1 Listeners
Five normal hearing German native speakers (four females, one male; aged 22-28 years; medianage of 25 years) participated in the experiments. They had hearing thresholds for pure tones better
than 20 dB HL at the frequencies 125, 250, 500, 750, 1000, 1500, 2000, 3000, 4000, 6000 and 8000
Hz, except for one listener with a hearing threshold of 30 dB HL at 500 Hz and 25 dB HL at 750 Hz.
All listeners received an hourly compensation for their participation in the experiment.
2.2 Acoustic environment and conditions
A typical living room and its virtual acoustic reproduction were used in the measurements (seeFig. 1).
The living room was built and designed as a lab environment at the University of Oldenburg toclosely resemble a typical German living room with an adjacent smaller (kitchen) room, connected bya door. In the living room lab, a seating group, a sofa table, a carpet, and a TV on a TV board arepresent among other furniture items. The flooring is laminate.

Figure 1 – Living room lab at the university Oldenburg with indicated target and receiver positions
The living room has the dimensions 4.97 m x 3.78 m x 2.71 m (width x length x height), while the
kitchen has the dimensions 4.97 m x 2.00 m x 2.71 m. The overall volume of booth rooms is 77.85 m³.
The receiver is positioned at R1, seated on the couch. The masker has a fixed position at S4. The
target is presented from one of three target positions: in the coupled room without a direct line of sight(S7), to the right of the receiver on a chair (S5), and directly in front of the receiver (S-TV). For thetarget and masker, Genelec 8030 loudspeakers were located at the respective positions (10, 11). The
upper row in Table 1 shows the reverberation time (specified as T30) of the living room with open
door to the connected kitchen.
Table 1 – Reverberation time T30 in the living room (door open) in seconds:
125 HzMeasured
Simulated

( )

( )

250 Hz 500 Hz

1000 Hz2000 Hz

4000 Hz

8000 Hz

0.49

0.5

0.49

0.53

0.54

0.47

0.38

0.47

0.45

0.48

0.48

0.48

0.46

0.34

2.3 Apparatus and procedure
Speech intelligibility (50% speech recognition threshold, SRT) was measured using the AFC
framework (12) with the OLSA matrix sentence test (13) in the presence of an interfering nonsense
speech talker using a male-transformed version (14) of the international speech test signal (ISTS; 15).
While the masker level was fixed, the level of the target was varied depending on the number ofcorrectly identified words. The OLSA was presented as an open test, and listeners repeated theunderstood words or sentences.
The target S-TV was calibrated to 63 dB SPL at the receiver position using a stationary masking
noise with the same spectrum as the sentences of the OLSA matrix test.
Speech perception was compared in the real world living room lab (LRL) and in three different
acoustic reproductions (see Table 2). The “ground truth” setup in the real living room consisted of 4
loudspeakers (from which 2 were active in each measurement) at the positions S-TV, S7, S5 and S4.
Here, the listener was seated at the receiver position R1 on the sofa. Seated in the same position, thelisteners were additionally presented with the simulation via binaural rendering on headphones (HP).
The second setup was a minimum (small-scale), horizontal 4-channel loudspeaker array (45°, 135°,225°, 315°) with a radius of 1.0 m located in a listening booth. Additionally, the simulation waspresented to the listeners via headphones in the listening booth. The third setup was a 3-dimensional
spherical 86-channel loudspeaker array in an anechoic chamber. Again, the simulation was
additionally presented via headphones.
Simulated binaural room impulse responses (BRIRs) and loudspeaker renderings were obtainedusing the room acoustic simulator RAZR (16, 17). Here, early reflections are calculated using animage source model and the reverberant tail is based on a feedback delay network.
For measurements with headphones, Sennheiser HD 650 was used, while setups with loudspeakers
used Genelec 8030 CP. In the measurement in the listening booth, a RME Fireface UCX, in the livingroom a Fireface UFX, and in the VR Lab a Madiface XT audio interface was used.
All listeners visited the living room for familiarization prior to starting the measurements.
Table 2 – Overview measurement setups
Lab
Listening booth

presentation mode

renderingabbreviation

Horizontal 4-channel Loudspeaker arraysimulated

LB 4ch LS

Headphones

LB HP

Headphones

simulated
real
acoustics
simulated

LRL HP

3-D 86-channel Loudspeaker arraysimulated

VR 86ch LS

Headphones

simulated

VR HP

2 Loudspeakers active

Living Room Lab

Virtual reality Lab

LRLRESULTS AND DISCUSSION
Figure 2 shows the SRTs in dB SNR of all presentation methods for the two target positions in theliving room (STV, S5) and connected kitchen (S7) in the different panels (left to right; triangles, dots,
and squares, respectively). Different acoustic presentation modes are indicated at the x-axis and color
coded. It is obvious that SRTs are considerably higher for target S7, for which no direct line of sight(or sound path) exists, thus involving sound diffraction through the door opening.
A two-way repeated-measures ANOVA (target position x presentation mode) showed a significant
main effect of target position [F(2,8) >100 , p < 0.001], a significant effect of presentation mode
[F(5,20) = 3.823, p = 0.014], and a significant interaction [F(10,40) = 2.173, p = 0.040].
For presentation mode, post-hoc pairwise comparisons with Bonferroni correction only showed asignificant difference (p=0.036) between headphone simulation in the living room lab (LRL HP) andthe reference living room lab condition (LRL).
For target position, post-hoc pairwise comparisons with Bonferroni correction showed a highly
significant difference (p < 0.001) between S-TV and S5, S-TV and S7, as well as S5 and S7. The

difference of S5 and S7, is however, considerably smaller. Lower SRTs in S5 (as observed for thesimulations) are in line with expectations from better-ear listening, given that for S5 the target is at
the side opposed to the masker, while for S-TV the target is in the front, spatially less separated fromthe masker. Somewhat unexpected, a trend in the reversed direction is observed for the SRTs in theliving room lab (LRL, black symbols). However, the relatively small sample size of the listener groupand potential training effects should be noted.
There is a trend that SRTs for the living room lab are higher than for the simulated conditions fortargets S7 and S5. Here, uncontrollable additional background noise from outside might have had asmall effect (birds outside or a train passing by in the far distance).

Figure 2 – SRTs of all presentation methods for the three target positions STV, S5, S7 (left to right;
triangles, dots, and squares). In each panel, the loudspeaker setup in the living room lab is shown in black(LRL). Headphone and loudspeaker presentations with simulated RIRs are shown in red and in blue color
tones (LB HP – Listening booth headphones, 4-ch LS – booth 4-ch loudspeaker array, LRL HP – Livingroom headphones, VR HP – headphones in 86 channel array lab, VR LS – 86 channel array)

3. CONCLUSION
A lab environment resembling a typical living room with a connected (kitchen) room has been
established. The living room lab enables reproducible measurements in realistic everyday conditions.
Here, preliminary SRT measurements in different acoustic reproduction modes for three different
target positions with a fixed masker are presented. It was demonstrated that SRTs depend on the threedifferent target positions, with highly elevated SRTs for the target in the connected room, reflecting achallenging, realistic communication situation. Except for one condition with headphone reproduction,
no significant differences between SRTs in the real room and for room acoustics simulations of theliving room lab with RAZR were found. Accordingly, SRTs obtained with the minimum set of four
horizontally arranged loudspeakers in the listening booth well match the SRTs in the living room lab
and those for the large-scale loudspeaker array. This promising result needs to be validated with morelisteners, as this would offer the possibility to move challenging listening situations from the real

world to a small-scale loudspeaker setup in the lab.
Moreover, regarding visual cues, it would be interesting to repeat the measurement with a mixed
reality setup using a Head Mounted Display (HMD), while listening to the loudspeakers in the room.

ACKNOWLEDGEMENTS
This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) –Projektnummer 352015383 –SFB 1330 C4.

REFERENCES
1. Keidser G, Naylor G, Brungart DS, Caduff A, Campos J, Carlile S, u. a. The Quest for Ecological Validity
in Hearing Science: What It Is, Why It Matters, and How to Advance It. Ear & Hearing. 2020.
41(Supplement 1):5S-19S.
2. Cord MT, Surr RK, Walden BE, Dyrlund O. Relationship between Laboratory Measures of Directional
Advantage and Everyday Success with Directional Microphone Hearing Aids. J Am Acad Audiol. 2004.
15(05):353–64.
3. Wolters F, Smeds K, Schmidt E, Christensen EK, Norup C. Common Sound Scenarios: A Context-Driven
Categorization of Everyday Sound Environments for Application in Hearing-Device Research. J Am Acad
Audiol. 2016. 27(07):527–40.
4. Weisser A, Buchholz JM, Oreinos C, Badajoz-Davila J, Galloway J, Beechey T, u. a. The Ambisonic
Recordings of Typical Environments (ARTE) Database. Acta Acustica united with Acustica. 2019.
105(4):695–713.
5. Hládek Ľ, Ewert SD, Seeber BU. Communication Conditions in Virtual Acoustic Scenes in an
Underground Station. In: 2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA). 2021.
S. 1–8.
6. Mansour N, Marschall M, May T, Westermann A, Dau T. Speech intelligibility in a realistic virtual soundenvironment. The Journal of the Acoustical Society of America. 2021. 149(4):2791–801.
7. Best V, Keidser G, Buchholz JM, Freeston K. An examination of speech reception thresholds measured ina simulated reverberant cafeteria environment. International Journal of Audiology. 2015. 54(10):682–90.
8. Cubick J, Dau T. Validation of a Virtual Sound Environment System for Testing Hearing Aids. Acta
Acustica united with Acustica. 2016. 102(3):547–57.
9. Ahrens A, Marschall M, Dau T. Measuring and modeling speech intelligibility in real and loudspeakerbased virtual sound environments. Hearing Research. 2019. 377:307–17.
10. Schütze, Julia, Kirsch, Christoph, Wagener, Kirsten C., Kollmeier, Birger, & Ewert, Stephan D. Livingroom environment (1.1) [Data set]. Zenodo. 2021. https://doi.org/10.5281/zenodo.5747753
11. van de Par, S, Ewert, S D, Hladek, L, Kirsch, C, Schütze, J, Llorca-Bofí, J, Grimm, G, Hendrikse, M M
E, Kollmeier, B, Seeber, B U. Auditory-visual scenes for hearing research. 2022. Acta Acustica united
with Acustica.
12. Ewert SD. AFC—A modular framework for running psychoacoustic experiments and computational
perception models. In: Proceedings of the international conference on acoustics AIA-DAGA. 2013. S.
1326–9.
13. Wagener K, Brand T, Kollmeier B. Development and evaluation of a German sentence test part III:
Evaluation of the Oldenburg sentence test. Zeitschrift für Audiologie. 1999. 38:86–95.
14. Schubotz W, Brand T, Kollmeier B, Ewert S D. Monaural speech intelligibility and detection in maskers
with varying amounts of spectro-temporal speech features. The Journal of the Acoustical Society ofAmerica. 2016. 140(1):524.
15. Holube I, Fredelake S, Vlaming M, Kollmeier B. Development and analysis of an International Speech
Test Signal (ISTS). International Journal of Audiology. 2010;49(12):891–903.
16. Wendt T, van de Par S, Ewert S D. A Computationally-Efficient and Perceptually-Plausible Algorithm for
Binaural Room Impulse Response Simulation. J Audio Eng Soc. 2014. 62(11):748–66.
17. Kirsch, C, Wendt, T, van de Par, S, Hu, H, Ewert, S D. Computationally-Efficient Simulation of Late
Reverberation for Inhomogeneous Boundary Conditions and Coupled Rooms. J. Audio Eng. Soc. 2022.
In press.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0481

Comparing headphone- and loudspeaker-playback using spatial
scene analysis in virtual audio-visual reverberant environments
Axel AHRENS1,2; Tobias NEHER1; Torsten DAU2
1

Research unit of Oto Rhino Laryngology, Department of Clinical Research, University of Southern
Denmark, Denmark

2

Hearing Systems Section, Department of Health Technology, Technical University of Denmark, Denmark

ABSTRACTIn multi-talker situations, listeners face the challenge of having to identify a target speech source out of a
mixture of interfering sources. To test these realistic situations in the laboratory it is common to employ
arrays with many loudspeakers. Allowing clinics to use such experimental paradigms, headphone playback
can be an alternative. However, headphone reproduction often leads to different outcomes because of the lackof individual head-related transfer functions (HRTFs). Here, we investigated how normal-hearing listeners
analyze virtual audio-visual scenes that differed in terms of the number of concurrent talkers and the amountof reverberation using either head-tracked headphones using generic HRTFs or via a loudspeaker-array.
Listeners were asked to identify and locate an ongoing story in a mixture of other stories. The visualcomponents of the scenarios were reproduced via virtual reality headset. Differences between loudspeakerand headphone-based reproduction were evaluated using the response time and the localization accuracy. The
number of talkers as well as the amount of reverberation affected the ability to analyze an audio-visual scene,in terms of response time and localization error. Preliminary data suggest only slightly elevated response
times when employing headphone reproduction compared to loudspeakers, making them a valid tool forclinical research.
Keywords: Hearing, Speech, Virtual Reality

1. INTRODUCTION
When multiple talkers are present in a scene, listeners face the challenge to perceptually separate
them, to be able to understand the speech. In a previous study a method to analyze a listener’s ability
to analyze a scene consisting of multiple talkers has been presented [1]. The listeners’ task was to
identify and locate a spoken story in the presence of other stories. Within the experiment, the numberof simultaneous talkers (stories) and the amount of reverberation were varied. The acoustic
information was presented via a spherical 64-channel loudspeaker array (see [2] for details) and visual
information via virtual reality glasses. However, in clinical practice as well as in research suchloudspeaker array is often not available. Thus, the sound reproduction via headphones is desirable.
When employing headphones for the reproduction of realistic spatial audio, head-related transfer
functions (HRTFs) are needed. HRTFs reflect the acoustical properties of the human torso, head andpinna. While the use of individual HRTFs allow for an accurate spatial reproduction of a sound, themeasurement of individual HRTFs needs either a large loudspeaker setup or long time when using few
loudspeakers. Thus, using a generic HRTF that are representative of a number of listeners, for example
one measured on a dummy head is of advantage. However, using a non-individual HRTF has beenshown to affect sound perception (e.g., [3]).
The aim of the current study was to investigate the difference between loudspeaker reproduction
and headphone reproduction using generic HRTFs in a speech perception task that closer reflects
listening in real-life situations. Visual stimuli were presented via virtual reality glasses. The twooutcome measures that were analyzed are the response time to identify and locate a target story andthe localization error. The target stories were presented between ±105°, in 15° steps.

2. RESULTS
Figure 1 shows the response time to identify and locate the target story. The response time increases

with the number of simultaneously presented talkers as well as with reverberation. No effect of the
reproduction method can be seen. Figure 2 shows the localization error. As for the response time, aneffect of the number of talkers can be seen. Furthermore, more and larger errors can be seen for theheadphone reproduction than for the loudspeaker reproduction.

Figure 1. Response time from audio onset until the listener identified and located the target story. The colors indicate thereverberation conditions. Top panel: Headphone reproduction. Bottom panel: Loudspeaker reproduction.

Figure 2. Absolute localization error of the target story in the azimuthal plane. The bubble sizes in each column visualize theerror distribution and add up to the same size. The colors indicate the reverberation conditions (see Figure 1). Top panel:
Headphone reproduction. Bottom panel: Loudspeaker reproduction.

3. CONCLUSIONS
Headphone- and loudspeaker-reproductions were analyzed within a speech comprehension andlocalization task. Two outcome measures showed different findings. The response time to identify andlocate a speech source was similar with both reproduction methods. The localization error was worsewith the headphone reproduction compared to the loudspeaker reproduction.

REFERENCES
1. Ahrens A, Duemose Lund, K. Auditory spatial analysis in reverberant audio-visual multi-talker
environments with congruent and incongruent visual room information. bioRxiv 2022.
2022.04.30.490125. https://doi.org/10.1101/2022.04.30.490125.
2. Ahrens A, Marschall M, Dau T. Measuring and modeling speech intelligibility in real andloudspeaker-based virtual sound environments. Hearing research 2019; 377. p. 307-317.
3. Møller H, Sørensen MF, Jensen CB, Hammershøi D. Binaural technique: Do we need individual
recordings?. Journal of the Audio Engineering Society 1996; 44(6). p. 451-464.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0782

Measuring auditory attention effort in virtual audio-visualenvironments using pupillometry
Bernhard U. SEEBER1; Tim WACKLER
Audio Information Processing, Technical University of Munich, Germany

ABSTRACT
Pupillometry is a common approach to measure cognitive effort due to attending auditory targets in noise.
The enhanced attention to sound sources required in difficult listening situations is associated with cognitiveeffort, which is linked with an increased pupil diameter. However, pupil diameter is far more affected by theluminance of the focused visual object. Hence, pupillometry is done with low illumination or by focusing ona region with constant luminance. In ecologically valid audio-visual environments, visual objects appear
interactively, dynamically in view and luminance changes constantly. We present a method that compensates
for these ‘environmental luminance effects’ such that the measurement of attention through pupil dilation ispossible. Using a head-mounted world-camera and an eye-tracker, luminance in the focused area is measured.
A system of differential equations is solved to dynamically predict a baseline pupil diameter as a function of
momentary luminance. The approach is verified in a listening experiment with cued attention to one of two
spatialized talkers and varying difficulty, while luminance on a video projection screen is varied. Results
show that the large effect of luminance on pupil diameter can predicted with sufficient accuracy in order tomeasure the residual small effect of attentional effort. This paves the way for dynamic attention
measurements in virtual communication scenarios.
Keywords: Spatial attention, eye-tracking, communication

1. INTRODUCTION
Listening to speech in noisy spaces is an everyday task, which we master seemingly with ease inyoung years. Nevertheless, listening creates a cognitive workload that becomes apparent as we age or
acquire hearing loss, particularly in noisy situations (1). One approach to measure the effect ofcognitive load are dual-task paradigms in which participants listen to speech while conducting anothernon-auditory task, e.g. adding numbers (2, 3). The dual-task approach is demanding and a laboratory
task – it cannot be easily embedded into natural communication situations without interrupting thecommunication.
In order to increase ecological validity in auditory testing, communication and hearing ability arerecently tested in more natural tasks with the help of virtual reality (4-6). For example, Hladek et al.
(7) presented two virtual audiovisual scenes for communication testing in a virtual undergroundstation and verified the auralization quality with room-acoustical, binaural hearing and speech
intelligibility measures against the real-world counterpart. Subsequently, Hládek and Seeber (8)conducted a triadic conversation experiment in which three participants were free to talk about
anything while they were standing in the Simulated Open Field Environment (SOFE) (9) in an
anechoic chamber and their voices were auralized with the reverberation of the virtual undergroundstation in real time. The presence of interfering speech-shaped noise sources reduced theirinterpersonal distance. While the position changes were too small to affect the signal-to-noise ratio
noticeably, getting closer to the source may indicate an enhanced attention to the source and hence
listening effort.
Measuring attention and effort with high temporal resolution in a non-intrusive way in dynamic
listening situations will be key to understanding natural communication. Pupillometry is anothermeasure for cognitive effort and it appears correlated with perceived listening effort. For example,
Zekveld et al. (10) showed that peak pupil diameter amplitude, peak latency and mean pupil diameter1

seeber@tum.desystematically increase with decreasing speech intelligibility at adverse signal-to-noise ratios (SNRs).
Beyond energetic masking, the results by Woodcock et al. (11) indicate that informational masking
likewise affects pupil dilation, thus supporting the view that pupillometry can yield one proxy measure
for listening effort. The complexity of the task also affects pupil dilation in a dynamic manner, inagreement with the extra effort needed to complete the task (12). Hence, pupil dilation appears like a
good measure for effort in natural communication situations.
However, the pupil’s diameter is primarily affected by the luminance of the visual scene while theeffect of cognitive effort is small in comparison. It is thus common to use highly controlled, constantlighting in pupillometry experiments measuring cognitive effort. In a natural communication scenario
in audiovisual space, luminance changes substantially depending on the person looked at, or if focus
is on the body or the face or on objects in the environment or even the sky. The aim of the present
study is to investigate if the pupil response due to luminance can be predicted dynamically to such ahigh accuracy that permits the measurement of the smaller “residual” effect of listening effort in aswitching attention auditory task.

2. METHODS
2.1

System Overview

Figure 1: Overview of the real-time recording and prediction software and system setup.
Figure 1 gives an overview of the total system for time-stamped real-time processing of the eye
and luminance data including its embedding in the test environment (“light stimulus”). The overall
assumption is that the pupil dilation change due to listening effort can be computed as the residual of
the observed pupil diameter in the task and a predicted average pupil diameter based on the luminanceof the focused part of the visual scene. Pupil diameter, gaze position and world camera data aremeasured with the Pupil Core eye tracker (Pupil Labs, Berlin, Germany). It consists of two eye
cameras for the left and right eye, respectively, and a world camera. The eye tracker ’s dll is used tocompute the pupil view vector at 120 Hz rate which is mapped using individual calibration data to theimage captured by the world camera (at 30 Hz). Thus, the x-y coordinates in the world camera image
are known where the subject is looking. The luminance in candela per meter squared is computed froman area around the view coordinates with the help of luminance calibration data for the world camera.
The luminance data are then fed into a differential equation system which is solved to predict the pupil

diameter.
2.2 Luminance measurement, calibration and computation
The luminance reaching the foveal area is considered most relevant for pupil dilation. It thus
depends strongly on where someone is looking. The dll of the pupil lab eye tracker computes the x,ycoordinates of the point of view in the world camera image (stream) of 1080x720 pixels. The pixels
around that position are taken into account for luminance calculation with a decreasing weightaccording to a two-dimensional Gauss-function depending on radial distance, as depicted in Figure 2.

Figure 2: Eye-position dependent weighting function for luminancecomputation from the world camera images.
The world camera of the Pupil Labs eye tracker uses an automatic exposure correction which canbe controlled by setting a fixed exposure time >1/30 sec. This slow exposure fits well with our lab
environment with its (low) luminance from video projection in the anechoic chamber. The camera’s
8-bit image data are then non-linearly related to the actual luminance in the field of view of the subject.
In order to obtain the transfer function of luminance in cd/m 2 and the camera’s 8-bit image
luminance data, gray scale images with varying luminance (RGB values) were projected using thefrontal video projector of the SOFE. The luminance of the projected images was measured with theluminance meter RS-137 (RS Pro by RS Components Ltd.). Figure 3 gives the transfer function of
luminance in cd/m 2 and the camera’s 8-bit image pixel values and the exponential fit to the data.

Figure 3: Transfer function of the eye-tracker world camera pixel values and luminance.
2.3 Dynamic pupil dilation computation
Figure 4 presents measured pupil diameter as a function of computed luminance around the view
point for projected gray images with increasing luminance. Pupil diameter changes after every
luminance step increase: it decreases on average, but the response shows a substantial temporal
behavior like an overshoot with return to a luminance-dependent baseline, the pupil light reflex. Thisrequires a dynamic prediction model.
We base our prediction of pupil diameter on the set of differential equations by Pamplona et al.
(13) which are solved to obtain the time course of pupil diameter. Additionally, we identify momentsof strong luminance change and add an extra template element to model the overshoot phenomenon.

Figure 4: Pupil diameter as a function of luminance.
2.4 Spatial attention experiment
The aim for the approach is to measure cognitive effort in a multi-source speech situation. Thisrequires that the effect size of the pupil diameter change due to cognitive effort is larger than the error
for predicting the pupil diameter when luminance changes. The pupil diameter prediction for variablebackground luminance was evaluated in a preliminary experiment with the second author as a subject.
We have chosen a speech intelligibility task where attention had to be directed to one of two
simultaneous, spatialized speech sources presented from the SOFE loudspeakers at -60° and +60° in
the horizontal plane (9), with the target being presented at random from the left or the rightloudspeaker. Target and interferer speech were 5-word OLSA sentences (14) from the same speaker
which were selected at random from 40 word lists. Since a trial lasted 35 sec, only the first fivesentences from each list were used. Target and interferer speech sources were presented each at 60 dB
SPL (0 dB SNR). Two 150ms-long bursts of pink noise separated by 100 ms indicated the target
location prior to the onset of speech. The task of the subject was either to report the last of the five
OLSA words (“easy task”) or the first and the last of the five words (“difficult task”).
Video projection to the frontal screen of the SOFE was used to present different luminance levels
and the subject was instructed to keep their eyes to the frontal screen (no head turns allowed) . In the
low luminance (“dark”) condition, a luminance of ~0.1 cd/m2 was presented to allow for a reliable
eye diameter estimation of the tracker. In the changing luminance condition, luminance of the
projected gray image was changed to a random image RGB-value in [90-255] every 4 sec while
acoustic stimuli were presented.
The following conditions were tested:
1. Acoustic stimulus in constant, very low luminance (audio, dark),
2. No acoustic stimulus in constant, very low luminance (no audio, dark),
3. Acoustic stimulus and varying luminance (audio, luminance change),
4. No acoustic stimulus and varying luminance (audio, luminance change).
For each condition, ten measurements were conducted in random order. Each measurement lasted
35 seconds and contained five trials (5 sentences each with response). The audio conditions wererepeated for the easy and the difficult task.

3. RESULTS
3.1

Dynamic pupil diameter prediction

Figure 5: Dynamic prediction of pupil diameter based on eye-tracker measured luminance.
Figure 5 shows results of the dynamic pupil diameter prediction model for random luminance steps
presented anew every four seconds. The green and blue dashed lines show measured pupil diameterof the left and right eyes, respectively. The red curve, the prediction, fits the measured right eye
diameter well in general and on average, albeit the time course deviates in detail after each luminancestep. The left eye has a larger diameter in general, but the prediction is nevertheless good. The model’s
prediction could be improved with individual parameters for the diameter and for time constants,
which has not been attempted here.
3.2

Spatial attention experiment

Figure 6: Pupil diameter measured in the four test conditions of the spatial attention experiment.
Figure 6 presents the measured pupil diameter for the different conditions of the preliminary spatialattention experiment. Luminance had by far the largest effect on pupil diameter, as expected. Pupildiameter changed from around 5.7 mm in the dark conditions to 3.5 -4.0 mm in the bright conditions.
The effect of cognitive effort to conduct the speech task (“audio”) is comparatively small in thesepreliminary data and potentially visible only as a 0.2 mm diameter change in the bright condition.

Figure 7: Analysis of the preliminary results with regard to task complexity: Pupil diameter is presented forluminance changes without a task, for the easy task to report only the final word of the sentence, and for thecomplex task to report both the first and the last word of the sentence.
This is analyzed further in Figure 7 with respect to the two task-complexity levels investigated.
For the complex task, pupil diameter is increased for the right eye only compared to the easy task, but
for both tasks pupil diameter is increased over the baseline without a task. This suggests that – withthe experimental design used here with only one subject – the effect of task complexity is difficult tomeasure against the variance imposed by the luminance changes. However, the effect of a task isclearly visible over having no task. The limited sensitivity to the task may be due to at least two
factors: 1) the ±60° spacing of the two speakers made both tasks relatively easy, 2) pupil diameter
was averaged across the whole duration of the recording, i.e. including moment s in quiet between
sentences or when results were reported. Further analysis around selected moments during and shortly
after sentence presentation yielded far stronger effects on pupil diameter, but are not reported here.

4. CONCLUSIONSA prediction of pupil diameter as a function of the luminance of the visual object in focus wasdeveloped and verified in an experiment with random luminance changes. The pupil diameter
prediction was applied in a task in which the subject switched attention between two cued s peakers.
Results show that the effect of pupil dilation due to listening effort can be captured despite the largechanges of pupil diameter due to luminance changes, which can be compensated by the new method.
This lays the basis for measuring effort in communication scenarios in realistically simulated
audiovisual environments.

ACKNOWLEDGEMENTS
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project
ID 352015383 – SFB 1330, project C5. The SOFE was funded by the BMBF Bernstein Center for
Computational Neuroscience, Munich (BMBF 01 GQ 1004B).

REFERENCES
1.

Ohlenforst B, Zekveld AA, Jansma EP, Wang Y, Naylor G, Lorens A, et al. Effects of Hearing
Impairment and Hearing Aid Amplification on Listening Effort: A Systematic Review. Ear Hear.

2017; 38(3):267-81.
Gagne J-P, Besser J, Lemke U. Behavioral assessment of listening effort using a dual -task
paradigm: A review. Trends in hearing. 2017; 21:2331216516687287.
3. Sarampalis A, Kalluri S, Edwards B, Hafter E. Objective Measures of Listening Effort: Effects ofBackground Noise and Noise Reduction. J Speech Lang Hear Res. 2009; 52:1230-40.
4. Hendrikse MME, Llorach G, Hohmann V, Grimm G. Movement and Gaze Behavior in Virtual
Audiovisual Listening Environments Resembling Everyday Life. Trends Hear. 2019;23:2331216519872362.
5. Hadley LV, Brimijoin WO, Whitmer WM. Speech, movement, and gaze behaviours during dyadic
conversation in noise. Scientific Reports. 2019; 9:10451.
6. Hadley LV, Naylor G, Hamilton AFdC. A review of theories and methods in the science of faceto-face social interaction. Nature Reviews Psychology. 2022; 1(1):42-54.
7. Hládek H, Ewert S, Seeber BU. Communication conditions in virtual acoustic scenes in anunderground station. 2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA)
2021; Bologna, Italy: IEEE. DOI: 10.1109/i3da48870.2021.9610843.
8. Hládek H, Seeber BU. Effects of noise presence and noise position on interpersonal distance in a
triadic conversation. Internoise 2022; Glasgow, Scotland 2022. p. 6.
9. Seeber BU, Kerber S, Hafter ER. A System to Simulate and Reproduce Audio-Visual
Environments for Spatial Hearing Research. Hear Res. 2010; 260(1-2):1-10.
10. Zekveld AA, Kramer SE, Festen JM. Pupil response as an indication of effortful listening: The
influence of sentence intelligibility. Ear and Hearing. 2010; 31(4):480-90.
11. Woodcock JS, Fazenda BM, Cox TJ, Davies WJ, editors. Pupil dilation reveals changes inlistening effort due to energetic and informational masking. Proceedings of the 23rd International
Congress on Acoustics, integrating 4th EAA Euroregio 2019; 2019: Deutsche Gesellschaft für
Akustik eV (DEGA), pp 6193-6198.
12. Kosch T, Hassib M, Buschek D, Schmidt A, editors. Look into my eyes: using pupil dilation toestimate mental workload for task complexity adaptation. Extended Abstracts of the 2018 CHI
Conference on Human Factors in Computing Systems; 2018, pp 1-6.
13. Pamplona VF, Oliveira MM, Baranoski GV. Photorealistic models for pupil light reflex and iridal
pattern deformation. ACM Transactions on Graphics (TOG). 2009; 28(4):1-12.
14. Wagener K, Kollmeier B, Kühnel V. Entwicklung und Evaluation eines Satztests in d eutscher
Sprache I: Design des Oldenburger Satztests. Z f Audiol. 1999; 38:4-15.
2.

ABS-0834

Interactive virtual environments for hearing research across disciplines
Jamilla BALINT(1) , Janina FELS(1)(1) Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany, jamilla.balint@akustik.rwth-aachen.de,janina.fels@akustik.rwth-aachen.de

ABSTRACTIn the German priority program SPP2236 - Auditory Cognition in interactive virtual environments (AUDICTIVE), the aim is to significantly improve the realism and vibrancy of virtual environments, both in terms ofaudiovisual representation as well as user interaction by means of innovative audio techniques and findings fromauditory cognition research. Only a few studies until now have addressed complex mental models and effects
that could help answering the question of how much immersion and interaction is sufficient to investigate cognitive performances in virtual reality, and transfer the results to real-life. AUDICTIVE brings together researchers
from different disciplines - cognitive psychology, acoustics, and computer science/virtual reality - by encouraging joint research efforts to enhance our understanding of and competence in the field of auditory cognition ininteractive virtual environments, as a proxy to the real world. This work presents an overview of all paradigms,
tasks, scenes, and quality evaluation methods that are currently used in AUDICTIVE. The long-term goal is toanalyze the different research approaches and research questions related to interactive virtual scenarios and todevelop a better understanding of the criteria and requirements of virtual scenes for hearing research.

Keywords: binaural audio, cognitive psychology, acoustics

1

INTRODUCTION

Audiovisual virtual reality has reached a high level of realism in the last decades and is used widely in many
areas, e.g. video games, education, and research. Moreover, concepts like the Metaverse1 even go further byimagining the internet as an immersive virtual rality (VR) and augmented reality (AR) environment focusedon social connections. For research purposes, VR enables controlled research environments and thus yield thepossibility to extend investigations to contextual factors. In case of auditory cognition research2 , we expectthat interactive VR technology helps us to understand the underlying cognitive and neural processes in complex
acoustic and visual contexts like classrooms, open-plan offices, or multi-party communication. The German
priority program SPP2236 - Auditory Cognition in Interactive Virtual Environments (AUDICTIVE) connects three
disciplines – acoustics, cognitive psychology and computer science/virtual reality – to address the topics of (1)how the realism of VEs can be elevated, (2) to what extent are the theories of auditory cognition applicablein IVEs and (3) what are suitable quality evaluation methods to assess VR systems.3 Eleven projects are
dedicating their research topic to those questions by working in interdisciplinary teams. By combining twoor three disciplines, we aim at a holistic approach for deeper understanding in auditory cognition processes.
In this paper, we introduce the experiments conducted within AUDICTIVE, discussing the ecological validityand benefits of experimental control of IVEs as well as the quality assessment methods for VEs, including theconcepts of research data management.
1 The term orginated from the science fiction novel Snow Crash by Neal Stephenson in 1992.
2 Auditory cognition addresses the interplay of cognitive processes and the auditory input condition.
3 https://www.spp2236-audictive.de

2

VIRTUAL REALITY FOR AUDITORY COGNITION

In AUDICTIVE, we transfer already existing paradigms to VR, extend them and develop new approaches forauditory cognition experiments. In this section a brief overview of the paradigms and conducted experiments isgiven.
The auditory serial recall task is a well established paradigm to examine short-term memory performance.
The participants listen to a series of spoken digits and have to repeat them in the same order after a shortretention period of several seconds. The long term goal is to understand the cognitive-psychological factors
which are inherent when someone is listening to and remembering content of spoken language. This task can
be extended from a paper-and-pencil assessment to a fully immersive VR task, where it is possible to add
audiovisual features and spatial cues. As a first step, spatial information is added to the auditory cues insteadof diotic headphone reproduction. [1] Gradually, the audiovisual realism of the task is increased, eventuallyleading to a fully immersive VR experience. Non-verbal cues of virtual humans are incorporated to increase
the perception of social presence. [2] The integration of multi-modal stimuli is important for the realism andtherefore the external validity of the task, since multiple channels are involved in multi-talker interactions thatinvolve auditory perception of social cues (i.e. prosody), visual perception (i.e. gestures) and plausibility of thescene. As a first attempt to use state of the art conversational agents, animated characters were generated with
the MetaHuman Creator agents who spoke the digits, increasing the overall realism of the scene and feeling ofpresence. [3]
In VR, the speech perception and listening comprehension can be studied in realistic environments, whentasks are extended to close to real-life settings, i.e. a virtual classroom or multi-talker scenarios, with multiple
sound sources, visual and auditory cues. Suitable paradigms are the auditory selective attention task or thelistening comprehension task, where the realism of the virtual environment is increased from computer generated images such as Unity 3D to immersive 360◦ video, where photo-realistic representation of a scene can beachieved. [4, 5, 6, 7]
Audiovisual virtual environments provide a controlled environment for studying the distance perception ofsound sources in rooms and underlying attentional processes. One experiment focuses on a setup, where multiple robots are distributed in a reverberant room. [8, 9] With the help of auditory cues from the robots, thedistance perception is investigated by measuring psychological and physiological measures. Another experimentfocuses on investigating the auditory perception of pedestrians in car pass-by scenarios in VEs. [10, 11] Bycomparing time-to-collision estimations in wave field synthesis and higher order ambisonics setups, the influence of different reproduction system can be identified for conducting further research in navigation tasks. The
impact of emotions on perception in the auditory domain is investigated in [12]. Participants are exposed toaffective voice and have to place virtual loudspeakers or agents at the perceived location of the source. Distanceestimates vary depending on the type of voice (i.e. angry or neutral).
The concept of plausibility plays an important role when it comes to interactive tasks, such as shared experiences and communication in a multi-talker scenario. In an interactive conversation task, factors influencing
audiovisual plausibility are investigated. [13] The communication behaviour is analysed with regard to spatial and non-spatial audio setups as well as embodied conversational agents with different audio reproduction
techniques.
Speech perception depends not only on the audio signal but also on the visual information that supports theunderstanding of spoken language. Lip-syncing and facial expression pose a major challenge in VR because of
the high complexity and increased computational demand. The question is, if current findings can be generalized
to experiments in VE. By measuring the cortical tracking of speech with a mobile EEG, different lip animationsof real and animated characters are investigated. [14] Furthermore, the underlying brain mechanisms of multimodal speech processing is investigated with the help of EEG studies in [15].
In the navigation task, participants are asked to navigate through a virtual environment without any visualcues, only by listening to the direct sound, early reflections from the walls and the reverberance. [16] This way
it is possible to understand how humans process acoustic cues to orient and navigate in space.

3

QUALITY EVALUATION OF INTERACTIVE VIRTUAL ENVIRONMENTS

In order to evaluate if VR is a suitable tool to investigate phenomenon in auditory cognition research, a framework for the quality evaluation has to be designed. [17] Current methods in quality evaluation from the virtual
reality community include questionnaires on presence, social and co-presence, simulator sickness, immersion,
coherence, illusion and plausibility. The question is, if those concepts can be adapted to gain a better understanding of how VR can be improved for auditory cognition research. Furthermore, combining assessments
methods (i.e. questionnaires with physiological measures) could be beneficial in understanding the mechanismsthat lead to increased immersion or higher validity regarding research results. Jung and Lindemann [18] suggest
metrics to evaluate the VR experience by introducing a model connecting the context (coherence and realism),
the system (immersion) and the concept of illusion (presence). In the following paragraphs we adapt thosemetrics and suggest further fidelity levels in the evaluation procedure. Figure 1 illustrates the components of
the quality evaluation framework from an interdisciplinary viewpoint with the focus on auditory cognition ininteractive virtual environments.

Figure 1. Quality evaluation framework from an interdisciplinary viewpoint for auditory cognition in interactive
virtual environments. Blue: disciplines, yellow: quality components, red: tasks.
Immersion has multiple definition, among others it can be divided into perceptual immersion and psychological immersion. [19] According to [18], objective immersion can be evaluated with the following metrics:(1) Wide and high-resolution field of view (HMD vs. screen vs. CAVE(like) systems), (2) 6DoF tracking, (3)spatial audio cues, and finally (4) additional sensory cues to enhance immersion. Spatial audio cues are notfurther elaborated and a more detailed distinction is necessary for specific task or contexts. With the diversityof audio reproduction systems in AUDICTIVE, we hope that we can contribute to the understanding of aspects
influencing immersion, including spatial audio cues.
Realism can be regarded as the extent to which a virtual environment is similar to the real world. Oneaspect that has a major influence on realism is the stimulus fidelity, which presents an option for an objective
metric of the VE. Studies show that highly realistic systems produce sensory stimuli that are very similar
to real-world stimuli. [20] In case of auditory cognition research, realistic auditory stimuli including spatial
information as well as an adequate reproduction system depending on the task are necessary. With the tasks in
AUDICTIVE, we aim at identifying fidelity characteristics of reproduction systems as an objective metric forrealism.
Presence is defined as the sense of being there in the virtual space, co-presence as the sense of perceiving
and being perceived by others, and social presence relates to the feeling of being there together. [21] Components influencing presence are the (1) place illusion, (2) plausibility illusion and (3) embodiment. They can beevaluated by standardizes questionnaires.
In order to evaluate the components of the quality framework, additional information regarding the specifictasks and the overall research goal has to be collected or provided beforehand. In this context, tools andmethods from the field of research data management appear to be helpful (i.e. glossary, metadata collection, data
documentation). Working in an interdisciplinary project requires a basic understanding of the terms, culture, andworking habits of the involved disciplines. Therefore, everyone involved can benefit from an interdisciplinary
glossary set up at the beginning of the project runtime. Furthermore, metadata and data documentation is
crucial when it comes to collaborating across disciplines and exchange of data. Within AUDICTIVE, data
documentation procedures in every discipline have to be combined in order to properly describe the investigatedtask, used methods, acquired data and results. Finally, the overall technical system involving all aspects of eachdiscipline has to be summarized in order to evaluate the influence of each component on the quality level.

4

GENERAL DISCUSSION AND CONCLUSION

Obvious advantages of conducting experiments in VR are the controllable environment, the scalable conditions
and eventually the gained reproducibility (although increased complexity). Nevertheless, the question remains
on how we evaluate the quality of the VR experience and how the findings from labs are transferable to reallife situations. In the majority of experiments, where multi-modal interactive environments are employed, thecongruency of acoustic and visual stimuli play a major role. Therefore, the auralization of sound sources has
a great impact when VR is used in auditory cognition research. The diversity of technical systems has the advantage of investigating multiple aspects from different viewpoints (i.e., different audio reproduction techniques
like higher order ambisonics and wave field synthesis could lead to different results in specific tasks or results
might vary depending on the visual environment used, like HMD vs. 360◦ ). On the contrary, the diversity also
poses the challenge of reproducibility and investigated level of detail. Here, the fidelity aspects of auditory andvisual stimuli play a major role when it comes to realism. The question is if the level of realism (evaluated via
the stimulus fidelity) is a suitable indicator for VR environments for experiments in auditory cognition and if it
is necessary to ensure a high level of quality of experience in order to obtain valid results?
Another key component in VR is the realism of the virtual representation of the interlocutor. The naturalness of characters or virtual agents and their behaviour influences the plausibility of the scene. A face-to-faceconversation in reality is accompanied by various aspects, i.e. gestures, facial expressions, eye movements, body
postures, etc., leading to an overall communication experience. All those aspects are desirable when it comesto embodied virtual agents as communication partners. This poses a major challenge, since multi-modal aspects
have to be combined (i.e. auditory and visual cues, gestures, prosody, etc.). Experiments with a communication
loop further increase the complexity, although the increased interactivity might lead to a higher plausibility of
the VR scene.
The initial experiments confirm the validity of VR experiments by replicating results to paper-and-pencil
assessments. Unfortunately, only little is gained from the use of VR, if no additional insights are obtained. On
the contrary, many experiments yield the potential to include measures that move beyond one discipline andprovide close to real life environments like classrooms or multi-talker settings. The benefit of interdisciplinary
work is the mutual assistance when it comes to detailed research question in the specific field. Cooperation between disciplines are helpful to assess auditory cognition with regard to spatial audio techniques, physiologicalmeasures, psychological evaluation methods, and advanced VR systems from computer science. With current
technology, it is possible to track head and eye movements, gestures and investigate the correlation between
multi-modal stimuli and physiological responses. For the future is is desirable if further questions can be discussed from an interdisciplinary and multi-modal point of view: (1) Are differences between the field and thelab expected or not, and does it depends on what we are measuring or how we assess the metrics? (2) When it
comes to the detailed design of an audiovisual VR scene, combinations like audio only, video only, combined
audio and video, can lead to different outcomes. How can a multi-modal approach be applied that comprises
all important aspects regarding a specific task? (3) Further components influencing the VR experience, whichhave not been discussed in this work, are the haptic feedback systems and additional sensory cues like olfactoric, tactile, and taste feedback. Is it even possible to create a plausible, real, and immersive VR environment
without considering all senses?

ACKNOWLEDGEMENTS
This work is funded by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) under the
project ID 444532506, SPP2236 - AUDICTIVE - Auditory Cognition in Interactive Virtual Environments.

REFERENCES[1] Ermert, C. A., Mohanathasan, C., Schlittmeier, S. J., Fels, J. Examining the Influence of Varying Audio Reproduction Methods on Verbal Short-Term Memory. In: 48. Jahrestagung für Akustik. Deutsche Gesellschaft
für Akustik e.V. (DEGA); p. 1177-1180, 2022. RWTH Aachen University.
[2] Fels, J., Ermert, C. A., Ehret, J., Mohanathasan, C., Bönsch, A., Kuhlen, T., Schlittmeier, S. J. Listening to,
and Remembering Conversations between Two Talkers: Cognitive Research using Embodied Conversational
Agents in Audiovisual Virtual Environments. In: 47. Jahrestagung für Akustik. Deutsche Gesellschaft für
Akustik e.V. (DEGA); 2021.
[3] Ermert, C. A., Ehret, J., Kuhle, T., Mohanathasan, C., Schlittmeier, S. J., Fels, J. Spatial audio-visualcongruency effects in virtual reality environments. Proc. of the 24th Int. Congress on Acoustics, Gyeongju,
Korea, 2022.
[4] Reimers, C., Fremerey, S., Leist, L., Klatte, M., Fels, J., Raake, A. Examining Auditory Selective Attention in a Classroom Setting using Audiovisual Virtual Reality. In: 47. Jahrestagung für Akustik. Deutsche
Gesellschaft für Akustik e.V. (DEGA); 2021.
[5] Reimers, C., Loh, K., Leist, L., Fremerey, S., Raake, A., Klatte, M., Fels, J. Investigating Different Cueing
Methods for Auditory Selective Attention in Virtual Reality. In: 48. Jahrestagung für Akustik. Deutsche
Gesellschaft für Akustik e.V. (DEGA); p. 1173-1176, 2022.
[6] Leist, L., Reimers, C., Fremerey, S., Fels, J., Raake, A., Lachmann, T., Klatte, M. Effects of binauralclassroom noise scenarios on speech perception and listening comprehension in primary school children.In:
48. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2022.
[7] Fremerey, S., Reimers, C., Leist, L., Spilski, J., Klatte, M., Fels, J., Raake, A. Generation of audiovisual immersive virtual environments to evaluate cognitive performance in classroom type scenarios. In: 47.
Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2021.
[8] Neudek, D., Nagathil, A., Wissmann, A., Getzmann, S., Martin, R. On Least-squares-based Auditory Attention Decoding with Individual Neural Latency Compensation. In: 47. Jahrestagung für Akustik. Deutsche
Gesellschaft für Akustik e.V. (DEGA); 2021.

[9] Neudek, D., Stodt, B., Getzmann, S., Martin, R. Calibration of an acoustic virtual environment for plausible
distance perception. In: 48. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2022.
[10] Oberfeld-Twistel, D., Wessels, M., Büttner, D. Auditory and audiovisual time-to-collision judgments forelectric and conventional vehicles. In: 47. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V.
(DEGA); 2021.
[11] Steinbach, L., Beyer, F., Altinsoy, E., Oberfeld-Twistel, D., Wessels, M. Auditory and audiovisual timeto-collision judgments for electric and conventional vehicles. In: 48. Jahrestagung für Akustik. Deutsche
Gesellschaft für Akustik e.V. (DEGA); 2022.
[12] Kroczek, L., Rosskopf, S., Staerz, F., Blau, M., Van de Par, S., Muehlberger, A. The influence of affective
voice on sound distance perception. In: 48. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik
e.V. (DEGA); 2022.
[13] Immohr, F., Rendle, G., Huebner, E., Neidhardt, A., Sinani, L., Froehlich, B., Raake, B. Exploring Factors
Influencing Audiovisual Plausibility and Co-Presence in Multi-Modal VR Communication. In: 48. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2022.
[14] Daeglau, M., Otten, J., Mirkovic, B., Grimm, G., Hohmann, V., Debener, S. The Influence of LipMovement Modelling on Cortical Speech Tracking in Virtual Environments with Mobile EEG. In: 47.
Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2021.
[15] Begau, A., Klatt, L., Schneider, D., Wascher, E., Getzmann, S. Age-Related Differences in Post-Stimulus
Alpha and Theta Oscillations: An EEG Study on Audiovisual Speech Processing in a Multi-Talker Scenario.
In: 47. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2021.
[16] Steffens, H., Schutte, M., Ewert, S. Navigation via Acoustical Cues through Enclosed Spaces without
Visual Information. In: 47. Jahrestagung für Akustik. Deutsche Gesellschaft für Akustik e.V. (DEGA); 2021.
[17] Raake, A., Rummukainen, O., Habets, E., Robotham, T., Singla, A. QoEvaVE - QoE Evaluation of Interactive Virtual Environments with Audiovisual Scenes. In: 47. Jahrestagung für Akustik. Deutsche Gesellschaft
für Akustik e.V. (DEGA); 2021.
[18] Jung, S., Lindeman, Robert. W. Perspective: Does Realism Improve Presence in VR? Suggesting a Modeland Metric for VR Experience Evaluation. Frontiers in Virtual Reality, 2, p. 693327, 2021.
[19] Lombard, M., Ditton, T. B., Crane, D., Davis, B., Gil-Egui, G., Horvath, K., Rossman, J. Measuring
presence: A literature-based approach to the development of a standardized paper-and-pencil instrument. 3rd
Int. Workshop on presence, Delft, Netherlands, 2000.
[20] Stoffregen, T., Pagulayan, R., Smart, L., Bardy, B.. On the Nature and Evaluation of Fidelity in Virtual
Environments. In L. Hettinger & M. Haas (Hrsg.), Virtual and Adaptive Environments, p. 111–128, CRC
Press, 2003.
[21] Nowak, K. Defining and Differentiating Copresence, Social Presence and Presence as Transportation. 4 th
Annual International Workshop, Philadelphia, Rettie, R.M., 2003, A Comparison of Four New Communication Technologies, Proceedings of HCI International Conference on Human-Computer Interaction, Lawrence
Erlbaum Associates, p. 686–690, 2001.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0837

Using real group conversations in laboratory testing
Karolina SMEDS1; Petra HERRLIN1; Eline Borch PETERSEN2; Frédéric MARMEL1;
Florian WOLTERS1; Anja Kofoed PEDERSEN3
1
2
3

ORCA Europe, WS Audiology, Sweden

Scientific Audiology, WS Audiology, Denmark

Applied Audiological Research, WS Audiology, Denmark

ABSTRACT
There is currently scientific focus on improving the ecological validity of laboratory hearing assessments.
Most work has highlighted the selection of listening scenarios and playback using multi-speaker setups. Our
work has instead centered on investigating the listening tasks that can be used, specifically real conversations.
Various outcomes can be obtained during a conversation. The actual conversation may be analyzed
acoustically (e.g., turn-taking timing), linguistically (e.g., repetition behavior) or behaviorally (e.g., head andeye movements). Another option is to stage a conversation and use traditional psychoacoustical tests whileconversing (e.g., paired comparisons of two hearing-aid settings). There are also various ways to elicit a
conversation. Collaborative puzzle-solving, map-navigation, and spot-the-difference (Diapix) tasks havebeen used in conversations between two interlocutors. Here, we report on aspects of using real conversations
in small groups of interlocutors. In one study, groups of three interlocutors subjectively evaluated four
different ways to elicit group conversations, while acoustical analyses were used to explore how theelicitation method affected the conversations. In another study, groups of four interlocutors evaluated fourstaged scenarios in terms of perceived realism and conversation success. The experimental considerations
for using group conversations in the laboratory will be discussed.
Keywords: Group conversation, Outcome measures, Conversation sparkers

1

INTRODUCTION

We want hearing-aid users to experience everyday benefit from their hearing aids. Therefore,
evaluation methods that show a high degree of ecological validity, i.e., methods that are indicative of
everyday performance, must be used when the aids are developed and fitted. To develop such
evaluation methods, knowledge about peoples’ auditory reality is important. Which listening situations are frequently experienced, and which are the listening demands in these situations?
In previous work, we investigated the signal-to-noise ratios (SNRs) experienced in the everyday
life of 20 hearing-aid users. Generally, the SNRs were higher than in traditional laboratory testing,and many situations that were experienced to vary in listening difficulty, had similar SNRs (1). We
further investigated common listening situations that people are encountering, and in the Common
Sound Scenarios (CoSS) framework these situations were categorized into seven “task categories” (2).
In subsequent studies, using Ecological Momentary Assessments (EMA), we have learned about how
often these various types of situations occur in peoples’ everyday life, how important they are to hear
well in, and how difficult it is to hear in the situations (3).
We have also used EMA to evaluate people’s preferred hearing-aid settings in everyday life (3).
Two hearing-aid settings were implemented in two hearing-aid programs, and the preferred program
was indicated when the participants were prompted to report. The advantage of using EMA is that
evaluations are made in the participants self-selected everyday listening situations. However, there
are also drawbacks with using EMA. Some difficult and important situations do not occur very often(3), which makes EMA inefficient. It may also be difficult or socially unacceptable to report in certain
situations, especially in situations with social interaction (4).
Therefore, there is still a need for laboratory testing for these important, often difficult social
interaction situations. When looking at the CoSS framework, it becomes apparent that traditional

laboratory testing only targets “focused listening”, i.e., listening intently and sometimes repeatingback what was heard. Neither real conversations, nor more passive listening situati ons are normallyused in laboratory testing. The current paper will explore ways to broadening laboratory testing toinclude real conversations.
Nicoras et al. (5) investigated the concept conversation success for people with normal and impaired hearing using group concept mapping. Seven factors important for conversation success wereidentified: 1. Being able to listen easily; 2. Being spoken to in a helpful way; 3. Being engaged andaccepted; 5. Sharing information as desired; 5. Perceiving flowing and balanced interaction; 6. Feeling
positive emotions; 7. Not having to engage coping mechanisms. These factors can potentially be used
when developing outcome measures for hearing-aid satisfaction or benefit in conversation situations.
Here, we will report on some of the activities we have been involved in that contribute to theunderstanding of conversations and which outcome measures we can develop. First, we will give an
overview of ways to study conversations, with a focus on finding outcome measures for hearing -aid
performance. Second, we will discuss different ways to spark a conversation in the laboratory, and wewill present a study evaluating four conversation sparkers. Third, we will present a pilot study
focusing on perceived realism and conversation success in group conversations. Last, someexperimental considerations for using group conversations in the laboratory will be discussed.
In the following, the term “scenario” refers to listening situations implemented in the laboratory
or clinic. Several aspects of these scenarios need to be designed carefully for laboratory test ing to
produce ecologically valid findings. A scenario includes an activity (for instance described using the
CoSS task categories), an acoustic environment and its implementation. The scenarios should sound(and preferably also look) realistic, but it is also important that both the test participants and thehearing aids behave as they would in a corresponding real-life situation. The activities performed in
the scenarios govern the outcome measures used.

2

OUTCOME MEASURES FOR CONVERSATIONSConversations are studied by researchers from many different fields, such as linguistics, social
psychology, cognitive neuroscience, engineering, and audiology. The purposes of studying conversations differ and different research methods have developed in the various disciplines.
In a linguistic analysis of conversations, researchers may for instance investigate breakdown andrepair in a conversation (6), lexical or syntactical alignment between interlocutors, or the use of
backchanneling (verbal or nonverbal responses to a talker indicating for instance attention or
agreement). The acoustic speech signals can be analyzed by looking for instance at speech levels,
speech rate, and turn-taking timing (7). Body posture, gestures (8), head movements, eye gaze (9) andcoordination of body movements can indicate difficulties or ease in a conversation. With the use of
EEG or other electrophysiological correlates, selective attention (10), attention switching (11), andattention drifting (12) can be investigated.
Some of these investigation methods have indicated differences in conversation behavior between
test participants with normal and impaired hearing. Researchers have shown that conversations
between one young interlocutor with normal hearing and one older interlocutor with impaired hearing
can work well, but that the communication effort was higher when the interlocutor with impaired
hearing was unaided than when hearing-aids were used (7, 13).
In general, the investigation methods mentioned above can be difficult to use for instance during
hearing-aid development, when a new hearing-aid feature will be compared to an old one. However,by tying some of the measures mentioned above to perceived conversation success (5), we will learn
more about which measures could be useful as outcome measures.
Another outcome measure strategy is to stage a conversation situation and use traditional
psychoacoustical methods to evaluate for instance hearing-aid preference. In our research group we
did so when we developed the Live Evaluation of Auditory Preference (LEAP) laboratory test (14).
The test used six mandatory test scenarios, selected based on the CoSS framework, and a few selfselected scenarios (derived from experienced listening situations during an EMA study). The scenarios
were implemented using a basic loudspeaker setup. For the conversation situations, we used real
conversations between the test participant and one or two test leaders. During the conversations, thetest participants compared two hearing-aid programs using a paired-comparison paradigm and reported on their preferred setting.

3

CONVERSATION SPARKERS

Different methods have been used to initiate or spark conversations in the laboratory. For twotalkers, Baker and Hazan (15) and Petersen et al. (7) used Diapix material, where the interlocutors
have one version each of similar pictures and they together solve a “spot the difference task”. Beechey
et al. (16) elicited naturalistic conversation by providing the interlocutors with one version each of apuzzle with 10 by 10 items, where the task was to find a way from the upper left corner of the puzzle
to the lower right corner by combining the information from the two sets of the puzzle. DohertySneddon et al. (17) used two versions of a map and the task was to give and take instructions about aroute to find a treasure. These tasks seem to have worked well in the mentioned studies. However,they are probably less suitable for group conversation. Further, with a strong visual focus on a picture,a puzzle or a map, the interlocutors might not pick up visual cues from the rest of the group or might
exhibit an unnatural behavior regarding for instance eye contact.
We therefore suggested and evaluated four other group conversation sparkers: 1. Photographs withkeywords (example: a photograph of a huge pan of paella with the key words “food, restaurant,
vacation”), 2. Consensus tasks (example: “Agree on a dinner menu exclusively with dishes none ofyou like”), 3. Timeline tasks (order four historical events chronologically), and 4. Free conversation.
Ten examples of each sparker were evaluated by four groups with three conversation partners in eachgroup. Test participants were recruited among unacquainted colleagues from different departments
and without insights into the project. The four types of sparkers were evaluated subjectively using
questionnaires and ranking of the sparkers. In addition, outcome measures such as speaking time,utterance length, turn-taking information, and speech levels were used.
Overall, the participants judged all sparkers to be natural, engaging and with a good flow (7-8 on
a 10-point scale). The results showed that the Timeline task was least suitable. It had significantly
lower ratings on the question “Did the task make you feel that you could contribute to the conversation”(p<0.05). It also created the most unbalanced conversation in terms of test participants’ speaking time
(Figure 1). The task created more silence, more very short utterances, longer gaps in turn -taking, andfewer turn-takings than the other sparkers. The fact that there is a “right answer” to the task probably
contributed to it being less successful as a sparker. Although the outcomes of the Free speech task
were not found to differ from the remaining tasks, the participants ranked the Free speech as the least
suitable conversational sparker (although not statistically significant). Participants commented thatnot providing any conversational sparker would make it difficult for some participants/groups to holda conversation. Both the Photos with key words and the Consensus task seemed appropriate as groupconversation sparkers.

Figure 1 – The left panel shows the speaking time for the three participants in each group. For
each conversation sparker, the data for the participants who spoke least are presented first and thedata for the participants who spoke most are presented last. The right panel shows the overall
proportion of speaking time for each sparker. Medians are represented by the horizontal line withineach box. Outliers are defined as values outside 1.5 times the box length, and the whiskers extend tothe highest and lowest values when outliers are excluded.

4

LEAP Group Conversation

In a pilot study, a group conversation version of the LEAP test was investigated. The study aimed
at producing a set of group conversation scenarios in an office setup and evaluate perceived realism
and conversation success. In each conversation group four people participated, two with hearing impairment and two with normal hearing.
4.1

Method

4.1.1 Research participants
20 test participants (5 groups) were recruited for the experiment. The inclusion criteria were age
between 65 and 79 years, fluent Swedish speakers with good eyesight (after correction). The inclusion
criteria for participants with impaired hearing were moderate hearing loss in both ears and hearingaid users for more than 6 months. The normal-hearing participants had self-reported normal hearing
and had passed an online speech-in-noise hearing test (horseltestaren.se). In one of the five groupconversations, one of the participants with normal hearing was unable to attend and was replaced bya colleague. Her data were excluded from the analyses. Audiograms for the two participant groups areshown in Figure 2.
Frequency (Hz)125

Sound Level (dB HL)

0
10
20
30
40
50
60
70
80
90
100

250

500 1000 2000 4000 8000

ONH

OHI

Figure 2 – Average audiogram (best ear) and standard deviation for older normal-hearing (ONH)
and older hearing-impaired (OHI) test participants.
4.1.2 Setup and Sound Files
Four listening scenarios were implemented. Sound files were selected from the Ambisonic Recordings of Typical Environments (ARTE) database (18). Background sound levels were set based on thelevels reported when the sound files were recorded or sound levels reported in our SNR study (1), andinformal listening by a group of colleagues with normal hearing.
1.
2.
3.
4.

Dinner at home (ARTE: 04_Living_Room_MOA_31ch, 52-55 dBA)
Dinner in restaurant (ARTE: 12_Food_Court_1_MOA_31ch, 57-60 dBA)
Business meeting (only video projector noise, 44-46 dBA)
Party (ARTE: 09_Dinner_party_MOA_31ch, 70 dBA)

Three of the scenarios were implemented in a meeting room (345×693 cm). The room wasacoustically treated with a suspended mineral fiber ceiling (pending down 55 cm from the 379 cm
inner ceiling) and a wall-hanging treatment (238×116 cm). Thin curtains (floor to ceiling) covered a
3 m long window section on one wall. Sound files were played from a laptop equipped with a RME
Fireface 802 sound card. Four loudspeakers (Genelec 8030 CP), placed in and oriented towards thecorners, were used. The test participants were placed at a table, two persons (one with impaired
hearing and one with normal hearing) at each side of the table.
The Party scenario was implemented in a kitchenette area with an adjacent room and corridors.
The main area was 354×336 cm with a ceiling height (suspended mineral fiber) of 324 cm. In one
section (80 cm closest to the kitchen part) the ceiling was lower (220 cm). Two loudspeakers placedat the two ends of the kitchenette area were used. In this scenario, the participants were standing andcould place themselves freely.

4.1.3 Conversation Sparkers
Conversation sparkers were selected based on the study described above. For scenarios 1, 2, and 4
Photos with key words were used. For scenario 3, a Consensus task was used, and all test participants
were asked to act as “secretaries” for the business meeting and note down what the group agreed on.
4.1.4 Subjective Evaluations
After each conversation, the participants used a tablet (Samsung Galaxy Tab S6 Lite) to rate realism
and conversation success using a 5- or 6-step Likert scale (if not otherwise indicated below).
Realism:
1. How realistic did you perceive the situation?
Conversation success:2. How easy was it to hear in the situation?
3. How engaged did you feel in the conversation?
4. How well were you able to say what you wanted to say?
5. How well did you think that the conversation was flowing?
6. How successful did you think that the conversation was?
After the last test scenario, the group discussed how realism could be increased in the scenarios
and what constitutes conversation success.
4.2 Results
The focus of this pilot study was to evaluate perceived realism and conversation success in thefour scenarios.
4.2.1 Realism
The Party was judged to be most realistic and the Dinner at home least realistic (Figure 3). In thediscussions after the testing, the participants commented that standing up in the kitchen area and being
able to move freely in relation to the other interlocutors contributed to the perceived realism in the
Party situation. The background noise type and level also contributed to the realism, but a few
participants with hearing impairment commented that these situations can be even louder in real life.
The Business meeting was judged to be realistic because the testing took place in an actual meetingroom.
Realism, Dinner at home
6
5
4
3
2
1
0
Not at all realistic

Somewhatrealistic

Rather realistic
OHI

Realistic

Very realistic

ONH

Figure 3. Perceived realism in the four scenarios used. OHI = Older participants with hearing
impairment. ONH = Older participants with normal hearing.

The two dinner situations were judged to be less realistic, and in the subsequent discussions theparticipants said this mainly was related to the type and level of the background noise. Test participants, especially those with impaired hearing, commented that these situations are usually more difficult in real life. They thought that the background noise was too static and that it should include
transient sounds from silverware, porcelain, and glasses. It could also include someone cough ing or
yelling, a chair being pushed, a telephone ringing, a baby crying or similar. The background soundlevels were also judged to be too low.
4.2.2 Conversation success
Here, the results of all questions related to conversation success are presented. Questions 2 -5 are
related to four of the factors found to be important for conversation success in the study by Nicoras
et al. (5), whereas question 6 asks for conversation success directly.
The results of the question “How difficult/easy was it to hear in the situation?” (question 2) showed
the expected difference between the two participant groups (Figure 4). The participants with normal
hearing generally rated the question higher than the participants with impaired hearing. The situations
that were perceived easiest to hear in were Dinner at home and Business meeting, where the noiselevels were low. None of the situations were described as “very difficult” to hear in. The participants ,in particular those with impaired hearing, commented that listening is more difficult in real life. It
was also slightly easier for the participants with normal hearing to say what they wanted (question 4)
and they perceived slightly higher conversation flow (question 5). The conversation flow was similar
for all scenarios.
The test participants showed the highest engagement (question 3) in the Business meeting (where
the consensus task was used as a sparker) and Dinner in public (where picture cards with keywords
were used). Both participant groups showed similar degree of engagement.
In terms of conversation success (question 6), dinner at home was judged to be the least successful
conversation. This scenario was presented first for all groups, which might explain the result. There
was no difference in judged conversation success between the two participant groups. In thesubsequent discussion about the conversations, the test participants agreed that the most importantfactor for conversation success was that everybody joins, but it was not judged important thateverybody spoke equally much. They also mentioned that it is important that all interlocutors “giveand take”, i.e., both listen and speak and that everybody should be included in the conversation in anatural manner.
Listen easily, Dinner at home
6
5
4
3
2
1
0
Very difficult

Difficult

Ratherdifficult
OHI

Rather easy

Easy

Very easy

ONH

Figure 4. Perceived ability to listen easily in the four scenarios. OHI = Older participants withhearing impairment. ONH = Older participants with normal hearing.

5

EXPERIMENTAL CONSIDERATIONSIn the two experiments described above, office rooms and a limited number of loudspeakers wereused. In the LEAP Group Conversation study, perceived realism was investigated, but detailed
physical measurements were not made to ensure that the sound fields in the four scenarios resembled
the sound fields in corresponding everyday listening situations.
The obvious way to improve the acoustical representation of the background sounds would be touse an anechoic test room with many loudspeakers, using for instance higher order ambisonics toplayback multi-microphone recordings faithfully (19). There are two problems with this approach forgroup conversations. First, it will be difficult to place a group of interlocutors in the sound field insuch a way that the acoustic representation is accurate for all participants. The so-called “sweet spot”,
where the reproduction error is small, is usually too small to host all participants. It might be worse
to sit far outside the sweet spot in an ambisonics playback situation than in a more simplistic setup ina room with some reverberation, which smooths the sound field. Second, the voices of the interlocutors will sound as if they do not belong to the intended scenario if the background sounds were recorded in a reverberant environment. If the scenario is one that could take place in an environment similar
to, for instance, the office and kitchenette rooms used in the Group LEAP study, it might be better totest where the representation of the interlocutors’ voices is more realistic.
Another option is to use so called Variable Room Acoustics Systems (VRAS), where microphones
are picking up the interlocutors’ voices, which are then processed so that the voices match the scenario(20). Researchers need to ensure not only that the scenario sounds realistic, but also that the physicalsound field is accurate, since the interlocutors will hear their communication partners as a combinationof direct sound from the communication partners’ mouth and the processed version from the loudspeakers.
Another question is related to the visual realism. In the LEAP Group Conversation study describedabove, the Party and the Business meeting were described as the most realistic. For these situations,the visuals of the scenario corresponded roughly to what it would be in real life. Standing with a groupof people in a kitchen area at a party or sitting down at a business meeting in a room that is used forbusiness meetings is realistic. For the dinner situations, further visual input could potentially haveincreased the perceived realism.

6

DISCUSSION AND CONCLUSIONS

We have presented some of our ongoing work where we include real conversations in laboratory
testing and strive to develop outcome measures indicating how hearing aids function in real groupconversations. Many ways to study conversations have been presented. Some of these can potentiallybe used as outcome measures during hearing-aid development if they can be tied to conversation
success. Alternatively, traditional psychoacoustical test methods, such as ratings or paired
comparisons of certain attributes, can be used in staged group conversations. Different ways to elicit
or spark group conversations have been evaluated. Photos with keywords and Consensus questions
seemed to work well. We further staged four group conversation scenarios, with four interlocutors ineach scenario. The scenarios were evaluated in terms of perceived realism and conversation success.
The results will be used to refine our conversation scenarios. Finally, we discussed the experimental
considerations when creating group conversations in the laboratory.

ACKNOWLEDGEMENTS
We want to thank the participants who contributed to this research.

REFERENCES
1. Smeds K, Wolters F, Rung M. Estimation of signal-to-noise ratios in realistic sound scenarios. J Am Acad
Audiol. 2015;26(2):183-96.
2. Wolters F, Smeds K, Schmidt E, Christensen EK, Norup C. Common Sound Scenarios: A context-driven
categorization of everyday sound environments for application in hearing-device research. J Am Acad
Audiol. 2016;27(7):527-40.
3. Smeds K, Gotowiec S, Wolters F, Herrlin P, Larsson J, Dahlquist M. Selecting scenarios for hearingrelated laboratory testing. Ear Hear. 2020;41 Suppl 1:20S-30S.
4. Schinkel-Bielefeld N, Kunz P, Zutz A, Buder B. Evaluation of Hearing Aids in Everyday Life UsingEcological Momentary Assessment: What Situations Are We Missing? Am J Audiol. 2020;29(3S):591609.
5. Nicoras R, Gotowiec S, Hadley LV, Smeds K, Naylor G. Conversation success in one-to-one and groupconversation: a group concept mapping study of adults with normal and impaired hearing. International
Journal of Audiology. 2022:1-9.
6. Lind C, Hickson L, Erber NP. Conversation repair and adult cochlear implantation: A qualitative casestudy. Cochlear Implants International. 2006;7(1):33-48.
7. Petersen EB, MacDonald EN, Josefine Munch Sorensen A. The Effects of Hearing-Aid Amplification andNoise on Conversational Dynamics Between Normal-Hearing and Hearing-Impaired Talkers. Trends Hear.
2022;26:23312165221103340.
8. Ter Bekke M, Drijvers L, Holler J. The predictive potential of hand gestures during conversation: An
investigation of the timing of gestures in relation to speech. 2020.
9. Hadley LV, Brimijoin WO, Whitmer WM. Speech, movement, and gaze behaviours during dyadic
conversation in noise. Scientific reports. 2019;9(1):1-8.
10. Ding N, Simon JZ. Emergence of neural encoding of auditory objects while listening to competing
speakers. Proceedings of the National Academy of Sciences. 2012;109(29):11854-9.
11. Miran S, Akram S, Sheikhattar A, Simon JZ, Zhang T, Babadi B. Real-time tracking of selective auditory
attention from M/EEG: A bayesian filtering approach. Frontiers in neuroscience. 2018;12:262.
12. Lesenfants D, Francart T. The interplay of top-down focal attention and the cortical tracking of speech.
Scientific reports. 2020;10(1):1-10.
13. Beechey T, Buchholz JM, Keidser G. Hearing Aid Amplification Reduces Communication Effort ofPeople With Hearing Impairment and Their Conversation Partners. J Speech Lang Hear Res.
2020;63(4):1299-311.
14. Smeds K, Larsson J, Dahlquist M, Wolters F, Herrlin P. Live Evaluation of Auditory Preference, aLaboratory Test for Evaluating Auditory Preference. J Am Acad Audiol. 2021;32(8):487-500.
15. Baker R, Hazan V. DiapixUK: task materials for the elicitation of multiple spontaneous speech dialogs.
Behavior research methods. 2011;43(3):761-70.
16. Beechey T, Buchholz JM, Keidser G. Eliciting naturalistic conversations: A method for assessing
communication ability, subjective experience, and the impacts of noise and hearing impairment. J Speech
Lang Hear Res. 2019;62(2):470-84.
17. Doherty-Sneddon G, Anderson A, O'malley C, Langton S, Garrod S, Bruce V. Face-to-face and videomediated communication: A comparison of dialogue structure and task performance. Journal ofexperimental psychology: applied. 1997;3(2):105.
18. Weisser A, Buchholz JM, Oreinos C, Badajoz-Davila J, Galloway J, Beechey T, et al. The Ambisonic
Recordings of Typical Environments (ARTE) Database. Acta Acustica united with Acustica.
2019;105(4):695-713.
19. Hohmann V, Paluch R, Krueger M, Meis M, Grimm G. The Virtual Reality Lab: Realization andApplication of Virtual Sound Environments. Ear Hear. 2020;41 Suppl 1:31S-8S.
20. Klatte M, Lachmann T, Meis M. Effects of noise and reverberation on speech perception and listening
comprehension of children and adults in a classroom-like setting. Noise Health. 2010;12(49):270-82.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0852

Evaluation of spatial tasks in virtual acoustic environments by meansof modeling individual localization performances
Roberto BARUMERLI1; Piotr MAJDAK1; Michele GERONAZZO2; David MEIJER1; Federico
AVANZINI3; Robert BAUMGARTNER1
1

Acoustic Research Institute, Austrian Academy of Sciences, Austria
Dept. of Management and Engineering, University of Padova, Italy
3
Dept. of Computer Science, University of Milano, Italy

2

ABSTRACT
Virtual acoustic environments (VAEs) are an excellent tool in hearing research, especially in the context ofinvestigating spatial-hearing abilities. On the one hand, the development of VAEs requires a solid
evaluation, which can be simplified by applying auditory models. On the other hand, VAE research
provides data, which can support the further improvement of auditory models. Here, we describe how
Bayesian inference can predict listeners' behavior when estimating the spatial direction of a static soundsource presented in a VAE experiment. We show which components of the behavioral process are reflectedin the model structure. Importantly, we highlight which acoustic cues are important to obtain accurate
model predictions of listeners’ localization performance in VAE. Moreover, we describe the influence of
spatial priors and sensorimotor noise on response behavior. To account for inter-individual differences, we
further demonstrate the necessity of individual calibration of sensory noise parameters in addition to theindividual acoustic properties captured in head-related transfer functions.
Keywords: sound localization, binaural sounds, probabilistic approach

1

INTRODUCTION

The fidelity of virtual acoustic environments (VAEs) is commonly assessed via psychoacoustic
experiments. However, these procedures are usually time-consuming and limited to small sample
sizes (1). A more efficient alternative utilizes computational models to evaluate VAEs for a larger
population (2). In particular, auditory localization models can be applied to investigate the various
attributes of spatial audio quality in VAEs (3). However, models that investigate sound localization
are usually limited to either the horizontal (4), vertical (5) or radial (6) dimension while VAEs
provide the users with a 3D sound experience.
To overcome this limitation, we here present a Bayesian inference framework that is capable ofincorporating multiple spatial features as well as prior knowledge. The probabilistic approach isfurther designed to accommodate inter-individual differences in perceptual uncertainty with the aim
to match the localization performance of a human individual to the best possible degree. Ourevaluation of the framework mainly focused on the vertical dimension where inter-individual
differences are quite pronounced (7).

roberto.barumerli@oeaw.ac.at1

Figure 1: (A) Structure of the Bayesian spatial auditory model framework; (B) directcomparison of subject and model performances in terms of lateral, polar and quadrant errors(LE, PE and QE, respectively) as defined in (1).
2

THE MODELThe basic structure of the model framework is depicted in Fig. 1A. The model first extracts
spatial features such as interaural time differences (ITDs), interaural level differences (ILDs), andmonaural spectral-shape cues from the binaural sound. These features are corrupted by additive
Gaussian noise with individual variance in order to limit the sensitivity to individual cues. Then,
those noisy input features are compared with location-specific templates to obtain a spatial
likelihood function. Bayesian inference then combines the likelihood with priors to form a posterior
distribution. The maximum of this distribution comprises the optimal choice in a spatial
identification task. Finally, a sensorimotor scatter is introduced to predict the listener’s spatial
pointing response in a localization experiment.
By using gradient extraction to extract monaural spectral-shape cues (6), a very simplistic
approach to extract ITDs and ILDs (8), and a persistent spatial prior reducing the likelihood of
elevated sources (9), the model can be fitted to listener-specific localization performance reasonably
well. Fig. 1B shows results for five listeners and three commonly used performance metrics,
evaluating localization accuracy on the horizontal dimension (LE, top row), on the verticaldimension (PE, middle row), and with respect to the frequency of large errors such as front/backconfusions (QE, bottom row). While the metrics of good localizers such as NH12 (most left column)
can be fitted very well, deviations between actual and simulation results remain especially for poor
localizers such as NH18 (most right column).
The model implementation is named as barumerli2022 in the Auditory Modelling Toolbox (10)and it can be tested by running the script demo_barumerli2022.

ACKNOWLEDGEMENTS

This work was supported by the European Union (EU) within the project SONICOM (grant
number: 101017743, RIA action of Horizon 2020) and the Austrian Science Fund (FWF) within the
project Dynamates (grant number: ZK66).

REFERENCES
1. Majdak P, Goupell MJ, Laback B. 3-D localization of virtual sound sources: Effects of visualenvironment, pointing method, and training. Attention, Perception, & Psychophysics. 2010
Feb;72(2):454–69.

2. Geronazzo M, Spagnol S, Avanzini F. Do We Need Individual Head-Related Transfer Functions for
Vertical Localization? The Case Study of a Spectral Notch Distance Metric. IEEE ACM Trans Audio
Speech Lang Process. 2018 Jul;26(7):1247–60.
3. Lindau A, Erbes V, Lepa S, Maempel HJ, Brinkman F, Weinzierl S. A Spatial Audio Quality Inventory(SAQI). Acta Acustica united with Acustica. 2014 Sep 1;100(5):984–94.
4. Dietz M, Ewert SD, Hohmann V. Auditory model based direction estimation of concurrent speakers
from binaural signals. Speech Communication. 2011 May;53(5):592–605.
5. Baumgartner R, Majdak P, Laback B. Modeling sound-source localization in sagittal planes for humanlisteners. The Journal of the Acoustical Society of America. 2014;136(2):791–802.
6. Baumgartner R, Majdak P. Decision making in auditory externalization perception: model predictions
for static conditions. Acta Acust. 2021;5:59.

7. Majdak P, Baumgartner R, Laback B. Acoustic and non-acoustic factors in modeling listener-specificperformance of sagittal-plane sound localization. Front Psychol. 2014 Apr 23;5.

8. Xie B. Head-Related Transfer Function and Virtual Auditory Display. Plantatation, FL: J. Ross
Publishing; 2013.

9. Ege R, van Opstal AJ, van Wanrooij MM. Accuracy-Precision Trade-off in Human Sound Localisation.
Sci Rep. 2018 Dec;8(1):16399.

10.Majdak P, Hollomey C, Baumgartner R. AMT 1.x: A toolbox for reproducible research in auditorymodeling. Acta Acust. 2022;6:19.

ABS-0631

A spatially-aware companion system for language learning andforeign-language dialogue
Albert Chang(1) , Samuel Chabot(2) , Jonathan Mathews(3) , Shannon Briggs(4) , Tomek Strzalkowski(5) , Mei Si(6) , Jonas
Braasch(7)(1) School of Architecture, Rensselaer Polytechnic Institute, USA, changa4@rpi.edu(2) School of Architecture, Rensselaer Polytechnic Institute, USA, chabos2@rpi.edu(3) School of Architecture, Rensselaer Polytechnic Institute, USA, mathej4@rpi.edu(4) Cognitive and Immersive Systems LaboratoryRensselaer Polytechnic Institute, USA, briggs4@rpi.edu(5) Cognitive Science, Rensselaer Polytechnic Institute, USA, tomek@rpi.edu(6) Cognitive Science, Rensselaer Polytechnic Institute, USA, sim@rpi.edu(7) School of Architecture, Rensselaer Polytechnic Institute, USA, braasj@rpi.edu

ABSTRACT
Dialogue systems, combined with deep learning technologies, have increased the potential for practical applications across many disciplines. One such vein of applications involves multi-modal dialogue systems deployed
in interactive spaces that seek to provide an immersive experience for participants. This project proposes a combination of spatial awareness with a multi-modal, immersive dialogue system as a potential interactive medium
to provide an additional layer of immersion. The system employs an array of audio/visual sensors that track
participants within the interactive space. It responds contextually depending on the application and information
domain, for example, by displaying and sonifying conversational agents at accurate spatial locations. The current application of this system involves foreign language learning (i.e., Mandarin Chinese), in which the system
will act as both a learning medium and conversation augmentation system to provide students with an immersive
environment to learn a language and provide real-time feedback during the learning process. This project aims
to provide insight into interactive spaces for education and general conversation applications and demonstrate
the capabilities of combining spatial awareness with a multi-modal dialogue system.

Keywords: Dialogue, Immersion

1

INTRODUCTION

Virtual reality systems have found their place in education and training scenarios, with studies finding significant
benefits over traditional learning methods [26, 20, 14, 19, 15, 23]. A remaining problem for long-term trainingis the effect of cybersickness when using head-mounted displays [21, 34]. A recent study showed that evenwhen watching a 3D movie, about 55% of the viewers complain about the side effects [30]. As an alternative
approach, Rensselaer has developed, erected, and operated two large-scale collaborative, immersive cognitive
laboratories with panorama screens, the CRAIVE-Lab [5, 3, 28] and the Cognitive Immersive Room (CIR) at
the Experimental Media and Performing Arts Center (EMPAC) [4, 12, 11]. We define collaborative, immersive
cognitive systems as environments where:
1. Groups of human collaborators can communicate naturally with each other without obstruction from wearable devices (collaborative),
2. Collaborators are exposed to situational and multi-sensory virtual environments facilitated by human-scale
panoramic projections with spatially-aware audiovisual rendering capabilities (immersive),
3. Users can draw from AI-based computing capabilities (cognitive).

Figure 1. CAD model of the CRAIVE-Lab (usable floor area: 12×10 sqm).
Both labs can serve as classrooms providing access to up to 49 students while mitigating cybersickness. The
CRAIVE-Lab has a useable floor space of 12×10 sqm (Fig. 1); the Cognitive Immersive Room (CIR) has adiameter of 12 m. The CIR hosts several projects including the Mandarin Project [10], which aims to teachstudents Chinese language in an immersive way. Several scenes have been developed for the project including
the restaurant scene shown in Fig. 2. Both labs provide a new form of embodied learning, one where thelearned material no longer needs to be scaled to the real world because situated learning is very similar to thereal world experience. One no longer needs to recall what was written in the textbook for what to say to theChinese customs officer because one learned the appropriated phrases in an immersive, human-scale environment
talking to a life-size interactive avatar presented on screen [11].
The theories of embodied cognition suggest that people naturally construct and utilize environmental cues tohelp them reason and lower cognitive workload [1, 9, 16]. An important feature of working in an immersive
system is the affordance of using body movements and gestures that are intuitive to communicate and performtasks and therefore leave the interface transparent. For example, the user can “grab” and “drag” new information
and “throw away” irrelevant items. In contrast, most existing computer-based learning systems require the users
to sit in front of a computer and use a keyboard and mouse to interact. The users’ body movements are genericfor operating a computer and have nothing to do with their tasks. Immersive cognitive systems thus can allow
users to interact with the digital environment in a similar way as they interact with the physical world. By
turning the meaningless hand movements of mouse click and pulling down a drop-down menu into something
consistent with the operation’s effects, we expect immersive cognitive systems to be more engaging and natural
to use.

Figure 2. Scene from the Mandarin Project Class. The restaurant environment was created using Unity.
This paper describes a method of using a dialogue system with an immersive educational environment tofacilitate interactions between students and automated services, for example, a tutor avatar that teaches a student
vocabulary. The immersive dialogue system for educational use should possess the following components: (i)A dialogue system that receives input from the user, decides on a response, and sends the response to theappropriate visual and audio output systems, (ii) Audio and visual hardware to receive input from participants
and output their respective responses, (iii) A database/third-party resource to draw answers from regarding thedomain(s) of interest, (iv) A spatial tracking system to identify where participants are in the space and identify
individual participants, and (v) A visual avatar system that is displayed to the participants.

2

IMMERSIVE SYSTEM INFRASTRUCTURE

The CRAIVE-Lab addresses the need for a specialized virtual-reality (VR) system for the study and enabling ofcommunication-driven tasks with groups of users immersed in a high-fidelity multi-modal environment locatedin the same physical space [6]. It is therefore ideal for the integration of a dialog system. For the visual
domain, an eight-projector front-projection display to (re)create scenes on a seamless screen has been created.
A DMX-controlled lighting system can tune the color and intensity of the interior light, which illuminates
the participants without much spill to the screen area. For the acoustic domain, a 134-loudspeaker-channel
system has been designed and installed for Wave Field Synthesis (WFS) with the support of Higher-OrderAmbisonic (HoA) sound projection to render inhomogeneous acoustic fields (Fig. 3). The screen material is
micro-perforated to minimize acoustical reflections. An intelligent position-tracking system estimates current
user locations and head orientations as well as positioning data for other objects. For the tracking system, ahybrid visual/acoustic sensor system is being used to emulate the humans’ ability to extract robust information
by relying simultaneously on different modalities. A network of six cameras has been installed in the CRAIVELab as well as a time-of-flight sensor array using six Microsoft Kinects. A 16-channel spherical ambisonicmicrophone with additional peripheral microphones is used for acoustical tracking – see Fig. 3.

Figure 3. Audio system of the CRAIVE-Lab, from left to right: Audio rack, loudspeaker system, sphericalmicrophones.

3

SPATIAL TRACKING

In order to localize participants, an acoustic tracking system has been developed and implemented into the
CRAIVE-Lab [25, 24]. The audio tracking system uses a 16-channel spherical microphone array in conjunction
with a sparse iterative beamforming algorithm to enable low-latency estimation of acoustical sound sources at alow computational cost. Our Sparse Iterative Search (SIS) method starts out by analyzing the whole sphericalarea with equal-area grid size. The algorithm computes the received energy in each area and contracts theanalyzed area and grid size based on the received energy. Using an iterative approach, the algorithm zones into
the energy-emitting sound sources with high accuracy due to the small grid size. Incorporating conditions basedon temporal smoothing and diffuse energy estimation further refines this process. This way, the algorithm cantrack up to four simultaneous static or moving sound sources.

Figure 4. Localization errors for different audio tracking algorithms as a function of reverberation time –from [24].

Figure 5. Architecture of the integrated audio-visual tracking/speech isolation system – from [24].
The system was successfully tested against existing algorithms, including Coarse-to-Fine Region Contraction (CFRC), Eigenbeam Estimation of Signal Parameters with Rotationally Invariant Techniques (EB-ESPRIT),
Pseudo-intensity Vector (PIV), Sparse Iterative Search (SIS), Stochastic Region Contraction (SRC), and Steered
Response (SRP). The Sparse Iterative Search (SIS) method maintains an average localization error of about 8◦
at 2 seconds of reverberation time (T60 ), while all other methods showed errors of more than 12◦ , typically ata much higher computational cost – see Fig. 5 and [24].
The audio tracking system has been integrated with a time-of-flight sensor-based tracking system as shownin Fig. 5. The hybrid tracking system received input from the spherical microphone (label:“HoA Array Data”)
using the higher-order ambisonics (HoA) format and six time-of-flight (ToA) sensors (Microsoft Kinect), seelabel: “ToF Array Data”. The acoustic data is then processed through the previously described Sparse Iterative
Search (SIS) method – see box “Acoustic Source Localization.” The time-of-flight sensor data is analyzed using
a Gaussian mixture model with background subtraction [33, 2, 27, 29, 18] – see box “Estimation Of Subject
Position.” Given that the ToF data is processed using a Cartesian Coordinate system and the HoA data usinga polar coordinate system, a transformation has to be applied to bring both data sets into the same coordinate
system – see box “Coordinate Transform.” Here, it is also noteworthy that the angles of the acoustic source
signal can be much better determined than the distance of the source from the spherical microphone. The
additional ToF data partially resolve the distance uncertainty. Alternatively, one could also use a second HoA
array to triangulate the sound source. Detected utterances are assigned to the estimated location of the soundsource (box: “Speech Activity Data Association”) and both can be used used to steer a beamformer (box:“Beamforming”). For details see [25].
Using this multi-modal approach, the participants can be tracked optimally. When a participant is talking, we
can identify the source of the utterance and assign this utterance to the visually tracked object. We can continue
to track the participants when moving in quiet. By tracking participants acoustically, the tracking process is alsoless susceptible to difficult lighting conditions, although the latter can be adjusted automatically using a DMX
lighting system. Under certain conditions, the participants benefit from a dark interior, for example, when being
presented with a movie or panoramic photography/videography.

4

DIALOGUE SYSTEM

4.1 Background
Dialogue systems involve human-computer conversations with at least one human and one computer system.
The applications of dialogue systems are ubiquitous, such as personal assistants available on smartphones such
as Google Assistant and Siri, non-player characters in video games, customer service bots that handle frequently
asked questions, and chatbots that employ machine learning to hold casual conversations. All of these systems
fundamentally use either or a combination of pre-defined choices and Natural Language Understanding as inputs
and employ a dialogue logic system to determine the best course of action in responding. Using their application
as a basis, dialogue systems can be categorized into two groups: task-oriented and non-task-oriented systems[8].

Third Party
Information

Dialogue
EngineParticipant information

Database

Participant information

Speech to
Text

Gesture/Body
Language
InterpreterGame Engine

Gesture/Body
Avatar Output/w Lipsync

Text toSpeech
OutputTracking data

User

Figure 6. Dialogue System Architecture.
Task-oriented systems, akin to their name, aim to assist users with a task of some sort, whether that bebooking travel tickets or finding products. These systems focus on a goal or set of goals to accomplish by theend of the conversation and may have additional mitigation to minimize off-topic conversation and task failure. These systems typically consist of Natural Language Understanding (NLU), dialogue state tracking, policy
learning, and Natural Language Generation (NLG). In a nutshell, each component, respectively, is responsible
for parsing inputs, predicting the user’s goal, generating the best system action, and finally responding withnatural language generation, e.g., see [8].
Non-tasked-oriented systems, on the other hand, focus on conversing on open domains where the conversation is steered by the human(s) interacting with the system. To accomplish open-domain discussion, non-taskedoriented systems use a retrieval-based, generative, or hybrid approach to dialogue responses [31]. Retrievalbased systems can employ more intelligent and fluent responses [17], while generative systems allow for more
context-aware responses that are not part of the original corpus [32].
4.2 Dialogue System Architecture
The immersive dialogue system in this project consists of a three-step process that handles visual and audio
inputs and outputs separately but uses both to provide the most amount of context needed to best respond in
the current dialogue. First, the systems uses a speech-to-text system, gesture/body language interpreter, andtracking system to handle inputs, processes the inputs in the dialogue engine. Second, working with the gameengine in tandem, the dialogue engine gathers information from third-party sources and an internal databaseand decides on a combined audio/visual response. Finally, the response is carried out through the avatar systemand text-to-speech output. The design of the immersive dialogue system builds upon the work done by Divekar
et al. [13] for the initial prototypes for the Mandarin Project by using the core architecture and extends its
functionality to include auditory spatial tracking enhancements in addition to a virtual human avatar that can
act as both the subject of a conversation and an assistant to group conversations.
Figure 6 shows the dialogue system design for the immersive dialogue agent that also includes an avatar forvisual representation. Another example, depicted in Fig. 7, shows students interacting within a garden scenario
in the Mandarin Project where they can talk to agents to participate in various activities, such as one that willteach basic Tai chi (white-clothed avatar pictured to the right).
Visual and audio components are utilized as described in the Immersive System Infrastructure and Spatial
Tracking sections to track participants as they use the system. The additional functionality of agent gaze canbe implemented as a standalone application using cameras that also takes in inputs from the visual and audio
tracking system, displays an image of an agent, and adjusts the agent’s gaze to look towards the head of theactive participant. Identifying the active speaker requires both a visual identification in the installation spacealong with the participant currently or most recently speaking.
A primarily task-oriented system was decided upon as the basis for the dialogue system due to its intended
purpose of acting as an assistant to language learning and foreign-language dialogue. The unpredictable anddemanding nature of multi-user interactions with real-world conversations also proposes a unique challenge fora multi-modal system, which makes the inherently unpredictable nature of generative dialogue systems lesssuitable for this application. Furthermore, there are currently no standardized designs nor testing proceduresfor comparison. As such, determining the effectiveness renders generative-leaning models too unpredictable for
evaluating initial user feedback to a system intended to educate or assist.

Figure 7. Students interacting with the Mandarin dialogue system.
5

DISCUSSION & OUTLOOKThe current application of the immersive system with the Mandarin Project demonstrates the potential reachand impact through enabling potential content creation by such a system. A successful pilot program has been
conducted using the space to supplement an undergraduate Chinese class held at Rensselaer Polytechnic Institute[10]. Both student feedback and learning performance were notably positive, which shows promising results for
immersive educational content. Future work for the project aims to integrate the Mandarin Project as a formal
addition to the Chinese curriculum at Rensselaer with expanded content covering more topics and interactablescenarios.
Additional future work will focus on a practical demo involving a game of “I Spy” where participants will
locate objects or points of interest in an environment by responding in Mandarin and/or pointing at an item.
The demo will employ additional spatial tracking features to enhance immersion while playing the game bybuilding upon the panoramic imagery system from Chabot et al. [7]. In addition, an investigation will be doneon incorporating the additional features of the new spatial tracking system into the dialogue, as adding multimodal components has been demonstrated to improve generative-based dialogue systems. In this context, Liaoet al. combined image data with text in the fashion domain and found their hybrid model outperformed otherstate-of-the-art models [22].

The potential for other applications outside of the education sector cannot be understated either, as potential
use cases such as remote monitoring and collaboration for business applications or site experiences for tourism
can exploit the spatial awareness and group engagement the immersive system allows.

ACKNOWLEDGEMENTS
This material is based upon work supported by the National Science Foundation under Grant No. HCC-1909229,
the United States Air Force under Contract No. FA8750-21-C-0075 and the IBM Corporation under the Artificial
Intelligence Research Collaboration Agreement No. W1771793 between IBM and Rensselaer. Any opinions,findings and conclusions or recommendations expressed in this material are those of the author(s) and do notnecessarily reflect the views of NSF, USAF, or IBM Corporation. The research presented in this paper was
carried out within Rensselaer Polytechnic Institute’s Cognitive and Immersive Systems Laboratory.

REFERENCES[1] L. W. Barsalou. Grounded cognition. Annu. Rev. Psychol., 59:617–645, 2008.
[2] I. Bhattacharya, M. Foley, N. Zhang, T. Zhang, C. Ku, C. Mine, H. Ji, C. Riedl, B. F. Welles, and R. J.
Radke. A multimodal-sensor-enabled room for unobtrusive group meeting analysis. In Proceedings of the20th ACM International Conference on Multimodal Interaction, pages 347–355, 2018.
[3] J. Braasch, J. Carter, S. Chabot, and J. Mathews. An immersive teleconferencing system using sphericalmicrophones and wave field synthesis. In 22nd International Congress on Acoustics, pages Paper ICA2016–
826, Buenos Aires, Argentina, September 2016.
[4] J. Braasch, J. Goebel, and T. Vos. A cinematic spatial sound display for panorama video applications.
Organised Sound, 15(3):260–270, 2010.
[5] J. Braasch (PI), R. Radke (Co-PI), B. Cutler (Co-PI), J. Goebel (Co-PI), and B. Chang (Co-PI). MRI: Development of the Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVELab), 2012–2015. NSF #1229391.
[6] S. Chabot and J. Braasch. Interactive application to control and rapid-prototype in a collaborative immersive environment. In Audio Engineering Society Convention 151. Audio Engineering Society, 2021.
[7] S. Chabot, J. Drozdal, M. Peveler, Y. Zhou, H. Su, and J. Braasch. A collaborative, immersive language
learning environment using augmented panoramic imagery. Proceedings of 6th International Conference of
the Immersive Learning Research Network, iLRN 2020, pages 225–229, 6 2020.
[8] H. Chen, X. Liu, D. Yin, and J. Tang. A survey on dialogue systems: Recent Advances and New Frontiers,
11 2017.
[9] A. Clark. Supersizing the mind: Embodiment, action, and cognitive extension. OUP USA, 2008.
[10] R. R. Divekar, J. Drozdal, S. Chabot, Y. Zhou, H. Su, Y. Chen, H. Zhu, J. A. Hendler, and J. Braasch.
Foreign language acquisition via artificial intelligence and extended reality: design and evaluation. Computer Assisted Language Learning, pages 1–29, 2 2021.
[11] R. R. Divekar, J. Drozdal, Y. Zhou, Z. Song, D. Allen, R. Rouhani, R. Zhao, S. Zheng, L. Balagyozyan,
and H. Su. Interaction challenges in ai equipped environments built to teach foreign languages throughdialogue and task-completion. In Proceedings of the 2018 Designing Interactive Systems Conference, DIS’18, pages 597–609, New York, NY, USA, 2018. ACM.
[12] R. R. Divekar, M. Peveler, R. Rouhani, R. Zhao, J. O. Kephart, D. Allen, K. Wang, Q. Ji, and H. Su.
CIRA: An architecture for building configurable immersive smart-rooms. In Proceedings of SAI Intelligent
Systems Conference, pages 76–95. Springer, 2018.

[13] R. R. Divekar, M. Peveler, R. Rouhani, R. Zhao, J. O. Kephart, D. Allen, K. Wang, Q. Ji, and H. Su.
Cira: An architecture for building configurable immersive smart-rooms. pages 76–95, 2019.
[14] H. A. El-Mounayri, C. Rogers, E. Fernandez, and J. C. Satterwhite. Assessment of stem e-learning in an
immersive virtual reality (VR) environment. American Society for Engineering Education, 2016.
[15] E. Fokides and P. Atsikpasi. Factors affecting primary school students’ learning experiences when using
muves: Development and validation of a scale. In Integrating Multi-User Virtual Environments in Modern
Classrooms, pages 185–206. IGI Global, 2018.
[16] S. Goldin-Meadow and M. W. Alibali. Gesture’s role in speaking, learning, and creating language. Annual
review of psychology, 64:257–283, 2013.
[17] Z. Ji, Z. Lu, and H. Li. An information retrieval approach to short text conversation. 8 2014.
[18] L. Jia and R. J. Radke. Using time-of-flight measurements for privacy-preserving tracking in a smart room.
IEEE Transactions on Industrial Informatics, 10(1):689–696, 2013.
[19] S. H. Kidd and H. Crompton. Augmented learning with augmented reality. In Mobile Learning Design,
pages 97–108. Springer, 2016.
[20] P. T. Kovács, N. Murray, G. Rozinaj, Y. Sulema, and R. Rybárová. Application of immersive technologies
for education: State of the art. In Interactive Mobile Communication Technologies and Learning (IMCL),2015 International Conference on, pages 283–288. IEEE, 2015.
[21] J. J. LaViola, Jr. A discussion of cybersickness in virtual environments. SIGCHI Bull., 32(1):47–56, Jan.
2000.
[22] L. Liao, Y. Ma, X. He, R. Hong, and T.-S. Chua. Knowledge-aware multimodal dialogue systems. pages
801–809. ACM, 10 2018.
[23] J. Lucas. Immersive VR in the construction classroom to increase student understanding of sequence,assembly, and space of wood frame construction. Journal of Information Technology in Construction (ITcon),23(9):179–194, 2018.
[24] J. Mathews. Development and evaluation of spherical microphone array-enabled systems for immersive multiuser environments. PhD thesis, Rensselaer Polytechnic Institute (RPI), Troy, NY, 2021.
[25] J. Mathews and J. Braasch. Sparse iterative beamforming using spherical microphone arrays for lowlatency direction of arrival estimation in reverberant environments. Journal of the Audio Engineering Society,69(12):967–977, 2021.
[26] M. Murray. Realizing the vision of a vr-based learning environment. Medicine Meets Virtual Reality: The
Convergence of Physical & Informational Technologies: Options for a New Era in Healthcare, 62:134, 1999.
[27] G. Sharma. Spatially aware interactions in large scale immersive environments, 2019.
[28] G. Sharma, J. Braasch, and R. Radke. Interactions in a human-scale immersive environment: The
CRAIVE-Lab. In Cross-Surface 2016, in conjunction with the ACM International Conference on Interactive
Surfaces and Spaces, 2017.
[29] G. Sharma, J. Braasch, and R. J. Radke. Interactions in a human-scale immersive environment: The craivelab. In Cross-Surface 2016, in conjunction with the ACM International Conference on Interactive Surfaces andSpaces, 2017.
[30] A. G. Solimini. Are there side effects to watching 3d movies? a prospective crossover observational study
on visually induced motion sickness. PloS one, 8(2):e56160, 2013.

[31] Y. Song, R. Yan, X. Li, D. Zhao, and M. Zhang. Two are better than one: An ensemble of retrieval- andgeneration-based dialog systems. 10 2016.
[32] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Y. Nie, J. Gao, and B. Dolan. A
neural network approach to context-sensitive generation of conversational responses. NAACL HLT 2015
- 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Proceedings of the Conference, pages 196–205, 6 2015.
[33] Z. Wu. Multi-object tracking and association with a camera network, 2014.
[34] P. Ziegler, D. Roth, A. Knots, M. Kreuzer, and S. von Mammen. Simulator sick but still immersed: A
comparison of head-object collision handling and their impact on fun, immersion, and simulator sickness.
In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 743–744. IEEE, 2018.

ABS-0807

A Case Study of Immersive Sound System in a Classical Concert- The unique musical production combining reverberation and sound image controls Hiromu Ohgi(1) , Dai Hashimoto(2) , Takayuki Watababe(3) , Hideo Miyazaki(4) , Tomomi Nishimoto(5)(1) Spatial Audio Group, Yamaha Corporation, Japan, hiromu.ohgi@music.yamaha.com(2) Spatial Audio Group, Yamaha Corporation, Japan, dai.hashimoto@music.yamaha.com(3) Spatial Audio Group, Yamaha Corporation, Japan, takayuki.watanabe@music.yamaha.com(4) Spatial Audio Group, Yamaha Corporation, Japan, hideo.miyazaki@music.yamaha.com(5) Conductor, Musicologist, Japan

Abstract
Immersive sound systems are now widely adopted in music performances, theater performances, and media arts.
We have regarded the immersive sound system as a system for controlling a sound field, rather than just a toolfor 3D productions. In general, a sound field comprises a sound environment influencing spatial impression
like a reverberance, and sound elements such as the type, location, and size of sound sources. To develop
the immersive sound system into a sound field creation system, we propose the combination of two systems:
reverberation and sound image controls. The system named active field control (AFC) has been developed tocontrol a sound environment as a tool for acoustic design, and has been widely installed at several venues.
Because an image control function has been implemented to the latest AFC system, the entire sound field canhave been controlled. In this study, a unique classical music concert, which was held using AFC system tooptimize the sound fields for each music piece, is investigated. The proposed system was also utilized to sharesound fields and the performances of players in different rooms.

Keywords: immersive sound system, reverberation control, sound image control

1

INTRODUCTIONOn October 8, 2021, Tomomi Nishimoto, artistic director and conductor, presented the classical concert “MARINART becomes a Forest” at the Shimizu Cultural Hall “MARINART” in Shizuoka city. This classical concert
was planned and directed by Nishimoto under the auspices of the “ARTS for the future project,” and supportedby the Agency for Cultural Affairs in Japan, to bolster the enhancement of cultural and artistic activities andovercome the Corona disaster.
Owing to the effects of COVID-19, distance restrictions have been enforced in various situations, and occasions to visit any places or experience anything have been limited. Therefore, in this performance, the total
stage production including lighting and sound was designed to realize a “Journey of the inner mind” in which
the audience could feel the different scenes corresponding to each performed music piece like traveling different spaces without leaving their seats. We provided technical support as members of the production team. A
summary of the performance is presented in Table 1.
In realizing the “Journey of the inner mind,” we considered that in addition to the music, the spatial impression of a sound field is important. Two factors represent the spatial information of a sound field: soundenvironment and sound elements [1]. The sound environment is the acoustic characteristics of a space and isperceived with some spatial extent. In contrast, the sound elements are sound sources presented in a space
and are perceived at a specific location within a space. Hence, in this performance, to enhance the audience’s
immersion in the music, we controlled the sound environment and sound elements according to the scenes ofthe pieces.

Table 1. Specifications of the concert
Name

MARINART becomes a Forest

Date

October 8, 2021

Venue

Shizuoka City Shimizu Cultural Hall “MARINART”

Artistic Director

Tomomi Nishimoto

Performer

Conductor

Tomomi Nishimoto

Recitation

Yoshiko Sakuma

Orchestra

IlluminArt Philharmonie Orchestra

ChorusIlluminArt Chorus, Narumi Yamaguchi(Sop.)

Puccini / “Suor Angelica” - Ave Maria, Intermezzo
Smetana / “Má vlast” - Vltava
ProgramGlazunov / Ballet “The Seasons” - Autumn, Winter highlight
Debussy, arr. Nishimoto / “Rêverie”<Yoshiko Sakuma, lyrics and recitation / A Tale of Beech Forest>

In live performances such as classical concerts, a sense of intimacy between the performers and audience isalso important to give the audience immersions in the music. This sense is usually created when the performers
and audience feel like they are located in the same space. However, for this concert, the chorus had to be in adifferent room from the orchestra and audience to prevent infection. Therefore, we attempted to share the same
sound field among the performers and audience even though they were in different rooms. To share the soundfield, we considered it would be effective to exchange the sound environment and sound elements rather thansimply transmit sound signals.
For this concert, Yamaha’s active field control (AFC) system [2, 3], a sound field control system that controls reverberation and sound image, was utilized to realize the aforementioned productions. In addition, thecharacteristics of the original sound field are also significant to control the sound field. Therefore, this facility,
which an acoustic design was developed by the authors [4], was selected as the concert venue. By utilizing
our knowledge of the acoustic design, we attempted to design a sound system while making the best use of
the hall’s characteristics and functions. This study reports an overview of the system adopted for the acoustic
productions and its effects.

2

AFC SYSTEM

AFC system is a sound field control system that combines a reverberation control system (Rev system) anda sound image control system (Image system). AFC system was originally developed as an acoustic design
tool to control the sound field in a room, and in 2021, the sound image control function was implemented inaddition to the conventional function. This enabled total control of the room’s sound field, including the spatial
localization of sound elements and reverberation.
2.1 Rev system
The Rev system is a system that naturally alters the main auditory impressions of the sound environment, suchas reverberation, early reflections, and loudness, by adopting electroacoustic technology. In general, acoustic
control systems can be divided into in-line and regenerative methods [5]. The in-line system is a method inwhich the processor adds reverberation to the signal recorded using a microphone placed at a distance close to
the sound source, and then plays them back from the loudspeakers while avoiding the acoustic feedback. In
contrast, the regenerative system actively utilizes acoustic feedback loop, in which diffuse sound is recordedby microphones placed at some distance away from the sound source and reproduced via a loudspeaker, which

Figure 1. Basic model diagram of Rev system

Figure 2. Section of the venue

again becomes the input signal for the microphones.
In the AFC Rev system, both the in-line and regenerative methods are implemented, and by combining
them, natural and highly variable reverberation control can be achieved. The basic model of the Rev system is
illustrated in Figure 1. The Rev system adopts impulse response data recorded in an actual hall. In particular,
the in-line method utilizes impulse response data recorded at the boundary of the hall walls, and by usingreflected sound data of different halls, it plays an important role in controlling the spatial impression of thesound field.
2.2 Image system
The Image system controls sound image localization using 3D amplitude panning, a highly realistic acoustic
technology based on psycho-acoustics [6]. The system supports object-based audio (OBA) [7] and can be
configured with any speaker placement and configuration. Although the sound object information of OBA is
generally normalized by a cubic shape, the Image system can adjust its information into polygons.
The Image system adopts open sound control (OSC) [8] to receive information such as the position andsize of the sound image. This makes it possible to control the sound image information from a digital audio
workstation(DAW), mixing console, or PC software that can handle OSC.

3

ACOUSTICAL FEATURES OF VENUE

The concert was held at shimizu cultural hall “MARINART.” This venue is a cultural complex, and the authors
were in charge of its acoustic design. This facility has two multi-purpose halls (a main hall and small hall), a
rehearsal room, and two practice rooms. The main hall, small hall, and rehearsal room were utilized for thisconcert. The halls specifications are presented in Table 2, and a layout of the venue is illustrated in Figure 2.
The main hall is designed primarily for musical use, with a sound field suitable for acoustic instrument
performances. The basic shape of this hall is a shoebox type, which provides a rich auditory source widthand listener envelopment. To realize a sufficient reverberance, the ceiling was raised to get a room volume perseat of more than 10m3 . Accordingly, the reverberation time is 2.1s (250Hz to 2kHz average value) with theunoccupied condition, which is suitable for musical performances. The stage is surrounded by movable acoustic
shells, which can be turned in to accommodate a various events, such as pop concerts, drama performances,and ceremonies. The reverberation frequency characteristics of the main hall are illustrated in Figure 3.
The small hall is a multipurpose hall for civic use, suitable for both theater and music. The small hall isalso equipped with movable acoustic shells to accommodate a variety of uses. In order to obtain a good soundimpression and reverberation for musical performances, the interior ceiling is not affixed to create room volume.
Furthermore, sound-absorbing curtains installed above the seating area provide a variable reverberation range of
0.1s, and can be used to adjust sound absorption. The reverberation time of the small hall is 1.4s with theunoccupied condition, and the reverberation frequency characteristics are presented in Figure 3.
The rehearsal room was designed to be used for stage rehearsals and small recitals. To achieve the same

Table 2. Specifications of the venue
Main hall
Configuration

ConcertCapacity (N)Drama

Small hall
Concert1,513

Reharsal room

Drama

292

-

Volume (V) [m3 ]15,30012,900

2,500

2,200

740

Surface area (S) [m2 ]

5,400

4,800

1,400

1,200

620

V/S [m]

2.8

2.7

1.8

1.8

1.2

V/N [m3 ]

10.1

8.5

8.6

7.4

-

Reverberation time [s] (Unoccupied)

2.1

1.6

1.4

1.0

0.9

Reverberation time [s] (Occupied)

2.0

1.5

1.3

0.9

0.7

* Reverberation time is 250Hz to 2kHz octave band average* Reverberation time at full occupancy is an estimated value

Figure 3. Reverberation time of the main hall and small hall
performance impression as on the stage of the main hall, the room volume was made as large as possible, andthe floor was constructed to be the same as that of the stage. The reverberation time of the rehearsal room is
0.9s with the unoccupied condition.
A distinctive feature of this facility is that these rooms are connected to each other by electroacoustic lines.
This allows for the combined use of these rooms. Furthermore, this facility was designed under the Private
Finance Initiative (PFI) program. Therefore, the staff and operators have high knowledge about the design
intent and functions of the facility. This enables them to provide accurate support for even large-scale events
that use multiple rooms in complex ways.
Because the sound field of the halls were suitable for classical music, the entire halls could be used flexibly,
and the staff could provide advanced support; hence, the concert was held at this facility.

4

SYSTEM DESIGN IN SOUND PRODUCTIONS

4.1 Outline of sound production
In this concert, we adopted the Rev and Image systems for two primary purposes: (1) creating sound spaceaccording to the scene of the piece and (2) sharing sound space in several rooms. For (1), to enhance a sense of
immersion in the scenery expressed in the pieces, we planned to change the acoustics of the venue synchronizedTable 3. Overview of sound productions
Concept

Production detail(A) Reverberation enhancement of the venue to match the scene in the piece(1) Creation of sound space(B) 3D Playback of environmental sounds representing scenes(2) Sharing sound space in multiple spaces(C) Transmission of sound to multiple spaces with low latency(D) Bringing impression of the reverberation closer between rooms.

Combination of (1) and (2)(E) Real-time rendering and 3D playback of sounds in another space

Improving stage support(F) Construction of a virtual acoustic shell in the hall

Table 4. Production List
Main hall

Small hall

Reharsal room

Audience

X--

Performers

Orchestra

ChorusSolo Instruments

(A)

X

-

-

(B)

X

-

-

(C)

X

X

X

(D)

-

X

X

(E)

X

-

-

(F)

X

-

-

Figure 4. System block diagramwith the pieces, and to position and play environmental sounds representing the scenery in 3D during and inbetween music. For (2), we planned to transmit sound signals with low latency, to enhance a sense of sharing
music, as well as close the impression of reverberance between rooms. Furthermore, to harness the fact thatbeing performed in separate spaces, we were able to place the sound image of each space in various areas in
the venue.
In this performance, because of the stage and lighting production, it was impossible to install the acoustic
shells. Therefore, to improve the play-ability of the stage, we attempted to construct a virtual acoustic shellthat assists in easing the performance by adopting the Rev system [9]. The details of the sound productions are
summarized in Table 3.
4.2 Overview of sound system
For this performance, the performers were divided into the main hall, small hall, and rehearsal room in thevenue. The orchestra performed in the main hall, and the chorus in the small hall, while the solo instruments
and piano performed in the rehearsal room. The details of the acoustic performance in each room are presented
in Table 4. To perform music in three different spaces simultaneously, a low-latency sound transmission methodthat can support multiple channels was required. Hence, signal transmission over a digital audio network wasused as the basis of the system. This enabled us to obtain the necessary signals for each room from the audio

Figure 5. Main hall loudspeaker layout

Figure 6. View of each room

network, as well as realize multi-channel signal transmission without complicated wiring. The system block
diagram is presented in Figure 4.
4.2.1 Main hallIn the main hall, the output system of AFC system was configured with a total of 56 loudspeakers combining
installed and brought-in loudspeakers. The layout of the equipment is illustrated in Figure 5.
For the production (A), in which the reverberation in the hall changes according to the music scene, a large
variable range of reverberation was required to enhance the effect of the production. Consequently, the in-line
method was adopted, with microphones signal near the performers as the input signal, to ensure safety against
feedback. This enabled a long reverberation-like huge cathedral effect for the piece that represented a church
scene, and an atmosphere in a forest for the piece that represented a forest scene. In addition, the regenerative
method was adopted to construct a virtual acoustic shell for the performance (F), and it also supported theplayer’s performance. Because the regenerative method uses the hall’s original reverberation, the sound energy
and reverberation on the stage were naturally enhanced. Hence, we created an acoustic field in which theperformers could easily perform with comfort.
For the 3D reproduction of environmental sounds in production (B), 3D acoustic content that had been
produced beforehand was reproduced on a DAW and rendered to loudspeakers in real time using the Image
system. For the production (E), in which the sound image in other spaces was moved, its position and size
were changed in real time to enhance the effect of the production. The sound images were smoothly moved
and positioned, which included moving a bird motif by piccolo as if a bird were flying, or arranging thehumming by chorus, such that it enveloped the hall.

Table 5. Specifications of Yamaha Ginza Studio
Capacity (N)Seats 96
Standing 250

Volume (V) [m3 ]1,000

Surface area (S) [m2 ]670

Reverberation time [s] (Unoccupied)

0.6* Reverberation time is 250Hz to 2kHz octave band average

Figure 7. Confirmation of sound production at
Yamaha Ginza Studio
4.2.2 Small hallIn the small hall, the Rev system realized a production (D) that makes the impression of the room reverberation
feel closer to that of the main hall. In the small hall where the chorus was performed, it was necessary to avoid
creating the uncomfortable feeling triggered by the varied reverberation. Therefore, the regenerative method was
adopted as the Rev system, which harnesses the original reverberation of the small hall. This allowed a naturalextension of the reverberation and created a sound similar to that of the main hall.
4.2.3 Rehearsal room
In the rehearsal room, similar to the small hall, the Rev system was adopted to realize production (D). Unlike the small hall, the rehearsal room has a small room volume, and its original reverberation time is short.
Therefore, the in-line method, which is a widely variable reverberation control method was used.

5

PERFORMANCE PREPARATION: Sound Productions Confirmation

To verify the effect of the sound productions beforehand, a demonstration of the sound productions was conducted in the studio of the Yamaha Ginza Building installed with AFC system. The picture taken that day isshown in Figure 7, and specifications of the studio are presented in Table 5
Because the size and shape of the rooms in the studio differed from those in the performance hall, it
was assumed that the sound effects would be heard differently in the studio. To create a sound space thatassumes a production venue, we attempted to create a sound space by adopting AFC system. First, the Rev
system was adopted to create a sound environment similar to that of the actual performance venue, and theregenerative method was used to enhance the reverberation of the entire studio. Next, the Image system wasused to reproduce the position and size of the sound elements. The Image system supports OBA; hence, it
can reproduce sound image information, such as position and size, even in a space with different loudspeakerconfigurations. This made it possible to verify the contents of the production even in a space that differed fromthe actual venue, which led to more effective adjustment work in the production.

6

PERFORMANCE: participants’ reactions to the performances

After the performance, the following comments were received from the audience.
- The synergistic effect of sound, stage, and lighting expressed the scene of the piece well.
- There were some parts where the sound effects did not complement the performance.
From these results, it is inferred that while the sound effects were effective in expressing the scenes, they did
not reach the level of naturalistic production that would have prevented the audience from feeling incongruous.

We also gathered feedback from the performers about the sound effects. In particular, we asked for theiropinions about the simultaneous performance from different spaces. The performers who performed in the main
hall gave the following comments- The ensemble did not uncomfortably feel the difference between the small hall and rehearsal room.
- The sound’s realistic air vibration was not felt; hence, that the sense of playing together was shut out.
The chorus members who performed in a small hall gave the following comments- As the orchestra sound was not live, some sounds were submerged and it was difficult to match theorchestra sound when we sang.
- Owing to the rich reverberation of the small hall, we were able to perform while relying on the voices of
the chorus surrounding us.
These comments suggest that there were no problems with simultaneous performance because no significant
delays occurred in the transmission; however, some level of unnaturalness was felt because the size and qualityof the sound image could not be reproduced realistically. Regarding the sound source position, it is hypothesizedthat moving the sound image within the auditorium for the production (E) was one of the factors that interferedwith the sense of simultaneous performance.
Further improvement is required for the realistic presentation of sound sources with a wide and deep soundimage, such as an orchestra. Nevertheless, there were several positive comments regarding the use of a reverberation control system to make the small hall sound more like the main hall. This will be a useful reference
when sharing music with other places in the future.

7

CONCLUSIONS

A novel production approach combining reverberation and sound image control technology was devised andattempted for a classical concert. Conductor Nishimoto commented, “For this concert, we planned a productionthat would allow the audience experience another dimension while still inside the hall, and would also connect
space to space. In the future, I would like to make it possible to transcend various restrictions and create amusical experience that connects us transparently and at a higher dimension.” We consider the sharing of soundspace to be the key to enjoying music together beyond space. We intend to investigate ways to further enhance
the sense of sharing the sound space using immersive sound technology.

REFERENCES[1] F. Rumsey, Spatial audio (Routledge, 2012).
[2] T. Watanabe, H. Miyazaki, S. Sawara, M. Ikeda, and R. Bakker, “Active field control in the teatr
wielki—opera narodowa,” in “Audio Engineering Society Convention 138,” (2015).
[3] T. Watanabe, D. Hashimoto, H. Miyazaki, and R. Bakker, “Comparison of effectiveness of acoustic enhancement systems—comparison of in-line,” in “Audio Engineering Society Convention 144,” (2018).
[4] Yamaha Corporation, “Acoustic design projects,” (2012). https://www.yamaha.co.jp/acoust/en/_cases/.
[5] F. Kawakami and Y. Shimizu, “Active field control in auditoria,” Applied Acoustics 31, 47–75 (1990).
[6] V. Pulkki et al., Spatial sound generation and perception by amplitude panning techniques (Helsinki Universityof Technology, 2001).
[7] M. Geier, J. Ahrens, and S. Spors, “Object-based audio reproduction and the audio scene description format,” Organised Sound 15, 219–227 (2010).
[8] M. Wright, A. Freed et al., “Open soundcontrol: A new protocol for communicating with sound synthesizers,” in “ICMC,” (1997).
[9] T. Watanabe, H. Miyazaki, and M. Ikeda, “Electronic shell—improvement of room acoustics without orchestra shell utilizing active field control,” in “Audio Engineering Society Convention 140,” (2016).

ABS-0047

Real-time spatial sound rendering system using LiDAR sensor forauditory augmented reality application
Rai SATO(1) , Takumi ITO(2) , Sungyoung KIM(3)(1) Embodied Cognitive Science Unit, Okinawa Institute of Science and Technology, Japan, rai.sato@oist.jp(2) GATARI Inc., Japan(3) College of Engineering Technology, Rochester Institute of Technology, USA

ABSTRACTThe advent of affordable technologies for interactive spatial audio has facilitated the delivery of auditory augmented reality (AAR) experiences. The basic idea behind AAR is to superimpose sounds on top of the realworld as its user moves through it, unlike games and VR where people experience pseudo-interactions in a
virtual space. However, the nature of AR prevents programmers from imagining in which space the user will
experience the AR application, making it more difﬁcult to process sounds based on spatiality than in games and
VR. Therefore, we developed a real-time spatialized audio reproduction system using space data automaticallyobtained by the AR application. This system provides dynamic spatial acoustics according to a space where
it operates by the following processes: 1) capture room shape from mesh data generated by LiDAR sensor;
2) calculate room impulse response on each sound and user position based on an image-source method; and 3)
rotate the sound image according to the user’s direction and provide spatialized sounds in binaural format. Since
these processes require only the game engine’s AR library and existing audio middleware to compute spatial
acoustics, it is possible to achieve personalized spatial acoustics and evaluate its plausibility for AR and audio
researchers.

Keywords: Augmented reality, Virtual acoustics, Game audio

1

INTRODUCTION

The fundamental concept of augmented reality (AR) is an interactive experience of a real-world environment
where the objects that reside in the real world are enhanced by computer-generated perceptual information,
typically 3D model visual information. According to Azuma et al. [1], an AR system should contain thesefollowing components that: 1) combine real and virtual objects in a real environment; 2) run interactively andin real-time; and 3) register real and virtual objects with each other. Whereas the concept of virtual reality(VR) is based on an entirely virtual interaction that does not correspond to the user’s physical environment,
AR is unique that computer-generated objects are superimposed on the actual world and such an augmented
experience is presented to the user instantaneously.
With the development and commercialization of technologies related to spatial acoustics, such as binauraltechnology and Ambisonics rendering system, the concept of auditory augmented reality (AAR) has emerged[2]. The basic AAR system is similar to a visual version of AR. Users who play the AAR can listen to
and interact with both virtual sound sources that are sequentially updated through spatialization ﬁlters such as
the dynamic head transfer function (HRTF) according to their head movements, allowing them to realize thedirection and presence of the sound sources in real time. In addition, the distance between the users and thesound sources can be calculated by motion sensors installed in the AR device, making it possible to present thecorresponding sound intensity in approximately the same way as in a physical space.
For a successful realization of AAR, the virtual acoustics objects have to be seamlessly integrated into
the real environment. There are several factors to match the sound objects with the environment; the criticalattribute would be how to accord with the target room’s acoustical information [3]. In a room sound ﬁeld,
not only direct sound, which is from the sound source directly arriving at the user’s ear, but also reﬂectedsounds from the room walls and ceiling will be heard simultaneously. The relative difference between direct
and reﬂected sounds can signiﬁcantly change the perceived impression of the sound ﬁeld and the sound itself.
Room reﬂection of sound is classiﬁed into two categories, early reﬂection and late reverberation, according to
the delay time from the direct sound. Early reﬂections affect auditory impressions such as loudness, articulation,
and apparent source width (ASW), while late reverberations create the impression of reverberance and listenerenvelopment (LEV) [4]. AAR makes us easily recognize acoustic differences between physical and AR spacebecause any generated experiences are tied to real space. If room reﬂection of sound is not properly handled
in the AR space and users experience a mismatch of auditory impressions, they may not be able to immersethemselves in the augmented interaction. Although the acoustic processing that an AR device can handle islimited, plausible match in the room acoustic reproduction is required to create a reasonable acoustic perception
for the AAR system.
However, the basic premise of AR is that the space where users experience AR application is inherent toeach individual, and the geometric and material information about the space is not available to the developer at
the time the application was developed. Some methods have been proposed to estimate the room acoustics ofthe space using a camera [5] or to compute some of the room acoustics by providing a simple impulse response
from a device [6] [7]. Yet, these methods still have issues such as real-time performance, and as Neidhardt etal. have stated that “perfectly transparent reproduction devices are not (yet) available [3].”We therefore implemented a research platform to investigate whether target room acoustic matching would
enhance plausible user experiences and subsequently overcome the aforementioned problems by combining existing AR application production technology with a commonly-used audio authoring tool for game development.
We envision that this platform will enable AR and acoustic researchers to achieve and evaluate reasonable spatial acoustics in AAR applications and will contribute signiﬁcantly to the development of immersive perception
research for not only existing applications but also future ones including but not limited to auditory perception
in the metaverse.

2

PLATFORM OVERVIEW

Augmented reality Room Acoustic Estimator (ARAE) is the state-of-the-art iOS platform to compute variablecharacteristics of room acoustics and produce spatialized sounds instantaneously using a game development
software, Unity and its audio middleware, Wwise. Figure 1 illustrates a typical usage of ARAE.
The users ﬁrst launch ARAE while wearing earphones and move the AR device measuring the shape of theroom. After placing a sound source at an arbitrary position, they can experience realistic listening of the soundsource in which the shape of the room and corresponding room acoustics are taken into account. The detailed
usage and procedure of this platform will be explained in the following sections.
2.1 Technical detail
ARAE requires a mobile phone with a LiDAR sensor, such as iPhone 12 pro (Apple Inc.) or higher. Moreover,earphone or headphone with a motion sensor is required to estimate a user’s head orientation. We use Airpods
pro (Apple Inc.) to meet this requirement, which can obtain the user’s motion data directly through Apple’s
Headphone Motion API (as shown in Figure 1).
For the AR rendering side, we decided to use Unity (Unity Software Inc.) and its AR library, AR Foundation. Unity is a cross-platform game engine and it can be used to develop two-dimensional and threedimensional games, as well as interactive experiences and physics engine simulation. AR Foundation, an extension of Unity, works with the Unity’s 3D rendering system to provide features for designing AR experiencesthat work efﬁciently on any device. This package only provides a system interface to design AR experiences
and it does not implement the AR functionality itself, requiring other sub-packages for each platform. Therefore, the functionality available in each sub-package may be limited. As for this platform, meshing, one of the
AR functions to capture a room shape, can only be available in iOS’s AR function (ARKit) at the time thispaper was published; thus, this platform cannot be used other than the iOS environment.

Figure 1. Typical usage of ARAE. A user uses the platform with with LiDAR-equipped iPhone and an earphonewith a motion-sensor in a small room, and listens to a virtual sound source rendered to account for a given
acoustics of the room.

The processing of the audio experience is implemented in Wwise (Audiokinetic Inc.). Wwise is a commonlyused audio middleware for game engines. It features an audio authoring tool and a sound processing engine
that enable to design and compute comprehensive sound expression that general game engines themselves cannot
achieve.
2.2 Procedure
ARAE provides dynamic spatial acoustics according to a space in which the platform operates by the following
processes: 1) capture room shape from mesh data generated by LiDAR sensor; 2) calculate room impulse response (RIR) on each sound and listener position using the image-source method; and 3) rotate the sound image
according to the listener’s direction and provide spatialized sounds in binaural format. This section describes
the detailed functionality of this platform in three parts. This section aims to let prospective AAR researchers
know what hardware preparation and software features should be required to study acoustics matching in AR
space.
2.2.1 Capture room shape
ARAE offers two features to obtain a geometrical room characteristic; Meshing and Re-emergence. Meshingrelies on AR mesh manager, one of the AR foundation functions, and employs the LiDAR sensor to create
a 3D polygon mesh model from the geometry of the real world. Once this feature is activated (by pushingthe “meshing” button at the top of the GUI [the left panel of Figure 2] in the case of this platform), the user
measures an entire space where the platform operates, relying on the blue-colored mesh displayed on the screen.
The platform stores the mesh model of the entire room. It is also possible to play sounds while capturing theshape of the room, depending on design of an AR application. The data will be saved in Wavefront OBJ format(.obj), but it can also be saved in any 3D model format that Unity supports, such as Standard Triangulated
Language (.stl) or Filmbox format (.fbx).
Re-emergence includes converting the mesh model to a format suitable for room acoustics calculations andthen sending the information to the audio rendering section. As the audio rendering software cannot handle
excessively detailed mesh models, it is necessary to simplify them using a mesh simpliﬁcation method (asa typical example in [8]). The quality of the simpliﬁcation required to properly compute its room acoustics

Figure 2. The platform GUI (left panel) and Wwise proﬁler (right panel). A user can obtain room shape andplace sound object through the GUI. Wwise proﬁler displays various acoustical information such as a shape ofthe room (orange-colored mesh) and trajectories of early reﬂections (yellow-colored lines)

depends on the volume of the room and the complexity of its geometry; therefore, a user may need to adjust
the parameter of its compression quality in response to the device’s processing capacity and adequacy of thecalculating reverberations. Once simpliﬁcation is complete, the mesh model will be shown again to match thephysical space, and at the same time as this process, the room model will be sent to Wwise.
2.2.2 Audio rendering
After completing these processes, the user locates the sound source at the desired position. With these processes,
all the necessary information for the audio rendering has been prepared. Wwise acquires the user’s position(relative three-dimensional coordinates of devices located on space in Unity), the orientation of the user’s head
(head tracking data from the Headphone Motion API), the position and direction of the sound sources, andthe geometric data of the room. An extensional library of Wwise, Wwise Reﬂect [9], uses this information tocompute early reﬂections of the room. This library allows features controlling properties such as the order ofthe early reﬂections, absorption coefﬁcient of each surface, and ﬁlters for each reﬂection (see [9] for details).
Wwise RoomVerb also allows controlling parameters for late reverberation of the room model. It can add
arbitrary late reverberation to the captured room model and it also provides the function to estimate the decayof the reverberation based on its volume and surface area. While this platform currently does not implement
an algorithm to estimate more precise late reverberation that takes room materials and complex interiors into
account carefully, it could estimate those parameters by capturing an impulse response of the room and recognizing the surface material by build-in microphone and camera while measuring the room geometry (see [5]and [7] for the possible implementation of this algorithm).
Furthermore, Wwise offers fundamental features to modify attributes of sound sources such as its intensityand frequency response, as well as parameters such as distance attenuation of direct sound and attenuation basedon the direction of the sound source (this attenuation function is called “cone attenuation” in Wwise).
ARAE itself does not have an ability to visualize audio information such as sound levels of each soundsource and propagation of the early reﬂections; however, Wwise proﬁler running on a PC can illustrate suchinformation (the right panel of Figure 2). With ARAE and the PC on the same network, Wwise proﬁler
can automatically retrieve the data used for acoustic processing and draw the illustration on a separate systemfrom the platform. Moreover, many of the parameters involved in the sound processing in Wwise can also becontrolled through the Wwise authoring API, which can be manipulated asynchronously by python or Node.js

as well as Unity.
2.2.3 Input and output
Sound sources can also be played not only from audio data embedded in the platform but also from thosestored separately on external storage using Unity Addressable. The platform supports not only 1-channel and2-channel sound sources, but also it can be potentially used for up to 13-channel surround audio formats andup to 5th-order Ambisonics. (As a side node, these are audio formats supported by Wwise itself and multichannel format and higher-order Ambisonics (HOA) may not be processed properly due to a user device’s CPU
capability.)
Sound sources are sent to the audio bus after applying EQ and other ﬁlters. Each audio bus can choose
whether to treat the sound sources as a channel-based, an Ambisonics format, or an object audio. In addition,
it is possible to determine at which stage the early reﬂections and late reverberations are to be combined.
These sounds are ﬁnally binauralized according to the user’s head orientation using Wwise Auro-Headphones
and played back through the earphones. Employing another binaural decoding algorithm or individual HRTF
ﬁltering is possible by implementing a custom plugin.
The platform’s GUI allows users to arbitrarily change the early reﬂections and reverberations, thereby comparing how the AR acoustic impression differs with or without the reﬂections at any point.
2.3 Future researches
Spatial acoustics in AAR that takes advantage of room characteristics have a chance to create a more precise
impression than otherwise. For example, the presence of sound is enhanced by early reﬂections, making iteasier to understand where the sound emanates from and the distance between them. This will be of great help
in a navigation system, which is one of typical use cases of AR (as a representative application; Google maps
AR navigation). Generally, AR navigation with only visual information is fraught with dangers, such as using asmartphone while walking. However, the spatial-aware AAR avoids such problems and could guide users safely.
We expect that the proposed platform will be instrumental in investigating what acoustic parameters would be
for precise and reasonable sound localization in various user-end applications including AR navigation.
ARAE has only constructed a system in which the various acoustic parameters are provided in a form
that developers and users can freely edit. Consequently, if those parameters are not optimized to provide an
acoustically natural listening experience, users may ﬁnd the platform less immersive than a simple AAR thatcan be achieved with existing AR systems themselves. There have not been many psychoacoustical studies on
AR experience conducted with AR applications as few systems have been implemented that allow experimenters
to properly control its experimental environment and take sufﬁcient physiological and behavioral data. Therefore,
it has been difﬁcult to deﬁne the optimal parameters at present. Our next research project with ARAE is toconduct subjective evaluation experiments on acoustic impressions in the AR spaces, and to suggest presetsof acoustical parameters that provide an acoustically reasonable listening experience in AR environments. Oncecompleted, we expect that this will not only extend the acoustic expressions in the existing AR systems but will
also enable a new sound experience that takes full advantage of AR’s potentiality by allowing free movement
through space in a six-degree of freedom (6DoF), thereby feasibly increasing the importance of acoustics in the
metaverse.

3

CONCLUSIONARAE has a potential as an experimental platform for AR and audio researchers to investigate the necessary
elements for natual acoustic representation in AR spaces. As noted in the review by Neidhardt et al., theopportunities for AAR usage are accelerating; still, psychophysical and physiological studies to ascertain its
acoustical plausibility have been difﬁcult to implement due to technical limitations [3]. Yet, we are conﬁdent
that this platform, which combines typical hardware and software used in game production, will overcome thetechnical limitation and could provide a multitude of research opportunities for experimenters who do not evenspecialize in the ﬁeld of AR.

ACKNOWLEDGEMENTS
This work was partially supported by the National Endowment for the Humanities (NEH) research and development award, “Digital Preservation and Access to Aural Heritage Via a Scalable, Extensible Method” (Award
No. PR-263931-19), and the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant Number
JP22K18163 and JP22K13862.

REFERENCES[1] Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, Macintyre B. Recent advances in augmented reality.
IEEE Computer Graphics and Applications. 2001;21:34–47.
[2] Härmä A, Jakka J, Tikander M, Karjalainen M, Lokki T, Hiipakka J, Lorho G. Augmented reality audio formobile and wearable appliances. Journal of the Audio Engineering Society. 2004;52(6):618–639.
[3] Neidhardt A, Schneiderwind C, Klein F. Perceptual matching of room acoustics for auditory augmented
reality in small rooms – literature review and theoretical framework. Trends in Hearing. 2022;26:1–22.
[4] Masahiro M, Hisayoshi F, Zyuniti M. Discrimination between auditory source width and envelopment. Journal of Acoustical Society of Japan. 1990;46:448–457. (in Japanese)[5] Colombo M. Vision-based acoustic information retrieval for interactive sound rendering. IEEE International
Symposium on Mixed and Augmented Reality Adjunct. 2021;487–490.
[6] Hideki K, Kohei Y. Safeguarding test signals for acoustic measurement using arbitrary sounds: Measuring
impulse response by playing music. Acoustical Science and Technology. 2022;43(3):209–212.
[7] Seetharaman P, Tarzia SP. The hand clap as an impulse source for measuring room acoustics. Audio Engineering Society Convention 132. 2012.
[8] Garland M, Heckbert PS. Simplifying surfaces with color and texture using quadric error metrics. Proceedings Visualization ’98. 1998;263–269.
[9] Audiokinetic Inc. Wwwise Reﬂect. https://www.audiokinetic.com/library/edge/?source=Help&
amp;id=wwise_reflect_plug_in_effect (accessed 2022.08.12).

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0109

A downmixing method of Multichannel three-dimensional spatial
sound with horizontal loudspeaker configuration
Lulu LIU1; Bosun XIE2
Acoustic lab, School of Physics and Optoelectronics, South China University of Technology, Guangzhou,China

1

1

Guangzhou Shiyuan Electronic Technology Company, Guangzhou,China

Acoustic lab, School of Physics and Optoelectronics, South China University of Technology, Guangzhou,China

2

ABSTRACTMultichannel three-dimensional spatial sound can improve auditory perceptions for the listener, but at thecost of more complex three-dimensional loudspeaker configuration. In order to adapt to some practicalapplications, multichannel three-dimensional spatial sound sometimes needs to be downmixed. However,vertical localization information is generally lost when multichannel three-dimensional spatial sound isdownmixed with horizontal loudspeaker configuration reproduction. In this paper, based on the mechanismof binaural dynamic cue acting on elevation perception, a downmixing method is proposed which enables
to recreate virtual source at various elevations with horizontal loudspeaker configuration. The proposed
method is implemented in the 5.1-channel horizontal loudspeaker configuration recommended by theInternational Telecommunications Union (ITU). A virtual source localization experiment validates that the
proposed method can generate virtual sources with different elevation angles within a certain range of theupper hemisphere, especially those with high elevation angles. Therefore, the method proposed in thispaper reduces the loss of vertical information in downmixng by only using horizontally arranged
loudspeakers.
Keywords: Downmixng; Spatial sound; Vertical localization; Dynamic cue

1. INTRODUCTION

Recently, multichannel sounds have been evolving from horizontal surround to(three-dimensional) spatial sounds (1). In comparison with horizontal surround sound, multichannel
spatial sound can recreate vertical localization information and thus improve the auditory perceptual
performance in reproduction. Multichannel spatial sounds usually uses layer-wise (three-dimensional)
loudspeaker configurations which include the horizontal, upper and sometimes bottom layers ofloudspeaker arrangement. However, these layer-wise loudspeaker configurations are complex andnot appropriate for some practical uses. For example, in some domestic uses, it is inconvenient toarrange the upper (high) layer of loudspeakers for multichannel spatial sound. In these cases, it isnecessary to downmix the multichannel spatial sound signals to fewer channel signals and then to
reproduce with some appropriate horizontal loudspeaker configurations. Traditional downmixing
methods for horizontal loudspeaker reproduction lose vertical localization information and therefore
the perceived virtual source is limited to the horizontal plane at most. A similar problem occurs inobject-based spatial sound (2), in which objects intended for being reproduced withthree-dimensional loudspeaker configurations may be rendered with a horizontal loudspeakerconfiguration. Therefore, a general problem is how to recreate vertical virtual source via a horizontal
loudspeaker configuration associated with a appropriate signal mixing or downmixing method.
Psychoacoustic principle can be utilized to realize aforementioned purpose to some extent.
Actually, the virtual source-elevated effect, a special psychoacoustic phenomenon, was observed inearly studies on stereophonic sound (3). For a two-channel stereophonic loudspeaker configuration
with a large span angle and an identical signal amplitude, the perceived virtual source is located at a12348802659@qq.com
Corresponding to: phbsxie@scut.edu.cn

high elevation in the median plane rather than in the direct front, especially when the head rotatesaround the vertical axis. In the end of the 1950s, Leakey interpreted this effect by an analysis ofinteraural time difference (ITD) variation caused by head rotation (4). Some recent studies also
provided experimental results with additional details (5).
According to psychoacoustic principle (6-8), both dynamic cue at low frequency (especiallydynamic variation of ITD caused by head turning) and spectral cue at high frequency (caused by thescattering and diffraction of pinna) contribute to front-back and vertical localization. But the
information provided by these two cues is somewhat redundant. When one cue is eliminated, anothercue alone still enables front-back and vertical localization to some extent (9).
In present work, the psychoacoustic principle of virtual source-elevated effect is utilized torecreate vertical virtual source using horizontal loudspeaker configuration. Taken the four horizontal
loudspeakers in ITU 5.1-channel loudspeaker configuration as an example, a signal mixing or
downmixng method is proposed. This method can create desired low-frequency ITD and its dynamic
variation with head rotation and thus can recreate virtual source in various elevation the medianplane. And a virtual source localization experiment is conducted to validate the proposed method.

2. PRINCIPLE OF SYNTHESIZING VERTICAL VIRTUAL SOURCE WITH
HORIZONTAL LOUDSPEAKER CONFIGURATION
2.1 System coordinates
For convenience in analyzing vertical localization, the interaural polar coordinate is used in this paper.
As shown in Fig. 1, the position of a sound source is specified by (r, Θ, Φ). Where 0 ≤ r < ∞ denotes the
distance; -90° ≤ Θ ≤ 90° denotes the interaural polar azimuth, e.g., the angle between the directional vectorof the sound source and the median plane, with Θ = -90°, 0°, and 90° being the left direction, median plane,
and right direction, respectively. A constant Θ represents the cone of confusion. The -90° ≤ Φ < 270°
denotes the interaural polar elevation, that is, the angle between the projection of the directional vector of
the sound source to the median plane and the frontal axis, with Φ = -90°, 0°, 90°, and 180° denote thebelow, front, above, and back directions, respectively.

Figure 1 – Default coordinates in the present work
2.2 Summing Localization Equations based on Simplified Head Model
It is known that low-frequency (f < 1.5 kHz) is a dominant cue for lateral localization (10), bywhich the interaural polar azimuth of a sound source can be determined. Early in 1940, Wallach
suggested that dynamic cue contribute to front-back and vertical localization (7). In detail, the ITD
variation caused by head turning around the vertical axis (rotation) allows the discrimination of thefront-back location as well as vertical displacement from the horizontal plane; and head turning
around the front-back axis (tilting) provides supplementary information for up-down discrimination.
Wallach’s classical hypothesis has been validated by some modern experiment (8).
Based on Wallach ’s hypothesis and a simplified (shadowless) headmodel, the perceived virtual
source direction in multichannel spatial sound reproduction can be analyzed by comparing the ITD
and its dynamic variation of reproduction with these of a target source, and then a set of summing
localization equations can be derived (11). The original equations are expressed in conventional
spherical coordinate system. Here, the equations are converted to the interaural polar coordinatesystem used in present paper.
Suppose that there are M loudspeakers arranged at a constant and far-field distance to the centerof head. The direction of the ith loudspeaker is (Θ i , Φ i) and the amplitude or gain of corresponding
loudspeaker signal is A i. The direction of perceived virtual source at low-frequency is denoted by (Θ I,Φ I) and can be evaluated by following three equations.
Comparing the ITD for a fixed head orientating to front in reproduction with that of target sourceyieldsM

 E sin Θiisin ΘI  i 1 M

(1)

 Ei
i 1

Comparing the variation rate of ITD against head rotation in reproduction with that of target
source yields
M E cos Θ cos Φ

cos ΘI cos Φ I  i 1

i

i

i

M

E
i 1

(2)

i

Comparing the variation rate of ITD with head tilting in reproduction with that of target sourceyieldsM

 E cos Θ sin Φ

cos ΘI sin ΦI  i 1

i

i

i

M

 Ei

(3)

i 1

Eq. (1) to Eq. (3) are applicable to the analysis of the summing localization in multichannel
spatial sound reproduction. Because a shadow-less head models is used in the derivation of theseequations, the equations are valid only at low frequencies below 700 Hz.
2.3 The principle of the proposed method
By using the theory presented in Section 2.2, we propose a method to recreate vertical virtual
source in median plane with horizontal loudspeaker configuration. The ITU 5.1-channel loudspeakerconfiguration is taken as an example (12). As shown in Fig.2, it involves five full-band channels andloudspeakers, namely the left (L), center (C), right (R), left-surround (LS) and right-surround (RS)loudspeakers. It also involves an optional low-frequency effect channel and subwoofer. We only use
the left, right, left-surround and right surround loudspeakers to recreate vertical virtual source in the
median plane. In interaural polar coordinate, the directions of these loudspeakers are:
Θ1  ΘL  30 Θ2  ΘR  30 Θ3  ΘLS  70 Θ4  ΘRS  70
(4)
Φ1  ΦL  0 Φ2  ΦR  0 Φ3  ΦLS  180 Φ4  ΦRS  180

Figure 2 – Loudspeaker configuration of 5.1-channel surround sound system
The amplitudes of loudspeaker signals are:E1  EL E2  ER

E3  ELS

E4  ERS(5)

In order to synthesize the virtual source in the median plane, the amplitudes of the loudspeakersignals should be left-right symmetric, thenE1  E2 E3  E4
(6)
Substituting Eq.(5) and Eq.(6) into Eq.(1) yields:
sinΘI  0

(7)

Therefore, when the head is fixed to orient to the front, the ITD in reproduction vanishes and theperceived polar azimuth is Θ I = 0°. Therefore, the synthetic virtual source is indeed in the medianplane.
To derive the loudspeakers signals for virtual source in the median plane, substituting Eq.(5) andEq.(6) into Eq.(2) yields:
E cos 30  E3 cos 70 cos 30  cos 70 E3 / E1
(8)cos Θ' I cos Φ' I  1
E1  E2
E3 / E1
Because Θ' I = 0° then cosΘ'I = 1, Eq.(8) becomescos 30  cos 70 E3 / E1
cos Φ' I 
E3 / E1
On the other hand, substituting Eq.(5) and Eq.(6) into Eq.(3) yields:cos Θ" I sin Φ" I  0

(9)
(10)

Because Θ'' I = 0°, we have(11)
sin Φ" I  0 Φ" I  0 or Φ" I  180Eq.(9) and Eq.(11) leads to inconsistent results. In this case, the localization of the perceived
virtual source is determined according to the psychoacoustic mechanism or hypothesis. Eq.(11)indicates that, for reproduction with four horizontal loudspeakers, head tilting falls to provides
effective information for up-down discrimination. The virtual source is located in the direct front orback in the horizontal plane. However, Eq.(9) indicates shows that the head rotation provides
information about the displacement of the virtual source from the horizontal plane. Since the verticaldisplacement of the virtual source from the horizontal plane is mainly determined by the ITD change
caused by the head rotation around the vertical axis, the elevation angle of the virtual source can beapproximately determined by Eq.(9). And in practice, the auditory system is used to the sound source
above the horizontal plane, and the sound source in the below direction is relatively rare. Therefore,
when the up-down localization cue is incomplete and conflicting, it will be used to sensing that thevirtual source is in the upper hemisphere.
Eq.(9) indicates that the polar elevation of the perceived virtual source is determined by the
amplitude ratio AR = E 3 / E 1 or interchannel level difference (ICLD) d = 20log10 (E 3 / E 1) (dB).
Theoretically, when the ICLD changes from -∞ dB to +∞ dB, and the polar elevation of the virtual
source changes from 30° to 110°. The variation of perceived polar evaluation with ICLD is plotted in
the following Fig.3. Therefore, the horizontal loudspeaker arrangement can produce virtual sources
with different polar elevation angles in the median plane.
From Eq. (9) and following condition of constant overall power of loudspeaker signals:(12)E12  E22  E32  E42  const
the amplitude of loudspeaker signals can be obtained:
E1  E2 

const
const
; E3  E4  AR
2
2(1  AR )2(1+ AR 2 )(13)

E3 cos 30°  cos Φ ' I

E1 cos 70+ cos Φ 'I

(14)

AR 

Eq.(13) and Eq.(14) can be regarded as a panning law or signal mixing method for the horizontal
loudspeaker configuration to recreate virtual source at different evaluation. When the target
elevation Φ'I is given, the amplitude or gain of loudspeaker signals can be calculated from these two
equations. When the multichannel spatial sound signals are downmixed for reproducing withhorizontal loudspeaker configuration, or when the signal of a vertical object in object-based spatial
sound is reproduced with horizontal loudspeaker configuration, the signals can be mixed to
loudspeakers according to these two equations.

3. VIRTUAL SOURCE LOCALIZATION EXPERIMENT
3.1 Experimental method
A virtual source localization experiments validate that the proposed method can recreate virtual sourcein different evaluation in the median plane. Experiment was conducted in a listening room withreverberation time 0.15s and background noise less than 30 dBA. Four loudspeakers (Philips MC1500H),
namely, the L, R, LS and RS loudspeakers were arranged in a circle with radius r = 1.45 m according to
Fig.2. The height of loudspeakers was 1.2 m. The loudspeaker signals were created according to Eq.(13)and Eq.(14), resulting to signals with EL= ER, ELS = ERS and ICLD of d = 20 log10 (ELS / EL) varying from-24 dB to +24 dB at an interval of 6 dB. Two kinds of stimuli, e.g., pink noise with full audible bandwidth(20 Hz ~ 20 kHz) and 3 kHz-low pass filtered pink noise were used in the experiment.
Eight subjects (3 male and 5 female), aging from 22 to 27 with normal hearing participated in theexperiment. Before the formal experiment, the subjects were trained in virtual source localization. In
the formal experiment, subject judged the perceived elevation of virtual source. During theexperiment, subjects were encouraged to turning their heads. Generally, rotation angle of headranged from ±10° to ±25°, and the tilting angle of head ranged from ±10° to ±20°, as monitored byan electromagnetic head tracker. Under each condition, the signals were reproduced three times in arandom order, and each subject judged three times repeatedly. Therefore, there were 3 repetitions × 8
subjects = 24 judgments in total under each condition.
Statistical analysis was applied to the result. Firstly, the consistency of the experimental data ischecked. And then the average perceived polar azimuth is used to judge whether the perceived
virtual source is near the median plane. Finally, the average perceived polar elevation andcorresponding standard deviation of 24 judgments under each condition were calculated as the final
statistical result.
3.2 Experimental results and discussion
Kruskal Wallis H test was performed on the experimental data. The test showed that the results ofall subjects were consistent with the repeated results at a significance level of 0.05, indicating thatthe experimental data were reliable and stable. Preliminary experimental statistical results showedthat all perceptual virtual sources were in the upper hemisphere, and the average perceived polar
azimuth was less than 5°, that was, the perceived virtual sources were located near the median planeof the upper hemisphere.
In order to analyze the relationship between the actual perceived elevation of virtual source andthe target (theoretical) elevation of the virtual source given by Eq.(9), Fig.3 illustrates the average
perceived polar elevation and the corresponding standard deviation for different ICLD and two
stimuli with different bandwidths. At the same time, the figure plots curve of elevation of virtual
source vs the ICLD according to Eq.(9). It can be observed from the figure that the results of two
stimuli with different bandwidths are similar. The average perceived polar elevation of the two
stimuli increases with the increase of the ICLD, and the overall trend of the perceived polar
elevation is basically consistent with the calculation from Eq.(9). A detail observation of Fig. 3
indicates that, when the ICLD is less than 0 dB, the perceived polar elevation is smaller than thecalculation result of Eq.(9). When the ICLD is greater than 0 dB, the perceived polar elevation islarger than the theoretical calculation result of Eq.(9). Therefore, in comparison with the calculation
results, the average perceived elevation of virtual source exhibits a bias towards the horizontal plane
expect for the virtual source at the top of Φ' I = 90°.
Actually, Eq.(9) is derived by assuming that the ITD variation caused by head rotation dominate
the perceived of displacement of virtual source from the horizontal plane. On the other hand, Eq.(11)indicates that virtual source is located in the horizontal plane in spite of ICLD. This inconsistent
result indicates that head tilting provides different information. Actual perceived elevation may be
determined by a weighted combination of information providing by head rotation and tilting.
Although head rotation contribution more to the perception of the displacement of virtual sourcefrom the horizontal plane, a completely neglecting of the information provided by head tilting may
cause some error the analysis.
In addition, it can also be observed from Fig.3 that, for ICLD = -24 dB, the average perceived
evaluation of virtual source is near 25° rather than 0° in the horizontal plane. In this case, the signalsof LS and RS loudspeakers (Fig.2) almost vanish, and the virtual source is created by the L and R
loudspeakers with identical signal amplitude. Because head rotation is encouraged during theexperiment, the experimental results exhibit a conventional virtual source-elevated effect generated
by the L and R loudspeakers (although the span angle of these two loudspeakers is 60°).
Overall, the aforementioned experimental results indicate that the proposed method caneffectively synthesize vertical virtual sources with different elevation angles by adjusting the ICLD,
including the virtual source directly above.

Figure 3 – Average and the corresponding standard deviation of perceived polar elevation ofsynthetic virtual

4. CONCLUSION
Multichannel spatial sound improves the spatial perception performance, but requires more complex
three-dimensional loudspeaker configuration. In order to adapt to some practical applications andfacilitate actual loudspeaker arrangement, it is necessary to downmix multichannel spatial sound signals
for reproduction with horizontal loudspeaker configuration. However, spatial information, especially thevertical information is often lost in the conventional downmixing. A similar problem may occur in
object-based spatial sound.
To address above problem, the method of recreating vertical virtual source by using horizontal
loudspeakers configuration is proposed in present work. For a horizontal loudspeaker configuration withfour loudspeakers taken from the ITU 5.1-channel configuration, theoretical and experimental results
indicate that appropriate signal mixing can recreate virtual source in various evaluations in theupper-median plane. Therefore, the proposed method effectively reduces the loss of vertical localization
information and improves the performance the downmixing reproduction of multichannel spatial sound.
The method in this paper is not only suitable for reproduction with 5.1-channel loudspeakerconfiguration, but also suitable for other horizontal loudspeaker configuration. It provides a new method
for reproduction three-dimensional spatial virtual sources with horizontal loudspeaker configuration.

ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (No.12174118).

REFERENCES
1. ITU-R Report BS.2159-7. Multichannel sound technology in home and broadcasting applications,International Telecommunication Union, Geneva, Switzerland. 2015.
2. Herre J., Hilpert J., Kuntz A., et al. MPEG-H audio —The new standard for universal spatial/3D
audio coding. J.Audio Eng Soc. 2014; 62(12): 821-830.
3. Boer K.De. A remarkable phenomenon with stersophonic sound reproduction. Philips Tech Rev.1947;
1947(9): 8-13.
4. Leakey D.M. Some measurements on the effects of interchannel intensity and time differences in two
channel sound systems. J Acoust Soc Am.1959; 31 (7): 977-986.
5. Lee H. Sound source and loudspeaker base angle dependency of phantom image elevation effect. J
Audio Eng Soc. 2017; 65(9): 733-748.
6. Blauert J. Spatial hearing: the psychophysics of human sound localization (Revised edition). Cambridge,
MA, USA: MIT Press; 1997.

7. Wallach H. The role of head movement and vestibular and visual cue in sound localization. J Exp
Psychol. 1940; 27 (4): 339-368.
8. Perrett S., Noble W. The effect of head rotations on vertical plane sound localization. J Acoust Soc Am.
1997;102 (4): 2325-2332.
9. Jiang J L, Xie B S, Mai H M, et al. The role of dynamic cue in auditory vertical localization. Appied
Acoustics. 2019: 146: 398-408.
10. Wightman F.L., Kistler D.J. The dominant role of low-frequency interaural time difference in soundlocalization. J Acoust Soc Am. 1992; 91(3): 1648-1661.
11. Xie B.S., Mai H.M., Rao D., et al. Analysis of and experiments on vertical summing localization ofmultichannel sound reproduction with amplitude panning. J Audio Eng Soc. 2019; 67(6): 382-399.
12. ITU-R BS 775-3. Multichannel stereophonic sound system with and without accompanying picture.
International Telecommunication Union, Geneva, Switzerland. 2012.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0207

Binaural pressures control and localization performance intransaural reproduction using three loudspeakers
Meiqiang LI1; Bosun XIE2
1

School of Physics and Optoelectronics, South China University of Technology, China

2

School of Physics and Optoelectronics, South China University of Technology, China

ABSTRACTIn transaural reproduction, binaural pressures are controlled by loudspeakers to create virtual source invarious directions. Virtual source localization performance in reproduction is determined by the stability andeffort of binaural pressures control. The effort of binaural pressures control is related to the average powerof transaural filter responses. An increase of the effort of binaural pressures control leads to difficulty increating virtual source. Previous studies have investigated the problem in recreating lateral virtual source intransaural reproduction with two frontal loudspeakers. By evaluating the stability and effort of binauralpressures control, the performance of transaural reproduction with three frontal loudspeakers are analyzed inpresent work. Results indicate that, compared with two-loudspeaker configuration, three-loudspeakerconfiguration with half-span angle of 30° is more stable, while three-loudspeaker configuration with halfspan angle of 90° exhibits a less effort. Virtual source localization experiment also reveals that transaural
reproduction with three front loudspeakers exhibit slight better localization performance of lateral virtual
source.
Keywords: Transaural reproduction, Binaural pressures control, Localization

1. INTRODUCTION
Transaural reproduction is a special kind of spatial sound technique which creates the auditory
perception of virtual source by controlling the binaural pressures v ia a set of loudspeakers(1‒3). One
advantage of transaural reproduction is that it is simple, e.g., it requires a small set of (not fewer thantwo) loudspeakers for reproduction. Therefore, it is not only used as an independent reproduction
system, but also used to convert various multichannel sound signals (such as 5.1-channel signals) for
reproducing with fewer loudspeakers.
Transaural reproduction with two or more loudspeakers in frontal-horizontal plane cantheoretically duplicate the binaural pressures of an arbitrary target source. Therefore, it is expe cted to
be able to recreate virtual source in arbitrary direction. However, practical static transaural
reproduction with loudspeakers in frontal-horizontal plane usually exhibits following problems:
1) The perceived virtual source directions are limited to the frontal -horizontal plane, rather thanarbitrary spatial directions.
2) The size of listening region (sweet point) is very small. A slight translation of head will destroy theperceived effect.
3) Lateral virtual source moves towards the front.
The aforementioned three problems are related to one another. The reasons for the first problem
are that, in static transaural reproduction with frontal loudspeaker configuration, the dynamic cues forfront-back and vertical localization are incorrect, and the high-frequency spectral cues are unstable(4).
The second problem is related to the stability and robustness of binaural pressure control which
can be evaluated by the condition number of transfer matrix from loudspeakers to two ears (5). Some
loudspeaker configuration, such as two-frontal loudspeakers with narrow span angle (stereo dipole)(6), three-frontal loudspeaker configuration (2) and optimal source distribution (7), have beensuggested to improve the stability in reproduction.
1
22422660387@qq.com
Corresponding to: phbsxie@scut.edu.cn

Previous work indicated that the third problem is related to multiple factors, including mismatched
HRTFs, mismatched characteristics of loudspeakers pair, mismatched dynamic cues caused by headturning around the vertical axis (3). Hamdan and Fazi analyzed effectiveness of binaural pressure
control or crosstalk cancellation in sound reproduction using a modal method (8). Moreover, ourrecent work has indicated that the third problem is closely related to the effectiveness of binauralpressure control (9). In transaural reproduction with two frontal loudspeakers, a narrow span angle
between loudspeakers increases the effort (required total signal power) of binaural pressure control
for lateral virtual source.

2. Basic principles of transaural reproduction with a set of frontal loudspeakers
A horizontal polar coordinate is chosen. The origin of coordinate is located at the head center. The
horizontal position is specified by (r, θ), where r is the distance with respect to the origin; −180° < θ≤ 180° is the horizontal azimuth with θ = −90°,0° and 90° being the left, front and right, respectively.
For a target horizontal source at far-field distance rS ≥ 1.0 m and azimuth θ S , the binaural pressures
can be calculated by a pair of far-field head-related transfer function (HRTFs) which are
asymptotically independent of source distance:(1) PL   H L ( S , f )  P  =  H ( , f ) P0
 R  R S
where f is frequency; P 0 is the free-field pressure at the position of head center when the head is absent.
For transaural reproduction with a set of front loudspeakers, N ≥ 2 loudspeakers are arranged in asemi-circle in the frontal-horizontal plane. The distance from each loudspeaker to the head center
satisfies r ≥ 1.0 m. Then the binaural pressures are the superposition of those caused by all
loudspeakers:(2)

 E1 
 
 P' L   H L (1 , f ) H L ( 2 , f ) ... H L ( N , f )   E 2  P'  =  H ( , f ) H ( , f ) ... H ( , f )  : R
2RN R  R 1
 
E N 

where θ i with i = 1,2…N is the azimuth of the ith loudspeaker; H L(θ i,f) and H R(θ i,f) are the HRTFs
from the ith loudspeaker to the left and right, respectively. Ei is the signal of the ith loudspeaker.
Eq.(2) can also be written in the following simple form:(3)
P' = [ H ]E
where P’, E and [H] represents the vector of binaural pressures, vector of loudspeaker signals andtransfer matrix from loudspeakers to two ears, respectively.
Given the loudspeaker configuration, loudspeaker signals in transaural reproduction can be solved
by matching the binaural pressures of reproduction in Eq.(2) with those o f target source in Eq.(1) (2).
In the case of conventional reproduction with two loudspeakers, loudspeakers are arranged
symmetrically in the frontal-horizontal plane with respect to the listener. The azimuth of left and rightloudspeakers are θ L = θ 1 = -θ 0 and θ R = θ 2 = θ 0. The HRTFs from two loudspeakers to two ears satisfy
H L(θ L, f) = H R(θ R, f) = α and H L(θ R, f) = H R(θ L, f) = β. Except for some frequencies at which the matrix[H] is ill-conditioned, Eq.(1) and Eq.(2) yield an unique solution of loudspeaker signals:(4)EL = E1 = GL ( S , f ) E0 ( f )ER = E2 = GR ( S , f ) E0 ( f )where E 0 (f) = P 0 is the complex-valued input signal; GL (θ S, f) and GR(θS , f) are the responses of a pairof transaural filters:(5)
H −  H− H +  H

GL ( S , f ) =

L
2

 −

R

2

GR ( S , f ) =

L
2

 −

R

2

In the case of three loudspeaker reproduction, the three loudspeakers are also arranged
symmetrically in the frontal-horizontal plane with respect to the listener (Figure 1). The azimuth ofthe left, right and center loudspeakers are θ L = θ1 = -θ 0 , θ R = θ2 = θ 0 and θC = θ3 = 0º. The HRTFs
from three loudspeakers to two ears satisfy H L (θ L, f) = H R(θ R, f) = α, H L(θ R, f) = HR(θ L, f) =β and H L (θC ,f) = H R(θ C, f) = γ. Because fewer receiver points (two ears) than loudspeakers exist, the problem isunderdetermined. Therefore, there exist infinite sets of solution of loudspeaker signals from Eq.(1)and Eq.(2). Some restriction should be added to fine a unique solution. A conventional solution isderived from the pseudoinverse of transfer matrix [H] in Eq.(3), which is equivalent to minimizing
the total power of loudspeaker signals. To explore the effectiveness of binaural pressure control forlateral virtual source, another solution of loudspeaker signals are analyzed.
Similar to the sum-difference or mid-side transformation in traditional stereophonic sound (10),the sum-difference transformation of target binaural signals in Eq.(1) is given by(6)

EM ( f ) = E R ( f ) + EL ( f )ES ( f ) = ER ( f ) − E L ( f )The sum signal E M(f) includes the front localization information, which should be reproduced by
the center loudspeakers. The difference signal E M (f) includes the lateral localization information,
which should be reproduced by the left and right loudspeakers. Therefore, the three loudspeakersignals can be written as:(7)

EL = E1 = FS1 ( f ) ES ( f )E R = E2 = FS 2 ( f ) ES ( f )EC = E3 = FM ( f ) EM ( f )where FS1 (f), FS2 (f) and F M(f) are the responses of three filters to be determined.
Substituting Eq.(6) an Eq.(7) into Eq.(2) and matching the binaural pressures in reproductio n withthose of target in Eq.(1), we obtain the FS1 (f), FS2 (f) and F M (f) and then the loudspeaker signal:(8)

EL = E1 = GL ( S , f ) E0 ( f )ER = E2 = GR ( S , f ) E0 ( f )EC = E3 = GC ( S , f ) E0 ( f )where the responses of three transaural filters are given by (2):H − HRGL ( S , f ) = L
2( −  )H − HLGR ( S , f ) = R
2( −  )H + HR
GC ( S , f ) = L
2

(9)

E0
GC

GL

GR

EC

ER

EL
Loudspeaker 3
Loudspeaker 1







Loudspeaker 2


0





Figure 1 – Transaural reproduction with two or three frontal loudspeakers

3. Analysis on the stability and effort of binaural pressures control intransaural reproduction
3.1 Method of analysis
Eq.(4) and Eq.(8) indicate that, in transaural reproduction, binaural pressures can be controlled byfiltering the input signal with two or three transaural filters and then reproduc ing by two or threeloudspeakers. The binaural pressure control in transaural reproduction can be evaluated by somecriteria. The stability (robustness) of binaural pressures control against certain perturbations is usuallyevaluated by the condition number of the transmission matrix [H] in Eq.(3) [5], e.g.,.

 (H ) =

1
2

(10)

where σ1 and σ2 are the largest and smallest singular values of the transmission matrix [ H],
respectively. Under the ideal conditions, κ(H) = 1. A small κ(H) (close to 1) represents that binauralpressure control is robust to perturbations. However, condition number is not enough for evaluating
the ability of creating virtual source at various directions.
Here, the effort of binaural pressures control, which is defined as the average power of transauralfilter responses across certain frequency range as a function of target virtual source direction, is also
used to evaluate the ability of creating virtual source at various directions. An increase of the effortof binaural pressures control leads to difficult in creating virtual source. For transaural reproduction
with two loudspeakers, the effort of binaural pressure control is calculated by following formula:

K
2
2 
[| GL ( S , f k | + | GR ( S , f k ) | ]
W ( S ) = 10 log 10  k =1
Kf

(11)

(dB)

where the responses G L and G R of two transaural filters are given by Eq.(5), K and Δf are number andinterval of frequency samples within the concerned frequency range, respectively.
For transaural reproduction with three loudspeakers, the effort of binaural pressure control iscalculated by following formula:

K
2
2
2 
[| GL ( S , f k | + | GC ( S , f ) | + | GR ( S , f k ) | ]
W ( S ) = 10 log 10  k =1
Kf

(dB)

(12)

where the responses GL, GC and GR of three transaural filters are given by Eq.(9).
In the following, the three-loudspeaker configurations with half-span angle of θ0 = 30° and 90° (or2θ0 = 60° and 180°) are analyzed. For comparison, conventional two-loudspeaker configuration withhalf-span angle θ 0 = 30° (or 2θ 0 = 60°) are also analyzed. W(θ S ) is calculated over two frequency
ranges, 0.1~10 kHz and 0.1~1.5 kHz. The far-field HRTF data of the KEMAR artificial head with
DB060/061 pinnae but without torso obtained by boundary element calculation are used in analyzed(11). The sampling frequency of HRTF is 44.1 kHz, the length of HRTF is 882 points.
3.2

Results of the stability of binaural pressures control

Figure 2 – The condition number of different loudspeaker configurations
Figure 2 illustrates the condition numbers of the three-loudspeaker reproduction with different two
span angles. In the low frequency range below 1 kHz, the three-loudspeaker reproduction with halfspan angle of θ 0 = 90° exhibit a smaller condition number and is relatively stable. At mid-frequency
range, the condition number of the three-loudspeaker reproduction with half-span angle θ 0 = 30° is
smaller, indicating a better stability. Narrowing the span angle can improve the stability of the mid frequency binaural pressures control, but at the cost of deteriorating the stability of the low-frequency
binaural pressures control. In comparison with the case of two-loudspeaker reproduction with halfspan angle of θ 0 = 30°, the condition number of the three-loudspeaker reproduction with half-spanangle θ 0 = 30° is small, especially in the mid-to-high frequency range of 2~4 kHz, which has obvious

advantages over the other two configurations. The three-loudspeaker reproduction with half-spanangle θ 0 = 90° has better stability at low frequencies, and is inferior to the other two configurations athigh frequencies. The stability of the two-loudspeaker reproduction with half-span angle θ 0 = 30° is acompromise. The condition number depends on loudspeaker configuration only and is independent
from the direction of target virtual source. Therefore, it cannot account for the target direction dependent defect of localization (especially defect in lateral localization).
3.3 Results of the effort of binaural pressures control
Figure 3(a) illustrates the effort of binaural pressure control for reproduction with different
loudspeaker configurations within the frequency range of 0.1~10 kHz. For all loudspeakerconfiguration, the general trend is that the effort of binaural pressures increases when the target virtual
source deviates from the front (θ S=0°) towards the side, and maximizes at θ S = ±90°. This trend reveals
that it is more difficult to recreate lateral virtual source than to recreate frontal virtual source inreproduction. When target source azimuth exceeds θS = ±60º, the efforts of three loudspeakerreproduction with two different span angles are smaller than that of two-loudspeaker reproduction.
And in three-loudspeaker reproduction, the effort for configuration with half-span angle of θ 0 = 90°
is small than that with half-span angle of θ 0 = 30°. For example, when θS = 90º, effort for two
loudspeaker reproduction and three loudspeaker reproduction with half-span angles of θ 0 = 30° and
90° are -15.4 dB, -16.6 dB and -18.3 dB, respectively. Therefore, three loudspeaker reproduction
moderately improves the effort in binaural pressure control for lateral target virtual source. Moreover,in three-loudspeaker reproduction, a large half-span of loudspeaker (θ 0 = 90°) further improves theeffort in binaural pressure control for lateral target virtual source. This improvement of the effort may
enhance the localization of lateral virtual source in reproduction to some extent.
Figure 3(b) illustrates the effort of binaural control within the frequency range of 0.1~1.5kHz. The
general trend is similar to those within frequency range of 0.1~10kHz.

(a)Within the frequency range of 0.1 to10 kHz(b) Within the frequency range of 0.1 to 1.5 kHz

Figure 3 – Effort of binaural pressure control in two and three loudspeaker reproduction

4.

Virtual source localization experiment

4.1 Experimental design
A virtual source localization experiment was conducted to explore the effort of binaural pressure
control in virtual source localization, especially in lateral virtual source localization, in transaural
reproduction with three loudspeakers.
Localization experiment was conducted for three loudspeaker reproduction with half-span angle θ0= 30°, 90° and two loudspeaker reproduction with half-span angle θ 0 = 30°. The input stimuli included
pink noise with full audible bandwidth and 1.5 kHz low-pass-filtered pink noise. Including thestimulus of low-pass-filtered pink noise is for exploring the effect of different signal bandwidths on
localization.
The length of stimulus was 10s. Localization at thirteen target virtual source azimuths of θ S = 0°,±15°, ±30°, ±45°, ±60°, ±75°, ±90° was evaluated. Experimental signals were generated by a PC andfed to the loudspeakers via a Galaxy 64 channel sound card (only three channels were used) andcustom power amplifiers.

The experiment was conducted in a listening room with a reverberation time of 0.15s. The
background noise level in the listening room did not exceed 30 dBA. PhilipsMCI500H loudspeakersystems were used in experiment. All loudspeakers were arranged at a height of 1.5 m and at a distanceof 1.45m with respect to the center of the subject's head. The pressure level (evaluated in the positionof head center with the head absent) was about 75 dBA. A total of 8 subjects (6 male and 2 female)with normal hearing participated in the experiment. Subject judged the perceived virtual source
azimuth. In each condition, each signal is played with random order, and each subject judge 3 times
repeatedly. Statistical analysis was applied to the experimental data.
4.2 Experimental results and discussion
Figure 4 is the results of virtual source localization experiment for pink noise with full audible
bandwidth. The figure shows the average perceived azimuth and standard deviation of the virtual
source. Compared with two-loudspeaker reproduction with half-span angle θ 0 = 30º, when the target
virtual source azumith |θ S| ≥ 60°, the average perceived virtual source azimuth for a three-loudspeakerreproduction with a half-span angle θ 0 = 90° is slightly larger and closer to the target virtual source,e.g., the lateral virtual source localization is slight improved. Further statistical t-test also confirmed
the above conclusion.

Figure 4 – Results of virtual source localization experiments for pink noise with full audible bandwidth(a)three-loudspeaker configuration of θ0=30°(b)three-loudspeaker configuration of θ0 = 90°

Figure 5 – Results of virtual source localization experiments for 1.5 kHz low-pass filtered pink noise
Figures 5 show the results of virtual source localization experiment for low-pass-filtered pink noisein three-loudspeaker reproduction with half-span angle of θ 0 = 30° and 90°. For comparison, the resultsof pink noise with full audible bandwidth are redrawn in the Figure 5. A statistical t-test on the results
indicates that there is no significant difference between the localization results of the stimuli with two
bandwidths. Actually, interaural time difference (ITD) at low frequency (below 1.5 kHz) is d ominantcue for lateral localization, the aforementioned two stimuli lead to identical low-frequency ITD inreproduction, and then similar average perceived azimuth of virtual source.
The aforementioned experimental result is consistent with the analysis on the effort of binauralpressures. Actually, a similar analysis on effect of span angle of loudspeakers on the transaural
reproduction with two loudspeakers indicated that, a narrow span angle (such as in stereo dipole)increases the effort of binaural pressures control for lateral virtual source considerably (9), and thus
is less effective in recreating virtual source. Therefore, the localization performance and defect oflateral virtual source in transaural reproduction with two or three loudspeakers in the frontal horizontal plane is closely related to the effort of binaural pressure control, and the relation to
condition number of transfer matrix from loudspeaker to two ears is not obvious.

5. CONCLUSION
By controlling binaural pressures via two or more loudspeakers, transaural reproduction is
expected to recreate virtual source at different directions. The condition number of transfer matrix ispreviously used to evaluate the stability in transaural reproduction. The effort of binaural pressurescontrol is closely related to the localization performance in reproduction. An increase of the effort ofbinaural pressures control leads to difficulty in creating virtual source. Compared with conventional
two loudspeaker reproduction, three loudspeaker reproduction with span angle of 90º reduces theeffort of binaural pressure control and slight improve the effect of lateral virtual source localization.
Therefore, one way to improve the lateral virtual source localization in transaural reproduction is tochose an appropriate loudspeaker configuration and signal feeding to reduce the effort of binauralpressure control. Further work includes analyzes the effort of binaural pressure control in transaural
reproduction with different number, configuration and signal feeding of loudspeakers.

ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China ( No.12174118).

REFERENCES
1. Schroeder MR, Atal BS. Computer simulation of sound transmission in rooms. Proceedings of the IEEE.
1963; 51(3): 536-537.
2. Bauck J, Cooper DH. Generalized transaural stereo and applications. J Audio Eng Soc. 1996; 44(9): 683705.
3. Xie BS. Head-related transfer function and virtual auditory display (second edition). J Ross Publishing,USA, 2013.
4. Liu LL, Xie BS. Analysis and experiment on the limitations of static and dynamic transaural reproduction
with two frontal loudspeakers. Archives Acoustics. 2021; 46(2): 213-228.
5. Ward DB, Elko GW. Effect of loudspeaker position on the robustness of acoustic crosstalk cancellation.
IEEE Signal Processing Letters. 1999; 6(5): 106-108.
6. Kirkeby O, Nelson PA, Hamada H. The “Stereo Dipole”—a virtual source imaging system using two
closely spaced loudspeakers. J Audio Eng Soc. 1998; 46(5): 387-395.
7. Takeuchi T, Nelson PA. Optimal source distribution for binaural synthesis over loudspeakers. J Acoust
Soc Am. 2002; 112(6): 2786-2797.
8. Hamdan EC, Fazi FM. A modal analysis of multichannel crosstalk cancellation systems and theirrelationship to amplitude panning. J Sound and Vibration. 2021; 490: 115743.
9. Li MQ. Research on binaural pressures control and localization performance in loudspeaker-based virtual
sound reproduction. Thesis of Master Degree of Acoustic, South China University of Technology, China
2022.
10. Rumsey F. Spatial Audio, Focal Press, Oxford, England. 2001.
11. Rui YQ, Yu GZ, Xie BS, et al. Calculation of individualized near-field head-related transfer function
database using boundary element method. Audio Engineering Society 134th Convention, Rome, Italy
2013, Paper: 8901.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0236

Computationally efficient simulation of edge diffraction
in virtual acoustic environments
Christoph KIRSCH1; Stephan EWERT1
1Carl von Ossietzky Universität Oldenburg,

Medizinische Physik and Cluster of Excellence Hearing4All,
Germany

ABSTRACT
Virtual acoustic environments serve as essential tools for hearing research and development in complex
acoustic environments, where sound sources can be occluded by finite objects or corners in rooms withdiffracted sound reaching the listener. State of the art (real-time) room acoustics simulations are often basedon geometrical acoustics (GA). Integration of diffraction phenomena into GA has been suggested in literature,with different computational demands for a) determining the spectral effects of diffraction and b)
implementing filter representations.
Here, edge diffraction from flat, limited size objects (e.g., tables, boards) placed inside a reverberant room isrendered using computationally-efficient low-order recursive filters. The suggested method can wellapproximate diffraction from a plate in comparison to acoustic measurements. For different geometricalarrangements of a whiteboard and flipchart in a room with obstructed and unobstructed sound paths, room
acoustics simulations with varying complexity are assessed: Conditions with and without (approximated)edge diffraction are compared with varying number of diffraction paths. Renderings are perceptually
evaluated in listening test with measured room impulse responses serving as a reference. A perceivable effectof the presence of objects in the room is found, both in the measurements as well as in the simulation.
Keywords: Virtual Acoustics, Room Acoustics, Diffraction

1. INTRODUCTION
Virtual acoustic environments (VAE) can be created using acoustics simulation, to replicate
existing real-world environments or to construct arbitrary environments without a real counterpart.
Applications range from entertainment to architectural planning (1) and hearing research (2), where
ecological validity (3) in laboratory experiments might be improved by the use of realistic, complex
VAE (4,5).
Given that (real-time) simulation of complex acoustic environments contain multiple interactive
auditory objects, typically located in rooms, simplifications in the underlying room acoustics
simulation are required with the potential to reach high perceptual plausibility (6) at sufficiently low
computational complexity.
For enclosed spaces, where discrete early reflections and diffuse late reverberation occur in
addition to the direct sound, geometrical acoustics (GA) have become a de-facto standard (e.g., 7–9).
Here, ray-like propagation is assumed for sound, often implemented by means of an image source
model (10,11). Limits of GA occur for obstructed sound paths, where the wave properties of soundcause diffraction at, e.g., room corners or finite objects such as billboards and tables. Based on thegeometrical theory of diffraction (12), the uniform theory of diffraction (UTD, 13) and similarly (14)offer asymptotic diffraction solutions for infinite wedges in the context of GA. Finite and infinite
wedges can be described using a secondary source model (here referred to as BTMS, Biot-Tolstoy
Medwin, 15,16). These solutions are, however, computationally involved and have to be simplified
for application in VAEs.
Several approaches have integrated diffraction in GA-based acoustics simulation, using two basic
1

gf@aaa.com

processing steps: i) sound propagation paths including diffraction at edges have to be established (e.g.,
17,18), and ii) diffraction transfer functions of filters have to be derived and implemented , based on
the geometrical arrangement of the incident and excitant sound ray in relation to each edge. A
prominent simplification is to only consider diffraction in the shadow zone, where the (direct) soundpath is occluded. Here, the effect of diffraction is perceptually most prominent, resul ting in an
increasing attenuation of high frequencies for increasing deflection (bending) angle of the sound path(19). Diffraction paths with spectral properties derived from BTMS are rendered in (20) using finite
impulse response (FIR) filters. (21) used diffraction paths with spectral properties derived from UTD.
Similarly, UTD transfer functions are approximated by a number of bandpass filters in (18). (22)
proposed a volumetric approach to approximate the (coarse) spectral characteristics of diffraction at
finite objects, disregarding the spectral fine structure of such transfer functions. (23) evaluated aparametric low-order infinite impulse response (IIR) solution for which parameters were derived using
machine learning from the BTMS solution. (24) suggested a modified fractional-order filter function
to approximate infinite edge diffraction in the shadow zone. Recently, (25) suggested acomputationally efficient, universal filter representation of the Pierce, UTD, and BTMS solution,
suited for application in VAEs.
While several approaches with different simplifications and approximations of spectral diffraction
effects exists, often diffraction at sound barriers, (large scale) building- or room corners was in thefocus, and perceptual evaluation of simulated diffraction in comparison to real-world measurements
is scarce. (26) conducted ABX tests on the audibility of simulated diffraction with different source
stimuli and found that differences could be detected with a pulse, however, not for speech. Particularly,
perception of diffraction effects in the presence of other non-diffracted early reflections and latereverberation has not been assessed systematically. However, in many indoor communication
situations, diffraction at small-scale finite objects such as tables and whiteboards occurs in thepresence of reverberation.
Here, perception of diffraction from flat finite objects in reverberant rooms is investigated.
Binaural room impulse responses (BRIRs) have been measured in two rooms with different
reverberation times, containing either a reflecting table or an occluding whiteboard placed between
the sound source and receiver. Additionally, BRIRs for the same conditions without the table andwhiteboard were measured. Based on (24,25), diffraction at the edges of the finite object was
simulated, extending the room acoustics simulator (RAZR, 9,27). Perceptual ratings of overall
difference and for a subset of attributes from the spatial audio quality inventory (SAQI, 28) wereobtained for headphone auralizations in two spatial configurations in relation to the objects, with avarying number of additional diffraction paths in the acoustic simulation. The results yield
information regarding i) the impact of finite objects on the spatial perc eption in listeners, and ii)
technical requirements for room acoustics rendering of perceptually plausible virtual acoustic
environments. It was found that an occluding object results in a different spatial perception and that
including objects and diffraction in the simulation increases similarity to the measured reference.

2. METHODS
2.1 Listeners
Listening tests were conducted with 6 normal hearing listeners (pure tone threshold of < 20 dB HL,
aged 22 – 35, 2 male). All listeners reported some previous experience with listening tests.
2.2 Rooms and finite objects
Two approximately shoebox-shaped rooms were used for evaluation (see Figure 1). The smaller(5.2 m x 7.2 m x 2.7 m) conference room, referred to as KAS, has a frequency-dependent reverberation
time of approx. 0.5 s – 0.7 s between 125 Hz and 8 kHz. The floor is covered in short-pile carpet, andthere are foam baffles at the ceiling. One of the long walls consists of windows, with the panes angled
to avoid flutter echoes. On the opposing wall, there are diffusor elements. One of the short walls ismade of acoustically treated plaster with a big window. The other short wall is a semi-permanent room
divider. The receiver is placed at a distance of 2.3 m from the long wall and 3.5 meter from the shortwall at height of 1.3 m.
The slightly larger (6.0 m x 7.0 m x 3.2 m) acoustic laboratory room, referred to as VAR, has areverberation time of 0.9 s and 1.4 s between 125 Hz and 8 kHz. One side of the room consists ofwindows with angled panes. There is a vinyl floor and the ceiling is covered with box -shaped diffusors

with dimensions of about 0.2 m x 2.0 m x 0.5 m. Eleven of these diffusors are also distributed on theremaining walls. The receiver is placed at a distance of 2.6 m from the long wall and 3.4 m from theshort wall.
Two loudspeakers were placed at a radius of 2.1 m and 1.3 m height above the floor at 0° and 330°
azimuth in relation to the view direction of the listener (see grey dot and crosses in Figure 1). The
finite objects were a 1.4 m x 1.4 m table with a thickness of 25 mm at a height of 0.7 m and a portable
whiteboard (about 30 mm thick) with a main vertical surface that is 1.19 m in height, placed 0.9 m
above the floor, and a width of 1.8 m. Both objects were placed with their center at half of the distance
between the receiver and the 0° loudspeaker, with the whiteboard perpendicular to the view direction(see bold outlined shapes in Figure 1).
BRIRs were measured using a G.R.A.S. KEMAR 45 BB-3 head and torso simulator withanthropometric pinnae (KB0090 and KB0091) using swept sines (29) using Genelec 8020d
loudspeakers as sound sources.

Figure 1 – Shoebox room representations (light and dark blue) and locations of objects, sources, andreceivers. The listener position and direction are indicated by the grey dot and bar. The loudspeaker
positions are indicated by the black crosses and bars. The thickly outlined horizontal plane represents thetable, while the vertical plane represents the occluding whiteboard
2.3 Room acoustics simulation
An extended version of RAZR (9,27) was used. An image source model is used for the direct soundand discrete early reflections (up to the 3rd reflection order) followed by physically-based feedback
delay network (FDN) for late reverberation.
To account for diffraction in the extended version, diffraction paths are established for each of thefour edges of the finite object in the room according to Fermat’s principle with an apex point locatedon the physical edge, reflecting a node in the diffracted path (e.g., 21). If the apex point would belocated outside the physical edge, the path node is located at the respective corner (end point of theedge). Spectral characteristics for the diffraction paths were implemented using two second-ordersection (SOS) IIR filters, extending the filter implementation described in (24), and based on the twoterm filter representation of (25). Heuristically motivated, complementary lowpass and highpass
filters were applied to the direct sound and associated diffraction paths to account for effects of higherorder diffraction in the low-frequency region. The resulting simulated diffraction for a thin plateclosely matches according measurements available in the benchmark for room ac oustics simulation(BRAS, 30) for occluded and non-occluded sound paths in a frequency range between 200 Hz and 6
kHz (not shown here). Source directivity was modelled based on the data provided for the Genelec
8020 loudspeaker in (30) and receiver directivity was accounted for through the “kemar_s.sofa”
dataset from (31). Different renderings were obtained as described in Table 1.

Table 1: Overview of measured and simulated BRIRs
Abbreviation

Description

mea_obj

BRIR measured with object in placemea_empty

BRIR measured without object in place
Simulation with object, 3rd-order ISM and diffraction paths,

sim_full

including one diffraction node for all paths with up to 3 reflections
Simulation with object, 3rd-order ISM and diffraction paths,

sim_direct

including one diffraction node only for the direct sound path

sim_geo

Simulation with object, 3rd-order ISM, but no diffraction modeling

sim_empty

Simulation without object, 3rd-order ISM

sim_ism

Simulation with 15th-order shoebox ISM (Allen and Berkley, 1979)

2.4 Apparatus, stimuli, and procedure
Listeners were seated in a double-walled listening booth. A computer monitor was placed in frontof them, and responses were provided using a mouse. Stimuli were presented at a sound pressure levelof 65 dB using Sennheiser HD650 headphones, driven by an RME Fireface UCX at a sampling rateof 44.1 kHz.
Three stimuli were binaurally rendered by convolution with the different measured and simulated
BRIRs listed in Tab. 1. i) A sentence from the OLSA corpus (32), spoken by a male talker, about 2 s
in length. ii) A short excerpt of an electric bass played with slapping technique, about 2 s in length .
iii) A deterministic transient stimulus (referred to as pulse or pink pulse) of 0.5 s duration (decay from
0 dBFS to -60 dBFS in 36 ms), which was defined in the frequency domain by as a pink spectrum
between 50 Hz and the Nyquist frequency with minimum phase, and then transformed to the time
domain. To reject temporal aliasing artifacts in the resulting periodic impulse, only the first half theimpulse was cut and a raised-cosine off ramp was applied to the second half of that cut.
For all three stimuli, the overall difference of the renderings (see Table 1) was rated in comparison
to the measured reference (mea_obj) using a multi-stimulus test with hidden reference and anchor(MUSHRA, 33) -inspired paradigm. The sim_ism renderings served as an anchor. Differences wererated for the table and whiteboard in both rooms, with a loudspeaker positioned at 0°.
Additionally, ratings for overall difference and a subset of SAQI attributes (see Table 2) on abipolar scale (using an adapted MUSHRA procedure) were obtained for the pink pulse. Here, thewhiteboard was used in both rooms with a loudspeaker positioned at 330°, moving the sound source
closer to the shadow boundary and potentially increasing the prevalence of binaural cues.
Table 2: Subset of SAQI attributes for listening test. Reverberation was chosen as a compound attribute thatencompasses “level of reverberation” and “duration of reverberation”
Perceptual quality
Horiztontal direction
Source distanceSource width

Scale end labels
shifted anticlockwise – shifted clockwise
closer – more distantless wide – wider

Reverberation

less reverberant – more reverberant

Tone color

darker – brighter

3. RESULTS AND DISCUSSION

Figure 2 – Average overall difference ratings (0 indicates no perceived difference) and standard deviation
across listeners. Source position at 0° azimuth. The blue diamonds indicate average scores for VAR, the red
triangles correspond to KAS. For comparison, the grand average over all stimuli and rooms is plotted as thegrey horizontal bars
Figure 2 shows the results for the overall difference ratings. The grand average over all stimuli androoms (grey horizontal bars) indicates a reliable detection of the hidden reference and anchor. The
simulations including diffraction (sim_full and sim_direct) achieved similar ratings when compared
to mea_empty at about 27. Not including diffraction results in worse scores (about 45) and simulating
an empty shoebox (about 63) further decreases similarity to the reference mea_obj.
Different tendencies can be observed for the two different objects: For the table (lower row), thedifference ratings tend to be similar for mea_empty and sim_full and sim_direct for speech and bass
at about 25. In contrast, with the occluder, a larger difference of about 35 is found for mea_empty,
indicating a larger influence of the presence of the object on the spatial impression. Differences tend
to be rated higher for the less reverberant KAS (red triangles) when compared to VAR (blue diamonds).
For the occluder (upper row), the difference is rated at about 25 for the similarly performing
sim_full and sim_direct, smaller than for mea_empty. This hints towards a relatively accurate
reproduction of the spatial impression by the simulations including diffraction.
Deviating from the clear differences observed for the table, sim_geo, where the occluded sound iscompletely muted, does not evoke a much larger difference compared to sim_full and sim_direct.
However, such a simulation is problematic for transitions from occluded to unoccluded conditions,where the sound would abruptly change. For both rooms, sim_empty has been rated to be mostdifferent, only being surpassed the sim_ism anchor.
A four-way, repeated-measures analysis of variance (rmANOVA) with factors room, stimulus,
diffracting object, and rendering method revealed significant main effects of room [F(1, 5) = 8.78, p= 0.031] and rendering method [F(6, 30) = 57.29, p < 0.001]. The main effect of stimulus just missed
significance [F(2, 10) = 3.36, p = 0.077].
Figure 3 shows the ratings of the individual spatial quality attributes (and overall difference) forthe pulse stimulus. Again, the hidden reference (mea_obj) and anchor (sim_ism) are well detected. It
is obvious that tone color showed generally the largest deviations, likely affecting the overall
difference rating for all simulations. The anchor also showed large deviations for reverberation, source
width and source distance. For each attribute, a two-way rmANOVA was performed, with factors
rendering method and room. A significant main effect of the rendering method was found for overall
difference [F(6, 30) = 35.12, p < 0.001], source distance [F(6, 30) = 8.79, p < 0.001], reverberation[F(6, 30) = 8.12, p < 0.001], and coloration [F(6, 30) = 21.8, p < 0.001]. A significant main effect of
the room was only found for coloration [F(1, 5) = 21.99, p < 0.005].

Figure 3 - Results for overall difference and SAQI attributes for the pulse source at 330° azimuth, occludedby the whiteboard, in the same style as in Fig. 2.

4. CONCLUSION
Based on the tested scenarios and the limited sample size of the (non-expert) listener group, thefollowing (preliminary) conclusions may be drawn:• There is a perceivable effect of the presence of objects in the rooms, both in themeasurements as well as in the simulation. The effect tends to be bigger for occlusion thanfor reflection in the tested conditions.
• There are perceivable differences between measurements and simulations, with tone color
likely affecting overall difference ratings as a main cue.
• Simulations incorporating objects resulted in better perceptual agreement to themeasurements with objects than simulations without objects.
• Simulations including diffraction effects for all reflected paths (up to 3 rd order) did not
improve agreement with measurements over simulating diffraction for the direct soundonly. This is generally in line with the findings from (26).

ACKNOWLEDGEMENTS
This work was funded by the Deutsche Forschungsgemeinschaft, DFG – Project-ID 352015383 –SFB 1330 C5.

REFERENCES
1. Rindel J. The Use of Computer Modeling in Room Acoustics. J Vibroengineering. 2001; 3(4):219–24.
2. Aspöck L, Pausch F, Stienen J et al. Application of virtual acoustic environments in the scope of auditory
research. XXVIII Encontro da Sociedade Brasileira de Acústica, October 2018, Porto Alegre, Brasil.
3. Keidser G, Naylor G, Brungart DS, Caduff A, Campos J, Carlile S, et al. The Quest for Ecological Validity
in Hearing Science: What It Is, Why It Matters, and How to Advance It. Ear Hear. 2020; 41:5-19.
4. Jerger J. Ecologically Valid Measures of Hearing aid Performance. Starkey Audiol Ser. 2009;1(1):4.
5. Pausch F, Fels J. Localization Performance in a Binaural Real-Time Auralization System Extended to
Research Hearing Aids. Trends Hear. 2020;24:1-18.
6. Brinkmann F, Aspöck L, Ackermann D, Lepa S, Vorländer M, Weinzierl S. A round robin on room
acoustical simulation and auralization. J Acoust Soc Am. 2019;145(4):2746–60.
7. Huopaniemi J, Savioja L, Takala T. DIVA virtual audio reality system. Proc Int Conf Auditory Display(ICAD’96). Palo Alto, California; 1996. p. 111–6.

8. Seeber BU, Kerber S, Hafter ER. A system to simulate and reproduce audio-visual environments for
spatial hearing research. Hear Res. 2010;260(1–2):1–10.
9. Wendt T, van de Par S, Ewert S. A Computationally-Efficient and Perceptually-Plausible Algorithm for
Binaural Room Impulse Response Simulation. J Audio Eng Soc. 2014 Dec 4;62(11):748–66.
10. Allen JB, Berkley DA. Image method for efficiently simulating small‐room acoustics. J Acoust Soc Am.
1979 Apr 1;65(4):943–50.
11. Borish J. Extension of the image model to arbitrary polyhedra. J Acoust Soc Am. 1984;75(6):1827–36.
12. Keller JB. Geometrical Theory of Diffraction. J Opt Soc Am. 1962;52(2):116–30.
13. Kouyoumjian RG, Pathak PH. A uniform geometrical theory of diffraction for an edge in a perfectly
conducting surface. Proc IEEE. 1974;62(11):1448–61.
14. Pierce AD. Diffraction of sound around corners and over wide barriers. J Acoust Soc Am.
1974;55(5):941–55.
15. Svensson UP, Fred RI, Vanderkooy J. An analytic secondary source model of edge diffraction impulse
responses. J Acoust Soc Am. 1999;106(5):2331–44.
16.Svensson UP, Calamia PT, Nakanishi S. Frequency-Domain Edge Diffraction for Finite and Infinite Edges.
Acta Acust United Acust. 2009;95(3):568–72.
17. Erraji A, Stienen J, Vorländer M. The image edge model. Acta Acust. 2021;5:17.
18. Schissler C, Mückl G, Calamia P. Fast diffraction pathfinding for dynamic sound propagation. ACM Trans
Graph. 2021;40(4):1–13.
19. Maekawa Z. Noise reduction by screens. Appl Acoust. 1968;1(3):157–73.
20. Schröder D. Physically Based Real-Time Auralization of Interactive Virtual Environments. Dissertation
RWTH Aachen; 2011. Logos Verlag, Berlin.
21. Tsingos N, Funkhouser T, Ngan A, Carlbom I. Modeling acoustics in virtual environments using theuniform theory of diffraction. Proc 28th annual conference on Computer graphics and interactive
techniques - SIGGRAPH ’01. ACM Press; 2001. p. 545–52.
22. Pisha L, Atre S, Burnett J, Yadegari S. Approximate diffraction modeling for real-time sound propagation
simulation. J Acoust Soc Am. 2020;148(4):1922–33.
23. Pulkki V. Virtual Sound Source Positioning Using Vector Base Amplitude Panning. J Audio Eng Soc.
1997;45(6):456–66.
24. Kirsch C, Ewert SD. Low-Order Filter Approximation of Diffraction for Virtual Acoustics. Proc 2021
IEEE WASPAA. New Paltz, NY, USA. 2021. p. 341–5.
25. Ewert SD. A filter description of diffraction at infinite edges. JASA Express Lett. 2022;in press.
26. Torres RR, Svensson UP, Kleiner M. Computation of edge diffraction for more accurate room acoustics
auralization. J Acoust Soc Am. 2001;109(2):600–10.
27. Kirsch C, Wendt T, van de Par S, Hu H, Ewert SD. Computationally-Efficient Simulation of Late
Reverberation for Inhomogeneous Boundary Conditions and Coupled Rooms. J Audio Eng Soc. in press(Special Issue on Spatial Audio).
28. Lindau A, Erbes V, Lepa S, Maempel HJ, Brinkman F, Weinzierl S. A Spatial Audio Quality Inventory(SAQI). Acta Acust United Acust. 2014;100(5):984–94.
29. Farina A. Simultaneous Measurement of Impulse Response and Distortion With a Swept-Sine Technique.
Proc 108th AES convention, February 2000, Paris, France.
30. BRAS - Benchmark for Room Acoustical Simulation. Technische Universität Berlin; 2020. Available
from: https://depositonce.tu-berlin.de/handle/11303/7506.3
31. Thiemann J, van de Par S. A multiple model high-resolution head-related impulse response database for
aided and unaided ears. EURASIP J Adv Signal Process. 2019;2019(1):9.
32. Kuehnel V, Kollmeier B, Wagener K. Entwicklung und Evaluation eines Satztests für die deutsche
Sprache I: Design des Oldenburger Satztests. Z Für Audiol. 1999;38:4–15.
33. ITU BS.1534 : Method for the subjective assessment of intermediate quality level of audio systems.
Geneva; 2015. Available from: https://www.itu.int/rec/R-REC-BS.1534

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0404

Recalibration of the NASA Exterior Effects Room
E.K. Ellington SCOTT1; Andrew CHRISTIAN2; Matthew BOUCHER2; Stephen RIZZI3;
Aric AUMANN4; Haik MARTIROSYAN5
1

Graduate Program in Architectural Acoustics, Rensselaer Polytechnic Institute (NIFS Intern), USA
2

Structural Acoustics Branch, NASA Langley Research Center, USA
3

Aeroacoustics Branch, NASA Langley Research Center, USA
4
5

Analytical Services and Materials, USA

Zeiss Industrial Metrology (NIFS Intern), USA

ABSTRACTThe Exterior Effects Room (EER) is a psychoacoustic testing facility at NASA Langley Research Center
which primarily focuses on testing human response to aircraft noise. The 39-seat auditorium houses a realtime spatial audio system and flyover simulation environment. The audio server utilizes an implementationof three-dimensional vector base amplitude panning (VBAP), a perceptual spatial audio technique exploiting
loudspeaker triplets to place a virtual sound source at an arbitrary spatial position. Due to the irregular room
geometry and nonuniform loudspeaker setup, the audio server applies equalization filtering to compensate
for spectral coloration attributed to loudspeaker installation, crossover filtering, and delay/gain offsets. Thesefilters can incorporate measurements taken at multiple listening points, allowing a more extensive listening
area than filters derived from measurements taken at a single listening point. This work endeavors to update
the existing equalization filter generation process by producing a new set of filters for various human subject
testing scenarios.
Keywords: Equalization, Vector Base Amplitude Panning, Spatial Audio.

1. INTRODUCTION
NASA has a long history of performing laboratory psychoacoustic research for community noise
applications across the aerospace industry: from fixed wing aircraft and rotorcraft of all shapes andsizes (see Rizzi for a recent overview (1)), to current efforts to produce quiet supersonic aircraft (2),Figure 1 – Exterior Effects Room: (a) Floor plan, (b) Photographto early versions of wind turbines (3) and beyond. One of the psychoacoustic test facilities that has
seen a considerable amount of use over the past decade is the Exterior Effects Room (EER) at LangleyResearch Center. Originally designed as a small auditorium, it was retrofit with speakers andacoustical treatments to allow for use as a psychoacoustic testing facility (4). This facility was well
suited for presentation of recorded aircraft flyover noises given its seating arrangement. Originally,
single-event psychoacoustic testing required large panels of actual subjects sitting in the field (i.e.,
outdoors, on chairs, responding via pencil and paper, e.g., Beranek (5)). This required extensive
coordination between pilots attempting to fly different aircraft consistently in the prevailing (and everchanging) environmental conditions. In a laboratory environment such as the EER, researchers cancontrol various aspects of the acoustic scene and be sure that there is a high degree of reproducibility
between presentations (e.g., to different groups of subjects).
Originally, an array of 10 loudspeakers was used to render moving noise sources in the EER, andthese speakers were manually equalized to the center of the room using the electroacoustic equipment
available at the time. In the late 2000s the EER was completely renovated into its current configuration(6). The setup now utilizes a 31-channel playback system of twenty-seven Klein & Hummel O300 and
four Klein & Hummel O900 subwoofer loudspeakers (Figure 1). Satellites are arranged roughly inthree rings in the EER, with those in the lower two rings baffled to protect the loudspeakers from
accidental contact by researchers and subjects. The subwoofers are fully embedded in the corner facetsof the EER.
Although it is possible to use the facility with any multichannel audio playback device – patching
into the speakers can be accomplished either via a ¼” TRS patchbay or “soft patched” via AVBnetworked audio devices – the primary playback device is a Vectsonic AuSIM3D audio rendering
server (7). This system implements a Vector Base Amplitude Panning (VBAP) approach to spatialize
sounds within the EER. VBAP is an extension of stereophonic panning that can create the impressionof a sound coming from anywhere in the EER (6), and the server can apply this technique both torecorded/auralized sources (e.g., rotorcraft flyover noise that was captured in synchrony with GPS
data (8)) as well as to real-time inputs. This system is integrated with a three-dimensional visualenvironment that, while not typically used for psychoacoustic testing purposes, has great utility as acommunication tool (e.g., to stakeholders and the public).
In addition to the VBAP processing, the AuSIM3D server can apply compensatory equalization
filters for each speaker in the array. Originally, these filters (along with compensation for differingtime-of-flight delay and gain) were derived for two multichannel microphone arrays within the EER(6). Since that work, the renovated EER has been used for many psychoacoustic tests (e.g., mostrecently (8)). Subjects typically sit in 4 seats near the geometric center of the room, 1 however, theoriginal equalization process was undertaken before any of these tests. This fact, combined with recent
upgrades to the playback system – from an all-analog system to one built around an AVB audio
network, in addition to any effects of aging/replacement of audio equipment – make it a prudent time
to resurrect the equalization process and undertake it for the system as it exists today.
This paper details the procedure to recalibrate the EER multichannel audio system for different
subject configurations using the current hardware and physical configuration. First, the EER VBAP
system is described. Then the method of deriving equalization filters from recordings of pink noise atvarious locations (including subject seat locations) is developed. Example filter shapes are shown for
combinations of these recordings.

2. VECTOR BASE AMPLITUDE PANNING
The common left-right stereophonic panning scheme, as in most contemporary recorded music,
applies differential gain between two channels of audio in order to create the illusion that a sound iscoming from a particular azimuthal/horizontal direction, as shown in the left of Figure 2 (see e.g.,
Begault (10)). VBAP generalizes this concept to 3 Cartesian (or 2 polar) dimensions to derive gains
for triplets of speakers to generate localization cues in both the azimuthal and elevation/verticaldirections.
Given the unit vectors of the loudspeaker positions, 𝒍(⋅) = [𝑙$ 𝑙% 𝑙& ]' , a loudspeaker triplet isused to reproduce a virtual source within a triangle formed by the 3 locations (Figure 2R). The gain
for each loudspeaker is calculated using a linear combination of the loudspeaker vectors,
1

A series of recent tests have also used a single subject location near one of the subwoofers. See, mostrecently, Rafaelof et al. (9).

Figure 2 – VBAP loudspeaker arrangements where blue is the virtual source vector and black are theloudspeaker vectors. (a) Stereo pair. (b) Triplet.

𝒑 = 𝑔( 𝒍( + 𝑔) 𝒍) + 𝑔* 𝒍* ,𝒈 = 𝒑' 𝑳+,
()*

(1)

where 𝒈 is the gain vector, 𝒑 is the virtual source vector, and 𝑳()* = [𝒍( 𝒍) 𝒍* ]' .
The EER spatial audio system augments VBAP with virtual source positions to remedy triangle
ambiguity between loudspeaker triplet sets. The arrangement of the ceiling loudspeakers creates
indistinguishable triangles – loudspeakers arranged in a rectangle form two geometrically similar
triangles. This phenomenon leads to undesirable level changes when panning through the triplets.
Consequently, virtual loudspeakers were used to create unique triangle sets (Figure 3). Gains arecalculated using the virtual loudspeaker vectors and then distributed between the physicalloudspeakers within the neighboring loudspeaker triplets. For example, for a sound positioned within

Figure 3 – EER VBAP triangles of loudspeaker triplets. Physical loudspeaker positions are black.
Virtual loudspeakers positions are red. (a) Satellite triangle sets. (b) Subwoofer triangle sets.

the triangle 6R, 6L, and Virtual, the gains are calculated using normal VBAP. The gain assigned to
the virtual speaker is then shared equally among its neighbors (5L, 5R, 6L, 6R), so that 6L and 6R
will have more gain than 5L and 5R. The same technique was also applied to the subwoofers: virtual
sources were placed above and below the plane of the 4 subwoofers in order to render sounds comingfrom any elevation. EER VBAP processing also uses spread angle to minimize gain artifacts that canbecome apparent when a source location results in playback through a single loudspeaker. This is doneby defining a small ring of virtual sources around the current source position (6).

3. LOUDSPEAKER EQUALIZATION
Equalization for each loudspeaker is individually calculated given a measurement at a set oflistening locations; one speaker’s equalization is not dependent on another’s. To implement
equalization of each loudspeaker in the EER, three main components are required: equalization filtersin second-order sections structure, relative gain compensation, and time delay compensation. The
following sections describe the methods utilized to calculate these elements of equalization.

Figure 4 – Desired response of subwoofers (blue), satellites (orange, green), and the entire frequency
range of the EER system at a sampling frequency of 48 kHz (dashed).

3.1 Equalization Filter
Implementation of the equalization accounts for both spectral coloration and crossover filtering.
Conceptually, this is accomplished by inverting the difference between the measured response and thedesired response shown in Figure 4. An eighth-order Linkwitz-Riley filter 2 was used as the desired
response due to minimum reaction at the crossover positions.
While an exact inverse of the frequency response may be obtained for a single measurement withzero error, the limitations of this simplistic process may cause the resulting filter to exhibit undesirable
artifacts such as excessive gain when compensating for a null caused by a reflection. Spatial averaging
is considered in the equalization process by incorporating the frequency response of multiple
measurement locations within the listening space. The following sections will discuss the techniques
used to generate the equalization filters.
3.1.1 FIR Wiener Filters
Wiener filtering, a linear optimal filtering, allows for the estimation of a desired response assuming
a linear-time invariant system. The process minimizes the mean-squared error between the desired
and measured system response (11). This process is carried out via Toeplitz matrix operations (12).
The number of FIR taps used for the satellite and subwoofers are 8,192 (213 ) and 32,768 (215 ),respectively. More taps are needed for the subwoofers to generate FIR filters that have the necessary
resolution at low frequency. Incorporating multiple responses in the same filter improves theconditioning of the matrix in the Wiener filtering process. Unfortunately, using these high-order FIR
filters mean that their implementation is both computationally very expensive and would add aprohibitive amount of delay into the system for tracked sources. Therefore, the direct use of the FIR
filters is restricted to offline simulation.
3.1.2 IIR Filters
Lower order IIR filters which approximate the large FIR filters were computed to allow real-timeplayback within the Vectsonic system. Minimum-phase IIR filters which estimate the magnitude
responses of the FIR filters were generated via a Gauss/Newton spectral factorization method (13).
To improve frequency resolution of the satellite IIR approximation, the FIR filter approximation
divided the passband into two segments: 80 Hz to 500 Hz and 500 Hz to 18.75 kHz. 3 If not split up,
2

That is to say that this has an equivalent response to two cascaded 8th-order Butterworth filters.
The upper frequency limit is determined by the sampling frequency of the generated set. It is 𝑓- /2.56, or
18.75 kHz for the 48 kHz filters shown in this paper.
3the IIR estimator will tend to assign too many points of control to higher frequencies, as frequencies
are linearly weighted in the process. The two segments are then joined to form a single IIR filter.
Orders were reduced significantly to 25 second-order sections and 55 second-order sections for
subwoofers and satellite loudspeakers, respectively.
3.2 Time Delay Compensation
The system time delay between input and output was calculated using a peak detection algorithm
in the time domain for a single measurement point (typically the center of the room). The compensateddelay was calculated by subtracting each delay estimate from the maximum measured delay; the delaydifferences were then added to the uncompensated delays. These delays account for the full path,
including the digital signal path and geometric position relative to the selected measurement position.
This allows the AuSIM3D engine to process the delay in a single line for each loudspeaker (6).
3.3 Relative Gain Compensation
Gain compensation was estimated using the mean of the one-third octave band level of each
loudspeaker’s passband. The same compensation technique was utilized with relative gain as
mentioned with delay – levels were subtracted from the maximum mean gain, compensating each
loudspeaker with the difference from the maximum.

4. RECORDINGS AND RESULTS
During the development of the original equalization filters, frequency response measurements weretaken at fourteen microphone positions – twelve spaced throughout the designated listening area andtwo placed near the room center (6). The two-microphone array (2Mic) was used to obtain a small but
accurate sweet spot intended for human-subject experimentation (though before it was known whereexactly the subjects would sit). The twelve-microphone array was used to obtain a larger sweet spotthat was a compromise between the larger distribution of positions and could be used for facility
demonstrations to large groups.
During the current equalization process, 21 measurements were taken with locations strategicallyselected given experiences with past tests. The locations measured are indicated in Figure 5. Elevenseat location recordings were set roughly to a center-of-head position; consistency of this location
was maintained by using a jig that fit over the armrests of the seats. The main four subject locations
were included in this set (4Mic), along with the extrema of the seating space. Recordings were also
taken at locations “in the pit” (or “on the stage,” depending on your proclivities), at the ends of anaisle, and at several locations that matched those from the previous effort. All mics faced upwards,

Figure 5 – Frequency response measurements used in updated equalization framework. Positions
include center location (cyan), two-microphone configuration (orange), seat locations (green), listening
area positions (dark blue, beige, red).

Figure 6 – Comparison between old and new equalization filters: (a, c) Subwoofer 16R, (b, d) Satellite
8L. (a, b) Comparison between old 2Mic filters (blue) and 2Mic recreation with new measurements(orange). (c, d) Comparison between old 2Mic filters (blue) and new 4Mic filters derived from subject
seat location measurements (orange).

except for positions that were meant to match those from the previous effort.4
The mics used were GRAS 40AQ capsules on 26CA preamplifiers. Two GRAS 12AX ICP power
supplies were used to power the mics and apply a +20 dB gain within the EER. The microphonesignals were digitized using an RME ADI-8 QS in the nearby EER control room and were transmitted
via a MADI connection to a computer running the Reaper DAW. 8 channels were captured at once –7 microphone “response” signals and a direct reference “stimulus” signal which was a digital copy ofthe signals being sent to the speakers. As the speakers are self-powered, there are no amplifier settingsfor each channel that could add complications. Pure tone calibration recordings were captured using
a B&K Type 4231 pistonphone.
4

Prior to the first psychoacoustic test following the latest renovation, a set of white curtains were added to block
the view of one subject from another. These curtains, while mainly acoustically transparent, were not installed
during the original recording effort. They were included in this set, as they are always now present during humansubject testing in the EER.

4.1 Equalization Filters
The recording suite consists of two-minute pink noise excitations per-speaker and per-microphone
location. These recordings are processed into incoherently-averaged single-sided power spectral
densities via FFTs of 32,768 (215 ) samples. At a 48 kHz sampling rate, this results in a binwidth of
1.5 Hz. A 50% overlap is used, to generate 240 averages. This was done for all speaker-microphone
combinations – 651 responses. Choosing to include a response (or not) in the equalization processeffectively extends the intended “sweet spot” of the listening area to those locations to the greatest
extent possible (given the constraints of the other included responses).
Figures 6a and 6b show examples of the original 2Mic equalization filter of a satellite andsubwoofer with filters derived from the new measurements (and using the new workflow) that weremade to replicate the old 2Mic case. This shows a large amount of agreement between the two efforts.
Although there are some minor discrepancies, especially in the higher frequencies (where these sortsof measurements are more sensitive), the trends largely match. These results are typical of theagreement found for all of the speakers between the new and old measurements. Evidence of the linear
weighting of the IIR-estimation process is evident by the increasing “jaggedness” of the filter
responses towards the top of the partitioned passbands (consider the smoothness of the functions justbelow and above 500 Hz).
Figures 6c and 6d compare magnitudes of the old 2Mic equalization filters used for human subject
testing with the new 4Mic case derived from the measurements of the subject seat locations. As
expected, the crossover filter limits are consistent while the bandpass variations show some marked
differences in response. Because the typical psychoacoustic test positions the subjects in these fourseat locations, the new 4Mic equalization filters are expected to improve the sound reproduction overthe 2Mic equalization filters (new or old).

Figure 7 – Full-path time delay compensation for each satellite and subwoofer using a measurement
made in the geometric center of the room as the reference.

4.2 Delay and Gain
The same suite of recordings can be processed in order to determine the relative delays and overall
gain values for the speakers. In the past, a microphone at the geometric center of the room was used
to derive the delays, and this location was included in the current recording suite. When employed as
the timing reference point, the maximum delay is approximately 16 ms or 750 samples (see Figure 7).
In comparison to the original measurements, the most considerable difference is within 0.5 ms,
showing the updated framework is highly consistent with the original results. It is possible to use
another microphone location to derive these delays, for example to accommodate a single subject
sitting close to a subwoofer, but for typical EER usage going forward, the center mic will be used toderive the timing information.
Finally, the same recordings can also be used to set the relative gain offset between speakers. Theseresults also showed consistency with previously derived gains. The process by which the gains arederived is tied to the microphones selected for the filter derivation, and not that chosen for the delay.
That is, it is possible to use the center mic to derive the timing, and the 4 subject seat locations toderive the filter and gains, but unwise to split the filter and gain computations between different mics.

ACKNOWLEDGEMENTS
This project was initiated during Haik Martirosyan’s stay at NASA Langley as a NIFS intern andwas largely completed during Ellington Scott’s NIFS internship. Funding for this work was provided
by the Revolutionary Vertical Lift Technology project – thanks to Susan Gorton, Benny Lunsford,
Noah Schiller, and others. Thanks to Jaye Moen for building the jig that kept the placement of the
microphones consistent between seats in the EER.

REFERENCES[1] Rizzi, S. A., “Toward Reduced Aircraft Community Noise Impact via a Perception-Influenced Design
Approach,” InterNosie 2016, Hamburg, Germany, 2016. p 2-26.
[2] Rathsam, J., Klos, J. and A. Loubeau, “Effects of chair vibration on indoor annoyance ratings of sonicbooms,” The Journal of the Acoustical Society of America, Vol. 143, No. 1, 2018.
[3] Shepherd, K. P., “Detection of Low Frequency Impulsive Noise from Large Wind Turbine Generators,”
NASA CR-172511, 1985.
[4] Hubbard, H. H. and C. A. Powell, “Acoustic Facilities for Human Factors Research at NASA LangleyResearch Center,” NASA TM-84585, 1981.
[5] Beranek, L., “Riding the Waves: A Life in Sound, Science, and Industry,” MIT Press, Cambridge, MA,
2008.
[6] Faller K.J., Rizzi, S.A. and A. A. Aumann, “Acoustic Performance of a Real-Time Three-Dimensional
Sound-Reproduction System,” NASA TM-2013-218004, 2013.
[7] “AuSIM3D Vectsonic,” <http://www.ausim3d.com/products/Vectsonic/index.html> , accessed 8/2022.
[8] Boucher, M. A., Christian, A. W., Krishnamurthy, S. and S. A. Rizzi, “A perceptual evaluation of theefficacy of Sound Exposure Level in the rating of annoyance to helicopter noise,” Presented at the VFSInternational 76th Annual Forum & Technology Display, Virginia Beach, VA, 2020.
[9] Rafaelof, M., Christian, A. W., Shepherd, K. P., Rizzi, S. A. and Stephenson, J. H., “Audibility of Multiple,
Low-Frequency Tonal Signals in Noise,” NASA TM-2019-220398, 2019.
[10] Begault, D. R., “3D Sound: For Virtual Reality and Multimedia,” AP Professional, Boston, MA, 1994.
[11] Haykin, S. “Adaptive Filter Theory (5th Ed.),” Pearson, Upper Saddle River, NJ, 2014.
[12] Roberts, R. A. and C. T. Mullis, “Digital Signal Processing,” Addison-Wesley Longman Publishing Co.,
Inc., 1987.
[13] Dennis Jr., J. E. and R. B. Schnabel, “Numerical methods for unconstrained optimization and nonlinear
equations,” Society for Industrial and Applied Mathematics, 1996.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0616

Perceptual evaluation of simulated and real acoustic scenes withdifferent acoustic level of detail
Stefan FICHNA1; Christoph KIRSCH2; Bernhard U. SEEBER3; Stephan D. EWERT2
1,2

Medizinische Physik and Cluster of Excellence Hearing4all, Carl von Ossietzky University of Oldenburg
3

Audio Information Processing, Technical University of Munich, Germany

ABSTRACTThe creation and simulation of ecologically valid acoustic environments is becoming an integral task invirtual acoustics. Such environments enable simulation of realistic daily-life situations under laboratory
conditions in hearing research and audiology. Regarding the underlying room acoustics simulation, it isunclear which acoustic level of detail is required to capture the perceptually relevant effects. In this study,
different levels of detail in the acoustic simulation of three diverse environments, a typical living room witha connected kitchen, a pub, and an underground station were examined. In each environment, an acoustic
scene relevant for hearing research and audiology was tested. In the room acoustics simulation, the level ofdetail in the geometrical acoustics-based simulation of early reflections, the inclusion of diffusely reflectedenergy, and simplified rendering of late reverberation were varied, including conditions were only the direct
sound was presented. Using synthetic binaural room impulse responses and dummy-head recordings fromthe corresponding real environments, speech intelligibility was examined with a matrix sentence test and an
additional interferer in the scene. Spatial audio quality was evaluated using a pulse with a pink spectrum, amusic sample, and a speech signal. Differences observed for speech intelligibility and audio quality arecompared.
Keywords: Room acoustics simulation, virtual acoustic environments, spatial audio quality

1. INTRODUCTION
In psychoacoustics, it is often necessary to conduct listening experiments under strict laboratory
conditions, in which all relevant factors can be controlled. This is opposed to the fact that results fromsuch laboratory conditions, often using simplified acoustic conditions and abstracted synthetic stimuli
do not necessarily correspond to results measured in more complex real-life situations. Regarding
hearing aid performance and speech intelligibility, this discrepancy has already been demonstrated(e.g., 1). Therefore, ecological validity, referring to the degree to which research findings reflect real life hearing-related function, activity, or participation (2) has received increasing attention in hearing
research. Complex acoustic environments (CAEs, e.g., 3) with suited acoustical reproduction andrendering methods offer an opportunity to close the gap between simplified laboratory and real-life
measurements (e.g., 4). Hereby, the use of ecological valid CAEs allows for a better understanding ofauditory perception as well as for an situation targeted hearing aid fitting in listening conditions
closely resembling real live situations. Acoustic measurements, characterization, and documentationof real-life CAEs offer “ground truth” data for establishing lab-based measurements (e.g., 5, 6).
One relevant acoustical factor in realistic CAEs and embedded communication situations are soundreflections and reverberation, typically occurring in enclosed spaces. One way to generate CAEs withthe ability to directly control room acoustic properties, is the use of room acoustics simulation andvirtual acoustics. Several room acoustic simulation approaches exist (e.g., 7, 8, 9, 10, 11), all involving
simplifications of the underlying acoustic processes. While a high degree of perceptual plausibility
can be reached with state of the art approaches (12), it remains unclear which acoustic details of thesimulation are particularly relevant for speech intelligibility and how spatial audio quality of thesimulation is affected.
This study investigates how different acoustic levels of detail (ALOD) of the room acoustics
simulation affect speech intelligibility and sound quality in three acoustically diverse rooms. Binaural
1

stefan.fichna@uol.deroom impulse responses (BRIRs) recorded in a living room w ith a connected kitchen, a pub, and anundergroundstation(6,
publicly
available
on
the
open
Zenodoplatform,
https://zenodo.org/communities/audiovisual_scenes/) were used as reference for headphoneauralization. The Room Acoustics SimulatoR RAZR (10, 11) was used to generate synthetic BRIRs.
The ALOD in the simulation was systematically changed from a simple anechoic condition
maintaining (relative) level effects, over the use of a shoebox image source model (13), to using all
features of RAZR including simplified effects of scattering and diffraction.

2. Methods
2.1 Listeners
Three normal-hearing listeners aged between 29 and 37 years who were all native German speakers
participated in the experiment. One of the three participants received hourly compensation. The othertwo listeners were employed by the University of Oldenburg. All listeners had some prior experiencein psychoacoustic tests.
2.2 Acoustic scene
Three everyday life acoustic scenes in three different acoustic environments were used (see vander Par et al., 2022 for details): The living room scene is located in a furnished living room with thedimensions of 4.97 m x 3.78 m x 2.71 m, which results in a volume of 50.91 m³ and a reverberation
time (T30) of 0.54 s. Connected to the living room by an open door is a kitchen (4.97 m x 2.00 m x
2.71 m), resulting in a volume of 26.94 m³ with a reverberation time (T30) of 0.66 s. The scene in thisenvironment consists of a receiver seated on a sofa and the target in the coupled kitchen directed
towards the door between the two rooms (located to the front-left of the receiver). In the speech
intelligibility measurement, a masker sound source is located in the living room to the front -right of
the receiver at a distance of 1.6 m.
The pub consists of a single large room with dimensions 17.76 m x 10.2 m x 2.9 m, leading to a
volume of ~442 m³. The reverberation time (T30) is 0.7 s. The pub scene is characterized by arelatively low reverberation time for its large volume and several objects in the immediate vicinity of
the receiver position, such as a table and a chalkboard, resulting in nearby reflections. The receiver islocated at a table, while the target signal is located 1 m away on the opposite side of the table. The
masker for the speech intelligibility measurement is located to the right of the receiver, with a distanceof 0.83 m.
For the third scene, an underground station serves as the environment. It consists of a very large,
elongated space (containing the platform and two tracks) of about 120 m x 15.7 m x 4.16 m with a
total volume of about 11000 m³ and a reverberation time (T30) of 1.6 s. In addition to the alreadylarge volume of the underground station, the connected underground tunne ls and escalators provide
additional (coupled) volumes affecting the late reverberation. The target signal is located at a distanceof 6.37 m in front of the receiver. A masker for the speech intelligibility measurement is located to
the right of the receiver at a distance of 1.6 m.
2.3 Room acoustics simulation
To obtain acoustic simulations in the three scenes at different acoustic level of detail , synthetic
BRIRs were generated using the Room Acoustic Simulator RAZR (10, 11). RAZR combines a proxy“shoebox room” image source model (ISM) for calculation of early reflections ( 13) and a feedback
delay network (FDN) for diffuse reverberation (14) using physically-based delay times. Five sets ofsynthetic BRIRs were generated per scene with different levels of detail, by enabling/disabling
features of the simulation: i) The set with the lowest ALOD are the anechoic scenes. These simulations
only consider the distance between the target position and the receiver position. ii) The second set
referred to as high-order ISM uses a pure shoebox ISM (13) of tenth order. Here, a completely regularspatial and temporal pattern of (specular) reflections emerges. iii) The highest ALOD uses RAZR withall features including coupled rooms. A third-order ISM is used for the early reflections with jitteringof image source positions as well as temporal smearing of early diffuse reflections, effectively
simulating effects of scattering and multiple (inter-)reflections caused by geometric disturbances atwalls and by objects in the room (15). iv) The ALOD in the early reflections is lowered by a reductionof the ISM order from three to one. v) For the last set of synthetic BRIRs, referred to as RAZR simple,
for each room a specific feature of the acoustic simulation was disabled . For the living room, thecoupled room simulation was replaced by a virtual source in the center of the door which radiated the
response of the kitchen room, simulated in isolation. For the pub scene, nearby reflecting finite
surfaces representing the table top and a chalkboard were removed. In the underground station, a
coupled volume, representing the underground tunnels and escalators, was disregarded in thesimulation, removing a dual-slope characteristic in the energy decay curve.
2.4 Stimuli
Speech intelligibility was assessed using the Oldenburg Sentence Test (OLSA, 16). The OLSA
speech material consists of a large number of meaningless test sentences with correct grammaticalstructure but no semantic predictability, as all sentences were constructed from a total of 50 words
with ten alternatives for each word type (name-verb-numeral-adjective-object). The sentences werespoken by a male talker with only a very mild accent. A male-transformed version (17) of theInternational Speech Test Signal (ISTS, 18) was used as a masker for the speech intelligibility test.
ISTS reflects nonsense speech generated from six different talkers in different languages .
The sound quality measurement used three different target stimuli.
Additionally, one OLSA
sentence (duration about 2 s) was used as well as a short excerpt of an electric bass played withslapping technique, about 2 s in duration.
All stimuli were binaurally rendered to headphones using either recorded BRIRs from the real -life
environments as reference or synthetic BRIRs generated at different ALOD.
2.5 Apparatus and procedure
Listeners were seated in a double-walled, sound-proof booth and listened via Sennheiser HD 650
headphones driven by an RME Fireface UCX at a sampling rate of 44.1 kHz. A computer monitor wasplaced in front of them, and responses were provided using a computer mouse.
For the speech intelligibility measurement, the closed version of OLSA was used with all potential
words shown in matrix form on the computer screen. For each test, the SNR was determined at a 50%
word intelligibility (speech recognition threshold, SRT). Lists of twenty sentences were used. The
measured impulse responses, the anechoic condition, RAZR (with all features), and the high-order
ISM were used.
Sound quality ratings were obtained using a procedure similar to the Multi-Stimulus Test withHidden Reference and Anchor (MUSHRA). The listeners rated a number of stimuli in comparison toa reference using sliders on the computer screen. The signal convolved with the measured BRIR was
considered the reference, while the test signals were convolved with the four simulated BRIRs withdifferent ALODs. The measured stimulus was also presented as hidden reference and (based on pre tests) the high-order shoebox ISM condition served as anchor. Nine items from the Spatial Audio
Quality Inventory (SAQI, 19) were evaluated with ratings relative to the reference. The overall
difference was rated on an unipolar scale. A bipolar rating scale with respective descriptors on bothends of the scale (e.g., more pronounced – less pronounced) was used for tone color, metallic tone
color, reverberation (jointly representing reverberation level and time), envelopment (byreverberation), source distance, source width, horizontal direction, and externalization. For the overall
difference, all three target signals speech, pink pulse, and bass were used. For the remaining 8 items
only the pink pulse was used. This resulted in overall 33 tests. The participants first evaluated theoverall difference in random order before they evaluated the other quality items in random order.
Listeners were able to listen to the signals repeatedly, to listen only to self-selected time segments (ofat least 1 s), and to sort the stimuli (sliders) according to their rating.

3. RESULTS AND DISCUSSION
3.1 Speech Intelligibility
In Figure 1 the average SRT as signal-to-noise ratio (SNR) at a speech intelligibility of 50 % isshown. Error bars represent the standard deviation across listeners. The data are grouped according to
the three environments, living room, pub, underground, respectively. The results show that SRTs arelowest in the pub, followed by the underground station and the living room. In all rooms, the anechoic
scene leads to the lowest SRTs caused by the absence of reverberation. Nevertheless, the overall
difference between the environments is maintained, reflecting effects of distance attenuation in theFigure 1.

Average SRTs (including one standard deviation across listenres) for different ALODs(blue: measured; red: RAZR; yellow: anechoic; purple: high-order shoebox ISM) on different
environments (left: living room; middle: pub; right: underground station.
direct sound. In the pub and the underground station, speech intelligibility is very similar for themeasured reference, the RAZR simulation and the high-order shoebox ISM. In the living room,
however, SRTs are considerably higher for high-order ISM and somewhat lower for RAZR than for
the reference.
3.2 Speech Intelligibility
Figure 2 shows the overall difference ratings as box plots. The middle marker of a box shows themedian of the listeners’ data and the top and bottom edges show the 25th and 75th percentiles,
respectively. The whiskers extend to the most extreme data points. The le ft to right panels show theresults for speech, bass guitar and pink pulse, respectively. Each panel is divided into five sections,
one for each ALOD, separated by a vertical dashed line. In each section, the colors represent thedifferent environments (left to right): living room (blue), pub (red), underground (black). The y-axis
shows the rating scale, where 0 represents no difference between the test stimulus and the reference
while 100 represents very large differences.
The listeners reliably identified the hidden reference and often assigned the highest perceived
difference to the high-order shoebox ISM (which served as anchor). Variability in the data is quite
high. There are perceived differences between the measured reference and all simulations. For all
three stimuli and rooms, the differences between RAZR and RAZR with 1st-order ISM are relatively

Figure 2.

Overall perceived audio quality difference (0 represents no difference between the test

stimulus and the reference,100 represents very large differences) for speech (left), bass guitar (middle) andpink pulse (right). The three different colors represent the living room environment (blue), the pub (red)and the underground station (black).

Figure 3.

Spatial audio quality ratings for the pink pulse depending on ALOD. The verbal attributes

assigned to the scales are indicated on the top and bottom of each panel. Numerical values from -50 to 50
where assigned to the bipolar scales; 0 represents no difference to the reference. Different colors represent
the living room environment (blue), the pub (red) and the underground station (black).
small. For the living room (blue) and all three stimuli, the listeners rated RAZR simple, where theimpulse response of the coupled room was realized by convolution of the impulse responses of thetwo individual rooms, considerably more different than the reference and RAZR. For speech and thebass guitar in the pub, the difference of the high-order shoebox ISM (anchor) condition from thereference was comparable to that of the other ALOD conditions.
Figure 3 shows the scores for all quality items using the pink pulse in the same format as for theoverall difference in Fig. 2. Again, the hidden reference was reliably detected and rated with nodifference. Differences between RAZR and RAZR with 1 st -order ISM are small for all three rooms.
For externalization, horizontal direction, and metallic tone color, median differences between thereference, RAZR, and RAZR with 1 st-order ISM are small. For these ALOD conditions, the largest
differences and spread across listeners occur for tone color, source width, and envelopment, likely
contributing to the overall perceived difference (see upper right panel). D ifferences between RAZR
and RAZR simple are mostly small except for the living room, for which tone color, metallic tone
color, and reverberation showed larger differences. The high-order shoebox ISM differed most fromthe reference for most items. Only for horizontal direction and source width, the results are similar tothe other ALODs.

4. CONCLUSIONSPreliminary data for a small group of three listeners was presented comparing speech intelligibility
and spatial audio quality features between headphone auralizations using room acoustics simulation
with varying acoustic level of detail and acoustic (ground truth) measurements in three diverse
environments, a living room, a pub, and an underground station.
For speech intelligibility, an overall good agreement between binaural renderings based on
measured and simulated BRIRs was shown, with part of the difference between the environment
explained by an anechoic simulation, representing distance-dependent level and SNR effects in the
direct sound. For spatial audio quality, a pure high-order shoebox ISM showed the largest perceived
differences. There was no considerable difference whether early specular reflections of the shoebox
room approximation in RAZR were rendered up to the first or third order in combination with a FDN

for late diffuse reverberation.
The current (preliminary) data have to be extended using larger subject pool and more measurement
repetitions to draw stronger conclusions. More training is required for the spatial audio quality ratings,
which are recommended to be performed by expert listeners.

ACKNOWLEDGEMENTS
This work was supported by the Deutsche Forschungsgemeinschaft (DFG) – Project-ID 352015383– SFB 1330 C5 and DFG SPP Audictive - Project ID 444827755.

REFERENCES
1. Cord, M., Baskent, D., Kalluri, S., and Moore, B. (2007). Disparity between clinical assessment and realworld performance of hearing aids, Hearing Review, 14, 22.
2. Keidser, G., Naylor, G., Brungart, D. S., Caduff, A., Campos, J., Carlile, S., ... & Smeds, K. (2020). The
quest for ecological validity in hearing science: What it is, why it matters, and how to advance it. Ear andhearing, 41(Suppl 1), 5S.
4. Fichna, S., Biberger, T., Seeber, B. U., & Ewert, S. D. (2021, September). Effect of acoustic scene
complexity and visual scene representation on auditory perception in virtual audio-visual environments.
In 2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA) (pp. 1-9). IEEE.
5. Hládek, Ľ., Ewert, S. D., & Seeber, B. U. (2021, September). Communication conditions in virtual
acoustic scenes in an underground station. In 2021 Immersive and 3D Audio: from Architecture to
Automotive (I3DA) (pp. 1-8). IEEE.
6. van de Par, S., Ewert, S.D., Hladek, L., Kirsch, C., Schütze, J., Llorca-Bofí, J., Grimm, G., Hendrikse,
M.M.E., Kollmeier, B., Seeber, B.U., 2022. Auditory-visual scenes for hearing research. Acta Acoustca,
in press
7. Hafter, E., & Seeber, B. (2004). The simulated open field environment for auditory localization research.
In Proc. ICA 2004, 18th Int. Congress on Acoustics, Kyoto, Japan, 4.-9.04. 2004.
8. Seeber, B. U., Kerber, S., & Hafter, E. R. (2010). A system to simulate and reproduce audio–visualenvironments for spatial hearing research. Hearing research, 260(1-2), 1-10.
9. Schröder, D., & Vorländer, M. (2011, January). RAVEN: A real-time framework for the auralization ofinteractive virtual environments. In Forum acusticum (pp. 1541-1546). Denmark: Aalborg.
10. T. Wendt, S. van de Par, and S. D. Ewert, “A computationally-efficient and perceptually-plausible
algorithm for binaural room impulse response simulation,” J. Audio Eng. Soc., vol. 62, no. 11, pp. 748–
766, 2014.
11. Kirsch, C., Wendt, T., van de Par, S., Hu, H., Ewert, S., 2022 Computationally-Efficient Simulation of
Late Reverberation for Inhomogeneous Boundary Conditions and Coupled Rooms. J. Audio Eng. Soc. In
press.
12. Brinkmann, F., Aspöck, L., Ackermann, D., Lepa, S., Vorländer, M., & Weinzierl, S. (2019). A roundrobin on room acoustical simulation and auralization. The Journal of the Acoustical Society of America,
145(4), 2746-2760.
13. Allen, J. B., & Berkley, D. A. (1979). Image method for efficiently simulating small‐room acoustics. The
Journal of the Acoustical Society of America, 65(4), 943-950.
14. J.-M. Jot and A. Chaigne, “Digital delay networks for designing artificial reverberators,” AudioEngineering Society Convention 90, 1991.
15 Buttler, O., Wendt, T., van de Par, S., & Ewert, S. D. (2018). Perceptually plausible room acoustics
simulation including diffuse reflections. The Journal of the Acoustical Society of America, 143(3), 18291830.
16. Wagener K, Brand T, Kollmeier B. Development and evaluation of a German sentence test part III:
Evaluation of the Oldenburg sentence test. Zeitschrift für Audiologie. 1999. 38:86–95.
17. Schubotz, W., Brand, T., Kollmeier, B., & Ewert, S. D. (2016). Monaural speech intelligibility anddetection in maskers with varying amounts of spectro-temporal speech features. The Journal of the
Acoustical Society of America, 140(1), 524-540.
18. Holube, I., Fredelake, S., Vlaming, M., & Kollmeier, B. (2010). Development and analysis of aninternational speech test signal (ISTS). International journal of audiology, 49(12), 891-903.
19. Lindau, A., Erbes, V., Lepa, S., Maempel, H. J., Brinkman, F., & Weinzierl, S. (2014). A spatial audio
quality inventory (SAQI). Acta Acustica united with Acustica, 100(5), 984-994.

ABS-0624

Rapid 3D Auralization Of Historically Significant Buildings For Immersive
Classroom Activities
Jonas Braasch(1) , Samuel R.V. Chabot(2) , M. Jerry Huang(3) , E.K. Ellington Scott(4)(1) Rensselaer Polytechnic Institute, Troy, USA, braasj@rpi.edu(2) Rensselaer Polytechnic Institute, Troy, USA, chabos2@rpi.edu(3) Rensselaer Polytechnic Institute, Troy, USA, huangm5@rpi.edu(4) Rensselaer Polytechnic Institute, Troy, USA, scotte3@rpi.edu

ABSTRACTThis paper proposes an efficient method to create auralizations of acoustical landmarks using a 2D ray-tracing
algorithm and publicly available floor plans for a 128-channel wave field synthesis (WFS) system with 2.5D
approximation. Late reverberation parameters are calculated using additional volumetric data. The approachallows students to create auralizations from floorplans and other available information (e.g., building height,
material info, and acoustical parameters ) using Matlab or Octave. By using adequate sound sources, important
historical events can be recreated. The listeners can walk through these recreations over an extended user area,such as an immersive classroom, and the software suite can be used to calculate room acoustical parameters forvarious positions directly using a binaural rendering method or via the WFS simulation.

Keywords: Virtual Acoustics, Wave Field Synthesis, Raytracing

1

INTRODUCTION

The paper presented here aims to create walkable room simulations using a rapid prototyping approach. Usinga 2D ray-tracing algorithm, the program can render sound fields over an extended listening area for classroom
activities. The program is used regularly in a class titled Aural Architecture, where students create their own
final projects. The program can also render binaural impulse responses using Head-related Transfer Functions(HRTFs) for the automated analysis of acoustical parameters using binaural models [3]. The concept uses a
2D method to render the direct sound signals and early reflections, a feasible approach considering that wave
field synthesis (WFS) systems are usually 2D setups, where loudspeakers are placed along the perimeter of an
extended listener area [4]. The proposed method is compatible with the 2.5D operator for WFS.
The system is primarily used to recreate historic architectural venues from antiquity to today so that listeners can experience these venues’ acoustical footprints over the extended floor area (10×12 sqm) of Rensselaer’s
Collaborative Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab). The CRAIVE-Lab
uses 134 full-range loudspeakers (JBL 308, 128 speakers along the lab’s perimeter at ear height, plus six ceiling speakers). Eight short-throw front projectors reproduce congruent immersive 360-deg images, e.g., HDR
panoramic photos, on a screen that encloses the listening area up to a height of 4.3 m. The screen is microperforated for improved acoustic performance. This makes the CRAIVE-Lab a unique venue for auralizations,
given that typical CAVE systems have hard projection surfaces. The latter create audible room reflections thatsuperimpose with the simulated room reflections of the WFS signals. The floor area of the CRAIVE-Lab isalso more extensive than that of a typical CAVE. The lab can host up to 49 participants at a time (restrictedby fire code), providing the opportunity to experience an extended section of the simulated acoustical enclosure.
The lab has been optimized for user interaction with the panorama-screen environment [5].
The proposed auralization method allows users to render historic buildings from scaled floor plans, which
are often publicly available in contrast to full three-dimensional CAD models. The research started from an

Figure 1. Photographic panorama presentation of a marketplace (Weikersheim, Germany) in the CRAIVE-Lab.
educational angle, and much of the description in this paper still reflects this original goal.
The 2D approach has some educational advantages over traditional 3D methods. Firstly, it allows creating
a model in less than an hour to a few hours, given the complexity of the floor plan and additional volumetricestimations. A 3D model typically takes much more time to reconstruct, and often the necessary information
is not available. The rendering process is also much quicker, and it allows students to render a 128-channel
simulation in class, while a 3D method takes much longer for the same number of channels. Given that the
CRAIVE-Lab and most other WFS systems position all loudspeakers on a 2D plane, the compromises of thedimensional reduction are acceptable, as long as the volume of the enclosure is considered to calculate theparameters for the late reverberation. For three-dimensional loudspeaker setups, e.g., dome configurations forhigher-order ambisonics, the 2D approach has more setbacks, although the annotated floorplan approach might
still be useful for educational purposes. The biggest side effect of dome configuration will be the lack of ceilingreflections, which have been deemed important in concert-hall acoustics. For these setups, it might be usefulto estimate and add at least one first-order ceiling reflection, for example, using another 2D approach in thevertical plane that dissects a concert hall from front to back.
In the next section, the basic concept of our proposed auralization technique is discussed, followed byauralization examples that are described in Section 2. The paper concludes with a brief conclusion section.

2

METHODS AND IMPLEMENTATION

Figure 2 shows the system architecture of the ray-tracing program. All input variables and data are shown at
the bottom. The basis for auralizing a building or outdoor venue is a floor plan with a scale or known dimensions. Further, a wall and floor materials database is utilized to determine absorption coefficients [7]. The
user determines the locations of sound sources and receivers and provides sufficiently anechoic sound files forauralization. The receiver positions determine the locations of virtual microphones for the WFS loudspeaker locations, the location of a virtual dummy head (rendered through head-related transfer functions), or the positionof a virtual spherical microphone array.

amplitude calibration

3D volume
estimation
Floor area
estimation
2D floorplan
annotation

scaled floorplan
average height estimation

Wall area
estimation

Floorboard Wallelement catalog
with Material IDs
wall-material
database

RT estimation(Sabine equation)

IR processing
Late reverberation
using windowed
Gaussian noise
bursts×

Ray Tracing
2D approachup to xth orderIR estimation
Direct sound/Early
reflections+

critical distanceomni sources

source & receiverlocations

DAW*

anechoic soundsourcesFigure 2. System architecture for the auralization algorithm.
2.1 Floor-plan annotation
In order to allow fast calculations, the method presented here is confined to simulations in the horizontal plane(two-dimensional rendering method). This allows for the simulation of complete walk-throughs using a batchprocess. The ray-tracing algorithm was implemented in Matlab following common practices [14, 9, 2]. Additional features were added where needed, for example, an algorithm to create circular boundaries to simulate
tree trunks and a method to rapidly generate models from floor plans.
Coordinates of acoustic boundaries can be assigned to the ray-tracing algorithm in three different ways: (i)line segments with start and end points representing walls, (ii) squared pillars with the center coordinates andthe pillar width, (iii) circles represented by center and radius coordinates. Figure 3 shows an example of the
ray-tracing software for a concert hall in Eisenstadt that was recreated from an annotated floor plan. For thispurpose, publicly-available floor plans are marked up in a standard bitmap editor (GIMP, Photoshop, etc.). Red
dots are used for room corners, green dots for squared pillars, and blue dots for circles – see Fig. 3, left graph.
In the next step, the program plots annotated points on top of the map providing a unique index for eachannotated point – see Fig. 3, center graph. The user then creates a list of how the red points connect to walls.
The scale of the floor plan needs to be annotated, and the original dimensions need to be handed over to theprogram (e.g., 1 and 10 m for two scale points). The program then transforms these data points into an editable
list of geometrical objects that can be extended by a user, for example, by adding identifiers for wall materials.
The user corrects and edits these pairs to finalize the digitized floor plan.
The user can either assign a single wall-material ID to all walls or specify individual wall-material IDs
for each wall – see Fig. 2, bottom, 2nd -from-left box. Currently, wall material data is taken from a DIN
database, and the database contains a short description of the material, frequency-specific absorption coefficients,
and Finite-Impulse-Response (FIR) filter coefficients. The latter is needed to compute adequate room impulse
responses.
2.2 Ray-tracing algorithm
For each sound-source position, the program sends out rays at equidistant azimuth angles covering the fullhorizontal plane (360◦ circle) – see Fig. 2, 3rd -left bottom box. Intersection points are computed for each ray
and boundary object using the solution of the line–line intersection problem in Euclidean geometry. For each
ray, the closest boundary intersection is determined. At the intersections, the reflection angle is calculated using
Ibn Sahl/Snell’s law [12, 15]. Then the next-order ray is sent out into the new direction until the maximum
order (e.g., the number of reflections) specified by the user is reached or the ray exits the floor plan – see
Fig. 4. The outgoing rays are stored as a sequence of ray elements containing the intersection points andthe boundary material identifiers. Since the initial angles are stored with the rays, a source-specific directivity
pattern can be simulated after all rays have been traced.

Rendered Floor Plan
50

Length [m]

40

30

20

10
0

10
Width [m]20

Figure 3. Demonstration of the ray-tracing algorithm for the Haydn-Saal in Eisenstadt, where Joseph Haydn
was active. Left: Processed marked-up floor plan for visual inspection, Center: Assignment of wall-corner
identification numbers, Right: Geometric model with a sound source (red dot) and receiver (blue dot) andcalculated rays. The original floor plan was obtained from [10, p. 147].
2.2.1 Creating a room impulse response
In the following step, the rays are collected by receivers in the module IR estimation – see Fig. 2, rightmost
bottom box. Receivers can be located anywhere in the rendered room. For WFS, a virtual array is placed
into the floor plan section that should be auralized. The receivers (virtual microphones) are positioned at the
WFS loudspeaker positions. So for a 128-channel loudspeaker system, the same number of receivers are placedat the loudspeaker positions. For each receiver, a separate impulse response is created. A virtual circle withan adjustable diameter is positioned at each receiver location to catch the rays. Then, the algorithm calculates
which ray elements intersect the circle, and for all positive cases, the total ray distance between sound source
and receiver is calculated. All listed values are stored together with the final angle of incidence, the angle theray was initially sent out, the reflection order, and the sequence of wall identifiers. Based on these data, theimpulse response is created. The direct sound and reflections are computed as delta peaks at the delays that
correspond to the ray’s path length from the source to the receiver. In addition, each impulse is transformed thefollowing way:
1. The intensity of the ray is reduced based on the inverse-square law. Consequently, the sound pressure
magnitude decreases with 1/r.
2. The high frequencies are filtered out based on atmospheric attenuation.
3. The absorption effects of the walls and other boundaries are simulated using a cascaded Finite Impulse
Response (FIR) filter. The material-specific filters are chosen from a DIN database [7]. The number ofcascaded filters matches the order of the reflection.
4. In the final step, direct incoming sound and reflections are selected by their close proximity passing a
receiver position within the virtual circle that was placed around it. The virtual receivers are directionallysensitive using the positive lobe of a figure-of-eight pattern. The latter point outward the array to avoid

W2−a2 a2
x2
−a1
a1

x3

W1

x1

W3

a3

a0

ReceiverSourceW4

S

SourceDirectivity
a1

SourceDistance
x1 + x2 + x’3

Wall
Absorption
W1

Wall
Absorption
W2

Angle of
Incidencea3

R

Figure 4. Left: Example of a ray-tracing pathway, Right: Ray-tracing signal flow diagram for this pathway.
echo artifacts resulting from rays passing through the array at the opposite end of the array. An overlapadd method ensures that the delayed reflections can partially overlap, which can happen because of thecascading FIR filters’ prolonged effect that simulates the wall reflections.
2.2.2 Simulation of late reverberation
Late diffuse reverberation is computed in addition to the early reflections generated by the ray-tracing model.
The structure of the simulated reverb tail is constructed from a Gaussian noise sample with temporal windowing
to simulate the reverberation decay – for details, see [4]. This method reflects that the late reverberation tail isformed by a stochastic process with an underlying Gaussian distribution.

3

AURALIZATION CASE STUDIESThis paper aimed to test the feasibility of the rapid prototyping approach. For this purpose, several landmark
buildings were selected to see if a single person could model them within one afternoon, a typical time slot
available for preparing a class, such as in this case “Aural Architecture.”
3.1 Notre-Dame of Paris
The first building selected was the Notre Dame cathedral in Paris. Built between the years 1163–1260, thechurch is among the most notable examples of Gothic church architecture. Moreover, large cathedrals likely
had a crucial role in the development of polyphonic music, a musical movement spearheaded by medieval composers Léonin (1135–1201) and Perotin (1160–1230), who were both based in Notre Dame. One theory statesthat polyphonic music was a consequence of the long reverberation found at large churches. The reverberation
tail of approximately 6 seconds [11] prolongs a sung note to the extent that multiple sequentially sung notes
will overlap in time, leading to the idea of having three voices sing these accidentally overlapping tones simultaneously. It should be noted, though, that Léonin and Perotin’s work preceded the completion of Notre Dame.
The polyphonic movement is also closely tied to the development of music notation in part because it wasdifficult to remember each voice, especially the center line, which completes the chord structure and is typicallynot very memorable by itself.

Figure 5. Floor plans of two landmark buildings created using the rapid-prototyping approach – left: NotreDame (Paris), right: Westerkerk (Amsterdam). The blue rounded rectangle represents the CRAIVE-Lab’s spatial
footprint in 2D, while the red dots represent source position, and the connected red lines represent sensitivity
among all secondary sound sources (individual loudspeaker).
Name

LengthWidth

HeightVolumeNo. seats

RTFirst Gewandhaus

23 m

13 m

7.5 m

2,242.5 m3

400

1.3 s

Second Gewandhaus

38 m

18 m

15.5

11,200 m3

1,560

1.6 s

Third Gewandhaus

32.3 m36 m

19.8 m21,000 m3

1,900

2.0 s

Table 1. Dimensions for the three Gewandhaus concert halls (RT stands for reverberation time).
3.2 Westerkerk in Amsterdam
The second selected building is the Wester Church (Westerkerk) in Amsterdam, the largest protestant church in
the Netherlands. Hendrick de Keyser designed the church, which was built during the years 1620–1631. The
Renaissance church has an 85-m tall tower, a historic pipe organ from 1686, and is the burial site of Rembrandt
van Rijn. For this building, it is essential to remember that the church Reformation also had profound acousticalconsequences. By empowering ordinary citizens, it became necessary that everyone could follow the church
service, and local languages replaced Latin, which was only understood by a small part of the population. In
1708, Christopher Wren, who was commissioned to build 50 new churches, determined that the new churches
needed to be sufficiently small so everyone could follow what was being said by the preacher [6], [8, Page
9]. The produced floor plan exhibits how small the Wersterkerk is compared to Notre Dame. Here, the plotted
dimensions of our WFS array showed to be a valuable tool for grasping the overall scale of a building.
3.3 Gewandhaus in Leipzig
The acoustics and history of the Gewandhaus Orchestra is an ideal example of aural architecture in the immersive classroom. The Gewandhaus Orchestra is the first German orchestra formed by a civil movement andnot as part of court procedures. It started in 1743 under the label “Great Concerts” [Grosses Concert]. The
Gewandhausorchester was initially a small ensemble of 16 musicians, which grew over time to the size of afull late-romantic orchestra with 185 full-time musicians. Over time, the orchestra has been housed in different
buildings, reflecting the modern concert hall’s evolution. After performing for years in a regular tavern named

Figure 6. Simulated sound fields for the three Gewandhaus concert halls: First Gewandhaus (top-left panel), Second Gewandhaus (top-right panel), Third Gewandhaus for overall structure (bottom-left panel), Third Gewandhaus for effective early reflection surfaces (bottom-right panel).

Figure 7. Simulated 128-channel impulse responses for the three Gewandhaus concert halls: First Gewandhaus (top-left panel), Second Gewandhaus (top-right panel), Third Gewandhaus for overall structure (bottom-leftpanel), Third Gewandhaus for effective early reflection surfaces (bottom-right panel). Each graph shows thespatially aligned impulse responses for the CRAIVE-Lab configuration.
Zu den drey Schwanen [at the three swans], a dedicated concert hall was integrated into a former garment tradingbuilding. The hall was constructed in 1781 by Johann Carl Friedrich Dauthe. This hall is known as the first
Gewandhaus.
The floor-plan-based auralization process presented in this article is ideal for simulating the first Gewandhaus, as there is little data on this building; it was demolished before architectural acoustical design principles
were developed. The first Gewandhaus resembles the transition from court-based orchestral practices to privateorganizations. Given their transitional nature, these provisional solutions have been replaced by specifically designed concert halls, and floor plans, drawings, and verbal descriptions are the only evidence of these buildings.
These circumstances make it an ideal project for the proposed rapid prototyping method. The simulated soundfield shown in the top-left panel of Fig. 6 was created using an existing floor plan. The height of the building
can be estimated from contemporary drawings. The second Gewandhaus was designed by Martin Gropius andcompleted in 1884. It was destroyed in World War II. After a transitional period with a repurposed building(Kongreßhalle am Zoo), the current Gewandhaus opened in 1981 and was the only single-purpose concert hall
erected by the East German Government. Table 1 lists the dimensions for all three Gewandhaus concert halls– the data for the other two concert halls and the seat counts have been described in the literature – e.g., see[13, 1].

Various sources list measured or estimated reverberation times for all three halls, including data for thesecond Gewandhaus that Clement Sabine analyzed in a case study while planning the New Boston Music Hall.
The top-right panel in Fig. 6 depicts the simulated sound field for the second Gewandhaus. Note that thedimensions are much larger than the first hall, with a seating capacity that is about four times as high and a
volume that is about five times larger. The plotted location of the simulated wave field synthesis array in bothvenues provides a good reference for the scale. The current fan/rhombus-shaped concert hall has much larger
dimensions than its two predecessors – see Fig. 6, panel bottom left. However, when computing the reflecting
walls at the concert podium level, the effective reflecting walls are much closer to each other, which is a
strategic part of the acoustic design. Now the effective floor area is similar to that of the second Gewandhaus– see Fig. 6, panel bottom right.
Figure 7 shows the simulated impulse responses for the four simulated sound fields from Fig. 6. Given
its small dimensions, the early reflections arrive the earliest for the First Gewandhaus and are noticeable later
for the Second Gewandhaus. For the Third Gewandhaus, early reflections are absent in the graph if the outer
enclosure is used for the simulation, but they are present if the inner enclosure is simulated as well. In thelatter case, the strength of the reflections is similar to the second Gewandhaus simulation.
The growing library of simulated venues also includes the Cologne Cathedral, Savoy Ballroom Jazz Club,
Columbia 30th Street Studio, St. Mark’s Cathedral in Venice, St. Patrick’s Cathedral in New York City, Pantheonin Rome, Amphitheatre in Trier, to name a few.

4

CONCLUSION

A rapid auralization method was demonstrated in this paper. The process allows rapid prototyping of acousticalenclosures using annotated floor plans and a ray-tracing method. It usually takes less than an hour to annotate
the floor plan for a project and about another hour to render the impulse responses and create the auralizations
for a brief demo with eight sound sources and 128 loudspeakers on a typical desktop computer.

Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant No. HCC-1909229.
The research presented in this paper was carried out within Rensselaer Polytechnic Institute’s Cognitive and
Immersive Systems Laboratory.

References[1] L. Beranek. Concert Halls and Opera Houses. Springer-Verlag, New York, NY, 2004.
[2] J. Blauert, H. Lehnert, J. Sahrhage, and H. Strauss. An interactive virtual-environment generator for psychoacoustic research. i: Architecture and implementation. Acta Acustica united with Acustica, 86(1):94–102,
2000.
[3] J. Braasch. Binaural modeling from an evolving-habitat perspective. In J. Blauert and J. Braasch, editors,
The Technology of Binaural Understanding, pages 251–286. Springer, 2020.
[4] J. Braasch, S. Chabot, E. Chertok, M. Jonathan, and E. E. Scott. Auralizing concert venues over extended
listening areas using wave field synthesis. In Audio Engineering Society Convention 150, page 10465. AudioEngineering Society, 2021.
[5] S. Chabot and J. Braasch. Interactive application to control and rapid-prototype in a collaborative immersive environment. In Audio Engineering Society Convention 151, page 10538, Oct 2021.
[6] E. J. Davis. The parish churches of the city of london. Journal of the Royal Society of Arts, 83(4316):895–
915, 1935.
[7] Deutscher Normenausschuß, editor. Schallabsorptionsgrad-Tabelle. Beuth-Vertrieb GmbH, Berlin, 1968.

[8] M. Forsyth. Buildings for music: the architect, the musician, the listener from the seventeenth century to thepresent day. Cambridge University Press, 1985.
[9] H. Lehnert and J. Blauert. Aspects of auralization in binaural room simulation. In Audio Engineering
Society Convention 93. Audio Engineering Society, 1992.
[10] J. Meyer. Raumakustik und orchesterklang in den konzertsälen joseph haydns. Acta Acustica united with
Acustica, 41(3):145–162, 1978.
[11] B. N. Postma and B. F. Katz. Acoustics of Notre-Dame Cathedral de Paris. In Proceedings of the International Congress on Acoustics (ICA), Buenos Aires, Argentina, pages 5–9, 2016.
[12] R. Rashed. A pioneer in anaclastics: Ibn Sahl on burning mirrors and lenses. Isis, 81(3):464–491, 1990.
[13] W. C. Sabine. Collected Papers on Acoustics. Harvard University Press, 1922.
[14] M. Vorländer. Simulation of the transient and steady-state sound propagation in rooms using a new combined ray-tracing/image-source algorithm. The Journal of the Acoustical Society of America, 86(1):172–178,
1989.
[15] M. Zghal, H.-E. Bouali, Z. B. Lakhdar, and H. Hamam. The first steps for learning optics: Ibn Sahl’s,
Al-Haytham’s and Young’s works on refraction as typical examples. In Education and Training in Optics
and Photonics, page ESB2. Optical Society of America, 2007.

ABS-0691

Ear centering in the spatial and transform domains for near-fieldhead-related transfer functions
César SALVADOR(1) , Ayrton URVIOLA(1) , Shuichi SAKAMOTO(2)(1)

Perception Research, Peru, salvador@perception3d.com, aurviola@perception3d.com(2)

RIEC and GSIS, Tohoku University, Japan, saka@ais.riec.tohoku.ac.jp

ABSTRACTThe head-related transfer function (HRTF) is a major tool in spatial audio. The HRTF for a point
source is defined as the ratio between the sound pressure at the ear position and the free-field soundpressure at a reference position. The reference is typically placed at the center of the head. However,when using spherical Fourier transforms (SFT) and distance-varying filters (DVF) to synthesize HRTFs
for point sources very close to the head, the synthesized HRTF assumes that the measurement position
and the reference position are the same. Ear centering is a technique that overcomes this ambiguity.
Ear centering can be performed with translation operators in the spatial domain (the unit sphere) orwith DVFs in the transform domain (the SFT domain). This paper presents a review of ear centeringand shows that operating in the spatial domain is computationally more efficient than operating in
the transform domain. The reason behind this is that transform-domain ear centering requires DVFs
that depend on two source distances, from the reference and from the ear, whereas spatial-domain ear
centering requires DVFs that depend on the distance from the reference only. Moreover, operating inthe spatial domain is more accurate.

Keywords: Head-related transfer functions, ear centering, translation operator, spherical Fouriertransform, distance-varying filter.

1

INTRODUCTION

There is a growing interest in accurately synthesizing the head-related transfer function (HRTF) for
arbitrary points in the near field, that is, in the region of space within 1 m from the center of thehead [1, 2]. Interests include the design of near-field binaural recording systems [3], the development ofnear-field auditory displays [4], and the consideration of distance in auditory attention experiments [5, 6].
A promising synthesis approach extrapolates near-field HRTFs from far-field ones using the spherical Fourier transform (SFT), distance-varying filters (DVF), and the inverse spherical Fourier transform(ISFT) [7, 8, 9]. When using the SFT to represent spherical HRTF datasets, the default spherical symmetry of the SFT is specified with respect to the reference position. The default spherical symmetry,however, does not allow to distinguish between the reference position (the head center) and the measurement positions (the ears). Such mismatch produces a demand of a high number of basis functions in the
SFT representation and, therefore, affects the synthesis accuracy.
Ear centering overcomes the mismatch between the default SFT center and the ear position. Ear
centering is performed with acoustic operators that translate the SFT center from the head center to theears. Table 1 overviews research related to ear centering interpreted in terms of translation operators. In
summary, translation operators can be applied in the spatial domain (the unit sphere) [10, 11, 12, 13, 14,
15, 16] or in the transform domain (the SFT domain) [17], can be used in far-field [10, 11, 12, 13, 14] ornear-field [15, 16, 17] HRTF synthesis, and can consider acoustic propagation in the free field [13, 14, 15,
16, 17] or include an acoustically rigid scatterer that mimics a simple head [10, 11, 12].

Table 1. Overview of translation operators used for ear centering in HRTF synthesis.
Reference

DomainDistance

Translation model

Porschmann et al.,2019 [10, 11] andArend et al., 2021 [12]Spatial domain

Far field

Plane wave with rigid
sphereZaunschirm et al., 2018[13]

Spatial domain

Far field

Plane wave

Ben-Hur et al., 2019[14]

Spatial domain

Far field

Plane wave

Urviola et al., 2021 [15]and 2022 [16]Spatial domain

Near field

Spherical wave

Richter et al., 2014 [17]Transform domain

Near field

Spherical wave

In our previous papers [15, 16], a question remained open on whether translation operators performbetter in the spatial domain or in the transform domain when synthesizing near-field HRTFs. In thispaper, we address the question by contrasting the performance of the spatial-domain method proposedin [15, 16] and the transform-domain method proposed in [17].

2

EAR CENTERING IN THE SPATIAL AND TRANSFORM DOMAINS

The HRTF is defined as the sound pressure at the left or right ear position rear due to a point sourceat the source position r, divided by the free-field sound pressure at the reference position rref [18]. The
HRTF, denoted by H, is defined as
H(r, rear , rref ) =Ψ(r, rear ),ΨF F (r, rref )(1)

where Ψ(rs , rr ) denotes the pressure emanated from a source position rs measured at a receiver position
rr and the sub-index F F stands for "free-field", which indicates that the head is not present.
a
b

rear

rref

Figure 1. Geometry for near-field HRTF synthesis.
Figure 1 shows the top-view geometry for theoretical near-field HRTF synthesis. The center of thehead coincides with the reference position rref and the ear position is denoted by rear . The point a isa point in a continuous, spherical distribution at a far distance ∥a − rref ∥ from rref . The point b is anH(a, rear , rref )1 Direct translation operator: T (a, rref 7→ rear )H(a, rear , rear )2 Spherical Fourier transformHnm (∥a − rref ∥, rear , rear )3 Distance-varying filters: Dn (∥a − rref ∥ 7→ ∥b − rref ∥)Hnm (∥b − rref ∥, rear , rear )4 Inverse spherical Fourier transform
H(b, rear , rear )5 Inverse translation operator: T −1 (b, rear 7→ rref )H(b, rear , rref )Figure 2. Near-field HRTF synthesis with ear centering in the spatial domain [15, 16].

H(a, rear , rref )1 Spherical Fourier transformHnm (∥a − rref ∥, rear , rref )2 Distance-varying filters: Dn (∥a − rref ∥ 7→ ∥b − rref ∥)Hnm (∥b − rref ∥, rear , rref )3 Translation operator: Dn (∥b − rref ∥ 7→ ∥b − rear ∥)Hnm (∥b − rear ∥, rear , rref )4 Inverse spherical Fourier transform

H(b, rear , rref )Figure 3. Near-field HRTF synthesis with ear centering in the transform domain [17].

arbitrary point at a near distance ∥b − rref ∥ from rref . The distances to a and b from rear are respectively
denoted by ∥a − rear ∥ and ∥b − rear ∥.
Figure 2 overviews the process for near-field HRTF synthesis using ear centering in the spatial domain [15, 16]. The input is a continuous, spherical distribution of HRTFs denoted by H(a, rear , rref )whereas the output is a synthesized HRTF denoted by H(b, rear , rref ). The encoding stage comprises
the direct translation operator in block 1 and the SFT in block 2 . Range extrapolation from distance∥a − rref ∥ to distance ∥b − rref ∥ is performed by block 3 using DVFs; they are transform-domain acoustic
propagators based on a ratio of spherical Hankel functions [7, 8, 9]. The decoding stage comprises the
ISFT in 4 and the inverse translation operator in 5 . Note that spatial-domain ear centering is composedof blocks 1 and 5 which explicitly translate the reference position.
Figure 3 overviews the process for near-field HRTF synthesis using ear centering in the transformdomain [17]. The input and output of this process are identical to those shown in Fig. 2. The encoding
stage only contains the SFT in block 1 . Range extrapolation is performed by block 2 in a similarmanner to block 3 in Fig. 2. Ear centering is performed by the transform-domain translation operator
in block 3 which is implemented with an additional DVF now from distance ∥b − rref ∥ to distance∥b − rear ∥. Although the original proposal in [17] considered an additional optimization stage to modify
rear according to frequency, block 3 in our implementation operates with the real position rear . The
decoding stage only contains the ISFT in block 4 . Note that transform-domain ear centering does not
explicitly translate the reference position.
Operating in the spatial domain is computationally more efficient than operating in the transformdomain. The reason behind this is that implementing the DVF of block 3 in Fig. 2 only requires
distances from rref whereas implementing the transform-domain translation operator of block 3 in Fig. 3
requires DVFs that depend on distances from rref and rear .
The mathematical formulation of the blocks used in Figs. 2 and 3 is available in detail in [15, 16].

3

EVALUATION WITH AN INDIVIDUAL CALCULATED HRTF

This section compares the performance of near-field ear centering in the spatial [15, 16] and transform [17]domains. The DVF and SFT algorithms respectively described in [9] and [19] were adapted to numericallyevaluate four scenarios:• No ear centering• Transform-domain ear centering [17]• Spatial-domain ear centering [15, 16].
3.1 Conditions
The conditions for evaluation are similar to the ones used in [15, 16]. Among the existing calculated nearfield HRTF collections [20, 21], we chose the one in [21] because its data is open and its resolution acrossdistance is dense. Left-ear HRTFs for one individual head model without torso are used in evaluations.
The spatial features due to the torso are prominent at the lower frequencies and can extend up to
3 kHz, whereas the features due to the head and pinna span the middle and high frequencies [22]. Ourassessment focuses on head and pinna features, for this reason the lower frequencies are also covered;hence, the absence of torso does not limit our results. The HRIRs have 512 samples along time, were
sampled at 48 kHz, and c = 344 m/s. The left-ear positions were extracted from the head model.
The sound sources are distributed in spherical grids based on subdivisions of the edges of the icosahedron. Icosahedral samplings are chosen because they achieve bounded spherical integration errors distributed across all orders, whereas other samplings aiming at perfect quadrature at low SFT orders yieldlarge errors concentrated in the high SFT orders [19].
The maximum frequency up to which reliable synthesis is ensured is calculated as
fmax =

cNg,
2πrbound

(2)

-30

-24

-18

-12

-6

0

(a) No ear centering.

(b) Transform-domain ear centering [17].

(c) Spatial-domain ear centering [15, 16].

Figure 4. Synthesis error calculated with (3). Black-dashed curves indicate −3 dB values. Black-dashedlines indicate fmax in (2).

Figure 5. Synthesis error calculated with (4). The black-dashed line indicates fmax in (2).
where c is the speed of sound in air, Ng is the maximum SFT order of the icosahedral grid, and rboundis the radius of a sphere fully containing the head.
Datasets at distances b, ranging from 20 to 100 cm with 1 cm spacing, are used as target data. For
each distance, 642 directions on an icosahedral grid are considered. Datasets are also synthesized for these
distributions of points.
3.2 Error Metric
Target and synthesized HRTF datasets are respectively organized as H (bi , Ωj , fκ ) and Ĥ (bi , Ωj , fκ ). Index
i = 1, 2, ..., 81 indicates radial distances; index j = 1, 2, ..., 642, directions on the sphere; and index
κ = 1, 2, ..., 257, frequency bins. The overall synthesis error across angles is defined as
RMS{H − Ĥ }E(bi , fκ ) =Ωj

RMS{H }

(3)

,

Ωj

and the overall synthesis error across angles and distances as( RMS{H − Ĥ } )E(fκ ) = RMSbiΩjRMS{H }

,

(4)

Ωj

where RMS stands for root mean square along either directions Ωj or distances bi .
3.3 Results
Panels in Fig. 4 show synthesis errors calculated with (3) and displayed in a logarithmic scale, from−30 dB to 0 dB, to contrast with the perceivable HRTF dynamic range of around 30 dB, as reported
in [23]. The black-dashed curves highlight the −3 dB values and are used as an indicator to compare
among panels. We use the −3 dB indicator because this value is commonly considered as a perceivabledifference. The black-dashed lines indicate fmax as formulated in (2).
Contrasting panels (a) and (b) in Fig. 4, it is observed that synthesis without ear centering and withear centering in the transform domain yield similar results. Panel (c) in the same figure, on the other
hand, clearly shows that the most accurate synthesis is obtained when applying ear centering in thespatial domain.

Figure 5 shows the overall synthesis errors across angles and distances calculated with (4). These
results also indicate that spatial-domain ear centering yields a clear enhancement in near-field HRTF
synthesis.

4

CONCLUSIONSWe presented a review of ear-centering methods used in HRTF synthesis. Ear centering can be appliedin the spatial or transform domains, can be formulated for sources in the far field or near field, and canconsider acoustic propagation in the free field or include an acoustically rigid scatterer that mimics a
simple head.
When aiming at synthesizing near-field HRTFs, we showed that operating in the spatial domain iscomputationally more efficient than operating in the transform domain. Moreover, operating in thespatial domain is more accurate than operating in the transform domain.
Extensions to this work might consider detailed mathematical analyses to contrast the equivalences ofthe existing methods for ear centering in the spatial and transform domain. Assessments with top-down
auditory models and subjective tests could also provide more insight into the validity of ear centering.

ACKNOWLEDGEMENTS
This research was partially supported by JSPS KAKENHI Grant Numbers 19H04145 and 22H00523. The
authors wish to thank the SI audio algorithm team for fruitful discussions. The authors also wish to
thank Jorge Treviño for his insights and constructive debates.

REFERENCES[1] S. T. Prepeliţă, J. G. Bolaños, V. Pulkki, L. Savioja, and R. Mehra, “Numerical simulations of nearfield head-related transfer functions: Magnitude verification and validation with laser spark sources,”
J. Acoust. Soc. Am., vol. 148, no. 1, pp. 153–166, 2020.
[2] J. M. Arend, H. R. Liesefeld, and C. Pörschmann, “On the influence of non-individual binaural cuesand the impact of level normalization on auditory distance estimation of nearby sound sources,” Acta
Acust. United Ac., vol. 5, p. 10, 2021.
[3] C. D. Salvador, S. Sakamoto, J. Treviño, and Y. Suzuki, “Design theory for binaural synthesis:
Combining microphone array recordings and head-related transfer function datasets,” Acoust. Sci.
Technol., vol. 38, pp. 51–62, Mar. 2017.
[4] D. S. Brungart, “Near-Field Virtual Audio Displays,” Presence: Teleop. Virt. Env., vol. 11, pp. 93–
106, Feb. 2002.
[5] F. Monasterolo, S. Sakamoto, C. D. Salvador, Z. Cui, and Y. Suzuki, “The effect of target speech
distance on reaction time under multi-talker environment,” IEICE Tech. Rep., vol. 118, pp. 83–88,
Nov. 2018.
[6] S. Sakamoto, F. Monasterolo, C. D. Salvador, Z. Cui, and Y. Suzuki, “Effects of target speech distanceon auditory spatial attention in noisy environments,” in Proc. ICA 2019 and EAA Euroregio, (Aachen,
Germany), pp. 2177–2181, Sept. 2019.
[7] R. Duraiswami, D. N. Zotkin, and N. A. Gumerov, “Interpolation and range extrapolation of HRTFs,”
in Proc. IEEE ICASSP, vol. 4, pp. 45–48, May 2004.
[8] M. Pollow, K.-V. Nguyen, O. Warusfel, T. Carpentier, M. Müller-Trapet, M. Vorländer, and M. Noisternig, “Calculation of head-related transfer functions for arbitrary field points using spherical harmonics,” Acta Acust. United Ac., vol. 98, pp. 72–82, Jan. 2012.

[9] C. D. Salvador, S. Sakamoto, J. Treviño, and Y. Suzuki, “Distance-varying filters to synthesize
head-related transfer functions in the horizontal plane from circular boundary values,” Acoust. Sci.
Technol., vol. 38, pp. 1–13, Jan. 2017.
[10] C. Pörschmann, J. M. Arend, and F. Brinkmann, “Directional Equalization of Sparse Head-Related
Transfer Function Sets for Spatial Upsampling,” IEEE/ACM Trans. Audio, Speech, Language Process.,
vol. 27, no. 6, pp. 1060–1071, 2019.
[11] C. Pörschmann, J. M. Arend, and F. Brinkmann, “Correction to “Directional Equalization of Sparse
Head-Related Transfer Function Sets for Spatial Upsampling” [Jun 19 1060-1071],” IEEE/ACM Trans.
Audio, Speech, Language Process., vol. 28, pp. 2194–2194, 2020.
[12] J. M. Arend, F. Brinkmann, and C. Pörschmann, “Assessing spherical harmonics interpolation oftime-aligned head-related transfer functions,” J. Audio Eng. Soc., vol. 69, pp. 104–117, Jan. 2021.
[13] M. Zaunschirm, C. Schörkhuber, and R. Höldrich, “Binaural rendering of ambisonic signals by headrelated impulse response time alignment and a diffuseness constraint,” J. Acoust. Soc. Am., vol. 143,
no. 6, pp. 3616–3627, 2018.
[14] Z. Ben-Hur, D. L. Alon, R. Mehra, and B. Rafaely, “Efficient representation and sparse sampling ofhead-related transfer functions using phase-correction based on ear alignment,” IEEE Trans. Audio,
Speech, Language Process., pp. 2249–2262, 2019.
[15] A. Urviola, S. Sakamoto, and C. D. Salvador, “Ear centering for near-distance head-related transfer
functions,” in Proc. Int. Conf. Immersive and 3D Audio (I3DA): from Architecture to Automotive,(Bologna, Italy), IEEE, Sept. 2021.
[16] A. Urviola, S. Sakamoto, and C. D. Salvador, “Ear centering for accurate synthesis of near-fieldhead-related transfer functions,” Appl. Sci., vol. 12, no. 16, 2022.
[17] J.-G. Richter, M. Pollow, F. Wefers, and J. Fels, “Spherical harmonics based HRTF datasets: Implementation and evaluation for real-time auralization,” Acta Acust. United Ac., vol. 100, pp. 667–675,
July 2014.
[18] J. Blauert, Spatial hearing: The psychophysics of human sound localization. Cambridge, MA, USA;
London, England.: MIT Press, revised ed., 1997.
[19] C. D. Salvador, S. Sakamoto, J. Treviño, and Y. Suzuki, “Boundary matching filters for sphericalmicrophone and loudspeaker arrays,” IEEE/ACM Trans. Audio, Speech, Language Process., vol. 26,
pp. 461–474, Mar. 2018.
[20] Y. Rui, G. Yu, B. Xie, and Y. Liu, “Calculation of individualized near-field head-related transfer
function database using boundary element method,” in Proc. 134th Convention Audio Eng. Soc., May
2013.
[21] C. D. Salvador, S. Sakamoto, J. Treviño, and Y. Suzuki, “Dataset of near-distance head-relatedtransfer functions calculated using the boundary element method,” in Proc. Audio Eng. Soc. Int.
Conf. Spatial Reproduction —Aesthetics and Science—, (Tokyo, Japan), Audio Engineering Society,
Aug. 2018.
[22] V. R. Algazi, R. O. Duda, R. Duraiswami, N. A. Gumerov, and Z. Tang, “Approximating the headrelated transfer function using simple geometric models of the head and torso,” J. Acoust. Soc. Am.,
vol. 112, no. 5, pp. 2053–2064, 2002.
[23] E. Rasumow, M. Blau, M. Hansen, S. van de Par, S. Doclo, V. Mellert, and D. Püschel, “Smoothing
individual head-related transfer functions in the frequency and spatial domains,” J. Acoust. Soc. Am.,
vol. 135, no. 4, pp. 2012–2025, 2014.

ABS-0709

Individual assessments of realism in auralization and visualizations of urban
environments
Josep LLORCA-BOFí, Jonas HECK, Christian DREIER, Michael VORLÄNDER
Institute for Hearing Technology and Acoustics, RWTH Aachen University, 52074 Aachen, Germany
Correspondence: josep.llorca@akustik.rwth-aachen.de

ABSTRACT
Virtual reality offers attractive tools to assess the urban design and its consequences for the quality of life incities, especially in complex acoustic scenarios. However, the evaluation with regard to the level of realism ofsuch scenarios still is a challenge. In this study, individuals are assessing with their own words the realism ofsuch renderings in an audio-visual case study, the IHTApark, geolocated in Aachen, Germany. The multimodal
nature of this kind of study allows for different levels of detail in both acoustic and visual cues, ranging from
non-plausible reproductions to renderings closer to realism. In particular, the auralizations are computed using
geometrical acoustic approaches, the visualizations are evaluated using physically-based visual renderings. The
study aims to provide a personal-weighted vocabulary collection to define what is a realistic representation ofsuch scenarios.

Keywords: Auralization, Visualization, Audiovisual simulation, Urban Soundscapes, Virtual Acoustics

1

INTRODUCTION

The topic of realism in auralization has been present since the technology appeared [1], and has developed
towards other terms, such as plausibility, accuracy or credibility [2], and more recently towards ecologicalvalidity [3], [4], when it comes to capturing and reproducing the acoustic environment as closely as possible
to how it appears in the real context. All those terms are dealing with a common problem which might beresumed in the following question: "how close to reality are our environment representations?".
In order to assess the realism of audio-visual representations, we assume that the most salient features of anurban space need to be -at least- recognised, or even better, maintained. However, since the verbal expressions
highly differ between experts and non-experts, we intend to include qualitative descriptions of the environment
to later extract common perceptual patterns.
In this study we report what research results have been to this point. In particular, that soundscape evaluations of the same urban space have been rated differently under different psychoacoustic conditions [5]. Thisrequires a need for more reproducible approaches when studying the soundscapes, such as state-of-the-art simulations. They allow the parametric modification of the physically-based simulations. Additionally, the synchronized audio-visual renderings are used and evaluated, compared to the real experience of the same environment.
In order to extract the perceptual categories which can be mixing both acoustic and visual impressions, weuse a descriptive evaluation method called Individual Vocabulary Profiling, which reveals perceptual dimensions
from collections of personal attributes [6].

2

METHODS

2.1 Individual Vocabulary Profiling
The Individual Vocabulary Profiling (IVP) method consists of a descriptive analysis test, in which the assessors
perform individual word elicitation by listening and viewing to the stimuli. While experiencing the reproduced
environments, the assessors provide words to describe how they perceive the space quality and the differencesbetween them. Assessors can provide 8 to 12 words as they feel necessary to describe their perception. The
assumptions behind this approach considers that assessors will fundamentally perceive the stimuli in a similarmanner, but may use different adjectives to express their perception. The test consisted of two phases:• A. Elicitation of attributes. 10 assessors are asked to provide attributes which describe the qualities ofstimulus and the differences among them. A total of 4 to 8 attributes are required to complete thetask,including audio and visual qualities. The assessors are asked to provide for each attribute not only
one adjective, but also pairs of extremes which can be used in scale ratings later on. The list of adjectives with their extremes are discussed personally with the testers, in order to avoid inaccuracies andredundancies. The outcome of this phase is an individual collection of attributes.
• B. Ratings of scenes. The same assessors are asked to rate the stimuli with the collection of attributes they
personally developed in Phase A. The outcome of this phase is a personal-weighted attribute collection.
In order to extract words and ratings describing realism, the IVP paradigm is adapted to perform phase A(elicitation of attributes) in the real scenario (IHTA park) with a minimum of 4 words for the acoustic and thevisual attributes. Later, the participants provide other 4 words inside the laboratory comparing the three different
simulations of the IHTA park among them and with the real environment. In this way, each participant provides
a total of 8 auditory- and 8 visually-related attributes.
2.2 Simulated scenario
Figure 1. Visualization model (bottom left), auralization model (bottom right) and aerial photograph of the
IHTApark (up). In the auralization model, the sound sources, their trajectories and the receiver positions are
depicted. In the visualization model, the highest level of detail is shown as an example. The models are freely
accessible in [7].

The so-called benchmark IHTApark site is used as case study here. This environment has been previously used
as case study by the authors [8], [9], figure 1. The venue is located west of the German city of Aachen, closeby to the Institute for Hearing Technology and Acoustics, at RWTH Aachen University. The study site spans

2

an area of about 9840 m and is enclosed by the roads Kopernikusstraße to the north-west, the road SeffenterWeg to the north/east, and Mies-van-der-Rohe Straße to the south-east. The southwest border adjoins a footpath,and thallotment gards Familien-Dauergärten "Eschenallee" e.V. Estimated roughly from looking at the map, thesurface sealed degree in the area amounts to 45 per cent. The area surrounding the study site is characterised bymore open building structures and more unsealed and green spaces as opposed to the area within the city ring,which is much more densely built and sealed. In figure 1, the models used for visualization and auralization
are depicted.
For the auralization of the scenario, a mixed approach combining recorded and simulated sounds was used.
This approach is especially useful to increase the level of realism with background ambient sound when reproducing urban spaces and when studying the perception of complex scenes, as urban environments are. This is
one of the key outcomes of ISO 12913 on soundscapes, where context plays a crucial role for the representation of scenes for audio-visual soundscape studies. The question remains that sound sources are expected tocontribute a plausible acoustic environment in a specific context. In order to include recorded ambient soundswith virtual vehicle synthesized sound sources in the urban context, we follow the calibration method proposed
by Heck et. al. [10]. The following acoustic scenario is created:• Background sound recorded in the place in a quiet situation, discarding vehicle, machinery and people
sounds. The recording was done using a Sennheiser Ambeo first order ambisonics microphone.
• Two synthesized car sounds passing the road of Seffenter Weg. The auralization is generated with geometrical acosutics between a dynamic source position and a static receiver position, as indicated in 1. The
propagation paths have been calculated using the geometrical model shown in the same figure, following
the model by Dreier and Vorländer [11] and the vehicle pass-by noise synthesis by Dreier et al. [12].
• One synthesized aircraft sounds passing over the area, as shown in 1. The geometrical acoustics auralization is rendered following the model by Schäfer and Vorländer [13].
• A pre-rendered sequence can be found in the following link: https://youtu.be/rudxfV94UwA?t=15
Regarding the visual simulation, three different urban representations have been produced, modifying thevisual level of detail in each of them, while one single acoustic simulation is used. The visual levels of detailfollow the framework described by the authors [8] and the variables included in this study are described inTable 1 and a screenshot of each of the scenario can be seen in in Figure 2.
Table 1. Visual simulation levels of detail

Geometry

Materials

Effects

ViewerVariables

High

Med

Low

Near façade objects

<5cm

>10cm

>20cm

Far façade objects

<10cm

>2m

>2m

Grass

♦

♦

Diffusion maps♦♦

Roughness maps♦♦

Flying pollen♦♦

Wind on foliage♦

Walkable area

25m

2

25m

♦

2

2

25m

2.3 Acoustic and visual reproduction
The scenario was reproduced visually using the VR glasses HTC Vive, Unreal Engine 4.27, in an area allowing
a walkable space of 5x5 m. The acoustic reproduction was done via headphones Sennheiser HD650. Headphone compensation was included using the Heaphone Transfer Function measured on the Kemar test fixture.
The reproduction was calibrated by adjusting the levels and A-weighted levels between the reproduction andthe recorded levels in the real park. The levels and A-weighted levels were calculated for the backgroundsound (48dB and 40dBA, respectively) and for one pass-by vehicle combined with background sound (56dB
and 49dBA), as well as in the reproduction system.
Figure 2. The three levels of detail in comparison: high, medium and low (up-down respectively). Note thedifferences in the windows of the left building, the buildings in the background, and the grass, in accordance
to Table 1.

2.4 Attributes clustering
The attributes are later clustered according to similar individual profiles. A hierarchical clustering is performed
to recursively cluster two items at a time. The Euclidean distances are used to measure the distance between
the attribute ratings for each urban representation. Since three simulations and one real environment werestudied, four different distance matrices were generated accordingly. Taking the mean over all four distancematrices results in a final distance matrix. This matrix contains how close is the mean of attribute X (overall 4 environments) from attribute Y (over all 4 environments). The means are then clustered under Euclidean

distances. Finally, the number of clusters is chosen as soon as each cluster has at least 2 attributes.

3

RESULTSA total of 10 participants rated the three simulated and the real environments, giving 8-12 attributes each. This
resulted on a total of 162 attributes. 8 clusters can be clearly distinguished when analyzing all the data. This
information can be seen in figures 3 and 4.
Figure 3. All ratings and all clusters with all attributes.

In figure 3, the ratings are given for the evaluation of the real environment, as well as for the three levelof details: High, Medium and Low, being 0 the lowest rating given for each attribute, and 100 the maximum.
The attributes terminating with "- V" correspond to the visual modality, whereas the ones terminating with "A" correspond to the acoustic. Figure 4 depicts each of the clusters separately, with one representative word
for each, extracted from the cluster itself. One representative word for visual and one for acoustics has been
extracted for each cluster, which appears at the top of each cluster.

Figure 4. Clusters, ratings and representative attributes

4

DISCUSSION

After the exploratory inspection of the results, eight main dimensions can be distinguished describing the levelof realism for both the acoustic and the visual modalities. These eight dimensions can be divided into threegroups: properties of the simulations which have been maintained in respect to the real site (cluster 2 and 4);
decreased (6, 5 and 1); and increased (cluster 4, 7 and 3).
In terms of acoustics, the most critical properties to achieve a level of realism are the realism of the traffic
noise and the ambient loudness and mismatch (clusters 1 and 7). This points out to the importance of calibrated
data according to a real situation. Of major importance to be improved are the ambient sound localization (cluster 5), which might be achieved using higher order ambisonic recordings. However, the sound pleasantness andnaturalness, as well as ambient loudness are achieved with a decrease mean of 10 per cent (cluster 2). This
indicates that the three simulated levels (high, medium and low) are equally rated for the attributes pleasantness, naturalness and loudness. Finally, the most unrealistic evaluation change between the real and the threesimulations is recorded in the ambient localization (cluster 5), but with a tendency to be diminished when thevisual simulation is less accurate. This indicates that people might concentrate less on the acoustic part whenthe visual level of detail is high.

5

CONCLUSIONSIn this study, three levels of detail of an urban environment have been simulated visually and acousticallyto be compared among them and with the real environment. While maintaining the acoustic simulation thesame for all three visualizations, the differences among the representations have been captured using individual
vocabulary profiling. The results show that the most critical properties of the real scenario to be respresented
as realistic in the simulations where the following for the acoustic modality: realism of vehicle simulations,which highly depends on the source synthesizer, and vehicle sound loudness, which slightly increases when
visual level of detail decreases. This suggests that special attention needs to be paid into the visual and theacoustic simulations when dealing with vehicle sound sources. On the same time, a good sound naturalnessand coherence is achieved using the proposed recorded ambient sounds in the simulations, which indicates thathigh level realism supports them. The study also suggests that more robust and reliable methods for capturing
complex urban scenarios [14] are needed to achieve realistic auralization and visualizations.

ACKNOWLEDGEMENTS
This research was funded by the Federal Ministry of Education and Research (BMBF), the Federal Ministry for
Economic Affairs and Climate Action (BMWi) and the Ministry of Culture and Science of the German State ofNorth Rhine-Westphalia (MKW) under the Excellence Strategy of the Federal Government and the Länder.

REFERENCES[1] Kleiner M, Dalenbäck B, Svensson P.Auralization - An Overview. J. Audio Eng. Soc. 1993.
[2] Vorländer M. Auralization. Fundamentals of acoustics, modelling, simulation, algorithms and acoustic virtual
reality. Springer, Berlin. 2020.
[3] Guastavino C, Katz B, Polack J, Levitin D, Dubois D. Ecological validity of soundscape reproduction. Acta
Acustica. 2004, 50(2): doi:https://doi.org/10.3389/fbuil.2021.687237[4] Tarlao C, Steele D, Guastavino C. Assessing the ecological valitdy of soundsacpe reproduction in different
laboratory settings. Plos one. 2022: doi:https://doi.org/10.1371/journal.pone.0270401[5] Aletta F, Asdrubali F, De Balsiis M, Evangelisti L, Guattari C. Influence of soundsacpe in the experienceof an urban area: a case study in Rome. In: 13th international conference on pplied Human Factors and
Ergonomics. New York. 2022: doi:http://dx.doi.org/10.54941/ahfe1002347[6] Lokki T, Pätynen J, Kuusinen A, Vertanen H, Tervo S. Concert hall acoustics assessment with individuallyelicited attributes. J. Acoustic. Soc. Am. 2011; 130: 835. doi:https://doi.org/10.1121/1.3607422[7] Llorca-Bofí J, Vorländer M. IHTApark. Multi-detailed 3D architectural framework for sound perception research in virtual reality. [Dataset]. Zenodo. 2021. doi:https://doi.org/10.5281/zenodo.4629759[8] Llorca-Bofí J, Vorländer M. Multi-detailed 3D architectural framework for sound perception research invirtual reality. Front. Built Environ. 2021. doi:https://doi.org/10.3389/fbuil.2021.687237[9] Llorca-Bofí J, Dreier C, Heck J, Vorländer M. Urban sound auralization and visualization framework - casestudy at IHTApark. Sustainability. 2022; 14(4): 2026. doi:https://doi.org/10.3390/su14042026[10] Heck J, Dreier C. Vorländer M. Ambient sounds in auralization - calibration method for spherical microphones. In: DAGA 2022 Stuttgart. 2022. Url:https://www.researchgate.net/publication/359939547_
Ambient_Sounds_in_Auralization_-_Calibration_Method_for_Spherical_Microphones[11] Dreier C, Vorländer M. Aircraft noise - auralization-based assessment of weather-dependent effects on
loudness and sharpness. J. Acoustic. Soc. Am. 2021; 146: 3565-3575. doi: https://doi.org/10.1121/10.
0005040[12] Dreier C, Hahn J, Heck J, Llorca-Bofí J, Vorländer M. Real-time vehicle pass-by noise synthesis for urban
sound auralization. In: Proc. Int.Cong. Ac., Gyeongju, South Korea. 2022.
[13] Schäfer P, Vorländer M. Atmospheric ray tracing: an efficient, open-source framework for finding eigenrays
in a stratified, moving medium. Acta Acust. 2021; 5, 26. doi: https://doi.org/10.1051/aacus/2021018[14] Heck J, Dreier C, Llorca-Bofí J, Vorländer M. Copacabana revisited: Photogrammetry-based model for
urban sound auralization. In: Proc. Int.Cong. Ac., Gyeongju, South Korea. 2022.

ABS-0711

IHTApark - A lab for auralization and visualization
Josep LLORCA-BOFí, Jonas HECK, Christian DREIER, Michael VORLÄNDER
Institute for Hearing Technology and Acoustics, RWTH Aachen University, 52074 Aachen, Germany
Correspondence: josep.llorca@akustik.rwth-aachen.de

ABSTRACTThis poster presents the IHTApark lab, a real environment monitorized with acoustic and visual recordings, aswell as with weather indicators, together with a virtual reproduction of the same environment for real-time,realistic and physically-based renderings of such scenarios. The description in detail of how the virtual model
has been designed and simulated regarding the acoustic and the visual cues are included. Additionally, the poster
presents an online test paradigm for the individual assessment of urban sound in audio-visual environments, andthe possibility for the reader of the poster to directly participate in the test, with the aim of approach a diverse
group of tested people in such urban assessments. The results of the test will be compared with the same test
performed in laboratory conditions.

Keywords: Auralization, Visualization, Audiovisual simulation, Urban Soundscapes, Virtual Acoustics

ACKNOWLEDGEMENTS
This research was funded by the Federal Ministry of Education and Research (BMBF), the Federal Ministry for
Economic Affairs and Climate Action (BMWi) and the Ministry of Culture and Science of the German State ofNorth Rhine-Westphalia (MKW) under the Excellence Strategy of the Federal Government and the Länder.

REFERENCES[1] Vorländer M. Auralization. Fundamentals of acoustics, modelling, simulation, algorithms and acoustic virtual
reality. Springer, Berlin. 2020.
[2] Llorca-Bofí J, Vorländer M. Multi-detailed 3D architectural frework for sound perception research in virtual
reality. Front. Buitl Environ. 2021. doi:https://doi.org/10.3389/fbuil.2021.687237[3] Llorca-Bofí J, Dreier C, Heck J, Vorländer M. Urban sound auralization and visualization framework - casestudy at IHTApark. Sustainability. 2022; 14(4): 2026. doi: https://doi.org/10.3390/su14042026[4] Llorca-Bofí J, Vorländer M. IHTApark. Multi-detailed 3D architectural frework for sound perception researchin virtual reality. [Dataset]. Zenodo. 2021. doi: https://doi.org/10.5281/zenodo.4629759[5] Heck J, Dreier C. Vorländer M. Ambient sounds in auralization - calibration method for spherical microphones. DAGA 2022 Stuttgart.2022. Url: https://www.researchgate.net/publication/359939547_
Ambient_Sounds_in_Auralization_-_Calibration_Method_for_Spherical_Microphones[6] Dreier C, Vorländer M. Aircraft noise - auralization-based assessment of weather-dependent effects on loudness and sharpness. J. Acoustic. Soc. Am. 2021; 146: 3565-3575. doi: https://doi.org/10.1121/10.
0005040[7] Schäfer P, Vorländer M. Atmospheric ray tracing: an efficient, open-source framework for finding eigenrays
in a stratified, moving medium. Acta Acust.2021; 5, 26. doi: https://doi.org/10.1051/aacus/2021018

ABS-0786

Perceptual comparison between the real and the auralized room when being
presented with congruent visual stimuli via a head-mounted display
Felix STÄRZ(1) , Leon O.H. KROCZEK(2) , Sarah ROSSKOPF(2) , Andreas MÜHLBERGER(2) , Steven VAN DE PAR(3,4) ,Matthias BLAU(1,4)(1) Institut für Hörtechnik und Audiologie, Jade Hochschule Oldenburg, Germany, felix.staerz@jade-hs.de(2) Department of Clinical Psychology and Psychotherapy, Regensburg University, Germany,

Andreas.Muehlberger@psychologie.uni-regensburg.de(3) Acoustics Group, Carl von Ossietzky University Oldenburg, Germany, steven.van.de.par@uni-oldenburg.de(4) Cluster of Excellence "Hearing4All"

ABSTRACTInteractive virtual environments (iVEs) with binaural auralization over headphones and visualization via headmounted display (HMD) are a promising research tool to transfer real room experiments into the lab. For thisreason, there is a high demand on the visual model and the associated auralization of the represented room.
It has been shown that, using speech stimuli with head-tracked binaural auralizations based on measured andsimulated binaural room impulse responses (BRIRs), a convincing agreement between the real and the simulatedroom is possible, even when some approximations are made for the auralization (Blau et al. 2021). However,auralizations might suffer when presented in a reproduction room that deviates from the real room. This is known
as the “room divergence effect” (Brandenburg et al., 2016). It is expected that congruent visual cues presented
via HMD reduce this effect. In this study, a virtual model of the auralized room was built and integrated intoa gaming engine. Subjects sitting in the real room wearing a HMD and headphones had to compare different
auralizations based on measured and simulated BRIRs to a loudspeaker representation in the real room. Results
show that a convincing auralization is possible even if both auditory and visual representations are based on
simulations.

Keywords: virtual acoustics, head-mounted display, auralization

1

INTRODUCTION

Moving the real world into the lab is a trade-off between realism and control. Interactive virtual environments(iVEs) are a promising tool to combine ecologically valid environmental conditions with high control over theexperiment. The combination of controllable visual and auditory stimuli opens up new possibilities to explore
human perception. As a pendant to visualization, auralizations can be used to make sound fields audible byrendering them over headphones or loudspeakers [1, 2], resulting in the possibility to integrate advanced audio
in iVEs. Auralizations have the potential to replace experiments in real rooms [3]. The requirement is that they
be as close as possible to reality.
A common way to test auralizations is to compare simulated with measured sound fields, assuming thatmeasurements represent the reality accurately enough [4, 5, 6]. Alternatively, auralizations over headphones
can be compared to loudspeaker representations in the real room. A highly plausible auralization was foundby [7], comparing static binaural auralizations with a loudspeaker in an anechoic environment. Nevertheless,direct comparisons led to differences between real loudspeaker presentation and headphones, especially for noise
pulses. Head-tracked binaural auralizations compared to loudspeakers, while being present in the real room,
showed that subjects were able to find differences for noise but not for speech stimuli [8]. These findings wereconfirmed by [9] where head-tracked binaural auralizations over headphones were compared to loudspeakers ina typical classroom scenario. In addition to auralizations based on measured BRIRs, BRIRs simulated with theroom acoustical simulator (RAZR, [10]) were added to the comparison. Results showed that simulated BRIRs
were rated lower then measured BRIRs. In a follow up study the lower ratings were attributed to the use ofomnidirectional sources in RAZR. An extended RAZR version with the possibility to add speaker directivities
was built [11]. By using the directivity of the loudspeakers the ratings for simulated BRIRs improved. Results
showed that a convincing agreement between the real and the simulated room is possible when using speechstimuli with head-tracked binaural auralizations based both on measured and simulated BRIRs [12]. Subjects
compared different auralizations to a loudspeaker representation while sitting in the real room. In conclusion,
all but the auditory cues were perfectly preserved.
It is known that auralizations might suffer from a mismatch between the real and the auralized room due tothe so called "room divergence effect" [13]. As with auralization, visualization must be as close as possible toreality. By using visualizations via HMD, a mismatch between the real and simulated room might occur. Thus,
it remains unclear whether a convincing auralization is possible when using simulations for both visualization
and auralization.
In this study, plausibility tests were performed while subjects were sitting in the real room, comparing loudspeaker reproductions with measured or simulated BRIRs while wearing an HMD. While visual cues and theplayback over headphones were based on simulations, all except visual cues for the loudspeaker presentations
are perfectly preserved.

2

METHODS

2.1 Room setup
The underlying room acoustic is a small lecture room at Jade Hochschule in Oldenburg, Germany (7.12 x 11.94
x 2.98 m3 ). The room consists of three concrete walls and a window front with opened curtains. The floor iscovered with linoleum and the ceiling is fully covered with broad-band absorbing plaster board including 5 cmof mineral wool and approximate 30 cm air space.
Subjects were seated slightly off axis in the center of the room with an ear height of 1.3 m. Three source
positions were used for the audio playback. The first loudspeaker was 4.32 m straight in front of the subject
with a height of 1.6 m. The second loudspeaker was placed 90◦ on the left hand side of the subject with adistance of 3.6 m. A third loudspeaker was placed slightly behind the subject on the right hand side. For thelast two speakers the acoustic center was placed at 1.3 m, see Figure 1.
2.2 Visual room model
During the whole experiment, subjects saw a visual 3D model of the room under investigation, presented overa HMD. This 3D model was created by architecture students at Jade Hochschule Oldenburg using 3Ds Max(Autodesk, Inc., San Rafael, CA, USA). Subsequently, the model was imported to Unreal Engine 4.26 (Digital Extremes, London (Ontario), Canada and Epic Games, Inc., Raleigh, NC, USA) via the datasmith plugin.
Textures and lighting were fitted within Unreal to get close to the visual impression in the real room. The
Graphical User Interface (GUI) that was used to interact with the subject was presented on a blackboard in theroom, showing the possibilities to answer the question whether the current playback was "A: real" or "B: simulated". Answers were given with a Valve Index Controller (Valve Corporation, Bellevue, WA, USA).
2.3 Listening setup
To compare the real playback in the room to auralizations based on measured and simulated BRIRs, stimuli
were presented via loudspeakers or headphones. For the presentation in the real room three two-way active
loudspeakers (Genelec 8030b, Genelec Oy, Iisalmi, Finland) were used representing three different source positions. Auralizations were played back via headphones (AKG K1000, AKG Acoustics GmbH, Vienna, Austria)"hovering" over the ear. This was achieved by mounting them to the HMD (HTC model VIVE Pro eye, HTC
Corporation, Xindian, New Taipei, Taiwan) using a specifically designed 3D-printed support, see Figure 2. As
a result, there was no haptic cue indicating the presence of headphones.

Figure 1. Left: A commercial head-and-torso-simulator KEMAR sitting in the real room wearing a HMD
including headphones. Right: The visual model used for visualizations via HMD is shown.

Figure 2. A commercial head-and-torso-simulator KEMAR wearing the HMD with AKG K1000 headphones.
Those were mounted with a custom made 3D-printed support.
2.4 BRIR Sets
The auralization was based on five different BRIR sets, of which two were measured and three were simulated.
The first BRIR set, further called "meas:HATS", was measured in the real room using a commercial head-andtorso-simulator (HATS) (KEMAR type 45BB,GRAS Sound and Vibration A/S, Holte, Denmark). The headabove-torso orientations of the HATS was varied between -90 to 90◦ in 5◦ steps, resulting in 37 azimuthal
orientations. The elevation angle was fixed at 0◦ , the ear height was set to 1.30 m. An anthroprometric ear
plug called PIRATE [14], equipped with MEMS microphones (TDK type ICS-40619, TDK InvenSense, San
Jose, CA, USA) was inserted into the ear canal creating a blocked ear condition comparable to [15]. The
measurement was based on the multiple exponential sweep method [16], including the modification presented in[17]. The exponential sweep stimulus ranged from 20 Hz to the Nyquist frequency, shifted by 4 s for the usageof three consecutive loudspeakers. The sampling rate was set to 44.1 kHz. For the measurement procedure an
external audio interface (RME Fireface UC, Audio AG Haimhausen,Germany) and a laptop running Matlab (TheMathWorks, Inc., Natick, MA, USA) and Pure Data scripts [18] were used. As a post processing step, impulse
responses were cut to approximately 410 ms and windowed with a 50 samples half Hann window. The wholeprocedure is comparable to [12].
The second BRIR Set, in the following described as "meas:HATSHMD", is comparable to the previously
described meas:HATS. The difference is that the commercial HATS KEMAR wore a head mounted display
including the headphones described in subsection 2.3. This setup is comparable to a situation were the subject
is sitting in the room wearing the HMD and headphones. The BRIR measurement setup was as described above.
The remaining BRIR sets were based on the combination of measured head related impulse responses (HRIRs)
and a simulated room that used the measured HRIRs to create BRIRs. To simulate BRIRs, an improved RAZR
version (0.962b) was used [10, 19]. This version allows to use the directivity of a loudspeaker. Please refer to[12] for more details with respect to the directivity measurements. RAZR was used with a third order image
source model (ISM) combined with a feedback delay network (FDN) for late reverberation. The loudspeakerdirectivity was taken into account for both, ISM and FDN.
RAZR is based on a simplified shoe box shaped room model. Absorption coefficients are uniformly distributed and can be chosen in octave bands per room wall. For a realistic simulation, absorption coefficientswere manually fitted in a way that the simulated reverberation time T20 was within 5 % of a previously measured reverberation time of the real room while preserving the characteristic absorption coefficients of each wall.
It should be noted that the absorption adjustment is a global modification affecting all BRIRs.
HRIRs were measured with a custom-made rig, see [9]. This rig is based on an semicircular aluminum arc
with a radius of 1.25 m, equipped with 24 small active loudspeakers (Speedlink type SL-8902-GY Xilu, Jöllenbeck GmbH, Weertzen, Germany), twelve at each side. The arc was rotated in 5◦ steps between −90◦ to 90◦
around the subject resulting in 37 azimuthal orientations. To represent twelve elevation angles, loudspeakers
were placed ranging between −30◦ to 30◦ in 7.5◦ steps and additional at 45◦ , 60◦ and 75◦ . As a result, HRIR
sets were measured at 864 directions of incidence. Due to the speaker size, measurements were only valid to a
low frequency limit of approximately 350 Hz. In a post processing step the limitation was overcome by fitting
the magnitude of a spherical head model [20] with a spline interpolation to the magnitude of the measured headrelated transfer function (HRTF). To ensure the interaural time differences (ITD) of the subject, the sphericalhead model, computed with the open software toolbox AKTools [21], was calculated with respect to the head
diameter.
To calculate BRIRs, a subset of the HRIR set was used. By using the nearest neighbor method, directional
impulse responses computed by the room simulation were convolved with a matching HRIR based on the direction of incidence. BRIRs were obtained for 333 head orientations, nine elevation (−30◦ to 30◦ in 7.5◦ )and 37 azimuth angles (−90◦ to 90◦ in 5◦ steps). HRIRs were measured for a HATS, resulting in a BRIR
set labeled as "sim:HATS", and for individual subjects, further described as "sim:indivHRIRs". In addition, ananchor BRIR set called "sim:HATSomni" was used where a room simulation with an omnidirectional speakerdirectivity was used.
Individually measured headphone transfer functions (HPTFs) were used to equalize the influence of headphones and pinna in amplitude and phase. Measurements were performed right after the HRIR measurements
without removing the microphones. For using individual HPTFs a regularized inversion was done comparedto [22]. Since the headphones are permanently installed on the HMD, the test subjects were shown a visualpresentation of the room. The HMD was individually adjusted to ensure the best possible visual presentation.
By marking the lateral fastening as well as the headband, the glasses could be placed in a reproducible manner
before the listening tests.

3

LISTENING TEST - PLAUSIBILITY

To asses the plausibility of the used interactive virtual environment, the following listening test was designed
using the method proposed by [23]. This approach describes a criterion-free method to estimate plausibility byusing a Yes/No task. The analysis of resulting data was based on the signal detection theory (SDT). During thelistening test, subjects had to rate whether the stimulus heard was real or simulated. In the above mentioned
study, d ′ values were related to a two alternative forced choice (2AFC) detection rate Pc . The pure guessing
rate (Pc = 0.50) is related to d ′ = 0, a deviation of less than 5 % (Pc = 0.55) with d ′ = 0.1777. The detection
threshold i.e. the inflection point of the psychometric function is at Pc = 0.75 comparable with a d ′ = 0.9539.
Note that a d ′ value below indicates that subjects could not reliably distinguish between the real loudspeaker
and the measured or simulated stimulus. Thus, a d ′ i value below this value indicates a plausible rendering
method. The d ′ value was computed as an individual d ′ i as well as an averaged d ′ avg . For the estimationof stable individual SDT parameters at least 100 decisions are necessary [24]. To overcome perfect hit rates,
hit and false alarm rates were corrected by the so called 1/2N rule [25]. As an additional STD parameter
the response bias c was calculated. A neutral bias is given when c = 0 i.e. no response is favored. Positive
values of c can be interpreted as a tendency to answer with "real", negative values indicating a bias towards"simulated".

3.1 Stimuli
For each BRIR set 100 different guitar as well as male and female speech stimuli taken from the DEGA datasetsfor virtual acoustics were used, see [26]. Stimuli were cut into smaller samples (between 3.4 s and 8.1 s longwith a mean of 5.8 s). Cutted stimuli were Hann windowed to overcome cutting artifacts. Guitar samples varied
between slow, melancholic to fast staccato played excerpts [27, 28]. Speech stimuli representing parts of "TheFrog Prince" prerecorded by female and male, professional and layman speakers in German language [29]. To
get a balanced stimulus set, loudness was normalized in accordance with EBU R 128 by the integrated loudness
function taken from the Matlab audio toolbox [30]. Audio playback over headphones was done with a custom
software called VARy which is based on the time variant partitioned convolution algorithm [31]. Binaural
renderings were computed in real-time based on the head tracker data of the HMD. The A-weighted soundpressure level at the listener position was on average 56 dB when the closest speaker was used as source. The
A-weighted background noise level in the room, depending on outdoor conditions ranged between 20 and 25 dB.
Seven subjects (all male, average age 27.7 years) participated in the study. All of them had experiencein listening tests but were not explicitly tested as expert listeners. The actual playback over loudspeakers or
headphones (real vs. simulated) was drawn randomly for each individual subject.
3.2 Results
For each subject d ′ i and ci were calculated, as shown in Figure 3. The majority of values are below d ′ = 0.9539
except for the BRIR set "sim:HATSomni". This can also be seen in Table 1. The average d ′ avg value of"sim:HATSomni" is slightly above a rating of one. For auralizations based on "sim:indivHRIRs" and "sim:HATS"
negative d ′ i values occur. The average response bias cavg has a slight positive tendency for each BRIR set. Listeners tend to rate stimuli as "real". For auralizations based on "sim:HATSomni" cavg is close to zero i.e. nobias was present.
Because normality, determined by Kolmogorov-Smirnov test, is not given for every BRIR set, the Friedman test was conducted (p = 0.0018). As a post hoc test the non-parametric Wilcoxon signed rank test wasused, testing for statistical significance (p < 0.05). For auralizations based on "sim:HATSomni", d ′ i values
were found to be significantly higher than auralizations based on "meas:HATS" (p = .01563) as well as for"meas:HATSHMD" (p = .04688), "sim:indivHRIRs" (p = .01563) and "sim:HATS" (p = .01563).
Table 1. Values of d ′ avg and cavg for each BRIR set
meas:HATS

meas:HATSHMD

sim:indivHRIRs

sim:HATS

sim:HATSomni

avg

0.4681

0.58380.19930.40401.1213cavg

0.40150.40940.55150.50600.0961d′

3.3 Discussion
The results of the listening test showed that a plausible auralization is possible even when the visualization,
presented via HMD, is based on a simulation. Values smaller than d ′ avg = 0.9539 were found for both measured as well as simulated BRIRs. This can be interpreted in a way that subjects were not able to distinguishbetween loudspeaker and headphone representation. Note that a d ′ avg = 0.9539 corresponds to a 2AFC detection
rate Pc = 0.75 known as the detection threshold. The plausibility of auralizations was found for measured aswell as for simulated BRIRs with the lowest value for sim:indivHRIRs. However no significant difference formeasured vs. simulated and generic vs. individual BRIRs were found.
The proposed value for d ′ avg = 0.1777 by [23] comparable to a pure guessing rate in a 2AFC procedure exceeded by less than 5 % (Pc = 0.55), was not reached by any auralization. In this context it should be noted thatin [23] visual cues were not simulated. In our experiments, some subjects reported slight mismatches between
heard and seen speaker positions. This indicates that subject may no longer have based their judgments on
their internal reference exclusively, which would be in contradiction to the usual definition of plausibility [32].

2sensitivity d'

1

0

-1

-2
meas:HATS

meas:HATSHMD sim:indivHRIRs

sim:HATS

sim:HATSomni

meas:HATS

meas:HATSHMD sim:indivHRIRs

sim:HATS

sim:HATSomni

1

0

-1

real | simulated

response bias c2-2

Figure 3. Box plots of sensitivity d ′ avg and response bias cavg for each BRIR set. Individual values d ′ i and ci
are shown as circles. Statistical significant differences p < 0.05 are indicated by horizontal lines in the bottom
part.
Hence, the question is raised how plausibility should be defined and measured in audio-visual iVEs where bothauditory and visual stimuli are simulated. More research is needed to answer the question.
Significant differences (p < 0.05) for d ′ avg values were found for the anchor "sim:HATSomni" compared to"meas:HATS", "meas:HATSHMD", "sim:HATS" and "sim:indivHRIRs". With an average value of d ′ avg = 1.12,
subjects were able to distinguish between loudspeaker and headphone presentation for this type of auralization.
It is important to mention that the median d ′ value is below 1 as well as four of seven d ′ i values. As aconsequence, a plausible auralization with an omnidirectional loudspeaker directivity is possible for some ofthe subjects. A plausible anchor results in a difficult listening experiment because every stimulus might soundplausible. This was also reported by subjects in a short interview after the listing test. Possibly it is better to
use a slightly worse anchor to make it a little easier and less frustrating.

4

CONCLUSION AND OUTLOOKThe present study shows that plausible, close-to-reality binaural auralizations for speech and guitar samples canbe achieved by a combination of a HMD-presented 3D visual simulation and a headphone-presented binaural
auditory simulation, both head-tracked. This is valid for measured BRIRs, as well as BRIRs simulated with
RAZR, for both individual and generic HRIRs. Further work to get an even closer match between loudspeakers
in the visual model and perceived loudspeaker positions in the real room is necessary. Also, existing methods
to define and measure plausibility in iVEs where only the auditory stimuli are simulated need to be adapted toiVEs where both auditory and visual stimuli are simulated.

ACKNOWLEDGEMENTS
This work was funded by Deutsche Forschungsgemeinschaft (DFG, German research Foundation) project ID
444832396 (SPP 2236). We also want to thank our subjects for their participation in the study.

REFERENCES
1.

Kleiner M, Dalenbäck BI, and Svensson P. Auralization-an overview. Journal of the Audio Engineering
Society. 1993; 41. Publisher: Audio Engineering Society:861–75

2.

Lokki T, Savioja L, Vaananen R, Huopaniemi J, and Takala T. Creating interactive virtual auditory environments. Computer Graphics and Applications, IEEE. 2002 Aug 1; 22:49–57. DOI: 10.1109/MCG.
2002.1016698

3.

Vorländer M. Auralization: Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic
Virtual Reality. Springer Science & Business Media, 2007 Oct 27

4.

Lokki T and Pulkki V. Evaluation of the Geometry-Based Parametric Auralization. 2002 Jun 1:367–76

5.

Rindel J and Christensen C. Room Acoustic Simulation and Auralization – How Close can we get to theReal Room? Keynote lecture. 2003 Apr 7

6.

Postma BNJ and Katz BFG. Perceptive and objective evaluation of calibrated room acoustic simulation
auralizations. The Journal of the Acoustical Society of America. 2016 Dec; 140. Publisher: AcousticalSociety of America:4326–37. DOI: 10.1121/1.4971422

7.

Oberem J, Masiero B, and Fels J. Experiments on authenticity and plausibility of binaural reproduction
via headphones employing different recording methods. Applied Acoustics. 2016 Dec; 114:71–8. DOI:
10.1016/j.apacoust.2016.07.009

8.

Brinkmann F, Lindau A, and Weinzierl S. On the authenticity of individual dynamic binaural synthesis.
The Journal of the Acoustical Society of America. 2017 Oct; 142. Publisher: Acoustical Society ofAmerica:1784–95. DOI: 10.1121/1.5005606

9.

Blau M, Budnik A, and Par S van de. Assessment of perceptual attributes of classroom acoustics: Realversus simulated room. Proceedings of the Institute of Acoustics. 2018; 40

10.

Wendt T, Van De Par S, and Ewert SD. A computationally-efficient and perceptually-plausible algorithm
for binaural room impulse response simulation. Journal of the Audio Engineering Society. 2014; 62.
Publisher: Audio Engineering Society:748–66

11.

Steffens H, Par S van de, and Ewert SD. The role of early and late reflections on perception of source
orientation. The Journal of the Acoustical Society of America. 2021 Apr; 149. Publisher: AcousticalSociety of America:2255–69. DOI: 10.1121/10.0003823

12.

Blau M, Budnik A, Fallahi M, Steffens H, Ewert SD, and Par Svd. Toward realistic binaural auralizations– perceptual comparison between measurement and simulation-based auralizations and the real room for aclassroom scenario. Acta Acustica. 2021; 5. Publisher: EDP Sciences. DOI: 10.1051/aacus/2020034

13.

Brandenburg K, Werner S, Klein F, and Sladeczek C. Auditory illusion through headphones: History, challenges and new solutions. Proceedings of Meetings on Acoustics. 2016 Sep; 28. Publisher: AcousticalSociety of America. DOI: 10.1121/2.0000897

14.

Denk F, Brinkmann F, Stirnemann A, and Kollmeier B. The PIRATE: an anthropometric earPlug with exchangeable microphones for Individual Reliable Acquisition of Transfer functions at the Ear canal entrance.
2019 Mar 18. DOI: 10.5281/zenodo.2574395

15.

Poppitz J, Blau M, and Hansen M. Entwicklung und Evaluation eines Systems zur Messung individueller
HRTFs in privater Wohn-Umgebung. 201616.

Majdak P, Balazs P, and Laback B. Multiple exponential sweep method for fast measurement of headrelated transfer functions. Journal of the Audio Engineering Society. 2007 Jul 1; 55:623–37

17.

Novák A, Simon L, Kadlec F, and Lotton P. Nonlinear System Identification Using Exponential SweptSine Signal. IEEE Transactions on Instrumentation and Measurement. 2010 Aug; 59:2220–9. DOI: 10.
1109/TIM.2009.2031836

18.

Puckette MS. Pure Data: another integrated computer music environment. Proceedings of the International
Computer Music Conference, San Francisco (USA). 1996

19.

Steffens H, Par S van de, and Ewert S. Perceptual Relevance of Speaker Directivity Modelling in Virtual
Rooms. Proc. 23rd Int. Congr. on Acoustics. Aachen (Germany), 2019 :pp. 2651–2658

20.

Duda RO and Martens WL. Range dependence of the response of a spherical head model. The Journal of
the Acoustical Society of America. 1998 Nov; 104. Publisher: Acoustical Society of America:3048–58.
DOI : 10.1121/1.423886

21.

Brinkmann F and Weinzierl S. AKtools—An Open Software Toolbox for Signal Acquisition, Processing,and Inspection in Acoustics. Audio Engineering Society Convention 142. Audio Engineering Society, 2017
May 11

22.

Kirkeby O and Nelson P. Digital Filter Design for Inversion Problems in Sound Reproduction. Journal of
the Audio Engineering Society. 1999 Jan 1; 47

23.

Lindau A and Weinzierl S. Assessing the Plausibility of Virtual Acoustic Environments. Acta Acustica
united with Acustica. 2012 Sep 1; 98:804–10. DOI: 10.3813/AAA.918562

24.

Kadlec H. Statistical properties of d ’ and β estimates of signal detection theory. Psychological Methods.
1999; 4:22–43. DOI: 10.1037/1082-989X.4.1.22

25.

Stanislaw H and Todorov N. Calculation of signal detection theory measures. 1999

26.

Leckschat D, Epe C, Kob M, Seeber B, Spors S, Weinzierl S, and Zotter F. DEGA-Memorandum Nr. VA
1201 zur Durchführung und Dokumentation von Audio-Produktionen für wissenschaftliche Anwendungenin der Akustik. 2020 Jan 3. Publisher: Zenodo. DOI: 10.5281/zenodo.3597238

27.

Zotter F and Rudrich D. Dokumentation Aufnahmen Akustische Gitarre. 2018. Available from: https://phaidra.kug.ac.at/view/o:68022 [Accessed on: 2022 Aug 15]

28.

Zotter F and Rudrich D. Dokumentation Aufnahmen Akustische Gitarre, 2. Sitzung. 2018. Available from:https://phaidra.kug.ac.at/view/o:69273 [Accessed on: 2022 Aug 15]

29.

Leckschat D and Epe C. Aufnahmen von Sprecherinnen und Sprechern zur Verwendung in der Virtuellen
Akustik. 2020 Feb 20. DOI: 10.5281/zenodo.3601086

30.

MathWorks. Audio Toolbox User’s Guide. 2022 :1186

31.

Jaeger H, Bitzer J, Simmer U, and Blau M. Echtzeitfähiges Binaurales Rendering mit Bewegungssensoren
von 3D-Brillen. 2017

32.

Kuhn-Rahloff C. Realitätstreue, Natürlichkeit, Plausibilität. Berlin, Heidelberg: Springer Berlin Heidelberg,
2012. DOI: 10.1007/978-3-642-22072-2

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0790

Dynamic spatial rendering of early and late reverberation ofvirtual acoustic scenes
Bernhard U. SEEBER1; Ľuboš HLÁDEK; Tong WANG
Audio Information Processing, Technical University of Munich, Germany

ABSTRACTHearing out a target sound source in the presence of other sources is affected by the sources’ level, spectrumat the ears and binaural configuration of interaural coherence and phase. We focus on creating reproduciblelife-like, interactive laboratory test situations of spatial unmasking with moving sound sources andreverberation. The real-time Simulated Open Field Environment (rtSOFE) is an open-source software
package for simulating and auralizing static, moving and interactive sound sources in virtual rooms via
headphones or via loudspeaker arrays. In order to achieve high simulation update rates, early and latereverberation parts are rendered in different rtSOFE instances or taken pre-computed or measured from a file.
Configuration options and different use cases of rtSOFE are discussed. These include offline room acoustic
simulation, interactive virtual reality scenarios and auditory modeling in Matlab with on-the-fly computationof room impulse responses. New features in the upcoming rtSOFE version 1.2 are presented as well as an
approach for spatial rendering of scattered energy. It uses spatial channels which are fed with the scatteredenergy and which feed back energy to other spatial channels similar to a feedback-delay network.
The virtual acoustic scenario is described as a defined acoustic environment in which a particularconfiguration of sources and receivers characterizes an acoustic scene. We present two verified open-source
scenes for audiological research which take place on the platform of a virtual twin model of an undergroundstation in Munich.
Keywords: Auralization, room acoustic modeling, specular reflections, diffuse reverberation

1. INTRODUCTION
Spatial hearing research increasingly investigates in sophisticated realistic listening conditions
with the aim to translate research findings from abstracted psychoacoustic conditions with simple tone
or noise-like stimuli to the real world. Bridging the gap between the laboratory and the real world ispotentially even more important for audiological research and hearing aid fitting, since thereproducibility of patient’s real-world listening problems with past laboratory approaches is limited.
Virtual acoustics approaches, which simulate the acoustic effects of spatially distributed sound sourcesin rooms and spatially auralize the sources via headphones or loudspeakers, are thus becoming a key
research tool. Simulation approaches and capabilities differ widely between tools (1) and therequirements for physically or at least perceptually adequa te spatial rendering are not yet fully clear,particularly if computational resources are limited as it is the case with dynamic rendering of moving
or interactive sources (2). One approach to handle the computational complexity is to simulate earlyand late reflections with different update rates in different threads or distributed across different
computers, as used in the real-time Simulated Open Field Environment (rtSOFE) software tool.
Another approach is to use computationally simpler models particularly for late reverberation, as it ismore stochastic (e.g., Raytracing: (3), FDN: (4)). We will discuss developments which aim to improve
the physical simulation quality while keeping computational complexity within the requirements forreal-time simulation.

1

seeber@tum.de2. The real-time SIMULATED OPEN FIELD ENVIRONMENT (rtSOFE)
2.1 Overview
The real-time Simulated Open Field Environment (rtSOFE) is a real-time room acoustics
simulation and auralization tool developed in C++ and based on a modular, distributed structure.
rtSOFE is freely available along with documentation and example use cases (5-7). Room acoustic
simulation entities (‘rtSOFE.exe’) create a spatial room impulse response (RIRs) or a part of a RIR
which is send to the convolver (‘convolver.exe’), see Figure 1. The convolver creates the sound signal
for auralization, performs loudspeaker equalization and plays the sound.
The room acoustic simulation is based on the generalized mirror-image source method for arbitraryroom geometries (6, 8) and includes multi-threaded, optimized code for image source generation,
backtracking and validity checking. RIRs can be synthesized for binaural rendering and for multichannel loudspeaker setups using nearest-speaker mapping or horizontal Ambisonics.
The system achieves high update rates (360 Hz is common) by multi-threading and by using singleinstruction, multiple data commands in PC-processors (AVX2 instruction set). Since rtSOFE is used
to compute images sources to high order (100-200), computation time would nevertheless be too longfor real-time rendering of moving sources. Hence, only early reflections (e.g. to order 3) arerecomputed at high update rates, whereas medium and late reverberation (e.g. up to order 100) is
computed in a second rtSOFE instance (Figure 1). This permits updates of the reverberant tail every
few hundred milliseconds in interactive situations where the spatial distribution of late reverberation
can change, e.g. when the receiver comes close to surfaces or openin gs. The convolver can handle
variable update rates and will fuse the different RIR parts. An alternate configuration is to use a pre computed RIR in cases when the receiver positions are restricted such that little change in the statisticsof the late reverberation can be expected. Also, measured RIR tails can be used with simulated earlyreflections, permitting the combination of dynamic early specular reflections with highly authentic
late reverberation (“hybrid rendering”)(9). Current developments to simulate late, diffuse
reverberation are discussed in section 2.4.
The convolver can also be used with multiple sound sources. Using network-distributed rtSOFE
instances will permit the simulation of detailed reflections in real time for each source. This wa s
demonstrated with six sources, each with source directivity simulation, and live sound input andcaptured in a video 2.

Figure 1: Common setup of rtSOFE for real-time rendering. Positions and orientations of source andreceiver are streamed via OSC-messages to rtSOFE instances, which each compute a part of the RIR andsend it to the convolver. The convolver can use FIR or FFT-blocks of variable length to convolve the soundsignal with the RIR parts, add them, convolve with loudspeaker equlization filters and play the resulting
signal. RIR parts can overlap in time. The convolver also handles the fade-over when new RIR parts arrive.
2

See https://www.youtube.com/watch?v=yN7vD1khTOI

2.2 rtSOFE use cases
rtSOFE is developed as a research tool for (interactive) hearing research and hence features a wide
variety of potential uses. rtSOFE is published on zenodo.org (5) and the documentation contains
example code and tools for Matlab and Python and discusses the following use cases:
1. Offline rendering: Start of rtSOFE from the command line, single computation from 0th to10th order, write RIR to file (rtSOFE in ‘static mode’).
2. Start of rtSOFE from Matlab, compute image source (IS) structure up to order 3, save IS
structure and load in MATLAB, e.g. for later manipulation or for rendering (Static mode).
3. Model-in-the-loop pipeline: Start rtSOFE in Matlab, send positions to rtSOFE, receive RIR in
Matlab, plot RIR (Interactive mode).
4. Realtime auralization with early reflections: Start rtSOFE on 1 PC, connect to convolver on
localhost, use audio from file (.wav) for real-time binaural rendering. Uses source directivity
transfer functions and head-related impulse responses in SOFA format and computes ininteractive mode.
5. Realtime auralization of full RIR with multiple rtSOFE instances: Start two rtSOFEs on 1 PC,
connect to convolver on localhost, uses audio from file (.wav) for real -time binaural rendering(SOFA file) in interactive mode. rtSOFE instance #1 computes early reflections and instance#2 the late reflections.
6. Realtime auralization over loudspeakers: Start rtSOFE instances on 1 PC for early reflections(Ambisonics) and late reflections (nearest speaker mapping) rendering, convolver on localhost,use audio from file for real-time auralization.
7. Integration with VR headset with headtracker for real-time auralization over headphones: Start
rtSOFE instances on 1 PC for early and late reflections, convolver on localhost; use audio fromfile for real-time binaural rendering (SOFA file) and a connection to an HMD to get head
position and rotation via Python OpenVR library.
8. This demonstration computes late reverberation by rendering image order 11-100. The
parameters "source-movement-threshold" and "receiver-movement-threshold" are assigned ahigh value so that computed image sources are reused with position updates only, without
further validation and visibility checks. Position updates of visible ISs make the RIR rendering
dynamically position dependent with as little real-time computational cost as possible, as analternative to using a static reverb RIR rendered offline.
9. Demonstration of binaural rendering of multiple sound sources with the convolver, eachrendered with a convolution object with different RIRs from different rtsofe.exe processes. A
static speech source and a noise source rotating around the listener are rendered. The noise
source is also filtered with source directivity transfer function s from a SOFA file.
Additionally, an extension to the Unreal game engine exist, the “rtSOFE controller”, which sends
OSC messages from Unreal to start and control rtSOFE instances. The Unreal game engine is used totranslate the incoming position and orientation data stream from a motion tracker to sources or th ereceiver inside the simulated virtual environment and to send these translated coordinates to rtSOFE
instances for room acoustic simulation. Unreal also conducts collision checks such that the charactercannot walk through objects or walls, and it can also start and stop sound sources upon state changes
in the virtual space.
2.3 New developments for version 1.2
rtSOFE version 1.1 is publically available on zenodo.org (5). We plan to release version 1.2, which
is currently being internally tested, in autumn 2022. The new version will feature:- source directivity using SOFA-standard transfer functions,- a dynamic latency reduction which removes leading zeros in RIRs (e.g. the direct sound or
first reflection traveling delay) which is useful for live-speech experiments,- interactive walls which can be turned on or off dynamically via OSC messages, e.g. to simulate
open and closed doors,- the option to set the seed for the random temporal jitter of image sources. This allows thesimulation of exactly the same RIR including its random jitter with different source or receivercharacteristics, i.e. for the front and back microphones in hearing devices,- sped up computation by internal changes in rtSOFE.exe to image tree arrangement, visibility--

tests, update scheduling, and smaller transmitted matrices,
updates of convolver.exe to permit multiple sources and to turn them on and off individually.
A delayed-stop option lets the reverberation ring out when source input stops. Internal
convolver channels can be dynamically re-initialized for new sound sources and hence bereused in interactive settings,
FFT-only convolution schemes in convolver.exe to increase efficiency with off-line sources,
like pre-recorded background sounds,
extensively updated documentation.

2.4 Spatial rendering of scattered diffuse energy
Historically, rtSOFE simulates specular reflections to high orders and renders them spatially witha temporal jitter. This ensures that the spectral, spatial and temporal distribution (“reflection clusters”)
of reflected energy is reproduced across the RIR. The jittering distributes individual reflections intime and reduces comb filters from repetitive reflection patters occurring with parallel walls. The highnumber of reflections in higher reflection orders leads to a diffuse field. However, the computationalload, but more so the lack of scattered energy in an early diffuse field in parallel to specular reflections,
led us search for additional approaches to simulate the diffuse field.
We are experimenting with a diffusion approach, which is fed energy through the scattered part
upon each reflection. We introduce a scattering coefficient which distributes the reflected energy into
the specularly reflected part and into a diffuse part (10). In reality, scattered energy would reflect into
all other directions but the specular direction, and it would not immediately reach the receiver but
bounce off other surfaces first. If we stand near a wall opening or an absorbing surface, the diffuse
field will be spatially asymmetric, and the simulation should capture this.
We hence define internal channels for carrying spatial, diffuse reverberation, which can be spatiallycoincident with the loudspeaker directions or a subset of them. To associate absorption with eachchannel (direction), rays are cast outward from the receiver in order to find the intersections with theroom surfaces. The receiver-surrounding sphere is divided into Voronoi regions associated with eachspatial channel and the average absorption factor is computed for this spatial area from knowing thesurfaces intersected by the rays. From each reflection, the scattered energy is cast into several spatialchannels, where it is delayed and attenuated before being fed back to other spatial channels similar toa feedback-delay network. This will diffuse energy across spatial channels and spatial channels withmore absorption will also feature a higher damping and thus lo wer level output. Low-pass filtering
the channel energy-time signals will lead to channel energy envelopes to be applied to noise for nonisotropic diffuse field rendering (11).

3. AUDIO-VISUAL SCENES FOR HEARING RESEARCH
3.1 Environment and scene description
Audio-visual scenes for hearing research are defined as specific combinations of source-receiver
positions and orientations placed inside a particular audio-visual environment (12). Descriptions ofenvironments and scenes are collected in the zenodo.org community audiovisual scenes 3. For easy reuse of the environment in different labs, the description requires a file for the visual representation(e.g. in .obj format) and – in lack of universally accepted standards – a text file describing source andreceiver positions [x,y,z-locations] and orientation vectors in the same frame of reference as the visual
scene (13).
We have developed, verified and open-source published two scenes taking place in an undergroundstation environment, which we measured and modeled after an underground station near our lab (9,14). The lower panel of Figure 2 shows the definition of both scenes: Scene 1 (blue loudspeakersymbols) defines equidistant sources around the listener R1, 0° in the front and every 30°. This reflects
a commonly used setup in audiological research where sources are often placed at 0° and ±60° or ± 90°
or at 180°. Target and interferer sources can be chosen from any of these locations and have similar
direct-to-reverberant ratios and mostly differ in binaural cues. Additionally, interferers can also beplaced at positions outside the circle (sources 18-21, green loudspeaker symbols). Scene 2 (sources in3

https://zenodo.org/communities/audiovisual_scenes

red, #13-17 and source #1) are in front of the listener and change only in radial distance. This mimics
an approaching speaker and results in changing direct-to-reverberant ratios and direct sound energy.
Without reverberation, the target would be almost diotic, but the reflection of the notice board in theplatform center and the late reverberation leads to some decorrelation of interaural cues.

Figure 2: Upper panels: Cross section (top) and floorplan (middle) of the underground station environment.
Lower panel: View of the platform area with listener position R1 (receivers R2-5 are for room acoustic
measurements only) and sources (loudspeaker symbols; at 1.6 m height). The blue loudspeakers are part ofscene 1 and the red loudspeakers of scene 2. Green loudspeakers can be used in both scenes for, e.g.,interfering sources. (Figure 2 in (12)).

3.2 Scene auralization verification
The auralization of selected scene positions via free-field loudspeakers was verified both
numerically-acoustically and perceptually (9). The auralization of early reflections with rtSOFE used
the room-acoustic model of the underground station scene and rendered reflections to the 60
loudspeakers in the SOFE (6, 7). The later reflections were rendered via the loudspeakers using
Ambisonics processing of the spatial room impulse responses recorded with a 32-channel microphonearray (Eigenmike EM32). For details see (9).
The numerical evaluation included the room acoustical parameters T20, T30, C50, C80 and EDT(15), the speech-weighted direct-to-reverberant energy ratio, the interaural time difference, theinteraural level difference and the interaural coherence in different positions of the modelled andrecorded impulse responses. A high correspondence of these parameters for the modelled and therecorded baseline impulse responses was obtained (9).
Moreover, a perceptual evaluation with respect to speech intelligibility was conducted with six
participants. Speech reception thresholds (SRTs) were measured for modelled room impulse responses
auralized in the free-field via 60 loudspeakers (“loudspeakers”) as well as for recorded binaural
impulse responses (“headphones”) as a baseline. The Oldenburg Sentence test was used. The resultsof the speech test in Figure 3 show across different scene positions a high correspondence of speech
intelligibility in the loudspeaker-reproduced modelled scene to the baseline recorded in the real space.

Figure 3: Speech reception thresholds measured at the different scene positions for the free-field auralized
modelled spatial room impulse response (“loudspeaker”) and the binaural room impulse responses recordedat the scene positions in the real underground station. Figure reproduced from (9).

4. CONCLUSIONSThis article presents latest developments for simulating and auralizing room reverberation in realtime with rtSOFE: new features available with version 1.2 for improved dynamic, interactive andmulti-source rendering are introduced and ideas for rendering scattered energy into a spatiallydependent diffuse field auralization discussed. Two scenes taking place in a virtual undergroundstation are introduced. Their acoustical and psychoacoustical verification demonstrates that theauralization with rtSOFE agrees well with the original space.

ACKNOWLEDGEMENTS
Scene research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) – Project ID 352015383 – SFB 1330, project C5. The (rt)SOFE was funded by the BMBF
Bernstein Center for Computational Neuroscience, Munich (BMBF 01 GQ 1004B).
We thank Christoph Kiener and Marius Lambacher for their contributions in their student projects
to former approaches for diffuse field rendering.

REFERENCES
1.

Brinkmann F, Aspock L, Ackermann D, Lepa S, Vorlander M, Weinzierl S. A round robin on room

acoustical simulation and auralization. J Acoust Soc Am. 2019;145(4):2746.
2.

Fichna S, Kirsch C, Seeber BU, Ewert SD. Perceptual evaluation of simulated and real acoustic scenes withdifferent acoustic level of detail.
Acoustics; 2022.

24th Int Congress on Acoustics; Gyeongju, Korea: Int. Commission on

3.

Schröder D, Vorländer M. RAVEN: A real-time framework for the auralization of interactive virtualenvironments. Forum Acusticum; 2011: Aalborg Denmark, pp 1541-1546.
4.

Wendt T, Van De Par S, Ewert SD. A computationally-efficient and perceptually-plausible algorithm forbinaural room impulse response simulation. Journal of the Audio Engineering Society. 2014;62(11):748-66.
5.

Seeber BU, Wang T. real-time Simulated Open Field Environment (rtSOFE) software package. 2021. DOI:

10.5281/zenodo.5648305
6.

Seeber BU, Kerber S, Hafter ER. A System to Simulate and Reproduce Audio-Visual Environments for

Spatial Hearing Research. Hear Res. 2010;260(1-2):1-10.
7.

Seeber BU, Clapp SW. Interactive simulation and free-field auralization of acoustic space with the rtSOFE.

J Acoust Soc Am. 2017;141(5):3974.
8.

Borish J. Extension to the image model to arbitrary polyhedra. J Acoust Soc Am. 1984;75(6):1827-36.

9.

Hládek H, Ewert S, Seeber BU. Communication conditions in virtual acoustic scenes in an undergroundstation. 2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA) 2021; Bologna, Italy: IEEE.
DOI: 10.1109/i3da48870.2021.9610843.
10.

Vorländer M. Auralization: Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic

Virtual Reality. Berlin: Springer; 2008. 335 p.
11.

Lehmann EA, Johansson AM. Diffuse Reverberation Model for Efficient Image-Source Simulation of

Room Impulse Responses. IEEE T Audio Speech. 2010;18(6):1429-39.
12.

van de Par S, Ewert SD, Hládek L, Kirsch C, Schütze J, Llorca-Bofí J, et al. Auditory-visual scenes forhearing research. Acta Acust. 2021:arXiv:2111.01237; in print.
13.

Hládek L, van de Par S, Ewert SD, Seeber BU. Audio-visual scenes repository: How to contribute. 2021.

DOI: 10.5281/zenodo.5532673.
14.

Hládek L, Seeber BU. Underground station environment (data set and description). 2022. DOI:

10.5281/zenodo.6025631.
15.

Acoustics – Measurement of room acoustic parameters – Part 1: Performance spaces (ISO 3382-1:2009).

ABS-0907

Parametric pinna model for a realistic representation of listener-specific pinna
geometry
Katharina POLLACK*, Florian PAUSCH, Piotr MAJDAKAcoustics Research Institute, Austrian Academy of Sciences, Vienna, Austria*katharina.pollack@oeaw.ac.at

ABSTRACTPersonalised binaural audio requires individual head-related transfer functions (HRTFs). Nowadays, it is feasible to compute HRTFs numerically from meshes of a listener’s anatomy. These meshes require geometric detailsof the pinna which are difficult to capture and often corrupted by noise and outliers. The alignment of a highresolution template geometry to the captured listener geometry seems to be a promising approach to deal withthese corruptions. However, such an alignment represents a multi-dimensional optimisation problem. To tackle
this, we describe a parametric pinna model which reduces the representation of a non-pathological human pinna
geometry from a few thousand points in space to 144 parameter dimensions. The model was evaluated in the geometric and psychoacoustic domains in terms of the Hausdorff distance distribution and simulated localisation
errors, respectively. When aligned with individual ground-truth meshes, the parametric pinna model was able torepresent these meshes with a small geometric error. The predicted sound-localisation performance, using thealigned meshes as a basis for HRTF calculations, was similar to that of the corresponding listener. These results
indicate that the proposed parametric pinna model represents a versatile tool to create human pinna meshes witha detailed geometry for the calculation of individual HRTFs.

Keywords: head-related transfer functions, binaural audio, numerical calculation, shape alignment

1

INTRODUCTION

The acquisition of individual head-related transfer functions (HRTFs) for plausible binaural reproduction has
been subject of research for a few decades. Augmented reality and virtual reality (AR/VR) systems evolved
into commercial products, requiring an easy solution for individual HRTF acquisition. Besides the possibility
to obtain individual HRTFs via acoustic measurement systems [1], numerical calculation represents a promisingeasy-to-use acquisition method [2, 3].
In order to numerically calculate accurate HRTFs, a three-dimensional (3D) representation of a listener’s
geometry, i.e., a mesh, is essential. While the head, neck, and torso can be modelled by simpler geometricshapes, such as spheres and trapezoids [4, 2], the sampling of the pinna geometry must be accurate with anaverage edge length of 1 mm of the triangular elements, resulting in meshes consisting of thousands of vertices.
Because of this high dimensionality, a substantial reduction of the number of dimensions in the parametericrepresentation of the pinna geometry is required [5, 6, 7].
Non-individual HRTFs can be personalised by spectrally warping them using the pinna height, which isan easily measurable parameter [8]. Altough this method improves the degree of personalisation, it does not
fully achieve it, which motivates a refined parametric description. The pinna geometry has been parameterisedby means of additional anthropometric features, such as pinna height and width, cavum conchae depth, andothers [7, 9]. One problem with these parameterisations is that they do not facilitate an unambiguous synthesisof a complete 3D pinna. Because of the fast development of computer graphics and augmented reality systems
in recent years, there have been advances towards a more complicated parametric description of a pinna, one ofwhich is revised in this study.

Figure 1. A human pinna including technical terms of various regions. The convex cavities, i.e., cymba andcavum conchae, fossa triangularis and scapha, are important for the first peaks and notches according to [10].
Figure 1 shows distinguished pinna regions, emphasizing that the pinna is a complex biological structure
whose parametric description is not trivial. Based on this representation, one could use, e.g., these regions’prominences, i.e., depths, as parameters. In that case, the problem would be that these parameters are not easy
to define for concave and convex areas uniformly and that they do not control the curvature of the correspondingor adjacent regions. Another approach would be to use a few carefully selected landmarks on the pinna surface
that can be moved, with adjacent vertices of the mesh moving according to a proximity-based weighting. The
problem with such an approach is that the pinna cavities cannot be deformed dramatically, as they do for humansubjects (for example, the cavum conchae depth varies up to 30 % for inviduals [9]), and that the proximity
cannot be defined as a sphere because of the small distance between structures. These approaches and theirdisadvantages demand a parametric representation that accounts for curvature and depth of the pinna geometry.
In this article, we describe a biologically motivated model of a pinna aiming at bridging the gap between alow-dimensional representation of a pinna geometry and an accurate pinna mesh for the calculation of individual
HRTFs. We show that our model is capable of deforming a pinna mesh such that it approximates the pinnaeof human subjects with a small geometric error, resulting in numerically calculated HRTFs showing similarsound-localisation errors as for highly accurate scans.

2

MODEL STRUCTURE

The concept of the described parametric pinna model (PPM) roots in skeletal animation and morph target animation, both concepts being usually applied in the area of animating digital characters, where the skeleton
controlling the mesh is called "armature". Once the armature is constructed, it is attached to a template meshsuch that changes in the armature deform the mesh globally and local regions of it. The armature consistsof two elements: "bendy bones" and "control bones". Further, the PPM includes "shape keys" to fine-tune
the geometry of the template mesh. Because the PPM is implemented in the graphics suite Blender (version
3.1.2)1 [11], we use the software-specific terms.
2.1 Template mesh
The model’s underlying pinna mesh is the average mesh of the WiDESPREaD database [12], which was derived
from a geometric principal component analysis (PCA) performed on the left pinnae of 119 subjects. The subject
meshes were acquired using a commercial structured-light-based scanner. This average pinna mesh will bedenoted as X0 throughout the rest of this manuscript.
1 http://www.blender.org

Figure 2. Example of one Bézier curve in the armature: bendy bone with two control bones (shown as spheresat the start and end of the bendy bone) following the curvature of the lower part of the helix. The displayed
pinna mesh represents the template mesh X0 .
2.2 Armature elements: Bézier curves
The armature is defined by cubic Bézier curves [13] c(l) along the surface of the pinna:c(l) = (1 − l)3 p0 + 3(1 − l)2 lp1 + 3(1 − l)l 2 p2 + l 3 p3 ,(1)

with 0 ≤ l ≤ 1, l ∈ R, and pq , q = {0, 1, 2, 3}, representing the curve’s control points in 3D space. Points p0
and p3 correspond to the control points at the start and end of a Bézier curve, respectively. The rotation of theline between control points p0 and p1 , and between control points p2 and p3 determines the amount of curve
flexion. In the software interface, these relationships between the control points is simplified at the start andend of a Bézier curve with so-called "control bones" that are conveniently named "start" and "end", and thecurve connecting the control bones is called "bendy bone". In our PPM, control bones can be translated androtated, whereas bendy bones can be scaled in an isotropic way only. Figure 2 shows an example of such abendy bone and its associated control bones.
The armature consists of nine Bézier curves, each with a bendy bone, and a start and end control bone. The
armature additionally includes a special Bézier curve whose control points lie in line and whose control bones
are not exposed in the model. This particular curve controls all bendy bones, allowing to globally translate,rotate, and scale the pinna mesh. In the PPM, we call this Bézier curve "parent bone", because it’s a parent
object in the software. The parent bone is the only one that may be scaled in an anisotropic way, while
anisotropic scaling of bendy bones is not necessary to accurately describe a pinna mesh. In fact, because
we do not need all degrees of freedom the software offers, not all parameter dimensions are exposed. The
exposed parameters are rotation and translation of control bones, isotropic scaling of bendy bones, and rotation,translation and anistropic scaling of the parent bone. Table 1 gives an overview of the armature.
In order to fully control the pinna geometry, the armature is attached to the mesh by means of automatedweighting. Automated weighting is implemented as a distance-based mapping of the amount of the rigid transformation of the armature, deforming the vertices in proximity of the corresponding bone. Slight changes to theweighting function were made manually to avoid overlap of adjacent regions of the complex geometry. Figure 3
shows the uniform weighting of the parent bone, and two examples of the mesh weighting for selected bendy
bones.
The pinna mesh deformation ∆Xb as per armature modifications is described byM∆Xb = ∑ Ai X0 , pq ,

(2)

i=1

where M = 10 is the number of all Bézier curves (consisting of bendy and control bones), and Ai is a surjectivefunction f : R3 7→ R3 that maps the parameter space to the mesh space. The function depends on X0 , i.e., theTable 1. List of the Bézier curves in our parametric pinna model. The first nine entries consist of the exposedparameters of a Bézier curve described by two control bones and a bendy bone, controlled by 13 parameters
each. The tenth entry is the parent bone, controlled by 9 parameters.

(a)

Index

Name

1

Lobulus2

Helix low3

Helix middle

4

Helix up5

Tragus6

Antitragus

7

Antithelix

8

Crus inferius anthelicis

9

Crus superius anthelicis

10

Parent bone

(b)

(c)

Figure 3. (a) Uniform weighting of the parent bone. (b, c) two bendy bones, ranging from none (blue) tomaximum deformation (red). Note the overlap between the two weighting functions at the top of the helix.

(a)

(b)

(c)

Figure 4. (a) The armature of the PPM (green) attached to the template mesh. (b) Example translating androtating the lower-helix control bones, and scaling of the corresponding bendy bone. (c) Resulting deformed
template pinna, with hidden armature.
template mesh, and the parameter settings pq for the i-th Bézier curve. These parameters, i.e., the position of thecontrol points of the Bézier curve in 3D space, are controlled in the model by rotating and translating control
bones, isotropically scaling bendy bones, and rotating, translating and anisotropically scaling the parent bone.
Figure 4 displays the complete armature of the model and demonstrates a deformation of the helix as an
example. Note that there are no points of discontinuity, and that the mesh is deformed in a smooth way.
2.3 Shape keys
Spatially less extensive changes of pinna areas are controlled via shape keys. Shape keys are described as aweighted deformation of the mesh:N∆Xs = ∑ wj Bj ,

(3)

j=1

where ∆Xs is the resulting mesh deformation, N describes the number of shape keys, Bj describes the meshdeformation by the j-th shape key, and wj the weights controlling the prominence of the corresponding shapekey deformation. Table 2 presents the weight ranges for each of the shape keys within our PPM.
2.4 Full model
The superposition of Eq. (2) and (3) yields the complete equation describing the deformed mesh controlled by
the parameters of the PPM:M N
XPPM = X0 + ∆Xb + ∆Xs = X0 + ∑ Ai X0 , pq + ∑ wj Bj ,|{z}|{z} |{z}i=1
j=1template

bones

(4)

shape keys

with M = 10 Bézier curves and N = 18 shape keys. As for the total number of dimensions, we have ninedegrees of freedom (DOFs) for the parent bone (translation, rotation, anistropic scaling), six DOFs for eachcontrol bone (translation and rotation), one DOF for each bendy bone (isotropic scaling), and one DOF foreach shape key (weights wj ), resulting in a total of 144 dimensions:N= 144.
dim{XPPM } = 6 + 3 + (M − 1) · 1 + 2 · (M − 1) · 6 + |{z}| {z }| {z } |{z}parent bone

bendy bones

control bones

shape keys

(5)

Table 2. Shape keys and corresponding weight ranges available in our parametric pinna model.

3

Index

Shape-key name

Weight ranges

1

Antitragus-Crease−1 ≤ w1 ≤ 1

2

Cavum_conchae-Depth−1 ≤ w2 ≤ 1

3Cymba_conchae-Depth

0 ≤ w3 ≤ 1

4

Crus_helicis-Prominence−1 ≤ w4 ≤ 1

5Upper_helix-Depth

0 ≤ w5 ≤ 1

6Middle_helix-Depth

0 ≤ w6 ≤ 1

7Lower_helix-Depth

0 ≤ w7 ≤ 1

8Lobulus-Form

0 ≤ w8 ≤ 1

9Scapha-Depth

0 ≤ w9 ≤ 1

10Fossa_triangularis-Depth−1 ≤ w10 ≤ 1

11

Crus_inferius_anthelicis-Lower_crease

0 ≤ w11 ≤ 1

12

Crus_inferius_anthelicis-Upper_crease

0 ≤ w12 ≤ 1

13

Crus_superius_anthelicis-Lower_crease

0 ≤ w13 ≤ 1

14

Crus_superius_anthelicis-Upper_crease

0 ≤ w14 ≤ 1

15

Tragus-Upper_dent−1 ≤ w15 ≤ 1

16

Crus_helicis-Upper_dent

0 ≤ w16 ≤ 1

17

Crus_helicis-Lower_dent

0 ≤ w17 ≤ 1

18

Ear_canal-Diameter−1 ≤ w18 ≤ 1

EVALUATION

We evaluated the model by aligning the template pinna to six target pinnae by manually deforming the describedparametric pinna model. The target meshes were obtained from subjects of the databases maintained by the
Acoustics Research Institute (ARI), Austria, and the Sorbonne University (SOU), France. To adequately sample
the geometric space spanning possible non-pathological human pinnae, we chose six pinnae that have distinctive
features. For example, NH1060 shows a wide scapha and a small lobulus, NH130 shows opposed values forthese parameters. In order to explore the pinna dimensions and further develop the PPM, we exported thedeviations from the default parameter values.
The alignment process was as follows: A target mesh of another pinna was loaded into the Blender workspace.
First, the parent bone was used for globally setting the location, orientation and size of the PPM such that the
template pinna roughly aligns with the target mesh. After that, the control bones and bendy bones were sequentially translated, rotated and scaled respectively until the deformed template pinna was aligned with the target
pinna. In this step, we focused on high alignment accuracy in convex regions.
For further refinement of the alignment, the weights of the shape keys were adjusted to synthesize theconcave target-pinna structures. Due to the overlapping weighting of the bendy bones and the manual definitionof the shape keys, changing one parameter manipulates a region another, possibly multiple parameters influence
as well. Thus, transforming the armature and adjusting shape key weights was done in an alternating fashion,
representing an iterative process.

The alignment was completed once the mean geometric error between the deformed template pinna andthe target mesh was below an a-priori defined error margin of 1 mm. Note that certain pinna regions needed
to be aligned with particular care since larger errors may lead to perceivable effects when listening via thenumerically calculated HRTF [7].
For all six subjects, ground truth meshes (coming from a computed tomography scan of a head cast [14])were provided. These serve the evaluation for both the geometric and the psychoacoustic domain. It has beenshown that these meshes provide the geometric accuracy necessary for the calculated HRTFs to yield similarsound-localisation errors as HRTFs from a measurement of the corresponding listener [14, 15]. In this evaluation, we thus only consider results from numerically calculated HRTFs.
In the geometric domain, the alignment process was evaluated using the distance between two point sets,i.e., the deformed template mesh and the target mesh. A metric commonly used in medical image processing isthe Hausdorff distance [16], describing the largest smallest Euclidean distance between two meshes. However,this single-value metric does not sufficiently describe whether an alignment was successful or not. Thus, weused a point-wise single-sided Hausdorff distance, i.e., taking the smallest Euclidean distance for every point of
the target pinna to the deformed template pinna, and inspect its distribution.
In the psychoacoustic domain, the alignment process was evaluated by means of sound-localisation errors,
specifically quadrant error and polar error. We numerically calculated HRTFs from the ground-truth meshes,
which were used as reference HRTFs for the evaluation in the psychoacoustic domain. After the alignment
using our model, the left pinna was stitched back to the ground truth mesh and we calculated the HRTFs again.
This way, we could ensure the only change between the HRTFs is the influence of the left pinna. The HRTFs
were calculated using the the open-source software Mesh2HRTF v1.02 . Since the pinna is mostly responsible forelevation perception [17], we used the model predicting sound-localisation performance in sagittal planes [18],implemented as the function baumgartner2014 from the Auditory Modeling Toolbox (AMT) v1.23 [19]. Thismodel simulates how a subject would perform in a localisation experiments given the ground-truth HRTFs as areference versus the calculated HRTFs from the PPM-aligned mesh stitched to the ground-truth head mesh.

4

RESULTS

Figure 5 shows the range distribution of all PPM parameters for the six alignments as a function of deviationfrom the default state. Figure 6 shows the Euclidean distance distribution of the PPM-aligned mesh versus
the target mesh, i.e., the ground truth, calculated and visualised with Meshlab [20]4 . All alignments resulted
in geometric distance values largely below the one-millimeter threshold. It has been shown that an accuracyof distance values at approximately 1 mm lead to perceptually valid HRTFs by means of sound-localisation
errors [14, 21].
Figure 7 shows the results of the sound-localisation model predictions, i.e., computed quadrant error rates
and polar errors for the HRTFs resulting from five of the alignments. The sensitivities have been adapted suchthat the quadrant error of the reference is at approximately 8 % for each subject, according to [22]. In thepsychoacoustic domain, we showed that the HRTFs from PPM-aligned meshes yield similar sound-localisation
errors as HRTFs calculated from the the ground-truth meshes. The differences in the quadrant error for subjects
NH1059, NH1060, and NH1061 can be explained by the geometric error in the fossa triangularis and cymba
conchae for the bottom row in Fig. 6.

5

CONCLUSION

We revisited a recently proposed PPM and described its structure and key components in detail, focusing on
the mathematical description connecting to the software-dependent terminology. In order to investigate model
applicability for individualisation of HRTFs, the PPM was manually controlled to align the average mesh to six
human target pinnae, and the deformed meshes were evaluated in the geometric and psychoacoustic domains.
We showed that the PPM was able to approximate the target meshes with a geometric error largely below
2 https://github.com/Any2HRTF/Mesh2HRTF
3 https://amtoolbox.org/
4 https://www.meshlab.net/

Location

Rotation (Euler, ZYX)

ScaleParameter

TragusParentLobulus
Helix_up
Helix_middle
Helix_low
Crus_superius_anthelicis
Crus_inferius_anthelicis
Antitragus
Antihelix

X

NH5
NH130

TragusParentLobulus
Helix_up
Helix_middle
Helix_low
Crus_superius_anthelicis
Crus_inferius_anthelicis
Antitragus
Antihelix

NH131
NH1059Y

NH1060NH1061

Bendy

TragusParentLobulus
Helix_up
Helix_middle
Helix_low
Crus_superius_anthelicis
Crus_inferius_anthelicis
Antitragus
Antihelix

Start
End
Z

−4

−2

0

2

−100 −50

0

50

100

0.4

0.8

1.2

1.6

Shape key(a) Parent bone, bendy bones and control bones.

Antitragus−Crease
Cavum_conchae−Depth
Cymba_conchae−Depth
Crus_helicis−Prominence
Upper_helix−Depth
Middle_helix−Depth
Lower_helix−Depth
Lobulus−Form
Scapha−Depth
Fossa_triangularis−Depth
Crus_inferius_anthelicis−Lower_crease
Crus_inferius_anthelicis−Upper_crease
Crus_superius_anthelicis−Lower_crease
Crus_superius_anthelicis−Upper_crease
Tragus−Upper_dent
Crus_helicis−Upper_dent
Crus_helicis−Lower_dent
Ear_canal−Diameter

NH5
NH130
NH131
NH1059NH1060NH1061

−1.0

−0.5

0.0

0.5

1.0

(b) Shape keys.

Figure 5. Parameter ranges of (a) the parent bone, bendy bones and control bones, and (b) the shape keys.
Location and rotation are provided in mm and deg, respectively. Rotations are represented by Euler angles withaxis-rotation order ZYX.

(a) NH5, 0.81 mm median distance(b) NH130, 0.72 mm median distance(c) NH131, 0.83 mm median distance(d) NH1059, 1.04 mm median distance(e) NH1060, 0.72 mm median distance(f) NH1061, 0.59 mm median distance

Figure 6. Geometric error distribution between the PPM-aligned meshes and the corresponding ground truth
mesh, for six different subjects. Left of the coloured pinnae are the individual distance distributions in mm,
ranging from red (error at 1 mm or below) over green (error at approx. 3.5 mm) to blue (error over approx.
5 mm). Subcaptions indicate the median value of the corresponding distance distribution.

(a)

(b)

Figure 7. (a) Quadrant-error rate in % and (b) polar error in deg.

1 mm. In the psychoacoustic domain, predictions of the saggital-plane sound-localisation performance were obtained based on the individual HRTFs, which were numerically calculated from the ground-truth meshes andPPM-aligned meshes. Compared to the results from the corresponding ground-truth HRTFs, both the geometricerror and the predictions of the auditory model indicate that the PPM was able to accurately synthesize the considered human pinnae. With this geometric and psychoacoustic accuracy, it is likely that the HRTFs calculated
from PPM-aligned meshes yield similar localisation errors as measured HRTFs in a physical sound-localisation
experiment.
The pinna is a highly complex biological structure, and especially the cavities which geometric dimensions
are difficult to describe are the important regions of the pinna responsible for peaks and notches in HRTFs.
The problem of describing this structure is a multidimensional one, and it is not easy to solve. The proposed
PPM serves as a bridge for the gap between a low-dimensional parametric pinna geometry description andhigh-accurate mesh synthesis.
For future applications, the model control will be improved to enable an automated alignment process, forwhich the exploration of the possible parameter ranges will help. But already in its current state, the model
represents a versatile tool to achieve a parameteric description of the pinna geometry, facilitating access toindividual HRTFs for a wide audience.

ACKNOWLEDGEMENTS
We thank Mantas Tamulionis for his contributions to the implementation of the PPM Matlab interface. The workof K.P. and P.M. was supported by the Austrian Research Promotion Agency (FFG, grant no. 871263, project"softpinna"). All authors received funding from the European Union’s Horizon 2020 research and innovation
funding programme (grant no. 101017743, project "SONICOM"–Transforming auditory-based social interaction
and communication in AR/VR).

REFERENCES[1] S. Li and J. Peissig, “Measurement of Head-Related Transfer Functions: A Review,” Applied Sciences,vol. 10, p. 5014, Jan. 2020. Number: 14 Publisher: Multidisciplinary Digital Publishing Institute.
[2] K. Pollack, W. Kreuzer, and P. Majdak, “Modern acquisition of personalised head-related transfer functions:An overview,” in Advances in Fundamental and Applied Research on Spatial Audio (D. B. F. Katz and D. P.
Majdak, eds.), ch. 4, Rijeka: IntechOpen, 2022.
[3] W. Kreuzer, P. Majdak, and Z. Chen, “Fast multipole boundary element method to calculate head-relatedtransfer functions for a wide frequency range,” The Journal of the Acoustical Society of America, vol. 126,
pp. 1280–1290, Sept. 2009.
[4] V. R. Algazi, R. O. Duda, R. Duraiswami, N. A. Gumerov, and Z. Tang, “Approximating the head-relatedtransfer function using simple geometric models of the head and torso,” The Journal of the AcousticalSociety of America, vol. 112, no. 5, pp. 2053–2064, 2002.
[5] K. Pollack, F. Brinkmann, P. Majdak, and W. Kreuzer, “Von fotos zu personalisierter räumlicher audiowiedergabe,” e & i Elektrotechnik und Informationstechnik, pp. 1–6, 2021.
[6] K. Pollack, P. Majdak, and H. Furtado, “A Parametric Pinna Model for the Calculations of Head-Related
Transfer Functions,” in Proceedings of Forum Acusticum, (Lyon), pp. 1357–1360, 2020.
[7] P. Stitt and B. F. G. Katz, “Sensitivity analysis of pinna morphology on head-related transfer functions simulated via a parametric pinna model,” The Journal of the Acoustical Society of America, vol. 149, pp. 2559–
2572, Apr. 2021.
[8] J. C. Middlebrooks, “Individual differences in external-ear transfer functions reduced by scaling in frequency,” The Journal of the Acoustical Society of America, vol. 106, no. 3, pp. 1480–1492, 1999.

[9] V. R. Algazi, R. O. Duda, D. M. Thompson, and C. Avendano, “The CIPIC HRTF database,” in Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat.
No.01TH8575), (New York), pp. 99–102, 2001.
[10] H. Takemoto, P. Mokhtari, H. Kato, R. Nishimura, and K. Iida, “Mechanism for generating peaks andnotches of head-related transfer functions in the median plane,” The Journal of the Acoustical Society ofAmerica, vol. 132, pp. 3832–41, Dec. 2012.
[11] B. O. Community, Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender
Foundation, Amsterdam, 2018.
[12] C. Guezenoc and R. Seguier, “A wide dataset of ear shapes and pinna-related transfer functions generated
by random ear drawings,” The Journal of the Acoustical Society of America, vol. 147, no. 6, pp. 4087–4096,
2020.
[13] P. E. Bézier and S. Sioussiou, “Semi-automatic system for defining free-form curves and surfaces,”
Computer-Aided Design, vol. 15, no. 2, pp. 65–72, 1983.
[14] H. Ziegelwanger, A. Reichinger, and P. Majdak, “Calculation of listener-specific head-related transfer functions: Effect of mesh quality,” in Proceedings of Meetings on Acoustics, vol. 19, (Montreal, Canada),
p. 050017, 2013.
[15] H. Ziegelwanger, P. Majdak, and W. Kreuzer, “Numerical calculation of listener-specific head-related transfer functions and sound localization: Microphone model and mesh discretization,” The Journal of the Acoustical Society of America, vol. 138, pp. 208–222, July 2015.
[16] D. Pompeiu, “Sur la continuité des fonctions de variables complexes,” in Annales de la Faculté des sciencesde Toulouse: Mathématiques, vol. 7, pp. 265–315, 1905.
[17] D. W. Batteau, “The role of the pinna in human localization,” Proceedings of the Royal Society of London.
Series B. Biological Sciences, vol. 168, no. 1011, pp. 158–180, 1967.
[18] R. Baumgartner, P. Majdak, and B. Laback, “Modeling sound-source localization in sagittal planes forhuman listeners,” The Journal of the Acoustical Society of America, vol. 136, no. 2, pp. 791–802, 2014.
[19] P. Majdak, C. Hollomey, and R. Baumgartner, “Amt 1. x: A toolbox for reproducible research in auditorymodeling,” Acta Acustica, vol. 6, p. 19, 2022.
[20] P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, and G. Ranzuglia, “MeshLab: an OpenSource Mesh Processing Tool,” in Eurographics Italian Chapter Conference (V. Scarano, R. D. Chiara, and
U. Erra, eds.), pp. 129–136, The Eurographics Association, 2008.
[21] K. Pollack and P. Majdak, “Evaluation of a parametric pinna model for the calculation of head-relatedtransfer functions,” in Immersive and 3D Audio (I3DA) conference, 2021.
[22] J. C. Middlebrooks, “Virtual localization improved by scaling nonindividualized external-ear transfer functions in frequency,” The Journal of the Acoustical Society of America, vol. 106, no. 3, pp. 1493–1510, 1999.

ABS-0138

Aircraft noise auralization considering atmospheric and urban soundpropagation effects
Philipp SCHÄFER, Michael VORLÄNDER
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany, psc@akustik.rwth-aachen.de

ABSTRACT
When auralizing scenarios in the context of environmental noise, the simulation of sound propagation is an
important aspect. Typically, the respective simulation models base on the concept of geometrical acoustics
which enables processing a huge number of simulations even for complex scenes and higher frequencies in areasonable time. However, the respective underlying assumptions limit the scenarios to which those models
can be applied to. An example for challenging scenarios, are aircraft flyovers close to residential areas. While
there are models for urban sound propagation considering reflection and diffraction at buildings, those typicallyneglect the inhomogeneity of the atmosphere which leads to curved propagation paths. On the other hand, there
are models for aircraft scenarios considering those atmospheric effects but only consider a direct sound and aground reflection. In this contribution, we briefly present an approach which links the two propagation models.
Applications and limitations are discussed in terms of computation times and auralization quality.

Keywords: Auralization, Outdoor sound propagation, Inhomogeneous medium, Urban environment

1

INTERFACE BETWEEN ATMOSPHERIC AND URBAN MODEL

In a recent open-access publication [1], we introduced the so-called virtual source method (VSM). This approachallows to link between atmospheric and urban sound propagation models used in the context of aircraft noiseauralization. The general idea is to apply the properties of a curved free-field path calculated with the atmospheric model to sound paths of the urban model. For this purpose, a virtual source position is derived from theatmospheric path which is then used for the urban simulation as shown in Figure 1. This position is chosen, sothat the incident direction at the receiver as well as the atmospheric propagation delay τatmos are maintained forthe direct urban sound path. Thus, the distance between virtual source and receiver datmos is determined using
speed of sound of the urban domain curban :datmos = curban · τatmos(a) Deriving the virtual source position from the atmosphericfree-field path(1)(b) Urban sound paths between virtual source and receiver

Figure 1. Procedure of the virtual source method [1]The presented method is based on the following assumptions: Sound mainly propagates through the atmosphere outside the urban domain, i.e. the residual path length in the urban environment are significantly shorter.
Thus, atmospheric propagation effects caused by the inhomogeneity of the medium do not change significantly
between the different urban sound propagation paths. However, considering the scenario of an aircraft flyover,those assumptions are very reasonable as the distance between aircraft and urban environment is much larger
then the respective dimensions of the buildings.

2

AURALIZATION PARAMETERS

After carrying out the urban simulation, a set of geometric sound paths is available. However, for the auralization, we are interested in the respective acoustic properties. Thus, so-called auralization parameters are derived
from those paths, including:• Propagation delay• Geometric spreading• Air attenuation• Launch direction at source• Incident direction at receiverThose parameters are used to adjust the digital signal processing (DSP) elements during the auralization process.
To consider the properties of the atmospheric path, some parameters are calculated using specialized formulas.
For details, it is referred to the original paper [1].

3

PERFORMANCE

The method is tested using a simple flyover scenario with a receiver residing in the center of a street canyon
consisting of two box-shaped buildings as shown in Figure 2. For the atmospheric and urban simulations, theFlyover trajectory
Receiver position

2000
1000
0
-2000

0

2000

4000

6000

(a) Aircraft take-off trajectory(b) Urban model

Figure 2. Aircraft flyover scenario [1]Atmospheric Ray Tracing (ART) [2] framework and the image edge model (IEM) [3] are utilized, respectively.
The aim is to verify the main assumptions behind the VSM. Additionally, the computational speed of a combined simulation is measured. Finally, the scenario is auralized using the Virtual Acoustics (VA) [4] software.

3.1 Testing the main assumptions
The position of the virtual source based on a single atmospheric sound path and is chosen so that the acoustic
properties of the atmospheric and urban direct path match. The main assumption was that the auralization
parameters do not change significantly for the other urban sound paths. However, this might not be the case, if
the first point of interaction is far from the receiver as shown in Figure 3. To verify this assumption, additional

Figure 3. Using the virtual source for sound paths interacting with buildings might lead to a mismatch of theatmospheric properties, e.g. incident direction [1]atmospheric free-field simulations are carried out for each urban sound path using the first point of interaction
as receiver. Then, the auralization parameters are compared. The results for the incident elevation and the
spreading loss are shown in Figure 4.
Min / max
Quantiles
MedianLocation of city
Source position

2

Min / max
Quantiles
Median2000

Location of city
Source position

2000

1000

1000

0

0

2

1
0

1

-1
0
-2
-1
-3
-2
-2000

0

2000

4000

6000

(a) Elevation angle at the first interaction points

-2000

0

2000

4000

6000

(b) Spreading loss

Figure 4. Difference in auralization parameters between the atmospheric and urban paths to the first interaction
points for different aircraft positions along the trajectory [1]Generally, the parameters have the highest deviation at the beginning of the aircraft trajectory. At thisposition, the aircraft is at a low altitude, far away from the city leading to a strong curvature of the soundpaths caused by refraction. The incident elevation shows a maximum deviation of 2.5◦ . It is relevant for the
3D sound reproduction at the receiver, e.g. using HRTFs. As the localization blur is at least 9◦ [5], this does

not affect the localization. Also knowing that typical HRTF resolution for auralization are below 3◦ × 3◦ , thisdeviation seems to be insignificant especially considering that the overall result is composed by approx. 100
sound paths. Similar results can be seen for the spreading loss. Here, the deviation only comes close to thejust noticable difference (JND) of 1 dB [6] for a few aircraft positions and is otherwise close to zero.
The results of the other parameters are discussed in the original paper [1] including an additional psychoacoustic analysis. Summarizing, the uncertainties caused by the virtual source method (VSM) alone are negligible
compared to uncertainties of input parameters - like weather profiles - and the influence of time-variant changes
caused by turbulence. Thus, it is reasonable that the VSM does not affect the plausibility of the final auralization result.
3.2 Computational effort
Additionally, the computation time for the combined sound path simulation (ART + VSM + IEM) is measured.
As shown in Figure 5, the run-time is approx. 1 s for each aircraft position. The simulation of the overall
trajectory (83 s flight time, 3900 positions each referring to one audio block) takes roughly one hour. This
is not real-time capable but a reasonably fast considering the complexity of the simulation. Also, it should
be noted that while the individual simulation tools are implemented in C++, the VSM is currently a Matlab
implementation. Thus, a major portion of the computation time is spend on transferring data between thosedomains. A pure C++ implementation would be significantly faster.
Standard deviation
Mean

Location of city
Source position

2000
1000
0

1.4

1.2

1

0.8

-2000

0

2000

4000

6000

Figure 5. Computation time of combined sound path simulation for aircraft positions along trajectory [1]3.3 Auralization example
Finally, the discussed scenario is auralized using the Virtual Acoustics (VA) [4] framework. The result is presented in a video [7] that shows the aircraft, the virtual source position, as well as the urban sound paths. The
audio underlines the importance of considering urban structures for sound propagation in such scenarios. Thisis particularly relevant when the direct sound is occluded. For example, at approx. 25 s, the aircraft resides leftfrom the receiver but is perceived from the right as the major contributing sound path is the reflection on theright wall. Additionally, the huge distance between actual and virtual source position, which is up to 460 m,shows the relevance of considering the inhomogeneity of the atmosphere. Regarding the auralization quality,
there are currently some minor artifacts caused by the processing of diffracted sound paths which we like toremove in the future.

4

CONCLUSIONSThe presented virtual source method allows to auralize aircraft noise in complex environments such as urban
areas while considering the inhomogeneity of the atmosphere as well as reflection and diffraction at buildings.
This is done by linking between atmospheric and urban sound propagation model. The method was validated ina proof of concept showing that it is expected to produce plausible results. This is underlined by the provided
auralization example.
In future work, we would like to improve the auralization quality and reduce the discussed artifacts. Finally,
it is desired to apply this method to generate complex urban scenes with multiple sound sources, e.g. to be
used in listening experiments.

References[1] Schäfer P, Reich L, Vorländer M. Linking Atmospheric and Urban Auralization Models. Acta Acustica.
2022;6:28. https://acta-acustica.edpsciences.org/10.1051/aacus/2022021.
[2] Schäfer P, Vorländer M. Atmospheric Ray Tracing: An Efficient, Open-Source Framework for Finding Eigenrays in a Stratified, Moving Medium. Acta Acustica. 2021;5:26. https://acta-acustica.
edpsciences.org/10.1051/aacus/2021018.
[3] Erraji A, Stienen J, Vorländer M. The Image Edge Model.
acta-acustica.edpsciences.org/10.1051/aacus/2021010.

Acta Acustica. 2021;5:17.

https://[4] Institute for Hearing Technology and Acoustics. Virtual Acoustics – A Real-Time Auralization Framework
for Scientific Research; 2021. http://virtualacoustics.org/VA.
[5] Blauert J. Spatial Hearing: The Psychophysics of Human Sound Localization. Rev. ed ed. Cambridge,
Mass: MIT Press; 1997.
[6] Fastl H, Zwicker E. Psychoacoustics: Facts and Models. 3rd ed. No. 22 in Springer Series in Information
Sciences. Berlin ; New York: Springer; 2007.
[7] Institute for Hearing Technology and Acoustics. Virtual Source Method - Aicraft Flyover Auralization Considering Atmospheric and Urban Propagation; 2021. https://youtu.be/jNWJOKI9sTc.

ABS-0300

Laboratory setup for assessing physiological stress buildup and recovery
associated with noise annoyance using virtual reality and ambisonicloudspeaker reproduction
Fotis GEORGIOU(1) , Claudia KAWAI(2) , Reto PIEREN(3) , Beat SCHÄFFER(4)(1-4) Empa, Swiss Federal Laboratories for Materials Science and Technology, Laboratory for Acoustics/Noise Control,

Dübendorf, Switzerland, fotis.georgiou@empa.ch

ABSTRACT
Urbanization with continuously increasing population and mobility results in increasing noise pollution and a
decline of green spaces, although the latter are important for restoration from stress. The Swiss research project
RESTORE (Restorative potential of green spaces in noise-polluted environments) aims to assess the effects ofgreen spaces as facilitators and noise as impediment to recover from stress. Besides field surveys and experiments, highly controlled laboratory VR experiments are to be conducted. This paper presents a framework forthe design of a perceptual VR experiment inside a laboratory environment, to be used for evaluating the effects
of: 1) noise on stress buildup and 2) green spaces on reducing noise annoyance and promote stress recovery.
The framework is split into four parts: 1) 360° video and higher order ambisonic (HOA) field recordings; 2)
Calibrated spatial sound reproduction with a spherical loudspeaker array; 3) Coupling of HOA audio and video,
which will be played back via head mounded display inside a game engine and the design of a graphical user interface for a perceptual experimental questionnaire; 4) Measurement of physiological stress using saliva cortisoland skin conductance. For the latter, an outlook is given on the design of the planned experiments.

Keywords: Virtual reality, Ambisonics, Soundscapes

1

INTRODUCTION

The interdisciplinary Swiss research project RESTORE (Restorative potential of green spaces in noise-polluted
environments; www.restore-project.ch) aims at investigating the build-up of noise-related stress in combination with the restorative effects of green spaces and urban environments. The present work is part of
RESTORE. It uses a controlled VR environment to systematically investigate the build-up of and the restorationfrom short-term stress via skin conductance and salivary cortisol measurements.
The great advantage of using VR to investigate the effect of different environments on different physiologicalparameters is that all the measurements/experiments can be done under control laboratory conditions i.e. one
can control the sound levels, the visual information and systematically change them to investigate their effects
on stress. The challenge is to reproduce all the audiovisual information as accurate as possible in order toachieve a fully immersive experience.
The main focus of this paper is to present the methodology used to record the audiovisual information ondifferent urban environments and reproduce them inside the lab. Special attention is given on the spatial audio
reproduction, which was realised using ambisonics, and tested in a validation listening experiment to evaluate
different ambisonic decoders.
The paper is organised as follows. Section 2 outlines the theory for the audiovisual reproduction of urban
environment using ambisonics loudspeaker reproduction for the audio and 360 video for the visuals. Section 3
presents the evaluation of 4 different ambisonic decoders for urban soundscape reproduction. Section 4 presents
the methodology followed for the audiovisual recordings and the experimental setup for the future stress/restoration experiments including a short description of these experiments.

2

THEORY

2.1 Loudspeaker vs headphone reproduction of soundscapes
In this research it was decided to use loudspeaker reproduction. Recent work from Hong et al. (1) on thequality assessment of acoustic environment reproduction methods for VR in soundscape applications showed thatfirst order ambisonics (FOA) reproduction via 2D loudspeaker array was rated overall as a better reproduction
method compared to the FOA via headphones.
2.2 Introduction to ambisonics
Ambisonics has some properties which make it more attractive compared to other loudspeaker reproduction
techniques such as Wave Field Synthesis (2–4). These are the following: 1) There is an associated recording
technology, which is not the case with Wave Field Synthesis; 2) Encoding is independent of decoding meaningthat you can encode 3rd order ambisonics but if the speaker layout cannot support it you can decode it to alower order; 3) It can provide a smooth listening experience from all directions; 4) The number of required
speakers for the playback is only dependent on the maximum ambisonic order and not on the number of soundsources which are present on the soundfield; 5) Since both recording and reproduction are defined on a spherethe soundfield can be rotated. However, it has also some disadvantages, which have pushed back ambisonics
from becoming a more commonly used spatial audio method (4, 5). Firstly, the sweet spot is small. Secondly,
perceptual artifacts such as coloration can occur due to the fact that the reproduction is implemented by coherent
loudspeaker signals. A good overview of the ambisonics can be found in (6).
Ambisonics recordings are taken using spherical microphone arrays. There are various first order sphericalmicrophone arrays available in the market such as the SoundField SPS200 microphone and the RØDE NT-SF1
but only few higher order ones such as the Zylia ZM-1 (3rd order) and the mhacoustics Eigenmike EM32 (4thorder). The signals captured via these microphone arrays cannot be used directly for playback. They mustbe first processed in the spherical harmonics domain (encoding) to produce the B-Format signals. The main
challenges at this stage is the design of the radial filters to compensate for the radial structure of the sound field
captured with the microphone, which basically scale the amplification gain of the spherical harmonics modes.
The challenge of these radial filters is to achieve a balance between the recovery of the sound field coefficients
while maintaining the noise amplification of the encoded signals at acceptable levels (7, 8). Companies suchas SoundField, Zylia and mhacoustics provide their own software for encoding the signals captured from theirmicrophones. There are however, available toolboxes and plug-ins such as the SPARTA suite developed by
Aalto University (9) that allows the users to experiment with different encoding methods and decide which one
suits their needs the best.
Decoding of the ambisonic signals a into loudspeaker feeds s is implemented using the decoding matrix D
as follows:
s=Da

(1)

where s = [s1 , s2 . . . sL ]T , D is an L × (N + 1)2 matrix and a = [a1 , a2 , . . . a(N+1)2 ]T . The goal the decoder is toachieve the best possible reproduction for the specified loudspeaker layout (4). Various decoding techniques
have been developed. Two traditional decoding methods are the sampling decoding and the mode matching
decoding. These methods are straightforward to implement but are not robust when the speaker setup is irregular or incomplete, which is the case in most real speaker layouts. Recent decoding methods are the All-roundambisonic decoding (ALLRAD) and the Energy-preserving decoding (EPAD) (3,10). ALLRAD uses a combination of ambisonic decoding and vector based amplitude panning (VBAP (11)). Firstly, it decodes the ambisonics
signals to an optimal virtual loudspeaker layout (spherical t-designs (3)) and then uses VBAP to map the virtual loudspeaker signals to the real speakers. That property of ALLRAD gives this method a lot of flexibility.
EPAD’s main goal is to achieve constant energy across all directions. For incomplete spherical loudspeakerlayouts EPAD is based on the design of a new set of modified spherical harmonics, which are orthogonal on
incomplete spheres. These new basis functions need to be used in both the encoding and the decoding stage,
which adds to this method some complexity compared to ALLRAD. Results presented from Zotter et al. (6)show that for hemispherical or incomplete spherical loudspeaker arrays ALLRAD is the preferred method be-cause it provides lower directional error, greater flexibility and it is simpler to implement compared to EPAD.
The drawback is that ALLRAD can produce a loudness increase in the front and back direction.
The design of the decoders can be either frequency dependent or frequency independent. Some earlier firstorder decoding methods can incorporate frequency dependency by using self-filters or by using dual-band decoding matrices (one for the low and one for the high frequencies) together with phase-matched band-splitting
filters (12). The goals of such decoders are to achieve the following: 1) Constant amplitude gain (important for
low frequencies) and energy gain (important for high frequencies) across all angles; 2) Optimizing the velocity
localization vector rV , which is related to low frequency localization, and the energy localization vector rE,
which is related to high frequency localization (rV and rE have been defined by Gerzon (13)). HOA methods
such as EPAD and ALLRAD are not frequency dependent and their main focus is the optimization of the rE .
The performance of the decoders can be also improved with the use of the max-rE weights, which have been
introduced by Daniel (14). These are a set of weights that optimize the energy concentration in the expected
direction by weighting each spherical harmonic component by an order depended gain, which helps in preserving the total energy in the sweet spot (2). It is also suggested in literature to create multi-band HOA decoders.
For example one can use at low frequencies a non-weighted decoder that focuses on amplitude preservation andat high frequencies a decoder with max-rE weights. A method to estimate the cross-over frequency between
the decoders is suggested in (2). In cases when the objective is to decode ambisonics recordings one can use aseparate decoder for each of the frequency ranges of the specific ambisonics microphone. According to (15) the
Zylia mic delivers 2nd order signals above 250 Hz and 3rd order above 900 Hz. Based on these considerations,an ideal frequency dependent HOA decoder may consists of a 1st order non-weighted decoder for frequencies
up to 250 Hz, a 2nd order decoder with max-rE weights for frequencies between 250 Hz and 900 Hz anda 3rd order with max-rE weights above 900 Hz. These frequency limits are not exact since apart from thecharacteristic of the microphone array they also depend on the choice on the encoding filters.
2.3 Monoscopic vs stereoscopic 360 video of urban environments
Monoscopic video means that during playback via the HMD the same image is directed to both eyes by projecting a flat image to a sphere. As a result the depth perception is lost. Nowadays, there are cameras available
in the market (at much higher price) that offer stereoscopic video, which resembles how humans see. Although
the stereoscopic video can provide a more immersive experience, it is more challenging to capture and edit
(stitching etc) especially in dynamic scenes (16). Badly produced 360 video can cause big discomfort to theviewer such as nausea and headaches. For video reproduction of urban environments, where objects such as
vehicles and humans move at different distances and speeds, using a monoscopic camera is the safer choice.

3

AMBISONIC DECODERS EVALUATION

3.1 The reproduction room
At the Acoustics/Noise Control Lab at Empa we operate the listening test facility AuraLab (Figure 1) where
often psychoacoustic experiments are conducted. It is an acoustically-treated room with 20 satellite loudspeakers. The speakers are placed at 4 elevation angles (−28◦ ,0◦ ,30◦ ,60◦ ) and at each elevation plane there are 5
loudspeakers at 36◦ , 108◦ , 180◦ , 252◦ and 324◦ . Four distributed sub-woofers on the room’s corner are also
installed with a crossover frequency of 100 Hz. The reverberation time in the mid-frequencies is 0.11 s and thebackground noise level is below 7 dB(A).
3.2 Tested decoders for evaluation
The ability of 4 different decoders in creating plausible reproduction of urban soundscapes of relatively low
sound pressure levels (SPL) was evaluated. It should be noted that all the encoding was done with the ZYLIA
Ambisonics Converter plug-in using N3D normalisation and ACN channel ordering. For the FOA decoder theambisonic signals were converted to FuMa channel ordering and also the N3D weights were converted to FuMa
weights. The max-rE weights used in this paper were computed as in (3).
The decoders are the following:
1. 2D FOA max-rE decoder using only the 5 speakers at 0◦ elevation, which will be referred to as FOA2D

2. 3rd order HOA max-rE ALLRAD decoder for the hemispherical loudspeaker setup (15 speakers layout),
which will be refered to as HOAhemi
3. 3rd order HOA max-rE ALLRAD decoder making use also of the ground speakers at −28◦ elevation (20speakers layout), which will be referred to as HOA f ull
4. Dual-band decoder in which for frequencies up to 400 Hz a 1st order unweighted ALLRAD decoder isused and for higher frequencies a 3rd order max-rE ALLRAD decoder is used (20 speakers layout), which
will be referred to as HOA f ull,dualband
Equation 2 is used for the FOA2D decoder.
sn = K0 ·W + K1 · [X · cos(ϕn ) +Y · sin(ϕn )](2)thwhere sn are the loudspeaker√ signals for the n loudspeaker located at ϕn , W,Y, X are the B-format channels
and the weights K0 = K1 = 2/L, where L = 5 is the number of speakers. The reason for setting the weights
K0 = K1 is to yield a max-rE decoder, which works well for frequencies 400 Hz and above and also provides
the best reproduction if no frequency dependent FOA decoding is applied (17).

Figure 1. AuraLab listening test facility at the Laboratory for Acoustics/Noise Control at Empa.
The HOA decoding is implemented using the ambisonics Matlab library developed by Archontis Politis(7). In all 3 HOA decoders two imaginary speakers are inserted at ±90◦ elevation to fill in the gaps in the
loudspeaker layout as recommended in (6, 18). The signal of the imaginary speaker’ signal is disposed assuggested in (6). However, whether the signals from the imaginary speakers should be decorrelated and thenmixed into the nearby speaker remains an open question (18).
The minimum required amount of loudspeakers for 3rd order ambisonic reproduction is L = 16 speakers.
However, in hermispherical loudspeaker layouts, such us the one used with decoder HOAhemi , the bottom half
can be omitted when using the ALLRAD or the EPAD with the modified basis functions meaning that thislayout could produce up to 4th order ambisonics on the top hemisphere (6). However, since the Zylia ZM-1
can provide up to 3rd ambisonics signals there is no benefit in that. Some psychoacoustic evaluations in 2D
HOA in the literature (an overview of this research can be found in (7, 19)) have shown that artifacts such as
comb-filtering and spectral impairment occur if the number of the loudspeakers is larger than the one required(hence more interference) from the the ambisonic order. Therefore, one could claim that using 15 loudspeaker
for producing a 3rd order soundfield in the hemisphere might be a lot. However, the amplitude, energy, velocity
vector and energy vector plots showed that the results were better when all loudspeaker positions where used.
The HOAhemi and HOA f ull make use of the max-rE for the whole frequency range (3). In earlier studies,
there were no speakers bellow the ear level in Auralab. Analysis of the amplitude and energy plots of thedecoder HOAhemi vs HOA f ull in Figure 2 showed that the reproduction could potentially improved (also in thehorizontal plane). Another reason for adding the ground speakers at −28◦ is that in some recording locations

some noise sources were bellow the ear height but with the hemispherical array in some occasions they werereproduced above the ear height.
In HOA f ull,dualband the max-rE weights were only applied in the mid-high frequency decoder. The cross-overbetween the two decoders is done using a 4th order Linkwitz-Riley filter.

Figure 2. Amplitude (a) and energy (b) plots of HOA f ull and amplitude (c) and energy (d) plots of HOAhemi .
The dots indicate the locations of the speakers and the crosses the locations of the imaginary loudspeakers.
3.3 Decoder equalisation
Once a person sits in the center of a loudspeaker array his/her head affects the soundfield by shielding thesound waves traveling between the left and right side. This shielding effect becomes more dominant at higher
frequencies since the wavelengths become smaller compared to the size of the head. To remove any directionindependent effects of the ambisonic reproduction, diffuse-field equalisation is applied. More specifically, adiffuse-field equalisation shelf-filter is applied on all 4 decoders based on virtual loudspeaker simulation and
HRTFs (H). The computation of the equalisation shelf-filter is described in this subsection. Firstly, ambisonicchannel signals are computed for both 1st order FuMa channel ordering and weighting and 3rd order ACN
channel ordering and N3D weighting for multiple coordinates i on the surrounding sphere. The ambisonics
signals are computed for azimuth angles φi starting from 0◦ to 360◦ and for elevation angles θi from −30◦
to 90◦ . Both the azimuth and elevation increments are set to ∆φ = ∆θ = 10◦ . In total M = 468 positions
around the sphere are encoded (the encoded signal is a Dirac delta). The 1st order ambisonics signals for eachposition i are decoded for the horizontal 2D loudspeaker layout using the FOA2D decoder, the 3rd order signals
are decoded for the hemispherical loudspeaker layout using the HOAhemi and for the layout with the groundspeakers at −28◦ elevation using the HOA f ull and the HOA f ull,dualband . Afterwards, each of the loudspeakersignals s for each position i is convolved with the corresponding head related impulse response (h) of the
SADIE HRTF database (20) (KEMAR dummy head). The measured angles of the HRTF database do not
perfectly match the angle of all loudspeakers. They are on average 1.5◦ off, thus, the virtual representation of
the layout is not 100% correct. Next, the diffuse-field energetic average of the ambisonically rendered HRTFs(EDFamb ) is computed with the following equation:
1 M
EDFamb = · ∑ F
M i=0

L

!2

∑ (si,n ∗ h(ϕn , θn ))· cos(θi )

(3)

n=1

where L the number of speakers in the layout and si,n the nth loudspeaker (with coordinates (φn ,θn )) signal forthe ith position. The diffuse-field energetic average of the measured HRTFs at the M coordinates is computedwith equation 4. From the selected M = 468 positions in the above mentioned analysis 35 HRTFs were missing
from the SADIE HRTF database, so, they are excluded from the analysis. A cosine weight is applied to eachHRTF based on its elevation angle in order to ensure no specific direction will be over-represented in the
average. In binaural technology literature it is common to use the solid angle weight and not the cosine weight
but since in this case ∆θ = ∆φ = 10◦ the effect is the same. Finally, a second-order infinite impulse response(IIR) shelf-filter is designed (using Matlab’s designShelvingEQ function) for each decoder to equalise it andbring its diffuse-field spectrum closer to the measured spectrum as shown on Figure 3. The approach followed
here is very similar with the one from McKenzie et al. (21), which was developed for binaural reproduction ofambisonics. The main difference is that instead of using a shelf-filter they created an equalization filter basedon the inverse diffuse-field response in order to flatten it.
EDFmeas =1 M· ∑ |H(ϕi , θi )|2 · cos(θi )
M i=0

(a)

(b)

0

6

-5

Magnitude (dB)

Magnitude (dB)

(4)

-10
-15

4

2

-20
0
-25
102

3

10
Frequency (Hz)

10

4

102

103
Frequency (Hz)104

Figure 3. (a) Diffuse-field energetic average of the measured HRTF (solid), of the ambisonically rendered using
the HOA f ull (dotted) and its equalised version (dashed); (b) The shelf-filter used to equalise the loudspeakersignals computed with the HOA f ull decoder.
3.4 Validation experiment and decoder choice
We played 3 different urban soundscape recordings (audio snippets containing traffic, human voices, or naturesounds of 10 s in length) to 13 participants (MAge = 35.1±10.7 years; 10 male). Each participant heard thesoundscape recordings decoded with the four different decoders (FOA2D , HOAhemi , HOA f ull , HOA f ull,dualband ).
The task was to rank the differently decoded sound snippets as the 1st , 2nd and 3rd most plausible (the unranked
sample was considered as the least plausible). The first trial was the training trial and was always the samefor all subjects; it was not used for the analysis. The other 3 trials were randomised for each participant. The
subjects could listen to the 4 samples at each trial as many times as they wanted. The average duration of thetest was 7 min 40 sec. The results showed that the recordings decoded with HOA f ull,dualband were selected mostoften as the most plausible and least often as the least plausible. On the contrary, the recordings decoded with
FOA2D were selected least often as the most plausible and most often as the least plausible. In general, all the
HOA decoders performed very similarly but significantly better that the FOA2D .

4

Data collection and experimental setup

4.1 Audio and video recordings
The Kandao QooCam 360 camera was used to capture the video. The QooCam and the Zylia were placed
at 1 m distance between them (see Figure 4). The Zylia mic was connected to a HP Elitebook Core i7 8th
Gen laptop with Windows 10 operation system via a 20m USB 3.0 active extension cable and the recordings
were taken using REAPER. A reference mic (B&K 4006 with diffuse-field grid) was also placed 30 cm from
Zylia. The reference mic was connected via a 20 m XLR cable to an NTi Audio XL2-TA/Class 1 sound level
meter, which recorded both SPLs and the audio signal. The camera and both mics were placed at 1.20 m
height above ground, which is representative to the eye and ear height of a seated person. The air temperature,
relative humidity and wind speed were also monitored using the Rotronic A1 HYGROMER hydrometer and theWindmaster 2 wind speed meter. The Zylia recordings were taken at a samplerate of 48 kHz and the videos
at 8K resolution and 30 FPS. The sound level meter stored SPLs every 1 s and the audio was also recorded
at 48 kHz. Windscreen was always placed on both mics (the Zylia original windscreen was used for the Zyliamic). Before each recording the reference mic was calibrated. The 1 kHz calibration was also recorded in order
to use it at post-processing to calibrate the reference recording, which is then used to calibrate the ambisonicrecording.

Figure 4. Field measurement setup with the sound pressure sensor with a foam windscreen (left), the HOA
microphone with a fur windscreen (center) and the 360° camera (right).
In total 10 locations around Zurich have been recorded (7 quiet green spaces, 2 noisy green spaces, 1 nongreen quiet space). The LAeq in these locations varied from 45 dB to 66 dB. The duration of each recording
is between 30-35 min to have sufficiently long recordings for the planned laboratory VR experiments (seesubsection 4.3).
4.2 VR environment
The VR environment including the experimental GUI is implemented within the Unity game engine. The audio playback is carried using the BASS audio library (https://www.un4seen.com/). More specificaly the
BASS.dll and BASSASIO.dll are imported into the Unity project. The HOA f ull,dualband is used for the reproduction (see section 3.4). The video playback is done using the VideoPlayer class of Unity and video is playedback via the Oculus Quest 2.
4.3 Application: Planned stress/restoration experiments
We will utilize the developed VR methodology in a full-factorial between-participants laboratory study researching stress restoration. First, participants enter a 10-min stress phase, where they are exposed to different SPLsof road traffic noise. One half of participants attentively listens to the stimuli, while the other half completes
a series cognitive tasks during the noise presentation (mental arithmetic, Stroop task). By including this additional cognitive stress component, we can investigate the effect of noise on cognitive performance. After the
stress phase, participants enter the restoration phase, where they can relax in a virtual environment for 20 min.
Half of the participants are immersed in a natural green space recording, the other in a calm urban location.
A continuous electrodermal activity measure will be applied and salivary cortisol will be taken (baseline, poststress, and post-restoration) to compare physiological responses between the different noise SPLs and restoration
settings. The results will give insight into (1) stress build-up during noise exposure at different SPLs, and (2)the restorative potential of green vs urban spaces – information that will be useful to public authorities forestablishing noise regulation guidelines on the one hand, and to city planners for improving public health andwellbeing of urban populations on the other.

5

CONCLUSIONSIn this paper a framework for the design of a perceptual VR experiment inside a laboratory environment, tobe used for evaluating the effects of: 1) noise on stress buildup and 2) green spaces on reducing noise annoyance and promote stress recovery is presented. The audio reproduction in the VR environment is done usingambisonics. Listening tests were conducted to evaluate the plausibility of urban soundscape recordings decodedwith 4 different decoders. The results showed that the 3D HOA decoders performed significantly better than the
2D FOA decoder.

ACKNOWLEDGEMENTS
The study is funded by the Swiss National Science Foundation (project RESTORE, grant number CRSII5_193847/1).
We would also like to thank Dr. Archontis Politis for the valuable discussion and his advice on ambisonics.

REFERENCES
1. Hong JY, Lam B, Ong ZT, Ooi K, Gan WS, Kang J, et al. Quality assessment of acoustic environment reproduction methods for cinematic
virtual reality in soundscape applications. Build Environ. 2019;149:1–14.
2. Favrot S. A loudspeaker-based room auralization system for auditory perception research. PhD Thesis. Technical University of Denmark; 2010.
3. Zotter F, Frank M. All-round ambisonic panning and decoding. J Audio Eng Soc. 2012;60(10):807–20. .
4. Zotter F, Frank M, Haar C. Spherical microphone array equalization for Ambisonics. In: DAGA. Nürnberg, Germany; 2015. p. 486–9.
5. Scaini D, Arteaga D. Decoding of higher order Ambisonics to irregular periphonic loudspeaker arrays. In: Audio Engineering Society 55th
International Conference. Helsinki, Finland; 2014.
6. Zotter F, Frank M. Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality.
Springer Open; 2019
7. Politis A. Microphone array processing for parametric spatial audio techniques. PhD Thesis. Aalto University; 2016.
8. Politis A, Gamper H. Comparing modeled and measurement-based spherical harmonic encoding filters for spherical microphone arrays. In:
IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. New Paltz, New York; 2017. p. 224–8.
9. McCormack, L, Politis, A. SPARTA & COMPASS: Real-time implementations of linear and parametric spatial audio reproduction and processing methods. In: Audio Engineering Society International Conference on Immersive and Interactive Audio. York, U.K.; 2019.
10. Zotter F, Pomberger H, Noisternig M. Energy-preserving ambisonic decoding. Acta Acust united with Acust. 2012;98(1):37–47.
11. Pulkki V. Virtual Sound Source Positioning Using Vector Base Amplitude Panning. J Audio Eng Soc. 1997;45(6):456–66.
12. Heller AJ, Lee R, Benjamin EM. Is my decoder Ambisonic? In: Audio Engineering Society 125th Convention. San Francisco, California;
2008.
13. Gerzon MA. General Metatheory of Auditory Localization. In: Audio Engineering Society 92nd Convention. Vienna, Austria; 1992.
14. Daniel, J. Representation de champs acoustiques, application a la transmis- sion et a la reproduction de scenes sonores complexes dans uncontexte multimedia . PhD Thesis. Universite Paris 6; 2000.
15. McKenzie T, McCormack L, Hold C. Dataset of Spatial Room Impulse Responses in a Variable Acoustics Room for Six Degrees-of-Freedom
Rendering and Analysis. 2021.
16. Guillemaut, J. Y., Sarim, M., & Hilton, A. (2010, September). Stereoscopic content production of complex dynamic scenes using a widebaseline monoscopic camera set-up. In: IEEE International Conference on Image Processing. Hong Kong; 201017. Benjamin E, Lee R, Heller A. Localization in horizontal-only Ambisonic systems. In: Audio Engineering Society 121st Convention. San
Francisco, California; 2006
18. Heller AJ, Benjamin E. The Ambisonic Decoder Toolbox: Extensions for Partial-Coverage Loudspeaker Arrays. In: Linux Audio Conference.
Karlsruhe, Germany; 2014.
19. Zotter F, Pomberger H, Noisternig M. Ambisonic decoding with and without mode-matching: a case study using the hemisphere. In: International Symposium on Ambisonics and Spherical Acoustics. Paris, France; 2010.
20. Kearney, G, Doyle, T. An HRTF database for virtual loudspeaker rendering. In: Audio Engineering Society 139th Convention. Audio Engineering Society. New York, New York; 2015.
21. McKenzie T, Murphy DT, Kearney G. Diffuse-Field Equalisation of binaural ambisonic rendering. Appl Sci. 2018;8(10).

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0337

Using Spatial Water Sound Sequences for Traffic Noise
Masking：Correlation Analysis of Subjective Evaluation and Neural
Measurements
Jian LI; Luigi MAFFEI; Aniello PASCALE; Massimiliano MASULLO
Department of Architecture and Industrial Design,
Università degli Studi della Campania “Luigi Vanvitelli”, Italy

ABSTRACT
Water-based masking sound has been proven effective on environmental noise control. But less is known
about the effects and the process of the spatial configuration of water sounds on the perception of thesurrounding environment. Through an immersive Spatial audio system, a virtual traffic noise environment
base on an existing urban park was created in the lab. And three different spatial settings of water-soundsequences were added into the virtual acoustical environment to investigate the role of spatialization of watersound sequences on traffic noise masking. The neural responses and subjective evaluations of twenty
participants were collected with a portable electroencephalogram (EEG) device during the spatial soundplayback time. The correlation analysis between subjective ratings and EEG indicators including spectral
power, cognitive indexes and brain network connectivity were conducted.
The more positive effects on noise masking induced by the spatial setting of water sound sequences had been
illustrated from the collinear relationship between the objective descriptors, the brain networks activities in
the alpha band and positive emotional saliency scores.
Moreover, the spectral power of the gamma band and the theta alpha ratios used as the cognitive load index
had showed a significant linear relationship with the negative emotional saliency.
Keywords: Water Sound; Noise Control; Spatialisation; Traffic Noise; EEG

1. INTRODUCTION
Noise pollution is a big concern for urban designers and landscape managers since it has been
proven to impact public health physically and mentally (1). The introduction of natural sounds (e.g.
water sounds, bird songs) into noisy urban environments has been treated as an effective strategy fornoise reduction and abatement (2). Among them, water sounds are commonly used to mask traffic
noise varying in its sound features (3). Many studies have tested various water sounds at different
signal-to-noise ratios to optimise the soundscape quality and the desired sound levels to set the watersounds playback for noise mitigation (4–6). However, the spatial features of water sound are hardly
discussed. Several studies have found that the spatialisation of sounds could improve the perceived
sound quality (7,8) and spatial consciousness (8). Masullo and his colleagues (9) used Immersive
Virtual Reality technology to investigate the effects of combining audio and visual elements of
installations with water features on traffic noise mitigation in urban green parks. They confirmed theinformational masking with water's sounds at levels 3 dB lower than the road-traffic background noise
improved the subjective perception of the environmental quality of urban parks. Installations withwater features improve their restorativeness on escaping and fascination components. They also
compared the effect of the simulated and real water features on the restorativeness in urban parks. The
results showed that water features simulated with audio-visual installations have significant positiveeffects on the Fascination and Being-Away components of restorativeness (10,11). Hong and others(12) explored the effects of spatial separations between target noise and water sound on perceived

loudness of target noise and overall soundscape quality. The results indicated that the spatial
separation between traffic noise and water sound can improve the overall soundscape quality anddecrease the perceived loudness of the noise.
While there are already existed various studies about the neural responses of different urban space
including green space (13,14), indoor environments (15–17), contemplative landscape (18) withperceived soundscape qualities, most of them used Electrophysiological (EEG) measures as neuralindicators of sonic environments related to the comfort and restoration of individuals. But thecognitive process referring to auditory attention and noise masking and their relationship withperceived qualities and subjective responses has been less investigated. The changes of whole-brain
functional networks have been used for indicating the process of the spatial attention in speech field(19) and emotion regulation in marketing researches (20,21). Szalárdy et al.(22) used functional
connectivity of EEG signals between different brain regions to investigate the neuronal correlates of
informational and energetic masking in a multi-talker situation. They found energetic masking was
predominantly associated with stronger connectivity between the frontal and temporal regions at thelower alpha and gamma bands, and informational masking was associated with a distributed network
between parietal, frontal, and temporal regions at the theta and beta bands. Our previous study (23)investigated the overall mental effects and attentional process of spatialized water-sound sequences
related to informational masking of traffic noise through the spectral power and connectivity analysisof EEG signals. The results showed higher relative power of the alpha band and greater alpha-beta
ratio among water sound conditions compared to traffic noise. And different spatial settings of watersound sequences induced different attention network changes. The two position switching watersounds brought more attentional network activations than other water sound sequences related to noisemasking. But correlation between the mental processes induced by the mask sounds in noisy
environments and soundscape qualities with corresponding emotional outcomes are remain unknown.
In this paper, water-sound sequences with different spatial settings were used to create a traffic
noise environment base on an existing urban park. And the masking effects of spatial water sound onroad traffic noise perception were investigated by correlation analysis of the neural responses andsubjective evaluations collected from twenty participants. The correlation results could inspire theurban designer and policy maker to have more thoughts about urban soundscape design and noise
control regarding acoustic comfort, human health and well-being.

2. METHODS
2.1 Experimental Design
A within-subjects experimental design was used. The independent variable (IV) was the spatial
settings of the water sounds. Four levels of IV were defined: Frontal-fixed Position Water sound(FPW), a Two-position Switching Water sound (TSW) and a Four-position randomised Moving Water
sounds (FMW), and no water sound (RTN), all of them combined the Road Traffic Noise on frontal
position as background. The two-position pair of TSW included four spatial settings: frontal-left pair,frontal-right pair, back-left pair, back-right pair. Dependent variables (DV) were the neural responses
collected by a wearable electroencephalogram device during each condition and subjective reports
obtained by questionnaires at the end of each condition.
2.2 Sound Environment Settings
The sound sequences included a 3-minute traffic noise (LAeq: 65 dB(A), recorded by a Zoom H6
Hand-Recorder device with a Soundfield SPS200 microphone) as background noise (BGN), and a 5s
water stream sound (recorded by a Zoom H6 Hand-Recorder with a Rode NTG-2 microphone). The
sound equivalent level of the water stream sound was set at -3 dB with respect to the backgroundtraffic noise for optimizing noise masking. To match with the same duration of BGN, the water soundsequence combined repeated 5s of water stream sound with 2s fade-in and fade-out both ends, andthen was cross-splitted to two streams with 2s overlap (see Figure 1-Temporal settings). They wereplayed back within the Sens i-Lab, the multisensory laboratory of the Department of Architecture andIndustrial Design of the Università degli Studi della Campania "Luigi Vanvitelli" through the Astro
Spatial Audio, an object-based audio system (ASA) which drives 25 Adorn A55 Martin Audio and 2
Sx110 Martin Audio and rendered by SARA II Premium Rendering Engine. Two different kinds ofsound source objects were used for the playback: a plane wave object, reproducing the wavefront of
the sound propagation from road traffic noise, and a point source object, reproducing the water sound(Figure 1-Spatial settings). The spatial location of each sound source for each condition was set as
follow (Figure 1-Spatial Settings). The experimenter controlled the order of these sound sequences
via browser-based GUI during the listening test. The listener was sitting at the centre of the test roomof the Sens i-Lab, at about 3.5 meters from position of the virtual sound sources. The audio stimuli at
listener’s position were recorded using a dual channel system Sympnonie and an Mk1 Cortex manikin.
They reproduced realistic auditory scenarios of the sound level about 57 dB(A), as those measured
inside an existing urban park.

Figure 1-The spatial and temporal settings of the sound sources
2.3 Experimental Procedure
Twenty subjects gave informed consent and were instructed to sit in the center of the test room,
being immersed in virtual sound environments. Before the formal experiment, the subject fulfilled
two pages of the initial questionnaire, containing basic information such as age (average: 30; SD:5.90), gender (Male: 12; Female: 8), working environment, and noise sensitivity (Weinstein Noise
Sensitivity Scale) (24) (average score: 3.73; SD: 0.50). After wearing the portable EEG device andpassing the impedance check of EEG electrodes, the subject was asked to listen to five sequences (twoTSW condition randomly selected from four TSW conditions) with a comfortable sitting position andeyes open in the pre-defined balanced order. Each sequence lasted 3 minutes; then, the subject must
fill a self-reported questionnaire. The first part of the questionnaire was focused on assessing generalcharacteristics of the sound environment including naturalness, mechanicalness, smoothness,rhythmicalness, spaciousness, and familiarity, while the second investigated the emotional reactionsof sounds. The latter part of the questionnaire combined items deriving from the circumplex model ofsoundscape perception with others focused on the emotional feeling of the sound environment (25).
After finishing the questionnaire, the subject informed the experimenter to play the following soundsequence. Finally, the subject took 1-minute rest with his/her eyes closed. The neural activities during
this period were used for baseline correction for EEG analysis.
During the whole process, the brain data of each subject were continuously recorded by DSI-24
wireless EEG headset with 20 dry electrodes signals referenced to Pz electrode at locations
corresponding to the 10-20 International system (see Figure 2). The light and temperature in the lab
were kept constant during the test. The EEG data were sampled at 300 Hz and streamed from themeasurement device to the recording laptop using the Lab Recorder application based on the Lab
Streaming Layer protocol (LSL) to synchronize the neural data with sound sequences. The EthicalCommittee for Scientific Research of the Department approved the protocol.

Figure 2-The experiment scenario of data collection
2.4 Data Analysis
The continuous EEG data were imported into MATLAB and EEGLAB toolbox (26) for data
cleaning. The data of two subjects were excluded because of less clean data (both the percentage ofinvalid data were higher than 50%, the average of the valid data was 87.06%). Then a 1-45 Hz bandpass filter was applied for each subject’s data. After re-referencing the EEG signal to the average(except for A1 and A2 mastoid electrodes), their independent components were calculated using theInfomax algorithm. Eye-blink and ocular movements artifacts were deleted based on the standard
topographic profiles of the individual components and the distinctive temporal pattern. After theremoval of eye-movement artifacts, the EEG data during each sound's perception for each subject
were extracted.
The cleaned EEG data were analyzed using Matlab and FieldTrip toolbox (27). Time–frequencyresolved activities were obtained using the multitaper method (4 cycles width) based on Hanningsequences between 1 and 45 Hz (stepsize: 1 Hz), from which the average powers of each frequency
band (delta band was defined as the range of 1 to 4 Hz; theta band: 4-8 Hz; alpha: 8-13 Hz, beta: 1330 Hz, low-gamma: 30-45 Hz) were derived. The interested electrodes were divided the interested
electrodes into five regions: the frontal (Fp1, Fp2, F3, F4), left temporal (F7, T3, T5), central (Cz, C3,
C4), right temporal (F8, T4, T6) and posterior regions (P3, P4, O1, O2), respectively.
The relative power of each given band divided by the sum of power from 1 to 45 Hz was calculated
as follow formula:𝑹𝑷(𝒇𝟏, 𝒇𝟐, ) = 𝑷(𝒇𝟏, 𝒇𝟐, )/𝑷(𝟏, 𝟒𝟓) ∙ 𝟏𝟎𝟎

Where P(·) indicates the power, RP(·) indicates the relative power, and f1, f2 indicate the low andhigh frequency, respectively. The relative power for each band and the power ratios for different
frequency bands were averaged in each region. The ratios of power for different frequency bands ineach electrode was also computed for possible pairs of frequency bands, such as P(theta)/P(alpha) andP(alpha)/P(beta). The 3-minutes EEG data during each sound's condition were also epoched by 7s
fixed length and analyzed by MNE toolbox (28) using spectral connectivity algorithm. The spectralconnectivity was computed for the debiased weighted phase lag index (dwPLI). dwPLI is a debiased
estimator of the squared wPLI developed by Vinck et al.(29), correcting for sample-size bias in phasesynchronization indices.

3. RESULTS
3.1

Correlations between self-reported evaluations: objective descriptors and emotion

responses
The scores of positive items’ responses including pleasant, happy, stimulating, attractive, energetic,
calm were averaged to compute the positive component of the emotional saliency’s (ES+)(25). The
correlation analysis showed as followed (Figure 3.a). The linear regression analysis between eachobjective descriptor (naturalness, mechanicalness, smoothness, rhythmicalness, spaciousness, andfamiliarity) of each sound condition and the ES+ were conducted (R2 = 0.452, F= 5.69, p<0.001) . The
significant regression coefficient was found in mechanical descriptor (β= -0.185, F= 9.456, p=0.003 ).
And the linear relationship was also found between objective descriptors and negative component(ES-, averaged by the scores of boring, unpleasant, nervous, weak, sad, unattractive items)(25) (R2 =0.374, F= 4.11, p<0.001 ) (see Figure 3.b for the correlation results). The significant regression
coefficients were found in natural (β= -0.172, F= 4.871, p=0.031 ), mechanical (β= 0.269, F= 11.775,
p=0.001), smooth (β= 0.205, F= 7.688, p=0.007) and rhythmic descriptors (β= 0.163, F= 5.560,
p=0.022).

Figure 3-The correlation matrixes plots of objective descriptors with ES+ (a) and ES- (b) scores.
3.2 Correlations between the neural activities and self-reported emotional responses
The spectral power of the alpha band had a linear relationship with ES+ scores (R 2= 0.228, F=
2.330, p=0.029). The differences between four conditions were significantly observed (F=3.844,
p=0.014). But the alpha power of each region had no significant relationship with ES+. The
connectivity of the alpha band had a linear relationship with ES+ scores (R 2= 0.390, F= 1.88, p=0.039)
especially contributed by the left-right regional connectivity (β= 9.290, F=5.105, p=0.028) (Figure4a) and intra-right connectivity (β= -5.590, F=8.639, p=0.005) (Figure 4b). And the significant
differences between each condition were also observed (F=2.889, p=0.044).

Figure 4- The linear plots of connectivity of the alpha band in left-right inter-region (a) and intraright region (b) with ES+ scores
The theta alpha ratio index had a linear relationship with ES- (R 2= 0.290, F= 3.22, p=0.004).
Especially the theta alpha index of right region had a significant relation with ES- (β= -2.853, F=4.602,
p=0.036) (see Figure 5.a for the correlation results). But the differences between each condition were
not significant. The spectral power of the gamma band also had a strong connection with ES- scores(R 2= 0.438, F= 6.13, p<0.001). Among each region, the gamma power of the central and posterior
regions indicated inversed linear relationship with ES- (β=11.840, F=15.140, p<0.001; β=-17.592,
F=34.719, p<0.001) (see Figure 5.b for the correlation results). But the differences between eachcondition were not significant.

Fig. 5. The correlation matrixes of the theta alpha ratio index (a) and the gamma power (b) with
ES- scores

4. DISCUSSION
The mechanical road traffic noise influenced both the positive and negative components of theemotional saliency. More mechanical caused more negative feelings and less positive feelings. Moreobjective descriptors were linked to other negative aspects of the emotional feelings. Natural features
are inversely correlated to ES- contrary to smooth and rhythmic. The results confirmed theeffectiveness of introducing natural sounds for noise masking (30). Some controversial evidence
already indicates the complicated effects of the temporal characteristics of water sounds for noisemasking (4,31). Our results suggested that the design of spatial and temporal characteristics of watersounds needs more consideration for improving positive effects.
The overall effects of the alpha band power revealed the positive effects of spatial settings react
on ES+ scores, suggesting the more relaxed and positive state of the brain induced by water sounds,
especially with spatial settings (TSW and FMW conditions). The difference between left-right interregion and right intra-region alpha connectivity refers to the activation level of the default mode
network (DMN), which is a network of interacting brain regions that are active in a resting state otherthan performing a task accompanied by more inter-regions activity and less intra-region connectionsin brain activities (32) could relate to the ES+ scores differently no matter of the water soundconditions or only traffic noise.
The theta alpha ratio index, often used as task load of mental state (33), was surprisingly negatively
related to negative feelings observed in the right region (the partial correlation results showed theindex in the frontal region had a positive relationship with ES- after control the index in the rightregion, r=0.237, p=0.047). Some evidence suggested that the theta/alpha synchronization in thetemporal cortex could reflect successful auditory memory encoding (34,35). Combined with the workfrom Raufi and Longowhich (36), the results suggested that more investigation is needed to clarify
the relationship between different regions of the alpha theta ratio index in different contexts andenvironment settings. The gamma power strongly reflected the negative components of emotionalsaliency in central and posterior regions of the brain. The central gamma power was positively related
to ES-. Meanwhile, the posterior region was inversely related to the ES- scores. Some researchers
have demonstrated that the posterior gamma power is linked with visual information and memory
encoding, and the central gamma power reflects spatial attention and emotional processing (37,38).
The gamma power of different regions could bring more insights into brain activities regarding spatialattention and memory in a noisy environment, and more multisensory investigation is needed.

5. CONCLUSIONS
Traffic noise is considered a health threat for citizens in urban cities. Landscape designers andengineers are developing more effective strategies of noise mitigation. Through an immersive spatial
audio system, three different spatial settings of water-sound sequences were added into the virtual
acoustical environment to investigate the role of the spatialization of water-sound sequences on traffic
noise masking. The neural responses and subjective self-reported evaluations collected during theauditory experiment showed preliminary correlations between the self-report evaluation of theobjective features of the sound environment, the perceived emotions and EEG indicators, including
spectral power, cognitive indexes and brain network connectivity.
The more positive effects on noise masking induced by the spatial setting of water sound sequences
had been illustrated by the collinear relationship between the objective descriptors, the brain networks
activities in the alpha band and positive emotional saliency scores. Moreover, the spectral power ofthe gamma band and the theta alpha ratios used as the cognitive load index had shown relationshipwith the negative emotional saliency that need further and deeper investigations.
The results also suggest investigating the existing differences existing from the explicit andimplicit individual responses to noise.

ACKNOWLEDGEMENTS
Research supported by Programma V:ALERE 2019 “VALERE: VAnviteLli pEr la RicErca”.
Project MIELE, Multisensory Investigation for ELderly-centred design of common living urban
Environments.

REFERENCES
1. Passchier-Vermeer W, Passchier WF. Noise exposure and public health. Environ Health Perspect.
2000;108(suppl 1):123–31.
2. Cerwén G, Kreutzfeldt J, Wingren C. Soundscape actions: A tool for noise treatment based on threeworkshops in landscape architecture. Front Archit Res. 2017;6(4):504–18.
3. Rådsten Ekman M. Unwanted wanted sounds: perception of sounds from water structures in urban
soundscapes. 2015;
4. Galbrun L, Ali T. Acoustical and perceptual assessment of water sounds and their use over road traffic
noise. J Acoust Soc Am. 2013 Jan;133:227–37.
5. Jeon JY, Lee PJ, You J, Kang J. Acoustical characteristics of water sounds for soundscape enhancement
in urban open spaces. J Acoust Soc Am. 2012;131(3):2101–9.
6. Zhang Y, Ou D, Kang S. The effects of masking sound and signal-to-noise ratio on work performance inChinese open-plan offices. Appl Acoust. 2021;172:107657.
7. Lepa S, Weinzierl S, Maempel HJ, Ungeheuer E. Emotional Impact of Different Forms of Spatialization
in Everyday Mediatized Music Listening: Placebo or Technology Effects? In Berlin; 2014. p. Convention
Paper #9024.
8. Deng Z, Kang J, Wang D, Liu A, Kang JZ. Linear multivariate evaluation models for spatial perceptionof soundscape. J Acoust Soc Am. 2015;138(5):2860–70.
9. Masullo M, Pascale A. Effects of combinations of water sounds and visual elements on the traffic noise
mitigation in urban green parks. INTER-NOISE NOISE-CON Congr Conf Proc. 2016; August 21-24,
2016; Hamburg,Germany 2016.253(4):3910–5.
10. Masullo M, Maffei L, Pascale A, Senese VP. An alternative noise mitigation strategy in urban green park:a laboratory experiment. In: INTER-NOISE and NOISE-CON Congress and Conference Proceedings.
Institute of Noise Control Engineering 2017; 27-30 August 2017; Hong Kong, China 2017. p. 3006–12.
11. Masullo M, Maffei L, Pascale A, Senese VP, De Stefano S, Chau CK. Effects of evocative audio-visual
installations on the restorativeness in urban parks. Sustainability. 2021;13(15):8328.
12. Hong JY, Lam B, Ong ZT, Ooi K, Gan WS, Kang J, et al. The effects of spatial separations between watersound and traffic noise sources on soundscape assessment. Build Environ. 2020;167:106423.
13. Aspinall P, Mavros P, Coyne R, Roe J. The urban brain: analysing outdoor physical activity with mobile
EEG. Br J Sports Med. 2015;49(4):272–6.
14. Neale C, Aspinall P, Roe J, Tilley S, Mavros P, Cinderby S, et al. The impact of walking in different urban
environments on brain activity in older people. Cities Health. 2020;4(1):94–106.

15. Choi Y, Kim M, Chun C. Measurement of occupants’ stress based on electroencephalograms (EEG) in
twelve combined environments. Build Environ. 2015;88:65–72.
16. Guan H, Hu S, Lu M, He M, Zhang X, Liu G. Analysis of human electroencephalogram features indifferent indoor environments. Build Environ. 2020;186:107328.
17. Kim Y, Han J, Chun C. Evaluation of comfort in subway stations via electroencephalographymeasurements in field experiments. Build Environ. 2020;183:107130.
18. Olszewska-Guizzo AA, Paiva TO, Barbosa F. Effects of 3D Contemplative Landscape Videos on Brain
Activity in a Passive Exposure EEG Experiment. Front Psychiatry [Internet]. 2018;9. Available from:https://www.frontiersin.org/article/10.3389/fpsyt.2018.00317
19. Tóth B, Farkas D, Urbán G, Szalárdy O, Orosz G, Hunyadi L, et al. Attention and speech-processing
related functional brain networks activated in a multi-speaker environment. PloS One.
2019;14(2):e0212754.
20. Pei G, Li T. A literature review of EEG-based affective computing in marketing. Front Psychol.
2021;12:602843.
21. McInnes AN, Sung B, Hooshmand R. A practical review of electroencephalography’s value to consumer
research. Int J Mark Res. 2022;14707853221112622.
22. Szalárdy O, Tóth B, Farkas D, György E, Winkler I. Neuronal Correlates of Informational and Energetic
Masking in the Human Brain in a Multi-Talker Situation. Front Psychol [Internet]. 2019;10. Available
from: https://www.frontiersin.org/article/10.3389/fpsyg.2019.00786
23. Li J, Maffei L, Pascale A, Masullo M. Effects of spatialized water-sound sequences for traffic noisemasking on brain activities. J Acoust Soc Am. 2022;152(1):172–83.
24. Weinstein ND. Individual differences in reactions to noise: A longitudinal study in a college dormitory. J
Appl Psychol. 1978;63(4):458–66.
25. Masullo M, Maffei L, Iachini T, Rapuano M, Cioffi F, Ruggiero G, et al. A questionnaire investigating theemotional salience of sounds. Appl Acoust. 2021;182:108281.
26. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics
including independent component analysis. J Neurosci Methods. 2004;134(1):9–21.
27. Oostenveld R, Fries P, Maris E, Schoffelen JM. FieldTrip: Open Source Software for Advanced Analysis
of MEG, EEG, and Invasive Electrophysiological Data. Baillet S, editor. Comput Intell Neurosci. 2010
Dec 23;2011:156869.
28. Gramfort A, Luessi M, Larson E, Engemann D, Strohmeier D, Brodbeck C, et al. MEG and EEG data
analysiswith
MNE-Python.
Front
Neurosci[Internet].
2013;7. Available
from:https://www.frontiersin.org/article/10.3389/fnins.2013.00267
29. Vinck M, Oostenveld R, Wingerden M van, Battaglia F, Pennartz CMA. An improved index of phasesynchronization for electrophysiological data in the presence of volume-conduction, noise and samplesize bias. NeuroImage. 2011;55(4):1548–65.
30. Buxton RT, Pearson AL, Allou C, Fristrup K, Wittemyer G. A synthesis of health benefits of naturalsounds and their distribution in national parks. Proc Natl Acad Sci. 2021;118(14):e2013097118.
31. You J, Lee PJ, Jeon JY. Evaluating water sounds to improve the soundscape of urban areas affected bytraffic noise. Noise Control Eng J. 2010;58(5):477–83.
32. Zhou ZW, Lan XQ, Fang YT, Gong Y, Zang YF, Luo H, et al. The inter-regional connectivity within thedefault mode network during the attentional processes of internal focus and external focus: An fmri studyof continuous finger force feedback. Front Psychol. 2019;10:2198.
33. Puma S, Matton N, Paubel PV, Raufaste É, El-Yagoubi R. Using theta and alpha band power to assess
cognitive workload in multitasking environments. Int J Psychophysiol. 2018;123:111–20.
34. Fell J, Ludowig E, Staresina BP, Wagner T, Kranz T, Elger CE, et al. Medial temporal theta/alpha power
enhancement precedes successful memory encoding: evidence based on intracranial EEG. J Neurosci.
2011;31(14):5392–7.
35. Kern P, Assaneo MF, Endres D, Poeppel D, Rimmele JM. Preferred auditory temporal processing regimes
and auditory-motor synchronization. Psychon Bull Rev. 2021;28(6):1860–73.
36. Raufi B, Longo L. An Evaluation of the EEG alpha-to-theta and theta-to-alpha band Ratios as Indexes ofMental Workload. Front Neuroinformatics. 2022;16.
37. Ishii R, Canuet L, Ishihara T, Aoki Y, Ikeda S, Hata M, et al. Frontal midline theta rhythm and gamma
power changes during focused attention on mental calculation: an MEG beamformer analysis. Front Hum
Neurosci. 2014;8:406.
38. Park H, Lee DS, Kang E, Kang H, Hahm J, Kim JS, et al. Formation of visual memories controlled by
gamma power phase-locked to alpha oscillations. Sci Rep. 2016;6(1):1–10.

ABS-0621

Copacabana revisited: Photogrammetry-based model for urban soundauralization
Jonas HECK, Christian DREIER, Josep LLORCA-BOFÍ, Michael VORLÄNDER
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany,
Correspondence: jonas.heck@akustik.rwth-aachen.de

ABSTRACTThis work presents the procedure to create a model of the iconic scenery at Copacabana in Rio de Janeiro,
Brazil, for auralization-based noise assessment in virtual urban environments with the aid of photogrammetry.
The proposed procedure, firstly, delivers a description of the investigated space and of the photographic dataacquisition. Secondly, the post-processing with the help of photogrammetry software is described includingalignment, filtering, and noise removal. A textured mesh as well as a simplified model for acoustic simulation
are derived. The final models can be scaled down or complemented in various degrees of detail accounting forplausible immersion. This allows for using them as a framework for multi-modal perceptual studies in complex
environments under controlled conditions and for further showcases.

Keywords: Urban sound, Auralization, Acoustic model1

INTRODUCTION

1.1 Motivation
The interest in urban and outdoor noise assessment steadily increases. This does not only apply to establishedmethods like noise mapping but goes beyond. A way to involve people into planning processes is the useof auralizations in virtual reality where sound sources, sound propagation and receiver configurations can beseparately controlled [1]. Complemented with a visual representation, those models can serve as a plausible
representation of the space to study people’s perception and preferences under laboratory conditions.
Concert halls and lecture rooms have been shown to be representable by models derived from photogrammetry [2]. Furthermore, this procedure has already been applied to an outdoor scenario, i.e. a green space in
Aachen, Germany [3, 4]. The focus of this model was on the geographical properties of the green and thedetailed representation of two buildings. The area is of the size of about 170*110 m and it is only visited by
few pedestrians. In contrast, the present paper aims to assess the procedure in a more complex environment
with many moving sources such as pedestrians, cars, buses, bicycles, airplanes, ships etc.
1.2 Copacabana revisitedWho thinks of the album "Rio Revisited" by Antonio Carlos (Tom) Jobim and Gal Costa [5] will immediately
feel transported to the Copacabana promenade. Inspired by their unique Bossa Nova sounds, the idea was bornto not only study the traffic noise along Rio’s streets but to capture data that serve as the basis to build a fullvisual and acoustic model of the famous scenery.
This paper describes the capturing procedure, i.e. photography walks. The photography walks were complemented with 360° video and simultaneous audio recordings in first-order Ambisonics format for future reference.
They are, however, not further discussed here. This work focuses on the photogrammetric procedure to build
models for both the visual and the acoustical representation of the space. The derived models can be used for
future studies of urban soundscapes and to demonstrate acoustic simulation techniques developed at the Institutefor Hearing Technology and Acoustics, RWTH Aachen University.

1.2.1 Topography
Copacabana, part of the city of Rio de Janeiro, is situated near the Tropic of Capricorn and is subject to tropicalclimate and continuous trade winds. Copacabana is flanked by Ipanema in the South-West, by forest and hills
in the North, which divide the Southern part of the city of Rio de Janeiro from the Northern part, while it
opens to the Atlantic ocean in the South-East. During our acquisitions between November 29 and December
10, 2021, temperature was in the range of 21.5–32.6°C, the humidity in the range of 52.4–86.2%.
The urban space under study is shown in figure 1. The area of interest is ca. 240 m wide along Avenida
Atlântica (between Rua Santa Clara in the North and Rua Siqueira Campos in the South) and 450 m long(from the water front to beyond the intersection of Rua Figueiredo de Magalhães with Avenida Nossa Senhora
de Copacabana).

Figure 1. Aerial view of the city block under study.1

1.2.2 Acoustic features of the urban environment
The choice of the investigated space was made with the help of our project partners from the Federal Universityof Rio de Janeiro (UFRJ) who could support us with local knowledge of typical traffic flow etc. This helped
to prepare and schedule the photography walks as well as the use of natural lighting conditions. The detailed
description of the urban environment as well as its characteristics follow below.
Firstly, the scenery was selected with regard to prominent traffic noise for the following reasons:• Heavy motorized traffic on Avenida Atlântica on both lanes, in the morning mainly in northern direction(to the city center of Rio de Janeiro).
• One-way road Avenida Nossa Senhora de Copacabana. Faster traffic with a lot of buses passing.
1 https://www.google.com/maps, accessed on 2022-07-26

Furthermore, the space was selected for its special topography which results in key features for acoustic simulations. The main factors that influence acoustic sound propagation in this context are, see also figure 1:• Highly reflective water surface on the sea.
• Highly absorptive sand at the beach.
• Pavement and asphalt of Avenida Atlântica, its promenade + bicycle lane between street and beach, aswell as the opposite sideway between street and buildings.
• Facades of the 6-8 story buildings along Avenida Atlântica with various degree of absorption and scattering.
• Some smaller objects that influence sound propagation, i.e. kiosks with sunshades, chairs and tables.
• Parking, stopping and driving cars, buses and trucks.
The sound sources that could be observed at Copacabana are:• Road traffic noise (cars, buses, trucks, motorcycles, alarm signals of emergency services like ambulanceand police).
• Air traffic (small airplanes and helicopters).
• Music from the bars/kiosks along the beach.
• Generators at the beach which are used to pump the water to provisional showers.
• Ocean sounds (Waves and sea spray as well as birds).
• Babble of voices and voices of vendors.

2

DATA ACQUISITION

The photogrammetric model generation requires a big amount of sufficiently detailed photographs as input data.
For Copacabana, no high quality photographic data were available that could be used for both an acoustic anda visual model. Therefore, own photographs were taken between November 29 and December 10, 2021. The
path of the photography walks showing all accumulated camera positions is mapped in figure 2. The camera
was a Canon EOS R5 body with a Canon RF 24-105/4.0-7-1 IS STM lens. The photographs were taken witha resolution of 8192*5464 px in one of the two following configurations:i. Hand-held photographs at head height (see blue color in figure 2). The camera is looking to the inside of
the area and, thus, defining the boundary of the model space.
ii. High tripod to lift the camera 8-12 m height (see red color in figure 2). The camera is rotated 360° with± 30° step size looking to the surrounding of the rotation axis. It is slightly tilted in order to not only
capture the horizon.
The whole scene was divided into multiple sets in order to ensure comparable conditions with regard to
sunlight, cloud cover and traffic situation for all pictures of one set. The different sets, the applied configurationand their order are described in table 1. In total, we concluded with 4271 photographs.
As many pictures as possible were taken in early morning. By this, we hoped to benefit from the diffuse
light and less traffic than at a later daytime. Most problematic was direct light from a high-standing sun as it
threw long shadows that resulted in high contrast and, thus, in a bad color reconstruction in the post-processed
model’s texture.
For the camera settings, we first used fixed settings of shutter time, blend opening and ISO. But since
the weather and cloud cover, and thus the light, heavily changed over the days, we decided in a second step,Figure 2. Camera positions during acquisition walks at Copacabana. Blue: Hand-held photographs at headheight. Red: Photographs on a tripod at 8–12 m height, rotating 360° with ± 30° step size. Parts I & II
correspond to the two-step approach in the photogrammetric processing, see ch. 3.
Table 1. Protocol of the photograph capture walkSet no.

Space description

Capture date

Capture config.

Color

1

Beach (definition of the boundary)

29.22.2021

Hand-held, head height

blue

2a

Beach (within boundary)

30.11.2021

High tripod

red

2b

Beach (within boundary)01.12.2021

High tripod

red

3

Av. Atlântica

01.12.2021

Hand-held, head height

blue

4

Rua Figueiredo

02.12.2021

Hand-held, head heightred

5

Av. Atlântica (housefront)

02.12.2021

Hand-held, head height

blue

6

Av. Atlântica (center reserve)

02.12.2021

High tripod

red7

Rua Figueiredo (elevated camera)

06.12.2021

High tripod

red8

Rua Figueiredo (transition low to high)07.12.2021

Head height to elevated height

blue

9

Beach (North)07.12.2021

High tripod

red

10

Beach (closing inner loop)

09.12.2021

High tripod

red

11

Rua Figueiredo (Northern side)

09.12.2021

High tripod

red

12

Rua Figueiredo (Southern side)

10.12.2021

High tripod

red

13

Rua Figueiredo (around kiosks)

10.12.2021

Hand-held, head heightblue

continue taking photographs with the built-in automatic mode. This resulted in a more stable color reproduction
under various lighting conditions, i.e. cloudy sky compared with sunny, cloud-free weather.
Another problem that occurred when capturing the photographs comes with the moving sun. Over the day,the shadows change their position and length. As capturing the photographs took time (at least 1–2 hrs per set),light could heavily change during one walk. Thus, the lighting and the colors of the pictures vary a lot. If thedifferences are too big, the colors become blurred and look faded and dirty.

3

PHOTOGRAMMETRIC PROCESSING

3.1 From image alignment to a textured modelOnce the photographs were taken, the pictures were post-processed in order to build a photogrammetric model.
For this purpose, the software Agisoft Metashape (version 1.8.3) was used. It aligns multiple pictures by triangulation and allows to generate point clouds, meshes as well as textured object models [6, 7].

Figure 3. Photogrammetry workflow. A: Filtering and marker selection, B: Sparse point cloud, C: Dense point
cloud, D: Mesh generation, E: Texture mapping
After manually removing some ineligible photographs, the post-processing followed a two-step approach:
Steps A–D of the procedure were performed separately for part I (2064 pictures along the street Rua Figueiredode Magalhães) and part II (2074 pictures for the beach) and joined afterwards in order to economize computational power. In detail, the post-processing contained the following steps, cf. figure 3:
A. Filtering and marker selection:• Firstly, pictures with a non-sufficient quality (Metashape’s quality value of less than 0.5) were excluded.
• Markers (at least 4) were manually positioned on, e.g., manhole covers and road signs to align thosepictures that were not automatically aligned in first place.
• Masking was applied to remove parts of pictures that compromised the point cloud quality, i.e. mainly
the sky was removed from the pictures.
• Further filters were applied adjusting the parameter settings for "reconstruction uncertainty", "projection
precision" and "reproduction error" as well as an optimization of the camera positions.
B. Sparse point cloud
C. Dense point cloud:• A bounding box was defined to exclude distant outliers.
• Filtering dense point cloud according to the confidence level; noise (conf. level ≤ 2) was removed.
• With the "Selection" tool, ghost objects (cars, trees, road signs) were manually removed.
• Over-determined areas were reduced by defining a minimum spacing between points.
D. Mesh: The mesh was generated from the dense point cloud. Interpolation was activated to close remaining
holes. Smoothing was used to remove too complex polygon structures.

E. Texture: The full mesh was textured from the photographs with a texture map of 4096x1 px, see result infigure 4.

Figure 4. Textured model derived from authors’ terrestrial photographs

3.2 Amendment with flyover photographs
As seen in figure 4, the quality of the reproduced model was so far insufficient for an audiovisual immersion.
Furthermore, the holes in the model prevent the use of geometrical acoustic simulation techniques.
To close our model, we combined the model derived from the presented terrestrial photogrammetry with aphotogrammetry-based model derived from Google Earth2 flyover captures of the beach and Avenida Atlântica.
The result of this amendment, which is only incorporated into part II so far, is depicted in figure 5.

Figure 5. Combined model (terrestrial + flyover photographs) with texture
2 https://earth.google.com, accessed on 2022-06-13

4

SIMPLIFIED MODEL FOR ACOUSTIC SIMULATION

While it is, in principle, possible to automatically reduce the mesh size (the number of polygons that form themesh), we could not successfully keep important geometries stable and exact. Mainly, the details of facades,such as ventilators, heat pumps, windows, balconies, balustrade and projections, were not maintained. This is
why we decided on building a new CAD model for the use in real-time auralizations that complies with therequirement of a reduced number of mesh polygons. It was manually derived from the photogrammetry mesh.
The derived model uses dimensions measured from the photogrammetry model (cf. figure 5) and was built withsimple rectangular blocks. The CAD sketch of the model can be seen in figure 6. The materials were estimatedfrom the photographs and the corresponding properties were defined on the object surfaces. From this on, theacoustic model is ready to be simulated with established simulation tools, e.g. to auralize synthesized pass-by
noise in urban environment [8].

Figure 6. CAD model of the urban environment at Copacabana of reduced complexity for acoustic simulation

5

CONCLUSIONS & OUTLOOKWe presented a procedure of capturing and post-processing a complex urban environment for photogrammetry
modelling. This photogrammetry model was used to derive an acoustic model for geometrical acoustic simulations. Both together can be used in interactive audio-visual experiments to study architectural perception andfor auralization-based noise assessment. The models are available at: 10.5281/zenodo.6992946.
The drawbacks we encountered during the post-processing should be briefly discussed: In the camera configuration ii. (high tripod), we slightly tilted the camera to not only capture the horizon but mainly the scene ofinterest. This could sometimes have resulted in alignment problems because the horizon may not always havebeen clearly recognizable as a landmark. Through adding flyover pictures to our model, we achieved closing
the previously perforated mesh. But the resolution of geometrical details, especially of the facades, as well as
the texture is much lower than from the high-resolution pictures taken by the authors.
Future work will contain a refinement of the post-processing steps with more detailed picture basis (higher
amount of data points), other options for texturing the mesh, the incorporation of flyover photographs at multiple stages and the testing of filtering algorithms to delete disturbing objects from the photogrammetry model.
Furthermore, we will work on finding mesh reduction algorithms which maintain important geometric featuresof building facades that are of importance for the acoustic simulation while reducing the number of polygons

significantly. This would allow us to immediately derive the acoustic model from the photogrammetry model
without significant manual adjustments.

ACKNOWLEDGMENTS
The authors acknowledge the financial support from the German Academic Exchange Service (DAAD) and thekind help from the exchange counterparts from the Federal University of Rio de Janeiro (UFRJ), especiallythe on-site support from Prof. Mariane Rembold Petraglia and Prof. Julio Cesar Boscher Torres. Furthermore,
the authors like to thank Mr. Artur Fuchs for the time he spent building models from the big amount ofphotographs during his bachelor thesis.

REFERENCES[1] Vorländer, M. (2020). Auralization. Fundamentals of Acoustics, Modelling, Simulation, Algorithms andAcoustic Virtual Reality. 2nd edition. Springer Cham, 2020. https://doi.org/10.1007/978-3-030-51202-6.
[2] Llorca-Bofí J.; Witew, I.; Redondo, E. & Vorländer, M. (2018). 3D modelling photogrammetry to support
acoustic measurements and derive geometries for simulation. In: Proceedings of the Institute of Acoustics(Auditorium Acoustics 2018), 40 661-666.
[3] Llorca-Bofí J. & Vorländer M. (2021). Multi-Detailed 3D Architectural Framework for Sound Perception
Research in Virtual Reality. Front. Built Environ. 7:687237. https://doi.org/10.3389/fbuil.2021.687237.
[4] Llorca-Bofí, J.; Dreier, C.; Heck, J. & Vorländer, M. (2022). Urban Sound Auralization and Visualization
Framework — Case Study at IHTApark. Sustainability, 14(4), 2026. https://doi.org/10.3390/su14042026.
[5] Jobim, A. C. & Costa, G. (1987). Rio Revisited. Recorded Live at Bill Graham’s Wiltern Theatre, Los
Angeles. Labels: Verve / PolyGram / Universal.
[6] Mikhail, E. M.; Bethel, J. S.; McGlone, J. C. (2001). Introduction to Modern Photogrammetry. John Wiley& Sons, 2001. ISBN: 978-0-471-30924-6.
[7] Agisoft LLC. (2022). Agisoft Metashape User Manual: Professional Edition. Version 1.8. St. Petersburg,
Russia, 2022. URL: https://www.agisoft.com/downloads/user-manuals/.
[8] Dreier, C.; Hahn, J.; Heck, J.; Llorca-Bofí, J. & Vorländer, M. (2022). Real-time vehicle pass-by noise
synthesis for urban sound auralization. In: Proceedings of the 24th International Congress on Acoustics,
Gyeongju, South Korea (2022).

ABS-0622

Real-time vehicle pass-by noise synthesis for urban sound auralization
Christian DREIER, Johannes HAHN, Jonas HECK, Josep LLORCA-BOFÍ, Michael VORLÄNDER
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany
Correspondence: christian.dreier@akustik.rwth-aachen.de

ABSTRACTThis work presents the design and implementation of a perceptionally plausible vehicle pass-by noise synthesis
model for environmental noise auralization. The source model consists of two separate components for engine
noise and road-tyre noise synthesis. Additionally, the model accounts for spatial radiation characteristics byincluding higher-order directivity coding. With regard to real-time constraints and parametrical controllability,
the proposed synthesizer is implemented in C++ and relies on procedural audio synthesis using digital waveguide networks and spectral shaping. Striving for the auralization of urban scenarios in VR using multiple object
instances, the model integrates well being computationally efficient.

Keywords: Auralization, Environmental noise, Source modeling

1

INTRODUCTION

1.1 Motivation
Auralization of complex urban scenarios is a promising, but technically challenging approach. From a theoretical point of view, urban noise auralization deals with inherent non-LTI (i.e. linear time-variant) propertiesof the propagation medium (air) that occurs due to turbulence-induced advection as well as refraction due totemperature, humidity and pressure inhomogeneities. Beyond the limited scope of urban noise assessments using level metering based on the the A-weighted equivalent continuous sound level LA,eq , the consideration of
psychoacoustic measures is highly relevant for the acceptance in the population. Auralizations can be used to
predict these psychoacoustic measures at virtual receivers, apart from the opportunity to listen to the auralizedsounds. As exemplarily shown in a recent publication and demonstrated in an according 360◦ video1 , environmental noise studies can be performed in Virtual Reality (VR) [1]. An auralization usually consists of separated
simulation models for sources, propagation and receivers [2]. Especially for psychoacoustic perception studies ofnoise sources – such as aircrafts, cars or trains – the acoustic source models should reproduce spectro-temporalcharacteristics and enable dynamic parameter control for enhanced plausibility.
1.2 State of the art and contribution
Traffic noise auralization approaches based on simulated sources have been performed e.g. in the LISTEN
project [3]. The model combines a pre-calculated road-tyre noise component and additive harmonics for theengine component. Furthermore, the model is enhanced by horizontal directivities in the frequency range up to
8 kHz. Based on this signals propagation simulations have been performed in combination with measurementbased [4] or simulated propagation path impulse responses using the pseudo-spectral time domain method [5].
Recently, a framework that includes visualization was developed by researchers at the University of York [6].
With regard to compatibility to the open-source auralization software Virtual Acoustics (VA) [7] and according
real-time constraints, this work presents a vehicle pass-by noise model for dynamic, real-time auralizations.
To the research field on environmental noise auralizations, the presented approach further contributes a spatial
1 https://www.youtube.com/watch?v=rudxfV94UwA&list=PLjVMT5BkCe83h299e3FWNivk-MFtRnpw-&index=2

directivity encoding using the OpenDAFF file format [8]. The framework enables rendering to multiple spatial
headphone- or loudspeaker-based reproduction systems. Due to its server-based nature VA can be interfaced tovisualization engines, such as Unity3D or Unreal Engine.

2

FRAMEWORK FOR URBAN SOUND AURALIZATIONThe VA framework is following a three-step paradigm of discrete source, propagation and receiver models. The
following block diagram (fig.1) includes a parameter overview of environmental noise auralization in VA withthe focus of this paper being emphasized in green color. The depiction emphasizes the necessity of a source
model with free-field boundary conditions since all surface reflections are separately simulated by the outdoor
module – that is calculating the effects according to the ISO9613 standard [9] – and urban sound propagation
module.

Figure 1. Block diagram of environmental noise auralization in VA.

3

REAL-TIME VEHICLE PASS-BY NOISE SYNTHESISThe proposed synthesis model follows the overall goal to be a computationally efficient, reflecting acoustically
free field conditions, being spectrally accurate and customizable model representation of an individual car’s
pass-by noise emissions. With regard to maximum flexibility towards automation and multiple host software
compatibility, the synthesizer module is implemented in C++ as a Virtual Studio Technology (VST) plugin.
3.1 Concept
The model consists of two separate source components: Road-tyre noise (chapter 3.2) and engine noise (chapter
3.3). The concept of vehicle pass-by sound auralization is shown in fig.2. The propagation is simulated along
multiple propagation paths so that the source model should have spherical properties (yellow sphere). Practically,
this is implemented using the OpenDAFF format with spatio-temporal discretized information (green circles).

Figure 2. Digital twin model of vehicle pass-by noise auralization.

As shown in fig.3 the spatial sound source signal is the convolution result of a parametrically synthesized soundsignal with additional directivity.

Figure 3. Concept of real-time parametric sound synthesis: The omnidirectional component is implemented asprocedural processor and convolved with higher-order spatial components.

3.2 Autoregressive modeling of road-tyre noise
The proposed modeling approach assumes the tyre-road sound component – including all its complex aerodynamic and mechanic sound generation mechanisms – to be purely noisy, i.e. free of tones. In order to find acompact model description of this noise, a two-step analysis-and-synthesis procedure (chs. 3.2.1 and 3.2.2) is
discussed in the following and depicted in an according flowchart (fig.4):

Figure 4. Flowchart of the iterative filter optimization approach for finding an IIR filter representation of vehicle
pass-by noise.

3.2.1 Road-tyre noise analysis
In a first step the statistical properties of an inversely filtered and calibrated vehicle pass-by recording (parallelogram) are analyzed. The procedure of inverse filtering is described in [10] and is based on time-variant
backpropagation. The resulting signal contains information on the directivity that can be decomposed into an
omnidirectional and a higher-order components. After inversely filtering the backpropagated effects, the signal
must be furthermore equalized for angle-dependent receiver properties of the measurement microphone. Generally, two approaches are useful for the spectral fitting of the omnidirectional component: Autoregressive modeling (AR) and linear prediction. For both techniques the goal is to determine parameters of a linear filter whose
output theoretically could yield the same numerical results:• From linear prediction technique, a FIR filter that predicts future samples of an autoregressive process,which is based on a linear combination of past samples, is created.

• From AR modeling an all-pole IIR filter is created that, by convolution with white Gaussian noise, produces a signal with the same statistics as the autoregressive process to be modeled.
Since the phase response of the digital filter has no audible impact on the resulting convolved signal with thewhite noise excitation – and the filter’s group delay can be neglected since the excitation signal is stationary –the second approach based on recursive IIR filters is chosen to address the low-order, feature preservation andminimum storage requirements. The algorithmic procedure for finding the filter parameters is briefly describedin the following. The power spectral density (PSD) of a pth-order autoregressive process is [11]PAR (e jω ) =|b(0)|2p(1)

2

1 + ∑ a p (k)e− jkω
k=1with a and b being the IIR filter coefficients resulting by solving Yule-Walker equations with the Levinson
algorithm. The PSD at the output of the IIR filter is given by the magnitude-squared of its frequency response
multiplied by the variance of the white Gaussian noise input.
3.2.2 Road-tyre noise synthesis
For real-time synthesis, the resulting coefficient vectors from the analysis step (ch. 3.2.1) must be converted ina format conventional C++ libraries can handle. A common approach is the use of cascaded biquads: The IIR
filter block is a cascaded chain of second order sections (SOS) where each SOS consists of a limited numberof coefficients b and a from the solution of equation (1) or alternatively a high-pass, low-pass, parametric or
shelving filter defined by its parameters frequency f, gain G and quality factor Q (fig.5).

Figure 5. IIR filter design consisting of n cascaded second order sections (SOS).
The number of SOS elements depends on the coefficients to be handled. The required number is assessed bycomparison of the vehicle’s averaged near-field radiation spectra with synthesized noise spectra (fig.6). At order30, a sufficient approximation – that is indistinguishable in its audibility – of the original spectrum is achieved.

10
Original
Order 5
Order 20
Order 30

5Magnitude [dB]

0
-5
-10
-15
-20
-25
-30
-35
102

103
Frequency [Hz]104

Figure 6. Comparison of averaged near-field radiation spectra with synthesized noise spectra of different filter
orders.

3.3 Engine noise synthesis
In addition to the road-tyre noise component, synthesized engine sound are implemented. The real-time syntheses of intake and exhaust noise contributions are based on the Baldan model [12]. Corresponding to the timeof a full engine cycle (induction, compression, combustion, exhaust) a sawtooth-like trigger phasor linearly rises
from 0 to 1. Four processes (piston motion, fuel ignition, intake valve, exhaust valve) are derived from thistrigger signal and described by trigonometric functions:• Piston moves up and down two times during one cycle → Cosine wave.
• Intake valve and exhaust valve opening → Positive half cycle of a sine wave.
• Variable pressure increase due to fuel ignition → Parametrically controllable width of the fuel ignition
pulse.
The engine model fundamentally assumes the intake, cylinder and exhaust system to be a tube network witheach tube being realized by a digital waveguide [13]. The implementation of a digital waveguide is the foundation for the synthesis of the engine sound: The sound field inside a waveguide is composed of two time-discrete
input signals x0 [k] and x1 [k] and two output signals y0 [k] and y1 [k] (fig.7).

Figure 7. Schematic of the digital waveguide.
On each delay line the waves propagate in both directions: Due to its variable reflection coefficients α and βa certain part of an incoming wave is reflected at the end of a delay line. The total propagating wave resultsfrom the constructive or destructive interference of the outgoing and the return wave: For multiple integers ofhalf wave lengths being equal to the waveguide length results in a standing wave of high amplitude. Except
from this case the interference is destructive.
A digital waveguide consists of two delay lines that support very fast modulation of the implemented delay
length without any audible artifacts or clicking noise, which is essential for a plausible result. Each delay line
stores values of an audio stream. A simple storage in terms of a first-in-first-out (FIFO) memory would not
be sufficient, since the variable cylinder tube length due to the piston motion must be mimiced. Therefore, thedelay line is implemented as variable delay line (VDL, fig.8). Each VDL consists of one write position andtwo read positions where new values are written into, or read from the VDL.

Figure 8. Schematic of the variable delay line (VDL) implementation.

The sound synthesis originates from a cylinder and is spectrally filtered by the intake and outlet elements. The
latter are modeled as waveguides of fixed length, with modulated reflection at the valve end and a fixed reflection coefficient at the free end. Additionally, the intake path is fed with lowpass-filtered white noise tosimulate turbulences in the air and the fuel mixture. The exhaust path consists of a centrally positioned muffler
with inlet and outlet tubes. The spectrally complex modeling of the muffler attenuation is based on four digital
waveguides with length-tuned delay lines resulting in destructive interferences.
3.4 Directivity model
As mentioned in chapter 3.2.1 the presented modeling approach is based on inversely filtered and calibrated vehicle pass-by recording in order to capture information on the acoustic far-field directivities of moving sources.
In such a far-field directivity, some effects on the wave propagation are included that cannot be easily simulated,even with elaborated numerical methods, and therefore have to be measured:• Advection due to the object speed and the laminar flow around the moving object;• Refraction due to turbulent flow at body cavities;• Diffraction due to object geometry;• Scattering due to surface roughness (of asphalt) that in turn influences both sound generation mechanism
and the near-field propagation.
In order to obtain a directivity that is comparable to free-field conditions, the measured signals from the pass-by
are corrected for• an image source that appears due to the surface impedance,• the geometrical distance law-dependent loss, and• the air absorption between the measurement point and the virtual directivity hull of the moving object.
The higher-order directional component is the angle-dependent spectral difference between the omnidirectional
component and the according compensated near-field spectrum. The angle-dependent filter curve is stored as
impulse response using the openDAFF file format that is conveniently integratable to the VA auralization framework. According to the slice width of the backpropagated signals, the DAFF file contains magnitude spectrawith an angular resolution of 10◦ .

4Integration to urban auralization framework

The presented vehicle pass-by noise synthesis model yields a spatial and spectro-temporal description with freefield properties and therefore can be combined with any sound propagation simulation engine. A tool for simulating propagation paths in 3D models is the image edge model [14]. For vehicle pass-by auralizations information on acoustic properties of surface materials can be provided to the propagation algorithm by applicationof parameter-based material models of non-porous and porous asphalt [15]. As shown in fig.9 the second-orderreflection and diffraction sound propagation paths are simulated between a fixed vehicle and a virtual receiver
position inside an auralization model of the urban Copacabana site [16].

Figure 9. Simulated second-order reflection (green) and diffraction (yellow) ray paths between a vehicle and avirtual receiver in a Copacabana site auralization model [16].

Urban sound propagation loss [dB]

From the source’s point of view, the outgoing waves emanate in different horizontal and vertical directions.
The auralization renderer convolves every path’s impulse (defined by its delay τ and gain g) with the according
spatial radiation signal (see fig. 3). For the defined scenario, the overall impulse response consists of 59
impulses (fig. 10, left). The according transfer function (fig. 10, right) reflects the spectral influences of theoutdoor sound propagation effects as defined by [9], i.e. geometrical spreading loss, atmospheric attenuation,
ground effect and diffraction whereas theoretical influences due to turbulence in urban sound propagation aredeliberately neglected as they do not play a significant role.
1

Amplitude

0.5
0
-0.5
-1
0

0.2

0.4

0.6
Time [s]

0.8

1

-40
-50
-60
-70
-80
101

102

103

104

Frequency [Hz]Figure 10. Impulse response (left, normalized) and according transfer function (right) between the vehicle’s
position and the virtual receiver’s position from the scenario in fig.9.

5

CONCLUSIONS

Still of ongoing research, this paper presents the concept and details on a real-time implementation of a synthesizer of vehicle pass-by noise including spatial information of distinct road-tyre noise and engine noise components. The synthesizer code is efficiently implemented in C++ and compiled using the Virtual Studio Technology(VST). The synthesizer can be applied and parametrically automated in a wide range of host applications, suchas digital audio workstations (DAW) or real-time auralization renderers such as Virtual Acoustics (VA). Further
research will focus on the individualization of different types of cars and tyres, as well as the integration ofan Acoustic Vehicle Alerting System (AVAS) component to the synthesizer accounting for acoustic features ofelectric cars.

ACKNOWLEDGEMENTS
The authors like to thank the HEAD-Genuit foundation (ID P-22/02-W) for funding this research.

REFERENCES[1] Llorca-Bofí, J., Dreier, C., Heck, J., Vorländer, M.: Urban sound auralization and visualization framework– Case study at IHTApark. Sustainability 14(4), 2026 (2022) doi:10.3390/su14042026[2] Vorländer, M.: Auralization. Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic
Virtual Reality. 2nd ed. Springer Nature, Switzerland (2020) doi:10.1007/978-3-030-51202-6[3] Forssén, J., Kaczmarek, T., Lundén, P., Nilsson, M., Alvarsson, J.: Auralization of traffic noise within the
LISTEN project - Preliminary results for passenger car pass-by. In: Proc. Euronoise, Edinburgh, Scotland(2009)[4] Georgiou, F., Hornikx, M., Kohlrausch, A.: Auralization of a car pass-by inside an urban canyon using
measured impulse responses. Applied Acoustics, 183 (2021)[5] Georgiou, F., Hornikx, M., Kohlrausch, A.: Auralization of a car pass-by using impulse responses computedwith a wave-based method. Acta Acustica united with Acustica 105(2), 381–391 (2019)[6] Fu, Y.: Auralisation of traffic flow using procedural audio methods. PhD thesis, University of York (2021)[7] Virtual Acoustics – A real-time auralization framework for scientific research, URL:
http://www.virtualacoustics.de[8] OpenDAFF – An open source file format for directional audio content, URL:
http://www.opendaff.org[9] ISO 9613-2: Acoustics – Attenuation of sound during propagation outdoors – Part 2: General method of
calculation. International Organization for Standardization, 1996
[10] Dreier, C., Vorländer, M.: Sound source modelling by nonnegative matrix factorization for virtual reality
applications. In: Proc. INTER-NOISE, 1053–1061, Washington DC, USA (2021) doi:10.3397/IN-2021-1742[11] Hayes, M.: Spectrum estimation (Chapter 8). In: Statistical digital signal processing and modeling. John
Wiley and Sons, New York (1996)[12] Baldan, S., and Lachambre, H., Delle Monache, S., Boussard, P.: Physically informed car engine soundsynthesis for virtual and augmented environments. In: 2015 IEEE 2nd VR Workshop on Sonic Interactions for
Virtual Environments, 1–6, (2015) doi:10.1109/SIVE.2015.7361287[13] Smith, J. O.: Physical modeling using digital waveguides. Computer Music Journal 16(4), 74–91 (1992)[14] Erraji, A., Stienen, J., Vorländer, M.:
doi:10.1051/aacus/2021010

The image edge model. Acta Acustica 5(17) (2021)[15] Dreier, C., Vorländer, M.: Vehicle pass-by noise auralization in a virtual urban environment. In: Proc.
INTER-NOISE, Glasgow, Scotland (2022)[16] Heck, J., Dreier, C., Llorca-Bofí, J., Vorländer, M.: Copacabana revisited: Photogrammetry-based model
for urban sound auralization. In: Proc. Int. Cong. Ac., Gyeongju, South Korea (2022)

ABS-0630

Auralization of Wind Turbine Noise Using Physics-Based Emission and
Propagation Models
David Mascarenhas1(1) , Benjamin Cotté1(2) and Olivier Doaré1(3)1 UME, IMSIA, ENSTA Paris, CNRS, CEA, EDF, IP Paris, France(1) david.mascarenhas@ensta-paris.fr(2) benjamin.cotte@ensta-paris.fr(3) olivier.doare@ensta-paris.fr

ABSTRACTThe problem of noise annoyance that is due to the characteristics of wind turbine noise such as amplitude
modulation is an important factor against the building of wind farms. It would thus be beneficial to predict thenoise that is generated by the wind turbines in the design phase itself. The auralization tool discussed in thisarticle synthesizes the noise generated by wind turbine blades with the help of trailing edge noise and turbulent
inflow noise models. The parabolic equation is used to include propagation effects associated to the ground
impedance and the temperature and wind speed gradients. The effect of scattering due to atmospheric turbulencein the acoustic signal is added to this synthesis tool. To achieve a convincing signal of the environment of thewind turbine, pre-recorded background noise can be added to the time signals. The resulting signal corresponds
to a wind turbine noise in an outdoor environment which can be used readily. The influence of the various
parameters on the perception and annoyance due to wind turbine noise can be studied in detail using this tool.

Keywords: Auralization, wind turbine noise, Annoyance, Parabolic equation, Harmonoise model1

INTRODUCTIONWind energy is among a few green energy sources which serves as a replacement for conventional polluting
energy sources and evidently a steady rise in the number of wind farms can be seen [3]. However one of
the main issue due to the installation of wind turbines is the potential impact of the noise generated by themoving blades leading to potential causes of annoyance and disturbances in sleep patterns for people living in
the vicinity of wind farms [17, 9]. One way to avert this bargain between concerns on the generated noise andthe necessity for building wind farms is to predict the noise generated by the wind turbines in the design phaseand accordingly adapt them to the requirements. For an accurate prediction of the noise based on controllable
parameters, it is necessary for the model to be physics-based. Synthesis of this noise into time signals can also
be used to study psychoacoustic questions such as disturbance and annoyance caused by wind turbine noise.
The main sources of broadband wind turbine noise as known in the literature are turbulent inflow noise(TIN), trailing edge noise (TEN) and stall noise [22, 15, 1], among which stall noise is the most intense
and is produced only when the angle of attack of the blade is large. Thus, as a simplest case scenario the
TIN and TEN which are observed as dominant sources of wind turbine noise are considered here. Anotherimportant component that influences the received noise is the propagation of the generated sound through an
inhomogeneous atmosphere and its reflection off the semi-porous ground. In an outdoor environment, the soundreceived from a point source can be modelled based on the characteristics of the noise source that is modified
by the influence of the propagation effects of the environment. To account for the propagation effects throughthe atmosphere the parabolic equation (PE) methods have be used widely [2, 6, 23] and have also been applied
to the models of wind turbine noise [4, 10, 11, 5]. The new formulation of the parabolic equation as given
by Ostashev et al. [16] is used to compute the effect of the refraction and ground reflection of the sound in a

moving medium. The effect of scattering due to turbulence in the atmosphere that is accounted for through theHarmonoise model [19] which we implement here. In this study we synthesize noise from a Megawatts windturbine which includes the propagation effects while benefiting from the moving monopole model [5] coupledwith the synthesis tool for wind turbine noise [13]. We first describe briefly the wind turbine noise model in
Section 2 and present a few test cases in Section 3.

Figure 1. Schematics for the wind turbine sound synthesis tool with the receiver represented as a sphere.

2

WIND TURBINE NOISE SYNTHESIS MODEL

The sound pressure level (SPL) in dB observed by a receiver from a point source in the far-field is given by[20]:
SPL = SPL f f + ∆L − αabs R,(1)

where SPL f f (dB) is the sound pressure level observed at the receiver position in free-field, ∆L (dB) is thesound pressure level relative to free field, αabs (dB/m) is the atmospheric absorption coefficient [20] and R (m)is the direct distance between the point source and the receiver. The SPL f f is characteristic of the source while(∆L − αabs R) includes the effects observed due to propagation of the sound through the atmosphere. For thewind turbine system, the noise from the blades can be modelled as multiple point sources along the lengthof the blade, whose individual sound propagates through the atmosphere [22, 5]. In the wind turbine noisemodel, the blades are divided into a number of segments whose contribution to the total sound is accounted forseparately (Fig. 1) for the receiver at the position defined by the distance from the hub xR , the height of thereceiver zR and the orientation of the receiver with respect to the wind direction θ .
The wind turbine noise source model developed by Tian and Cotté [22] predicts the trailing edge noise(TEN) and turbulent inflow noise (TIN) generated by a segmented wind turbine blade based on Amiet’s theory.
The presented wind turbine noise model adapts this approach and predicts the SPL f f (dB) for the TIN and
TEN for a receiver in free field for each segment at each discrete angular positions. The TEN is computed
using Goody’s model of the wall pressure spectrum (WPS) [7] for the pressure side and Rozenberg and Lee’smodel of the WPS [12, 18] for the suction side of the airfoil. The TIN is computed using the Kolmogorov
energy spectrum which is determined based on the turbulent dissipation rate (TDR) ε(m2 /s3 ). The frequencydomain model approximates the complete rotation of the blade as a series of translations with discrete angular
positions. Following Sinayoko et al. [21], the convective amplification and Doppler effect observed on accountof the moving blades is considered to obtain the the instantaneous PSD.
A few of the propagation effects that have an influence on the sound in the outdoor environment involve
refraction due to the wind speed profile, ground reflection, atmospheric absorption, scattering due to turbulence

in the atmosphere [20]. We implement the wide angle parabolic equation in a moving medium (WAPE-MM) to
account for the effect of the sound refraction and ground reflection obtaining the term ∆LPE [16]. The influenceof the ground impedance parameters in the form of the normalized admittance for the boundary conditions of the
WAPE-MM is important in the interference patterns caused by the wave reflected off the specific ground. Theseground impedance parameters can be estimated for a real ground using the method suggested by Guillaume et.
al. [8] for the Miki model [14] which relate to the impedance observed in terms of the air flow resistivity of
the ground σ (kNs/m4 ) and the effective thickness e(m). The estimated impedance parameters for a few realistic
grounds are included in this model. For the effect of scattering due to turbulence, the Harmonoise model [19] is
implemented to obtain the term ∆Lscat . For consistency with the source model, the turbulence energy spectrum
used is the Kolmogorov spectrum. The total relative sound pressure observed by the receiver is thus given by:∆L = 10 log10 (10∆LPE /10 + 10∆Lscat /10 ),(2)which accounts for the refraction due to the wind speed profile, ground reflection and the scattering due to theatmospheric turbulence.
For this study, the WAPE-MM implemented is calculated using a grid resolution of ∆x=∆z < λ /20, where
λ is the wavelength of the propagated sound. A second-order Salomons initial starter at xS = 0 is used at therequired source height zS . The wind speed profile moving in the positive x-direction implemented follows thepower-law profile given by:Ux (z) = Ure f

z
zre f

α
(3)

where Ure f is the reference wind speed taken at the hub height zre f =80m and α is the wind shear exponent.
The effective wind speed profile observed for a receiver at the orientation θ is thus: U = Ux cos(θ ).
For the calculation of the ∆L, the source and the receiver positions are evidently important parameters in
the influence of the propagation effect. However, to be coupled with the emission model described above,the computation of the propagation effect for each segment of the blade at each height is expensive and time
consuming. We benefit from the moving monopole model which suggests the computation of ∆L for a finite
number of heights and approximates the height of each the noise source of the wind turbine model to thenearest-neighboring height [5]. Following the suggestion that the approximation of 7 heights or more in themoving monopole model provides satisfactory results for the implementation of the propagation effects, we
compute ∆L for 7 heights distributed between the highest and lowest heights of the blade noise sources.
With the SPL f f computed for the source model and ∆L − αabs R to account for the propagation effects, we
obtain the frequency response for each moving segment at each discrete angular position. To synthesize a time
signal from the frequency-domain of the wind turbine noise, we use the method of Mascarenhas et al. [13] bysynthesizing the contribution of each blade segment at every angular position by availing of the Inverse Discrete
Fourier transform. The individual time-signals of the segments at each angular position termed as grains are
multiplied with an adaptive window and arranged on the basis of their corresponding propagation time. The
adaptive window is used with an overlap of Ψ=100% to account for the different propagation times of eachsegment which is suggested [13]. As each of the blade segments in rotation are assumed to be uncorrelatedwith individual contribution to the total noise, the grains are synthesized separately for the TEN and TIN andthen summed together at the corresponding time step. The synthesis of the total wind turbine noise is donewith the inclusion of the propagation effects for a few test cases that are described in the next section.

3

TEST CASES

The modelled wind turbine has a hub height of H = 80 m with a blade span of 45 m divided into 8 segments
and rotational speed Ω = 1.47 rad/s for the blade moving with Nβ = 36 discrete angular blade positions. The
receiver is placed at a distance of xR and constant height zR = 2 m with the orientation θ with respect to thewind direction (Fig. 1). The implemented wind speed profile follows the power-law profile as in Eq. (3) withthe wind speed Ure f = 8m/s at the reference height zre f = 80m and wind shear exponents α. The SPL f f is

Table 1. Test cases done with xR = 500m, TDR ε = 0.0115m2 /s3 , Ground parameter for Grass-summer: σ =354kNs/m4 and e = 0.0157m, Grass-winter: σ = 631kNs/m4 and e = 0.006m.
Cases

θ (deg)GroundWind shear exponent α

A

80

Free field

-

B

180

Free field

-

C

80

Grass-summer

0.5

D

180

Grass-summer

0.2

E

180

Grass-winter

0.3

computed for the TEN and TIN between the frequencies between 100 Hz and 6000 Hz. For the propagation
model, the influence of the scattering due to the turbulence ∆Lscat and the atmospheric absorption αabs R are
calculated for the same frequencies and all the segments at each height. The ∆LPE is calculated from 100 Hz
until 2300 Hz using Nhts =7 heights which are equally distributed between 35m to 125m as per the moving
monopole model. The average value of ∆LPE of the 3rd octave frequency band at 2000 Hz is taken as the valueof ∆LPE for the frequencies between 2300 Hz and 6000 Hz. We synthesize the signals of the wind turbine noiseat the sampling frequency of fs =44.1kHz between the frequencies fmin = 100 Hz and fmax = 6000 Hz for the few
test cases which are described below. More signals are provided here: [link]. In the signals provided, 0 dBFS
corrsponds to 0.1Pa. The spectrograms of the test cases are shown in Fig. 2 to 4. For a quantitative comparisonof the different test cases, the time averaged SPL is plotted in Fig. 5a. The maximum computed difference inthe amplitude that is observed by the receiver during the rotation of the wind turbine blades for each frequency
is seen in Fig. 5b as the amplitude modulation (AM).

Figure 2. Spectrogram for cases A (left) and B (right)
From comparing the cases A, B with C, D and E it can be seen that the effect of the propagation issignificant in the spectra generated as well as the AM (Fig.5). From case A and C that effect of the ground canbe seen in the interference dips caused by the direct and ground reflected wave while the AM is the same formost frequencies but different for others. In the downwind condition as seen in cases B and D the interference
dip is also seen but the AM is significantly different. This is because the wind turbine blades rotating in
the inhomogeneous atmosphere experiences different amounts of propagation effects in the term ∆L − αabs R at
different heights. The effect of the type of ground and the wind shear exponent is seen in the cases D and
E where the SPL is not very different as compared to the computed AM. For all the cases that include thepropagation effects (C, D and E), it is seen that the generated AM is not the same for all frequencies.

Figure 3. Spectrogram for cases C (left) and D (right)

Figure 4. Spectrogram for case E

4

CONCLUSIONSWe present a physics-based synthesis model of wind turbine noise in a moving medium with ground effect andturbulence. The source model is based on Amiet’s theory for the TEN and TIN. The propagation model uses
the wide angle parabolic equation in a moving medium to consider the refraction due to the wind speed profileand ground reflection off the semi-porous ground. The effect of scattering due to turbulence is accounted forby using the Harmonoise model based on the Kolmogorv spectrum. The synthesis method of Mascarenhas etal. [13] is used to obtain time signals of the wind turbine noise under the specific desired conditions.
From the test cases it can be seen that the effect of the propagation significantly changes the response
observed in free-field. The model is able to capture the effect of the wind profile on the propagated sound as
well as the ground effect with realistic ground impedance parameters. The individual effects of the controllable
parameters that contribute to the predicted sound can be studied further with the help of this model.
The effect of the scattering due to turbulence is mostly significant in the shadow zone of a particular point
source. The computed result is a certain obtained level in the shadow zone depending on the amount of required
turbulence. However, in the wind turbine noise model as an extended source, the effect of scattering of thesound in the shadow zone is not the same for all the heights and is thus the overall effect is not very significant.
The turbulence effect on the phase and amplitude fluctuation may be important components which add realism
to the signal which are yet to be added to complete the model. This physics-based synthesis model is relatively
adaptive to the requirements of the study. This makes it an useful tool for further studies that can avail of
this model. One such application is that use of this model for psychoacoustic analysis of the contribution

(a)

(b)

Figure 5. (a) Time-averaged SPL and (b) AM of the test cases that are given in Table.1
of individual components of the model on the impact of wind turbine noise. With this model we achieve a
straightforward way to synthesize the noise observed from one wind turbine. We can proceed in this same
course to synthesize an entire wind farm with this tool and thus study the noise on wind farm in the designphase itself .

ACKNOWLEDGEMENTS
This project has received funding from the European Union’s Horizon 2020 research and innovation program
under the Marie Sklodowska-Curie grant agreement No 812719.

REFERENCES[1] F. Bertagnolio, H. A. Madsen, and A. Fischer. A combined aeroelastic-aeroacoustic model for wind turbine
noise: verification and analysis of field measurements. Wind Energy, 20(8):1331–1348, 2017.
[2] P. Blanc-Benon, L. Dallois, and D. Juvé. Long range sound propagation in a turbulent atmosphere within
the parabolic approximation. Acta Acustica united with Acustica, 87(6):659–669, 2001.
[3] P. Bórawski, A. Bełdycka-Bórawska, K. J. Jankowski, B. Dubis, and J. W. Dunn. Development of windenergy market in the european union. Renewable Energy, 161:691–700, 2020.
[4] B. Cotté. Coupling of an aeroacoustic model and a parabolic equation code for long range wind turbine
noise propagation. Journal of Sound and Vibration, 422:343–357, 2018.
[5] B. Cotté. Extended source models for wind turbine noise propagation. The Journal of the Acoustical Societyof America, 145(3):1363–1371, 2019.
[6] B. Gauvreau, M. Bérengier, P. Blanc-Benon, and C. Depollier. Traffic noise prediction with the parabolic
equation method: Validation of a split-step padé approach in complex environments. The Journal of the
Acoustical Society of America, 112(6):2680–2687, 2002.
[7] M. Goody. Empirical spectral model of surface pressure fluctuations. AIAA journal, 42(9):1788–1794,
2004.

[8] G. Guillaume, O. Faure, B. Gauvreau, F. Junker, M. Bérengier, and P. L’Hermite. Estimation of impedancemodel input parameters from in situ measurements: Principles and applications. Applied Acoustics, 95:27–
36, 2015.
[9] K. L. Hansen, P. Nguyen, G. Micic, B. Lechat, P. Catcheside, and B. Zajamšek. Amplitude modulated
wind farm noise relationship with annoyance: A year-long field study. The Journal of the Acoustical Societyof America, 150(2):1198–1208, 2021.
[10] B. Kayser, B. Gauvreau, and D. Ecotière. Sensitivity analysis of a parabolic equation model to ground
impedance and surface roughness for wind turbine noise. The Journal of the Acoustical Society of America,
146(5):3222–3231, 2019.
[11] B. Kayser, B. Gauvreau, D. Écotière, and V. Mallet. Wind turbine noise uncertainty quantification for
downwind conditions using metamodeling. The Journal of the Acoustical Society of America, 151(1):390–
401, 2022.
[12] S. Lee. Empirical wall-pressure spectral modeling for zero and adverse pressure gradient flows. AIAA
Journal, 56(5):1818–1829, 2018.
[13] D. Mascarenhas, B. Cotté, and O. Doaré. Synthesis of wind turbine trailing edge noise in free field. JASA
Express Letters, 2(3):033601, 2022.
[14] Y. Miki. Acoustical properties of porous materials-modifications of delany-bazley models. Journal of the
Acoustical Society of Japan (E), 11(1):19–24, 1990.
[15] S. Oerlemans and J. G. Schepers. Prediction of wind turbine noise and validation against experiment.
International journal of aeroacoustics, 8(6):555–584, 2009.
[16] V. E. Ostashev, D. K. Wilson, and M. B. Muhlestein. Wave and extra-wide-angle parabolic equations forsound propagation in a moving atmosphere. The Journal of the Acoustical Society of America, 147(6):3969–
3984, 2020.
[17] E. Pedersen and K. Persson Waye. Perception and annoyance due to wind turbine noise—a dose–response
relationship. The Journal of the Acoustical Society of America, 116(6):3460–3470, 2004.
[18] Y. Rozenberg, G. Robert, and S. Moreau. Wall-pressure spectral model including the adverse pressure
gradient effects. AIAA journal, 50(10):2168–2179, 2012.
[19] E. Salomons, D. Van Maercke, J. Defrance, and F. de Roo. The harmonoise sound propagation model.
Acta acustica united with acustica, 97(1):62–74, 2011.
[20] E. M. Salomons. Computational atmospheric acoustics. Springer Science & Business Media, 2001.
[21] S. Sinayoko, M. Kingan, and A. Agarwal. Trailing edge noise theory for rotating blades in uniform flow.
Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469(2157):20130065,
2013.
[22] Y. Tian and B. Cotté. Wind turbine noise modeling based on amiet’s theory: Effects of wind shear andatmospheric turbulence. Acta Acustica united with Acustica, 102(4):626–639, 2016.
[23] M. J. White and K. E. Gilbert. Application of the parabolic equation to the outdoor propagation of sound.
Applied Acoustics, 27(3):227–238, 1989.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0401

Roaming in the concert hall: a customized interactive immersive
system for professional musicians and appreciators
Junzhe Liu2,4, Jingyu Liu1,2,3,4*, Jiaxing Zhu5,6 and Zijin Li7
1State Key Laboratory of Media Convergence and Communication, Communication University of China, China
2Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, China
3Beijing Key Laboratory of Modern Entertainment Technology, Communication University of China, China
4School of Information and Communication Engineering, Communication University of China, China
5School of Architecture, Tianjin University, China
6Department of Musicology, China Conservatory of Music, China
7AI music and Music Information Technology Department, Central Conservatory of Music, China

ABSTRACT
Virtual concerts have become an emerging form of music performance, and it casts a perspective on thedevelopment and exploration of real-time interactive virtual spaces as well as the creation of digital twins.
This paper introduces a customizable interactive immersive concert hall system, which can meet the diverse
needs of different roles such as audiences, musicians, conductors and audio engineers. It supports switching
between different concert hall scenes, changing the placement of orchestra, adjusting instrument volume,and listening to selected instrument ensembles while performing in the concert hall. The system also
supports real-time switching between different music pickup types and microphone frequency responses.
The system simulates the architectural acoustic environment of outstanding concert halls around the world,
and is enabled with real-time auralization.
Keywords: Concert Hall Acoustic Simulation, Real-Time Auralization, Interactive Immersive Acoustic
Space

1. INTRODUCTION
With the development of computer science and digital media technology, virtual reality
technology is no longer used only as an image rendering tool. More and more interactive virtualenvironments are being applied across borders in the fields of architecture, industry, education, andtourism, creating results that drive the industry forward. Music performance, as one of the industries
with the highest demand for audio technology, should become an important object of research andapplication for virtual acoustics. A certain number of scholars have already launched related
research, some stage acoustic simulation techniques and real-time audible functions have beendeveloped, related acoustic evaluations have been carried out. The proven acoustic reliability ofvirtual scenes calls for more applied R&D. A development plan for an application-based system
which is close to the needs of the industry was created.
The need to recreate the architectural scenes of a real concert hall in a virtual system is atechnical requirement that is critical to the experience of concert hall audiences and music
appreciators. In addition, for musicians in the performance industry, the development of virtual
platform-based rehearsal and performance simulation systems can help artists shift the stage fromoffline to online to help solve the current situation of touring difficulties, talent loss, and difficultyin establishing cultural and artistic brands in the context of the epidemic.In addition, the public
appreciation mode is gradually changing from passive appreciation to active control, and as more
and more music applications add features such as frequency equalization and reverb switching, wehave seen the threshold of audio technology lowered and music applications are looking forward to atechnological and conceptual innovation. Instead of following the footsteps of increasinglyprofessional music lovers to upgrade product features, professional audio features should be added
directly to mass-market application products. A full chain of applications serving the music
performance industry should be designed, and more people should be encouraged to reach out to thespecifics of professional audio, regardless of their background.
In this paper, we propose a customized interactive immersive system serving professionalmusicians and appreciators based on auralization, virtual acoustics, and architectural acoustics, withan eye on the various dilemmas and needs of different players in the music performance industry inthe context of the times.

2. RELATED WORK AND BACKGROUND
In 1999, virtual acoustics was summarized as sound source modeling, transmission medium(room acoustics) modeling, and receiver (listener) modeling, and this concept ofsource-room-listener model has been used in the field of virtual acoustics until today (1).
Auralization belongs to a subordinate branch of virtual acoustics in the sense of acoustic modelingand playback, and the continuous integration of new simulation algorithms, sound synthesis andsignal processing tools has led to the rapid development of auralization (2). In this paper, theresearch work related to virtual acoustics involving audibility is organized into the development ofroom acoustic simulation technology and the construction of interactive virtual acoustic
environments for a variety of application scenarios, which will be presented in the following
sections.
Room acoustic simulation starts from sound propagation theory and builds a variety of physicalpropagation models including reflection, diffraction, and other acoustic phenomena through computer
simulation algorithms (3). Spatial sound rendering is the core technology used to simulate realistic
auditory environments, and a large number of innovative sound rendering algorithms have been proposedthat can be summarized as acoustic wave equation methods (also known as wave-based methods),
geometric methods, and hybrid methods. The advantage of wave-based techniques lies in the realism andaccuracy of sound propagation simulation calculations, such as WAVE (4). Geometric methods include
image-source methods, ray tracing, and beam tracing, which are flexible, high-speed, and easy to interact
with (5). Due to the development of software science, a framework for real-time simulation of room
acoustics based on geometric methods has been developed. Examples are RAVEN (6), EVERTims (7).
Hybrid methods select wave-based methods with the use of several of the geometric methods. Somehybrid methods using two or more geometric methods are used in commercial tools for acoustic
simulation, such as Odeon (8) and EASE (9) .
On the other hand, virtual reality technology enhances the immersion of virtual acoustic environments
and raises the expectation of two additional dimensions, i.e., multimodality and real-time interaction (10).
Common application scenarios include urban noise planning (11), architectural noise studies (12-14),
audio-visual sensory studies integrated into soundscapes (15), soundscape reproduction (16), and acoustic
reproduction and preservation of ancient buildings (17). There are also studies that integrate virtual
reality with music performances to create virtual performance environments (18, 19). McGill University
developed the VAT (The Virtual Acoustics Technology) system (20) and investigated the effect of virtual
acoustic systems on musicians' perception of room acoustics and their musical performance. The virtualenvironment was also used as a virtual acoustic practice room to study the vocal behavior of female
singers in different virtual acoustic environments and the effect of the room acoustic environment on
vocal perception (21).
Room acoustic simulations focus more on technical development and algorithmic innovation, andmost of their research results are presented in the form of plug-ins, frameworks and systems thatprovide strong technical support for subsequent research projects. Interactive virtual acoustic
environments, on the other hand, build on existing technological tools to further extend theapplication of virtual spaces to a wider range of scenarios. Based on the existing platform anddevelopment tools, this paper carries out the development of a virtual concert hall application systemwith integrated practical functions with human design at its core to solve practical situational
dilemmas.
Starting from the field of music performance, this paper proposes a multifunctional customizableinteractive immersive concert hall system that combines the usage needs of audiences, musicians,
conductors and audio engineers. Considering the performance requirements of fast real-time
computation, geometric acoustic-based simulation algorithms are the best choice to achieve acoustic
interaction, and a multi-engine based cross-platform development pipeline will be used to fill thetechnical gaps between professional development platforms through flexible switching and joint

collaboration between them to form a complete and comprehensive solution.

3. FUNCTIONAL DESIGNIn this paper, we develop a customized interactive immersive concert hall system in which people
can roam. Taking the Chinese Orchestra as an example, the system is equipped with eleveninstruments and simulates the real-time acoustic environment of an interactive immersive concert
hall during a musical performance. The system also incorporates a user interaction module that willallow users to make real-time customizations to the music being played in the system and the systemacoustic environment. The core design concept involving virtual reality, sound transmission, soundpropagation and other working steps is shown in Figure 1.

Figure 1 – Functional design of the system
3.1 For Concert Hall Audiences
Through sound, vision, touch, smell, and taste, multi-sensory integration into a scene can create
an immersive experience (22). In order to simulate the experience of being there, immersion
becomes the top issue to be considered in the design of audience-oriented concert hall system.The
system provides an immersive experience for the concert hall audience through a combination ofaudio and visual senses. In the system we develop a roaming function, and the camera is specially set
to the first-person view. The virtual character can be manipulated to roam in various locations of theconcert hall, standing and listening. To better restore the experience of identifying listening positionin the real world, the system adds the head turn function, audiences can control the virtual roaming
character's head turning through the keyboard direction keys, thus, the change in the orientation of
the sound source is perceived in the process.
Considering that comparing different part of the sitting area is the most urgent demand of theaudience, this concert hall system realizes the interaction between the listener and the sound field
through real-time audible simulation. During the roaming process, as the position of the characterchanges, the interactions between the character and the nearby building and sound sources arecalculated by the computer in real time and fed back as a ray carrying sound information. Whatmight happen is, the listener roam around to feel the effect of the concert in different positions, andthey can also walk to the stage to experience the working point of view of the orchestra conductor,
standing by each instrument up close and get to know its acoustic characteristics.
3.2 For Orchestra & Conductor
The system's functions for orchestra musicians are designed to focus on the dilemmas they face

when working on a global tour. Virtual rehearsals are one of the solutions to the time, travel, andmanpower challenges faced by touring orchestras, and this concert hall system can be used as a
virtual rehearsal hall to simulate the performance of music in different arrangements and genres. The
system is also suitable for open house events and online performance venues. To reproduce thearchitectural acoustics of a real concert hall, this system uses both virtual reality technology andbinaural room impulse response, simulates an acoustic environment based on sound propagation
principles, geometric feature models, and acoustic materials.
The system enables switching between different concert hall scenes with the click of a button.
Musicians can rehearse online without being physically present in the real world. This system is alsopreset with a variety of classic orchestra position modes for the operator to switch at a single click,and it enables dragging and dropping to adjust instrument positions during performances. Thisfunction can be used to research the stage acoustics of classical instruments, improved instruments,
and innovative instruments in different positional situations.
3.3 For Audio Engineers
The system is also designed with a range of audio functions for professionals to practice theirskills in the virtual workplace, as well as for music lovers to experience and explore. Given that
digital audio workstations are the most familiar working environment for audio professionals, aninteractive control system in the form of a digital audio workstation is the core requirement. This
concert hall system creates several comprehensive control panels that support customized
adjustments, and all parameter variable changes will have an effect while the virtual orchestra isperforming, without causing pauses or jams.
For mix engineers and music acoustics researchers, the volume of each instrument track in theorchestra is allowed to be adjusted, and the system supports selective playback and mute whenmixing specific instrument groups or adjusting the overall sound level of the piece. The system also
supports switching and bypass of the reverb effect.
The major work of recording engineers and stage technicians is to deal with various microphones,
and the selection and matching among multiple microphones need to be especially analyzed anddesigned. Recording engineers are tasked with delivering first-hand music, but it is difficult to growquickly without hands-on learning in the field, and one solution is to simulate recording online foradequate preparation. As a result, the system supports switching and critically listening to audio files
recorded with different pickup types and frequency responses of microphones. The function can also
contribute to building a digital microphone library.

4. TECHNICAL FRAMEWORK
4.1 Concert Hall Modeling
3D virtual reality technology has greatly assisted the room acoustics simulation needed forarchitectural acoustics and virtual acoustics research due to its real-time development and flexible
integration characteristics. Unity, the industry-leading real-time 3D development engine, is chosen
as the scene rendering platform, which has strong interactivity and cross-platform characteristics,
and can be integrated with professional modeling software and interactive audio engine, while thedeveloped UI system is also carried on the engine, and eventually the above is integrated into an
application system.
The system includes one concert hall model, one scene roaming character model, and eleven
Chinese Orchestra instrument models. The scripting of character roaming and instrument moving
functions is done through C#, a programming language supported by Unity.
4.2 Sound Field Simulation
The audio processing workstation for this system is built outside of Unity and is completed in theaudio middleware Wwise, a leading interactive audio design and development tool that supports
cross-platform development, can be flexibly embedded in the system production pipeline, and offerssome powerful spatial audio rendering plug-ins. As shown in Figure 2, the sound processing chain
includes the specific handling of direct sound, early reflection, and late reverberation.

Figure 2 – Technical process and framework of audio signal flow
4.2.1 Object-Based Direct Sound Setting
In this system, sounds are set up as point sources with their position coordinates given to the
instrument models in the scene and changed as the instrument models move. Attenuation modelingto simulate the natural sound decay is done in this step, and all sources are classified and thecorresponding distance decay curves are created. To simulate the airborne effect, recursive filters fordifferent frequencies are also used. The directional patterns of instruments in spatial audio isaccomplished by cone attenuation, as the listener's orientation changes, such as forward, sideways,
and backward, different degrees of volume attenuation and filtering effects occur, and the listener
can directly perceive the process of sound change with direction.
4.2.2 Early Reflection Simulation
Early reflections reveal the size and shape of the room, and are directly related to the distance,
direction, and material information of various reflective obstacles in the propagation path, which
together constitute the listener's judgment of spatial location, and this part should be detected andcalculated in real time. Wwise Reflect, an early sound reflection simulation plug-in developed by
Audiokinetic was chosen to help construct early sound reflection models based on geometric forms
with advanced algorithms. It dynamically presenting early reflections that change in real time basedon the distance and directional relationship between the sound sources and the listener. In this paper,plug-in algorithm, geometric modeling, filter design and other methods are used. The acoustic model
including the walls, acoustic panels, and audience seating area is established. Segmented matchedfilters are designed for simulating different sound absorption materials. The spatialized earlyreflection sound as well as the physical phenomena of diffraction and transmission of the concert
hall model are calculated and simulated based on the image-source method and real-time ray casting
technology.
4.2.3 Late Reverberation Simulation
The Wwise Convolution plug-in is chosen for the system, and the late reverberations of theconcert hall system are simulated by means of parameter-based control. We selected nine stereo
impulse response signals recorded from famous concert halls as well as natural, living scenes fromthe Bricasti M7 and EchoThief impulse response libraries. The binaural room reverberation sounds
are obtained after convolving the impulse responses with the music stem signals. Multiple auxiliary
buses mounted with convolution reverb effects are added to the audio system to form acoustic
perception of different concert halls, each one will be triggered by the corresponding operator
commands while the system is running.
One issue to be discussed is that the convolution reverb can only convolve one impulse response
signal with dry sound at a single time, and the impulse response signal is actually formed based on
one single point in the concert hall. Strictly speaking, what is perceived is the spatial audio
information standing at one particular point in the concert hall. When the reproduction environment

is changed from a flat 2D to a virtual reality 3D scene, especially when the scene characters canroam in the auditory environment, with the listening position changes, the convolution reverb will nolonger accurately reveal the real spatial listening sensation, which means that the offline renderedreverb effect is static and it lacks the spatial position information, nor does it support the renderingof sound involving head turn positioning. More sound interaction information related to the motion
state of the roaming character should be added to the reverb sound simulation, and one approach thatis used in this concert hall system is to process the dry sound separately from the wet sound. The
convolution reverberation auxiliary bus outputs only wet reverberation, and is set with an attenuation
curve that increases in amplitude as distance increasing. The difference in distance between thelistener and the sound source contributes to the spatial location information. As the wet and dry
sounds transmit through the virtual space, they are smoothly rendered into one reverberation withintegrated information.
4.3 Implementation of the System
As shown in Figure 3, the communication between Wwise and Unity is based on the logic of
Audio Event-Based Packaging, where all audio materials, events and state attributes are packaged
into soundbanks. Through API it can be sent to Unity, where a series of event commands can becalled using C# scripts.

Figure 3 – System implementation
The system outputs audio in a two-channel stereo format. For headphone reproduction, the audio
hardware Nx head tracker developed by Waves is suggested, which helps simulate the effect which is
closer to a speaker reproduction way with algorithms for channel interleaving, filtering and delay,
sound reflections, and sound field displacement.

5. PERFORMANCEThe user interface of this system includes the main control panel, the reverb switching panel, andthe position reset panel.
The main control panel includes volume control for four groups of instruments: wind, plucked,
string bowed and percussion, as well as group mute. Reverb bypass is available by checking
toggle.There is also a button to activate the reverb switching panel. The panel is shown in Figure 4.

Figure 4 – Main control panel
Reverb switching panel includes five classic concert halls in Amsterdam, Berlin, Boston, Chicago,
Vienna and four nature living scenes including bridge, glacier, cathedral and ball court. The panel isshown in Figure 5-6.

Figure 5 – Reverb switching panel - Halls

Figure 6 – Reverb switching panel - Others
Position reset panel includes three switching buttons. Three types of orchestra positionings are
selected by referring to the performances of the Hong Kong Chinese Orchestra and the China Radio
Nationalities Orchestra. The panel is shown in Figure 7.

Figure 7 – Position reset panel
A real-time profiler is used during the test work to verify the simulation of acoustic phenomena
such as reflections and diffractions based on geometric acoustics. The sound sources, virtual soundsources, reflection and diffraction rays are shown in Figure 8. The system tests show that the
instrument sources are accurately positioned, that the instrument groups are balanced and wellintegrated, and that the visual and aural coherence is largely achieved in terms of the breadth anddepth of the virtual space.

Figure 8 – Refections and diffractions

6. SUMMARY AND FUTURE WORKIn this paper, we develop a customized interactive immersive concert hall system by means ofdigital twins, virtual reality, sound field simulation, and interactive function development. Multiple
functions are designed for audience, musicians, conductors and audio engineers. We completes themodeling and scene rendering of the concert hall in the virtual reality engine, and integrates theinteractive audio development middleware into the system production pipeline, taking the physicalpropagation model of direct sound - early reflection - late reverb as the starting point, simulates theacoustic phenomena of reflections, diffractions, diffusions, absorptions, and transmissions based on
the geometric acoustic methods. In the system, we presents the major famous concert halls'
acoustical characteristics, and the real-time auralization functions of the music performance isrealized.
The audio implementation scheme used in this concert hall system is expected to be improved.
There is a degree of functional overlap between the early reflection and convolution reverb plug-ins
when they are used simultaneously. Currently, this blurred boundary can only be delineated bymanually cutting off the early reflection portion of the impulse response signal. In the future a
simpler and more versatile audio implementation process should be developed and used to achieve a
better immersion of the system. In the following research, models of concert halls with different
volumes and acoustic architectural features will be added to the system. The system will also enrich
the instrument library and music library, and the simulation of different instruments' acoustic
characteristics will be carried out. Basic parameters such as sound power level and directional
patterns of instruments' sound radiation will be benchmarked against professional measurement
results. Internal audio recording function will also be developed to enable virtual recording withcustomizable microphone types and placement. In the mean time, a multi-person online version will
be attempted in the future, allowing people in different places to meet and work together in a virtual
concert hall. The system will also extend its application scenarios. One possible proposal is todevelop an online educational system to fulfill the need for quality music education as well as
enlightenment in the subject of audio technology in colleges.

ACKNOWLEDGEMENTS
This work was supported by Funds for National Natural Science Foundation of China(62101514).

REFERENCES
1. Savioja, Lauri, et al. "Creating interactive virtual acoustic environments." Journal of the AudioEngineering Society 47.9 (1999): 675-705.
2. Vorländer, Michael, and Jason E. Summers. "Auralization: Fundamentals of acoustics, modelling,simulation, algorithms, and acoustic virtual reality." Acoustical Society of America Journal 123.6(2008): 4028.
3. Pind, Finnur, et al. "Acoustic virtual reality – methods and challenges." Proceedings of Baltic-Nordic

Acoustic Meeting (BNAM), Reykjavík, Iceland. 2018.
4. Mehra, Ravish, et al. "Wave: Interactive wave-based sound propagation for virtual environments." IEEE
transactions on visualization and computer graphics 21.4 (2015): 434-442.
5. Lakka, Eftychia, et al. "Spatial sound rendering–a survey." IJIMAI 5.3 (2018): 33-45.
6. Schröder, Dirk, and Michael Vorländer. "RAVEN: A real-time framework for the auralization ofinteractive virtual environments." Forum acusticum. Denmark: Aalborg, 2011.
7. Poirier-Quinot, David, Brian Katz, and Markus Noisternig. "EVERTims: Open source framework forreal-time auralization in architectural acoustics and virtual reality." 20th International Conference on
Digital Audio Effects (DAFx-17). 2017.
8. Naylor, Graham M. "ODEON—Another hybrid room acoustical model." Applied Acoustics 38.2-4(1993): 131-143.
9. Ahnert, Wolfgang, and Rainer Feistel. "EARS auralization software." Audio Engineering Society
Convention 93. Audio Engineering Society, 1992.
10. Vorländer, Michael. "Virtual acoustics: opportunities and limits of spatial sound reproduction."
Archives of Acoustics 33.4 (2008): 413-422.
11. Jiang, Like, Luigi Maffei, and Massimiliano Masullo. "Developing an online Virtual Reality applicationfor e-participation in urban sound planning." 9th Iberian Acoustics Congress, Porto, Portugal. 2016.
12. Muhammad, Imran, Michael Vorländer, and Sabine J. Schlittmeier. "Audio-video virtual reality
environments in building acoustics: An exemplary study reproducing performance results andsubjective ratings of a laboratory listening experiment." The Journal of the Acoustical Society ofAmerica 146.3 (2019): EL310-EL316.
13. Doggett, Rachel, et al. "Using virtual reality to evaluate the impact of room acoustics on cognitive
performance and well-being." Frontiers in Virtual Reality 2 (2021): 20.
14. Wyke, Simon Christian Swanström, et al. "Real-time Evaluation of Room Acoustics using IFC-based
Virtual Reality and Auralization Engines: Documentation Report." (2019).
15. Stevens, Francis, Damian T. Murphy, and Stephen L. Smith. "Soundscape auralisation and visualisation:a cross-modal approach to soundscape evaluation." DAFx 2018 (2018).
16. Fırat, Hasan Baran, et al. "The soundscape reconstructions of the early 20th century vendor cries instreets of Istanbul and Naples with two 3D sound spatialization approaches." INTER-NOISE andNOISE-CON Congress and Conference Proceedings. Vol. 261. No. 3. Institute of Noise Control
Engineering, 2020.
17. Postma, Barteld NJ, et al. "Virtual reality performance auralization in a calibrated model of
Notre-Dame Cathedral." EuroRegio2016, Porto, Portugal (2016).
18. Hamilton, Robert. "UDKOSC: An immersive musical environment." ICMC. 2011.
19. Maempel, Hans-Joachim, and Michael Horn. "The virtual concert hall: a research tool for theexperimental investigation of audiovisual room perception." (2017).
20. Ko, Doyuen. Virtual acoustics for musicians: Exploring the influence of an electronic acoustic
enhancement system on music performance. McGill University (Canada), 2016.
21. Nelson, Heather R. The effects of actual recital hall and four digitally-produced variable practice room
environments on phonatory, acoustical, and perceptual measures of vocal performances by experienced
female singers. Diss. University of Kansas, 2016.
22. Tsingos N. Immersive Sound: The Art and Science of Binaural and Multi-Channel Audio. Agnieska
Roginska and Paul Geluso,(Eds.) Routledge, New York and Londres. Chap. Object-Based Audio,
244–275[J]. 2018.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0659

Sound reproduction methods for source localization in virtual-reality-based
automotive simulators.
Eun Soo Jo1; Dong Chul Park2; Juan Camilo Gil-Carvajal3; Cheol-Ho Jeong4; Wookeun Song5
1,2

Hyundai Motor Group, South Korea

3

Hearing Systems, Department of health technology, Technical University of Denmark

4

Acoustic Technology, Department of electrical and photonics engineering, Technical University of Denmark5

Hottinger Brüel & Kjær A/S, Denmark

ABSTRACT
Vehicle driving simulators allow to recreate the experience of driving a real vehicle. The implementation ofsuch simulators requires the use of sound reproduction methods, which may influence the sound perception
differently inside the vehicles. This study investigated sound localization in a virtual-reality-based driving
simulator for three sound reproduction methods: higher-order Ambisonics (HOA) reproduced with aspherical array of 64 loudspeakers, the matrix inversion method reproduced with a circular array including
eight loudspeakers and the binaural reproduction with headphones. Ten experienced drivers provided soundlocalization judgments in terms of perceived distance and horizontal angle. Sound recordings were madeinside a real vehicle and presented at five source positions inside the car in the virtual sound environment.
The simulated scenarios included the operating conditions of idling and constant speed at 100 km/h, as well
as a warning and a speech signal. The binaural reproduction over headphones outperformed the reproduction
methods with loudspeakers in terms of perceived horizontal angle, whereas perceived distance was moreaccurate for the methods including loudspeakers. Moreover, distance perception was more affected for the
100 k/m than for the idling condition. The results highlight the need to consider the sound reproduction
method when developing and testing vehicle features.
Keywords: VR-based automotive simulator, Higher order ambisonics, Binaural reproduction, Matrix
inversion, Multi-modal perception

1. INTRODUCTION
The virtual environment should be able to reproduce the perceptual cues obtained while driving areal vehicle. Sound source localization is one important perceptual attribute 1. Moreover, the filteringof the sound due to the listener’s head, pinnae and torso, which is described by the head -relatedtransfer functions (HRTF), influence distance perception, often also known as externalization. In
enclosed spaces, such as inside a vehicle, distance perception depends on the intensity of the soundsource and the direct-to-reverberant ratio 2. Virtual sound systems can be reproduced over headphones
by considering the position-dependent filtering effect due to the listener's anatomy 3. However, any
deviation of the listeners' anatomy from the physical features of the head and torso simulator (HATS)
would affect the accuracy of the reproduction. The spatial resolution of such methods, however, canbe compromised due to the finite number of loudspeakers 4. Thus, in-vehicle sound source localization
for a VR-based driving simulator is expected to also depend on the source position and type of source.
In this study, three sound reproduction techniques were compared: HOA reproduced through aspherical array of 64 loudspeakers, matrix inversion method played back through a circular array of
goodwater@hyundai.com
dc.park@hyundai.com
3
jucc@demant.com
4
chj@elektro.dtu.dk
5
woo-keun.song@hbkworld.com
12

8 loudspeakers, and binaural recordings reproduced over headphones. Sound localization performancewas examined for two types of sources (a speech excerpt and a warning signal), five source positions
and two operating conditions (idling and constant speed at 100 km/h).

2. Methods
2.1 Subjects
Ten subjects participated in the experiment. All subjects were drivers with five or more years ofcar driving experience, who reported to have normal hearing and normal vision. All subjects provided
written consent before testing, and all experiments were approved by the Science-Ethics Committee
for the Capital Region of Denmark (reference H-16036391).
2.2 Reproduction system
A head mounted display HTC Vive Pro was used to present the visual scenes, which consisted ofa first-person view from the front passenger seat. Sound source localization was evaluated for awarning excerpt and a speech excerpt. The specific source positions relative to the listener’s position
were: Front Left (FL), Front Center (FC), Rear Right (RR), Rear Left (RL), and Far Rear Left (FRL).
The signals were recorded in the front passenger’s seat of a sport sedan in three ways: (1) with thebuilt-in microphones of a HATS, (2) with an Eigenmike, i.e. a 32-microphone spherical array, (3) with
an 8-microphone circular array placed around the HATS’ head. During the experiment, the recordings
from the HATS were reproduced over headphones. The recordings from the Eigenmike werereproduced through a full 64-loudspeaker spherical array using HOA7, 8. The recordings from the 8microphone array were reproduced through 8 loudspeakers of the spherical array using the matrix
inversion method 5, 6.
2.3 Experimental procedure
The experiment was conducted in the Audio-Visual Immersion Lab located at the Technical
University of Denmark. The experiment consisted of three experimental testing sessions with aduration of 15-20 minutes each. The three experimental sessions were: sound reproduction with 64
loudspeakers (64L), eight loudspeakers (8L) or headphones (H). Each session contained the trials
corresponding to two operating conditions (idling and constant speed at 100 km/h), two signals(warning or speech), and five source positions. The order of trials was counterba lanced within eachsession, and the session order was counterbalanced across subjects. In each trial, localization ratings
were obtained in terms of perceived horizontal angle and distance. For this, the subjects were provided
with a user interface presented through the HMD. This was repeated until finishing the experimental
session, after which the subjects were asked to take a break of about five minutes.

3. Data analysis and Results
To evaluate sound source localization, relative errors were calculated as the difference between thepresented and perceived source locations for both horizontal angle and distance ratings. Significant
effects were evaluated with analyses of variance performed after fitting linear mixed-effects models
to the relative error ratings.
Localization error in terms of perceived angle was poor for all sound reproduction methods
investigated. The poor localization performance for the loudspeaker reproduction may be caused by
the fact that the recordings were done in the near field while the loudspeaker reproduction was donein the far field, with mean errors ranging from about -90° to 90°. Angle errors were position-dependent,
and varied slightly with the sound reproduction method, operation condition and type of signal.
The position-dependent effect on perceived horizontal angle across sound reproduction methods wasmore prominent for rear source positions, for which the reproduction method over headphones was
generally more accurate. In contrast, for frontal positions all three methods performed similarly. Thiswas confirmed by post hoc analyses, which revealed significant differences between 8L and H when
the sound was delivered from positions RR and RL for both the warning and speech signals, and fromposition FRL for the warning signal. Post hoc analyses also showed significant differences between
64L and H for the source positions RR and FRL when the signal was speech and warning, respectively.
In contrast, no significant differences were found between the 8L and 64L reproduction methods. Post
hoc tests were also performed to further investigate the effect of the operating condition and signa l.

Angle errors were similar for both operating conditions, except when presented from the position RR
in the 8L reproduction method. Thus, sound source localization in terms of perceived horizontal angle
was the same for both operating conditions across most source positions and sound reproduction
methods. Furthermore, the localization of the warning signal was more accurate than for speech in theoperating condition of 100 km/h when the reproduction method included loudspeakers .

ACKNOWLEDGEMENTS
We would like to thank the visual developer from Perfect Storm for providing the visuals for the VR
simulator.

REFERENCES
1. Middlebrooks, J. C., & Green, D. M. (1991). Sound localization by human listeners. Annual review ofpsychology, 42(1), 135-159.
2. Zahorik, P., Brungart, D. S., & Bronkhorst, A. W. (2005). Auditory distance perception in humans: A
summary of past and present research. ACTA Acustica united with Acustica, 91(3), 409-420.
3. Kim, S. M., & Choi, W. (2005). On the externalization of virtual sound images in headphone reproduction:
A Wiener filter approach. The Journal of the Acoustical Society of America, 117(6), 3657-3665.
4. Grimm, G., Ewert, S., & Hohmann, V. (2015). Evaluation of spatial audio reproduction schemes forapplication in hearing aid research. Acta Acustica United with Acustica, 101(4), 842-854.
5. Kirkeby, O., Nelson, P. A., Orduna‐Bustamante, F., & Hamada, H. (1996). Local sound field reproduction
using digital signal processing. The Journal of the Acoustical Society of America, 100(3), 1584-1593.
6. Song, W., Marschall, M., & Corrales, J. D. G. (2015, September). Simulation of realistic background noiseusing multiple loudspeakers. In Proc. Int. Conf. on Spatial Audio (ICSA), Graz, Austria.
7. Favrot, S., & Buchholz, J. M. (2010). LoRA: A loudspeaker-based room auralization system. Acta acustica
united with Acustica, 96(2), 364-375.
8. Malham, D. G., & Myatt, A. (1995). 3-D sound spatialization using ambisonic techniques. Computer music
journal, 19(4), 58-70.