A22:EDUCATION, PUBLIC OUTREACH,
AND HISTORY IN ACOUSTICS

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0704

Neurofeedback training of selective attention improves
speech-in-noise understanding
Inyong Choi1; Hwan Shim2; Kayla Howerton1; Sungyoung Kim2
12

Department of Communication Sciences and Disorders, University of Iowa, USA

Department of Electrical and Computer Engineering Technology, Rochester Institute of Technology, USA

ABSTRACT
Selective attention enhances cortical responses to attended sensory inputs while suppressing others, which
can be an effective strategy for speech-in-noise understanding. Emerging evidence exhibits a large variancein attentional control during speech-in-noise tasks, even among normal-hearing listeners. Yet whether
training can enhance the efficacy of attentional control and, if so, whether the training effects can betransferred to performance on a speech-in-noise task have not been explicitly studied. Here, we introduce aneurofeedback training paradigm designed to reinforce the attentional modulation of auditory evoked
responses. Subjects attended one of two competing speech streams. Our electroencephalography-based
attention decoder classified every single trial using a template-matching method based on pre-definedpatterns of cortical auditory responses elicited by either stream. The result of decoding was provided on thescreen as neurofeedback. After four sessions of this neurofeedback training over four weeks, the subjects
exhibited improved attentional modulation of evoked responses to the training stimuli as well as enhanced
cortical responses to target speech and better performance during a post-training speech-in-noise task. These
results indicate that the neurofeedback training may reinforce the strength of attentional modulation, which
likely improves speech-in-noise understanding. Our finding suggests a potential rehabilitation strategy forspeech-in-noise deficits.
Keywords: Speech-in-noise, auditory selective attention, neurofeedback training

1. INTRODUCTION
Understanding speech in noisy situations is essential for verbal communications in social settings.
However, persistent variability in speech-in-noise perception exists even in young listeners withnormal hearing (NH) thresholds1 . Yet, no therapeutic method has been established for NH listeners
who struggle with speech-in-noise (SiN) understanding. Conventional hearing remediations throughamplification do not always improve SiN ability (even when equipped with noise reduction
algorithms)2 . Instead, perceptual training3,4 is one of the very few options currently available for such
SiN problems in NH listeners, although a dearth of information exists regarding its efficacy andmediating biological mechanisms. Here, we aim to validate the feasibility of a novel auditory training
protocol to improve SiN performance in NH listeners and investigate what biological changes suchtraining induces.

2. Method
Our neurofeedback training paradigm aims to enhance the attentional modulation of cortical
auditory evoked responses. Twenty subjects (i.e., ten in each of the Experimental and Placebo groups)
participated in the training program of 4-week training. Both Experimental and Placebo Groups
listened to two concurrent speech streams: A female voice repeating “Up” five times and a male voice
repeating “Down” four times. The female voice was played from the loudspeaker on the left (-45°azimuth), while the male voice was played from the right (+45°). No physical overlap between
competing speech streams in time and location was intended to maximize stream segregation. Before
12

inyong-choi@uiowa.edu
hxsiee@rit.edu

the sound begins, subjects were instructed to attend to one of those streams by a visual cue saying“Target: Up” or “Target: Down”. 64-channel EEG was recorded throughout the experiment. Rightafter the sound, auditory selective attention was decoded from single-trial EEG signals based on thetemplate-matching classification method5,6 . That means, if the single-trial EEG waveform shows
greater similarity with the “Template: Down” than the “Template: Up,” the EEG decoder determinesthat “Down” was attended. Only the Experimental Group subjects received neurofeedback (i.e., thefixation cross on the computer screen moves upward if the EEG decoder determines that the “up”
stream was attended; it moves down if “down” was attended). The Control Group underwent a similar
selective attention training but did not receive the neurofeedback. Since the accuracy of decoding
attention from single-trial EEG signals reflects the strength of attentional modulation on cortical
auditory evoked responses6 , providing the result of EEG-based attention decoding as neurofeedback
was expected to reinforce trainees' attentional modulation of cortical responses.

3. Result
During the last 4th week training, compared to the first week, Experimental Group participants
exhibited strengthened alpha (~10Hz) oscillation in the right intraparietal sulcus during the silence
after the visual cue and before the sound while they attended right (i.e., the “Down” stream) andignored left (the “Up” stream), indicating that spatial inhibitory processing7 to suppress sound inputs
from the left was enhanced. Also, mid-temporal gyrus and superior temporal sulcus exhibited greaterattentional modulation of beta (~25Hz) oscillation after the four weeks of training, indicating
enhanced neural activity to predict the target voice8,9 . The strength of attentional modulation on
cortical evoked responses was also strengthened after training. The accuracy during a separated SiN
task was improved after the training while the auditory cortex evoked response to the target speechduring the SiN task was enhanced. Placebo Group participants did not exhibit any of the above changes.

4. Conclusion
These results successfully prove the concept that 1) neurofeedback training effectively enhances
the cortical processing for auditory selective attention and 2) the training effect is transferrable to a
separated SiN task.

ACKNOWLEDGEMENTS
This work was supported by Department of Defense Hearing Restoration and Rehabilitation
Program grant awarded to Choi & Kim (W81XWH1910637).

REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.

Surprenant, A. M. & Watson, C. S. Individual differences in the processing of speech and nonspeechsounds by normal-hearing listeners. J Acoust Soc Am 110, 2085–2095 (2001).
Bentler, R., Wu, Y. H., Kettel, J. & Hurtig, R. Digital noise reduction: Outcomes from laboratory andfield studies. International Journal of Audiology 47, 447–460 (2008).
Song, J. H., Skoe, E., Banai, K. & Kraus, N. Training to improve hearing speech in noise: Biologicalmechanisms. Cerebral Cortex 22, 1180–1190 (2012).
Whitton, J. P., Hancock, K. E. & Polley, D. B. Immersive audiomotor game play enhances neural andperceptual salience of weak signals in noise. Proceedings of the National Academy of Sciences 111,
E2606–E2615 (2014).
Kerlin, J. R., Shahin, A. J. & Miller, L. M. Attentional gain control of ongoing cortical speech
representations in a “cocktail party”. The Journal of neuroscience 30, 620–8 (2010).
Choi, I., Rajaram, S., Varghese, L. A. & Shinn-Cunningham, B. G. Quantifying attentional
modulation of auditory-evoked cortical responses from single-trial electroencephalography. Frontiersin Human Neuroscience 7, 1–19 (2013).
Deng, Y., Reinhart, R. M., Choi, I. & Shinn-Cunningham, B. G. Causal links between parietal alphaactivity and spatial auditory attention. Elife 8, (2019).
Arnal, L. H. & Giraud, A.-L. Cortical oscillations and sensory predictions. Trends Cogn Sci 16, 390–
8 (2012).
Holmes, E. & Johnsrude, I. S. Speech-evoked brain activity is more robust to competing speech when
it is spoken by someone familiar. Neuroimage 237, 118107 (2021).

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0821

Individualized identification training to improve native Japanese
speakers’ pronunciation of American English vowels
William L. MARTENS1; Stephen G. LAMBACHER2
1
2National Acoustic Laboratories, Australia

School of Social Informatics, Aoyama Gakuin University, Japan

ABSTRACT
Native speakers of Japanese often have difficulty distinguishing certain American English vowels, such as
the mid vowels /æ/, /ɑ/, /ʌ/, /ɔ/, and/ɝ/, which are assimilated to native vowel categories. This paper proposesa speech training method whereby native Japanese learning English as a foreign language (EFL) are given
individualized training in pronouncing problematic American English vowel sounds. Having first determinedthe central tendency and range of the first two formant frequencies characterizing each learner’s vowel space,an individualized set of target utterances was synthesized to conform to each learner’s vowel space. This
enabled Computer-Assisted Pronunciation Training (CAPT) using examples of American English
monophthongs in a consonant-vowel-consonant (CVC) context that were intended to sound as if the learners
themselves had uttered them. The EFL learners were then trained to identify the vowel sound in thoseindividualized CVC stimuli in preparation for their subsequent training in producing the exemplified
American English monophthongs. A more satisfying result was achieved in this fashion, as learners were notforced to attempt to match the formant frequencies of their own utterances to those of the target utterances
as produced by a speaker exhibiting a different vowel space (i.e., a speaker with a different vocal tract length).
Keywords: Second language speech learning, Pronunciation training, Vowel space normalization

1. INTRODUCTION
A great deal of work has been done investigating the link between the perception and productionof the sounds of a second language (L2), with results having implications for L2 teaching and learning.
A recent review (1) has attempted to connect L2 perception-production research with practicalapplications, in hopes of advancing the state of the art in this area, and has done so by “revisiting
theoretical assumptions and methodological practices.” While the problems faced by the L2 learnerare well understood, the discipline remains, for the most part, pre-paradigmatic and has not adopted
a single predominant model to account for the development of L2 skills. The hypothesis under test in
the current work is one regarding how a specialized form of individualized training can benefit L2
learners in their perception and production of L2 speech sounds. A review of the literature reveals that
there are very few attempts to apply some identification of individual learners’ characteristics to theirpronunciation training. An exception to this review finding was that reported in (2), which described
the adaptive recognition of the learner’s vowel space in an individually customized approach toteaching English pronunciation for native speakers of Japanese studying English as a foreign language(aka English as a second language).
When Japanese EFL students learn to speak English, their pronunciation of American English (AE)vowels can be aided by presenting speech-sound targets that have been synthesized specifically foreach learner. This approach has been applied within a Computer-Assisted Language Learning (CALL)
environment that provides Japanese EFL learners with target utterances synthesized according to an
adaptive recognition of the learner’s vowel space, sounding as if they had been uttered by the learners
themselves (2). This adaptive process is used in Computer-Assisted Pronunciation Training (CAPT),
which eschews the practice of providing target utterances produced by a speaker exhibiting a different
vowel space (i.e., associated with a different vocal tract length). Such CAPT addresses the difficulties
1
2

william.martens@nal.gov.au
lambacher@si.aoyama.ac.jp

adults often have when learning a second language (L2), particularly the difficulty mastering specificphonemic contrasts in the target language (3,4). Adults’ perception of speech is tuned to contrastive
elements that serve to distinguish native phones during native language (L1) acquisition because they
are language-specific perceivers. Listeners may find it difficult to distinguish between sounds in theL2 or between L1 and L2 sounds. Non-native sounds are perceived and produced following theacoustic patterns of L1 categories, leading to mispronunciation of specific phones in the L2. This
could be due to the process of equivalence classification, whereby two L2 sounds are linked or
assimilated into a single native category due to the L1 sound system “filtering out” phoneticallyessential features of the L2 sounds (3).
Many cross-language studies that have examined the performance of native Japanese in perceiving
AE vowels have used both identification and discrimination tasks, with results showing that some AE
vowels are particularly difficult for Japanese EFL learners to distinguish and are assimilated to L1
vowel categories (e.g., 4,5,6). Teaching L2 vowel categories to learners whose L1 has a smaller vowel
inventory, such as Japanese, is difficult. The number of vowel sounds in English is large in comparison
to Japanese. In the so-called “General American English” dialect, there are ten or more monophthongal
vowels, including the “smaller” diphthongs shown in Table 1:
Table 1. Examples of the consonant-vowel-consonant (CVC) words used to identify the vowel
sounds presented in the experimental sessions described in this paper. The table’s first column
shows the corresponding symbols as defined in the International Phonetic Alphabet (IPA), which
are summarized online at the following URL: www.internationalphoneticassociation.org. The
English-language words appearing in the second column are CVC words often heard in AE
pronunciation training sessions. Note that the IPA symbols for the last two words indicate that they
are diphthongs [eI, oʊ] rather than monophthongs, but these two are regarded as “smaller”
diphthongs that involve less spectral movement than “true” diphthongs. Indeed, in the examples
produced for the current study, the central vowel sound in these two ‘CVC’ words was pronouncedwith relatively constant formant frequencies, consistent with those for the h-V-d utterances
reported by Hillenbrand, et al. (7).

IPA symbol

English-languageʌ

but‘CVC’ tokensɔːbought

i

beat

ʊ

book

ɪ

bit

u

boot

ɛ

bet

ɝ

bird

æ

bat

oʊ

boat

ɑ

bottleeI

bait

Although the Japanese syllabary appears to include only five monophthongal vowels
, the language does feature two sets of these five that contrast
long and short vowel sounds (mono-moraic or bi-moraic). Indeed, native English speakers learning
Japanese as a second language must learn to produce the non-native vowel-length contrasts to be surethat the intended word meanings are effectively communicated. For example, the long and short vowel
sounds differ between alternatives / ‘house’ and  ‘no’, and native English speakers must
learn such differences to improve their Japanese pronunciation. Interestingly, attempts at training
native English speakers to identify such Japanese vowel length contrasts have been shown to be moreeffective when the training stimuli included sentences produced with varied speaking rates ( 8). Whatis of greatest interest here, with implications for the current work, is that high variability in a training
procedure has clear benefits for the generalization of learning.
In another study of Japanese vowel length contrasts (9), it was observed that Japanese long vowels
averaged roughly three times longer in duration than the short vowels, whether utterances wererecorded in sentence form or citation form. Then, in a study that focused on the effectiveness of ahigh-variability identification training procedure in improving native Japanese production of AE mid
and low vowels exhibiting contrasts between vowel sounds that were non -native for the learners (6).
Superficially it might seem that the principle of high variability is at odds with the individualizedsound synthesis approach promoted in the current work. It will be argued herein that pronunciation
training using speech sounds tailored to match the vowel space derived for the individual EFL
student’s vowel space should be included in the set of speech sounds provided for each student’s
training. Many factors must, of course, be considered in optimizing a training program. However, theexperimental studies reported in this paper were undertaken to test the efficacy of the inclusion ofsuch individualized vowel-sound synthesis on learning to identify (and subsequently produce) AE
vowels.
Prior studies on the effects of laboratory training on L2 perception and production performance of
L2 vowels have incorporated, for the most part, discrimination or identification tasks (differentiating
between two or more stimuli) (assigning labels to individual stimuli). The majority of studies haveshown some gains in learning, with the amount of improvement depending on several factors such aslearner characteristics, the relationship of the L1 to L2, the nature and range of the L1 stimuli, as wellas the training methods used, and the use or non-use of variability in the stimuli. Again, the reader isreferred to (1) for an in-depth examination of theoretical and methodological issues in L2 perception production research. That review recognizes the importance of differences between individuals inmeasured L2 performance, but that review focused on how individual differences in short-term
phonological memory and auditory processing could affect task performance.
In contrast to other popular approaches to Computer-Assisted Pronunciation Training (CAPT),
distinct advantages are observed when using an approach based upon identification training withcarefully selected pronunciation examples. A popular alternative approach is that based upon thedisplay of visualized acoustic properties, such as those shown by a sound spectrogram (10). Providing
such visual feedback in L2 pronunciation training has been observed to exhibit two substantial
disadvantages (as summarized in (10)):“First, trainees with no knowledge of speech acoustics have difficulty in reading and interpretingthe visualized acoustic properties. Second, it is hard to correct articulation behavior from acoustic
properties, since there is often no simple correspondence between gesture and acoustic structure.”
The above-mentioned high-variability identification training study (6) employed speech-soundrecordings produced by native AE speakers for the vowel identification training of native Japanese
EFL students. That study produced results supporting the current approach using training that provides
L2 sound examples rather than visual feedback, at least in the case of a five-alternative vowel
classification task. Those results can be summarized as follows: Before and after a six -week
identification training period, performance in the production of five AE mid and low vowels wasassessed for 54 native Japanese participants. The rates of distinct pronunciations of those five vowels
were measured through blind assessment by AE native speakers. The results of that study (6) revealed
that identification training with feedback improved the students’ production of the target AE vowels.
In the current study, an alternative method of providing target utterances for the AE vowels was
employed in an effort to circumvent a problem in pronunciation training that stems from variation in
production between Japanese participants exhibiting differences in their vowel spaces (i.e.,
differences in the range of frequencies over which the first two vocal formants varied for thoseparticipants). This paper describes initial attempts at providing Japanese EFL learners with customized
target utterances of 10 short words that were synthesized according to an adaptive recognition of thelearner’s vowel space. These 10 words (from the Table-1 list) featured the five AE mid and low vowels
that were presented in the previous study (6); however, the analysis of classification responses in thecurrent study will focus only on responses to just those five vowels for which identification training
was done in the previous study (6), although additional vowels were also presented here.

2. METHODS AND RESULTS
A group of 47 native Japanese speaking EFL students were asked to identify ten AE vowels,
including /i/, / I/, /Ɛ /, /æ/, /ɑ/, /ɝ/, /ʌ/, /ɔ/, /ʊ/, and /u/, which were presented in three different consonant
contexts [bVt], [mVt], and [hVt], and spoken by five native speakers of a Midwestern regional dialectof AE. The listeners were all 2nd-year students of the University of Aizu in Fukushima Prefecture,
Japan. As is common in Japan, all the listeners had six years of prior English inst ruction at the juniorand senior high school levels, but the focus was on English reading and grammar. Having had little orno prior exposure to English listening or pronunciation training. These students were engaged in an
English language course in which they had been trained to identify the ten vowels as they appeared in
English-language words, and so could be expected to be able to classify each of the CVC stimuli into
one of ten response categories. These listeners were presented a total of 150 tokens (5 speakers x 30
stimuli) in a ten-alternative, forced-choice (10AFC) task.
An introduction to each of the ten vowels was given to the listeners before the identification testin the form of practice trials. The listeners were instructed to mark their responses on an answer sheet
containing the above-listed ten alternatives in regular orthography, identifying each word they heard
as containing one of those vowel sounds. No other response was allowed on each trial. In eachlistening session, 30 trials were completed in five randomized blocks with a brief rest period between
each block of trials. The stimuli were presented to the listeners with a pause of seven seconds between
each stimulus. Although the listeners were presented a total of 150 tokens (5 speakers x 30 stimuli) ina ten-alternative, forced-choice (10AFC) task, only a portion of the response data was submitted to
the following analysis. In order to focus on the perceptual structure underlying the confusion observed
between the five AE mid and low vowels presented in the previous study (6), the observed
identification performance of the Japanese EFL students was examined as follows: A subset of the
combined resulting 10-by-10 confusion matrix (shown in Table 2) was transformed to produce a 5by-5 matrix of estimated inter-stimulus dissimilarities (assuming an exponential relationship between
confusion rates and dissimilarity magnitudes). The estimated dissimilarities were submitted tomultidimensional scaling (MDS) analysis to reveal the underlying perceptual similarity structure.
Table 2. The obtained confusion matrix showing the combined classification frequencies of 47
Japanese EFL students in response to with the five indicated mid and low vowels, including /æ/,/ɑ/, /ʌ/, /ɔ/, and /ɝ/.

Responses/æ//ɑ/
Stimuli

/ʌ/
/ɔ/
/ɝ/

/æ/

/ɑ/

/ʌ/

/ɔ/

/ɝ/

337
121
123
29
44

94
248
183
200
64

213
188
347
38
54

13 25
60 66
5
4
309 106
60 416

The two graphs shown in Figure 1 depict the physical vowel space and the MDS-derived perceptual
vowel space for the five AE mid and low vowels under examination here. The right panel of Figure 1
plots the average values of the first two formant frequencies of those AE five vowels reported byHillenbrand, et al. (9). The left panel of Figure 1 plots the stimulus space coordinates for the fivevowels using the IPA symbols as labels for the plotting symbols.

Figure 1 – Right Panel: The average values of the first two formant frequencies reported in(11) of five AE vowels (labeled according to their IPA symbols). – Left Panel: The MDSderived stimulus space coordinates for those same five vowels, based on the transformed
confusion matrix shown in Table 2.
While the right panel of Figure 1 shows the configuration of the five vowels in a physical vowel
space that is defined in terms of their first two formant frequencie s, as is often used in characterizing
vowel space, the left panel of Figure 1 shows a perceptual structure for the five vowels oriented so as
to reveal how the dimensions underlying their confusability relates to the two-formant plane that ismost commonly presented. Considering that the AE /a/ vowel is most similar to a Japanese vowel, it
should be noted that the two non-native vowels that are most extreme physically from the /a/ vowel
are the /æ/ and the /ɔ/ vowel sounds of the AE physical space in terms of the values of their second
formant frequencies. Interestingly, these two non-native vowels are diametrically opposed in theperceptual space shown on the right, suggesting that how the /a/ sound is confusable with the /æ/sound differs from how the /a/ sound is confusable with the /ɔ/ sound. In the discussion section of thispaper, the implications of such a finding are placed into the context of how AE pronunciation might
be more effectively taught to Japanese EFL students.
Also interesting to note from these results is the close proximity of the /æ/ and the /ʌ/ vowel sounds
in the perceptual space, despite the relatively substantial distance in their second formant frequencies.
Suppose only the relative distances between formant frequencies were to be considered. In that case,the /ʌ/ vowel sound might be expected to be more confusable with the /ɑ/ vowel sound, which would
predict a smaller “identification distance” between these two vowels in perceptual space rather than
the relatively larger “identification distance” than that between the /æ/ and the /ʌ/ vowel sounds . In
fact, prior research has demonstrated that Japanese students have difficulty identifying andpronouncing AE /ʌ/, which is often replaced with Japanese /a/ (6). The following discussion places
these results in the context of current practice in auditory training of native Japanese EFL students.

3. DISCUSSION
The difficulty of the Japanese listeners in identifying the AE mid and low vowels /æ/ /ɑ/ /ʌ/ /ɔ/and /ɝ/ is not surprising. L2 researchers agree that L2 learners perceive non-native sounds in terms of
L1 phonemic categories. Being “attuned” to the phonetic and phonemic categories in their native
language, L2 learners frequently have trouble perceiving non-native sounds. Best’s (3) Perceptual
Assimilation Model does predict how such non-native sounds may be more or less difficult for
beginning L2 learners to identify correctly. Specific pairs of L2 sounds will be easier to discriminate
than other pairs. L2 sounds are presumedly identified according to articulatory similarities ordifferences of L1 sounds. Japanese learners of English must learn to identify the /ʌ/ sound as distinct
from the nearby /ɑ/ sound, which is likely assimilated to the Japanese /a/. In general, then, if multiple
AE vowels are assimilated to single native vowel sounds, those AE vowels will be more challenging
to identify than others, such as the case in which the Japanese vowel (/a/) occupies a space within thearea of the AE central and low vowels /æ/ /ɑ/ /ʌ/ and /ɔ/.

Japanese speakers typically substitute /a/ for these AE central and low vowels as spoken. When
English loanwords were adopted, the unusual English vowel sounds were replaced by the closest
sounds in the Japanese syllabary. Thus, when teaching the pronunciation of those loanwords according
to the AE dialect, it is instructive to examine where their vowel sounds lie relative to the native
Japanese vowels as they are typically pronounced. Note, however, that the range of formant
frequencies observed in everyday Japanese speech may be quite wide. For example, the ellipses shownin Figure 2 depict the range of formant frequencies measured from recordings of seven different
speakers’ readings of five prose passages taken from Japanese textbooks and magazines (10). Figure
2 shows the regions of that two-formant plane in which two AE vowels might be found when theseare produced in the CVC context of the English-language words “bought” and “bat” (which arecontained within the rectangles indicating a spread in the first formant frequency). The /æ/ sound in“bat” is most typically assimilated to the Japanese /a/, while the /ɔ/ sound in “b ought” is most typicallyassimilated to the Japanese /o/. As these two words incorporate non-native vowel sounds, they areoften assimilated to those native vowels closest in physical vowel space. These exceptional cases
demand the production of “new” rather than “similar” phones (4).

Figure 2 – The range of formant frequencies observed in Japanese speech depicted as ellipses forthe five vowel sounds extracted from recordings of readings of five prose passages taken fromJapanese textbooks and magazines, captured for seven different native Japanese speakers.

Recognition of individual differences in learning can help EFL teachers prioritize L2 speechtraining objectives and better understand the difficulties associated with specific training tasks.
Individual learning differences can be attributed to the amount of mental processing required by thelearning task at hand. As a result, the nature of the learning task must be carefully considered.
Depending on the nature of the training task, thinking and learning will naturally differ. Learners
typically have varying aptitudes for various learning outcomes. To be successful learners they must
be willing or motivated to learn, as well as having the ability to learn; however, learners must also be
exposed to a classroom environment in which training tasks are incorporated into a teaching plan withthe goal of effectively fostering learning (12).

How can individual variations in L2 speech learning be addressed in the classroom and a learning
environment be created that motivates students to be more sensitive to L2 vowel distinctions while
also considering the different between students so that all might benefit from the instruction? Recent
efforts in this regard have been made by Martens & Wang (2), who proposed a promisingindividualized training method that was instantiated in an online CALL environment that provided
native Japanese L2 learners with customized target utterances of 12 single -syllable words createdusing an adaptive vowel space recognition system. These customized target utterances were included
in recorded sentences that provided learners with samples of all 12 AE monophthongs (listed in Table
1) in a consonant-vowel-consonant (CVC) context (and simulated real-life speech). This adaptive
process offered learners more appropriate pronunciation targets instead of forcing them to try to match
the formant frequencies of their utterances to those of the target utterances produced by a speaker witha different vowel space (i.e., a speaker with a different vocal tract length). Indeed, it was found (2)that the ability of native Japanese EFL students could be improved by exposing them to training
stimuli featuring AE mid vowels with formant frequencies based on their own individualized vowel
spaces. The students were better able to identify and produce the phonetic differences of problematic
AE vowel contrasts, and succeeded in finding their own pronunciation of the words “bat” and “bought”
without the typical assimilation to Japanese /ɑ/ and /o/, respectively.
The premise upon which this work is based is that learning to perceive the phonological distinc tions
between native and nonnative vowels within the learner’s own vowel space is a critical first step insuccessfully producing those phonological distinctions the development of which otherwise might bequite problematic. Further experimental tests of the benefits of the proposed auditory training are
underway, as the completed case studies (2) provide only anecdotal evidence for improvement inproduction. Although observed improvements in identification performance support the premise on
which this work is based, there is yet to be any systematic study of the benefits of such auditory
training on production of AE vowel contrasts that are readily discriminated by experimental listeners.

4. CONCLUSIONSThis study was focused on determining whether individualized L2 speech training using
personalized vowel spaces can help improve the pronunciation of AE vowels by native speakers ofJapanese. Our goal has been to further expand our understanding of the effects of computer-assisted
training on the improved L2 production of AE vowels by Japanese EFL students by proposing an
individualized training method whereby personalized vowel spaces are the key component of the
training stimuli as introduced in Martens & Wang (2). Additional research on training Japanese
students to develop robust categories of AE vowels is underway, and is currently based on
accommodation of individual differences related to personalized vowel spaces, despite the impacts ofearlier L1 experience on each student’s phonological system. It is hoped that other researchers will
adopt such practices and thereby help improve the understanding of how adults can realign or adjust
their perceptual spaces for acquiring new L2 speech contrasts.

ACKNOWLEDGEMENTS
The authors acknowledge the research support provided by the School of Social Informatics,
Aoyama Gakuin University.

REFERENCES
1. Nagle CL, Baese-Berk MM. Advancing the state of the art in L2 speech perception-production research:
Revisiting theoretical assumptions and methodological practices. Studies in Second Language
Acquisition. 2022 May;44(2):580-605.
2. Martens WL, Wang R. Applying adaptive recognition of the learner’s vowel space to English pronunciation
training of native speakers of Japanese. In SHS Web of Conferences 2021 (Vol. 102, p. 01004). EDP
Sciences.

3. Best CT. A direct realist view of cross-language speech perception. Speech perception and linguistic
experience. 1995:171-206, Baltimore: York Press.
4. Flege JE. Second language speech learning: Theory, findings, and problems. Speech perception andlinguistic experience: Issues in cross-language research. 1995;92:233-277.
5. Strange W, Akahane-Yamada R, Kubo R, Trent SA, Nishi K, Jenkins JJ. Perceptual assimilation ofAmerican English vowels by Japanese listeners. Journal of Phonetics. 1998 Oct 1;26(4):311-344.
6. Lambacher SG, Martens WL, Kakehi K, Marasinghe CA, Molholt G. The effects of identification training
on the identification and production of American English vowels by native speakers of Japanese. Applied
Psycholinguistics. 2005 Apr;26(2):227-47.
7. Hillenbrand J, Getty LA, Clark MJ, Wheeler K. Acoustic characteristics of American English vowels. The
Journal of the Acoustical Society of America. 1995 May;97(5):3099-111.
8. Hirata Y, Whitehurst E, Cullings E. Training native English speakers to identify Japanese vowel length
contrast with sentences at varied speaking rates. The Journal of the Acoustical Society of America. 2007
Jun;121(6):3837-45.
9. Nishi K, Strange W, Akahane-Yamada R, Kubo R, Trent-Brown SA. Acoustic and perceptual similarity ofJapanese and American English vowels. The Journal of the Acoustical Society of America. 2008
Jul;124(1):576-88.
10. Akahane R, Yamada R, Adachi T, Kawahara H. Second language production training using spectrographic
representations as feedback. Journal of the Acoustical Society of Japan (E). 1997;18(6):341-3.
11. Keating PA, Huffman MK. Vowel variation in Japanese. Phonetica. 1984 Jul 1;41(4):191-207.
12. Akahane Jonassen, D., B. Grabowski. Individual differences and learning. Handbook of Individual
Differences-Learning & Instruction. 1993: 3-17.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0838

Spatial auditory training improves spatial localization of the soundof an unseen talker’s voice after unilateral hearing loss
William L. MARTENS1; Arun SEBASTIAN1; Jennifer CLEMESHA1
1

National Acoustic Laboratories, Australia

ABSTRACTThe speech-sound localization ability of normal hearing listeners was compared to that of listeners withmoderate to profound hearing loss in just one ear, under conditions in which source level and spectrum
varied from trial to trial. Headphone-based virtual acoustic simulation enabled blind listening tests that
initially were completed without feedback to the listener, revealing the great difficulty listeners had in
localizing such speech sounds after suffering a unilateral hearing loss, with directional judgments strongly
biased by the source spectra. The directional judgments of normal-hearing listeners were biased by source
spectra in similar ways, but analysis of responses in terms of receiver operating characteristic (ROC) curves
showed that the normal-hearing listeners had relatively greater sensitivity to directional cues based upon
differences in binaural head-related transfer functions (HRTFs). Although listeners with highly
asymmetric hearing sensitivities showed very poor localization performance on initial tests, subsequent
training with feedback resulted in a dramatic improvement in directional discrimination between virtual
sources incident from frontward versus rearward hemifields, revealing increased sensitivity to spectral
differences in monaural HRTFs.
Keywords: Unilateral hearing loss, Sound localization, Head-related transfer function, Monaural directional
hearing, Spatial offset task
1.

INTRODUCTION

A great deal of scientific evidence from laboratory-based research exists (1, 2, 3, 4) that has
been proffered in support of the conclusion that adults can learn to localize broadband noise pulses
after unilateral hearing loss (UHL); however, there is less evidence that auditory training can aid in
the recovery of spatial hearing skills used in everyday listening that are typically found to be
degraded after a suddenly acquired UHL. In fact, most scientific studies that have been done toreveal acquired-UHL effects on spatial hearing skills have focused upon changes in measurable
performance on simple absolute localization tasks in which brief bursts of noise were presented
from loudspeakers arrayed on a listener’s horizontal plane. The study reported in this paper
introduced a novel psychophysical task into the context of spatial hearing assessment following
UHL that emulates a more ecologically valid testing scenario in which listeners report on the spatialoffset between two successively presented human voices. The study addresses the UHL listener’sdifficulty in shifting focus from one talker’s voice to another, while not strictly focusing uponselective attention during presentation of multiple simultaneous voices, as in the multi-talker
scenario often used to investigate the phenomenon that has come to be termed the “cocktail party
problem” (5). Rather, the task employed for the work presented in this paper required listeners toreorient themselves to listen in the direction of an unseen second talker’s voice immediately after
hearing a first talker’s voice in an alternative direction. Listeners reported only the direction of therelative spatial offset between the two voices in this task, while a more active response might havebeen employed that would have required listeners to turn their heads to reorient themselves to facein the direction of the second talker’s voice (6). There is prior work (7) setting a precedent for the“spatial offset task” employed in the current study, the results of which support the use of the task
as a productive means for examining spatial hearing skills in a manner that co mplements an absolutelocalization task in which listeners report only the direction of a single isolated stimulus.

1.1 Motivation
There are serious spatial hearing problems faced by UHL listeners, who make up as much as 7%of the population of the U.S.A. (8), and the unmet needs for rehabilitation within this population are
regarded as substantial enough to warrant an effort to have some positive impact on this importantcomponent of their associated daily-life hearing difficulties (9). Recognizing the difficulties inspatial hearing that are experienced after an acquired hearing loss in a single ear, there was interestin the development of a spatial auditory training program that might improve spatial localization of
speech sounds for such listeners, including listeners with highly asymmetric hearing sensitivities,
particularly those listeners who are described as single-sided deaf (SSD). After sudden
sensorineural hearing loss (SSNL), these listeners are expected to adapt to their UHL through active
everyday listening to sounds in the space around them; however, it has been hypothesized thatstructured training in sound localization might potentially aid in aural rehabilitation for such SSD
listeners. Although binaural cues to direction are not available to SSD listeners, unilateral spectral
cues are certainly available to aid them in resolving directional confusions, enabling listeners todiscriminate sounds incident from frontward versus rearward hemifields (10). The training program
employed in the current study was designed to attune SSD listeners to unilateral acoustic cues that
could be used to localize speech sounds under conditions of uncertainty, in blind listening tests
featuring immediate feedback. Although such structured spatial auditory training may be usefulalso in the aural rehabilitation of children with congenital UHL (11), the current work has been
focused upon adult listeners who have suffered SSNL.
In addition to the spatial hearing skills that can be demonstrated via listening tests that are
executed via simulated cocktail parties (12), there are real-life situations in which spatial hearing
skills are needed for enabling social engagement and for maintaining selective auditory attention
without too much listening effort. Ultimately, this work was driven by the motivation to improve
quality of life for UHL clients, recognizing the problematic spatial disorientation and fatigue that
results from monaural listening. Nevertheless, the work described in this paper did not include any
direct investigation of hearing-related complaints such as spatial disorientation and fatigue, whichinvestigation could be enabled by the use of questionnaires such as the Speech, Spatial and Qualities
of Hearing Scale (SSQ) developed by Gatehouse & Noble (13) and the Spatial Hearing
Questionnaire (SHQ) developed by Tyler, et al. (14). Rather than investigating the psychologicaland social consequences of UHL and SSD, which have been well reviewed by Lucas, Katiri, &Kitterick (15), the current work primarily has been motivated by goals of developing an effective
rehabilitation method and providing an experimental test of its successful application.
1.2 Objective
The objective of the current work was to develop a spatial auditory training program that could
provide for the aural rehabilitation of listeners with acquired UHL. It is desired that the spatial
auditory training method be robust enough to show a positive transfer of the training to common
everyday listening situations, such as that encountered when listening to group conversations. For
example, UHL listeners often report that they have difficulty knowing where talkers are located duringgroup conversations. Indeed, this difficulty is directly addressed by an item on the SSQ self-reportquestionnaire (13), wherein the respondent makes a rating in response to the following question:“You are sitting around a table or at a meeting with several people. You can’t see everyone. Can you
tell where any person is as soon as they start speaking?”
What might enable a positive transfer of training in this context would be for listeners to engage
in a spatial hearing task within which they must repeatedly reorient themselves to listen in the
direction of the speech sound arriving from a second talker’s location that is spatially offset from thatof a first talker. Furthermore, rather than employing broadband noise stimuli, the task should present
recorded speech sounds that vary within each trial and between trials during the training. As transferof training has been identified as an important issue, it should be stated that it would not satisfy theobjective of this study for UHL clients to be trained only in a laboratory setting, focused upon
performing well in free-field absolute localization of single broadband noise stimuli delivered fromincidence angles spanned by a closed set of loudspeaker locations. It is also clear that the directiondependent change in timbre and loudness of such known stimuli can be learned successfully; however,
it is certainly questionable that such learning will generalize readily to the successful localization ofcommonly-heard sound stimuli that naturally vary in level and spectrum, as human speech signals do.

1.3

Background

There has been an increase in the amount of work reported recently that has shown the effects oftraining on localization abilities of UHL and SSD listeners, especially in the context of establishing
the benefits of unilateral cochlear implantation (CI) as a treatment for unilateral deafness (1). In fact,
there is ample evidence (11, 16) that switching on a unilateral cochlear implant can reduce error
observed on an absolute localization task for the same listeners when tested monaurally (i.e., with
their cochlear implant switched off). That said, there is a substantial population of UHL clients who
do not receive a unilateral CI for their affected ear. While only 2% of known UHL cases are reported
to use hearing aids, a popular alternative has been the use of a contralateral routing of signal (CROS)
device (15). Despite the benefits of transmitting a signal from the location of the affected ear over tothe contralateral normal-hearing (NL) ear, there has been little evidence accrued to suggest that thisreally aids the UHL client with their localization of unseen sounds (16). Therefore, the current study
has been focused only upon the influence of spatial auditory training upon spatial localization abilities
of UHL clients under unaided listening conditions, with no use of a CROS device or bone-anchoredhearing aid (BAHA) or other assistive device (17). Further introduction to the current research effort
requires additional backgrounding discussion regarding the question of how human spatial soundlocalization abilities might be assessed in a most reliable way that also is most appropriate to thisapplication context. It should also be stated at the outset that the relative localization task promoted
herein for assessing sound localization abilities has not been widely adopted in such UHL studies,
despite the demonstrated merits of the approach for evaluating frontward/rearward hemifield
discrimination for sound sources presented on the horizontal plane (18).
While absolute localization tasks are the most commonly employed means for assessing humansound localization abilities (19), this approach assumes that measures of localization accuracy are themost valid quantitative measure of merit. In contrast, few studies give localization precisionadequate consideration, and studies that compare absolute and relative localization are the rare
exception (20). Psychophysical studies generally indicate that most normal-hearing (NH) individuals
have relative localization thresholds of a few degrees of azimuth angle for sources displayed on thehorizontal plane, whereas the absolute localization of sounds is typically observed to be less acutethan this (20). Of course, when normal-hearing (NH) individuals are (artificially) limited to the useof a single ear on comparable tasks, their absolute localization error generally increases (21). Theseso-called “monauralized” listeners show an increased number of frontward/rearward reversals inreports of absolute source direction for sources incident from frontward versus rearward hemifields(22). When such judgments are made for sources processed using the listener’s own measured headrelated transfer functions (HRTFs) and then presented monotically via headphones (that is, reproduced
as a virtual source for a single ear), increases in frontward/rearward reversal rates again are often
observed, particularly when stimuli with unknown spectra are presented. Such results have beencharacterized as providing evidence that the “extraction of monaural spectral cues can be severely
disrupted by trial-to-trial fluctuations in the source spectrum” (22).
Because binaural sound-localization cues are absent (naturally) in SSD listeners, their ability tolocalize sound is assumed (naturally) to be based upon unilateral cues, and this acoustic information
is assumed to be encoded as spectral cues that are revealed by variation in the HRTFs that describe
how the head, torso and pinna transform incident sound. These cues are termed here unilateral rather
than monaural cues so as to identify these as the spectral cues separately available at each ear to anormal-hearing binaural (NHB) listener. This is an important distinction since the typical “plug andmuff” approach used in “monauralizing” NHB listeners results only in great attenuation at thecontralateral ear, meaning that unnatural interaural level differences (ILDs) will be present during
testing (noting that sound will reach the occluded ear via bone conduction). In the current research,
there was no attempt to “monauralize” the NHB listeners who participated in the pilot studies reported
herein, with results intended only to provide a frame of reference on how NHB results might lookwere those listeners presented via stereo headphone successive reproductions of the same soundsources as those presented to UHL listeners (noting that in both cases those sources had beenprocessed using each listener’s own individually-measured HRTFs).

A central research question addressed in the current study is that regarding w hat unilateral cues
are most salient in supporting directional perception for UHL listeners. The results of many studies
suggest that SSD listeners can rely on the spectral "pinna cues” of their normal-hearing ear to localize
sounds.
In addition to the "pinna cues” that have been shown to provide a basis forfrontward/rearward hemifield discrimination (23), the acoustic Head-Shadow Effect (HSE) can also
serve as an azimuth cue, despite the fact that the HSE appears to be ambiguous when absolute soundlevels (of the source) are unknown. In that study (23) broadband noise stimuli were presented from57 loudspeakers at seven intensities (pseudo-randomly varying over trials from 30 to 60 dBA SPL),
and it was concluded that this 30-dB “rove” in intensity made it impossible for their SSD listeners torely on absolute sound levels as the basis for their directional judgments.
Although the absence of binaural cues for SSD listeners might have been thought to impair theirability to report the lateral angle of sound sources that shift left and right along the interaural axis (onthe horizontal plane), relevant unilateral spectral cues at the normal-hearing ear are certainly evident
in the directional variation of HRTFs. Results of experimental listening tests employing a group of
9 SSD listeners (24) support the conclusion that those SSD listeners “could rely on monaural pinnainduced spectral-shape cues of their hearing ear for directional hearing” (on the horizontal plane).
While most such studies employ an absolute localization task, others (25) have observed relative
localization thresholds as the measure a listener’s ability to determine that a repeated stimulus haschanged location “left–right” along the interaural axis (termed here the lateral-angle discriminationthreshold). In addition, there are studies (22) that do occasionally resolve azimuth judgments into a“left–right” component and a “front–back” component. Participants in the current experiments were
instructed to report directly such “front–back” shifts along the axis orthogonal to the interaural axis.
While the task performed by participants in the current experiments resembled the minimum
audible angle (MAA) task that has been used to measure thresholds for the just discriminable spatialoffset between sequentially presented stimuli, that task classically required participants to indicate
whether the second stimulus was to the right or left of the first (26). The current experimental taskdiffered from the classic MMA task in two fundamental ways, the first being that no threshold was
being measured here, and the second being the axis along which discrimination was required.
Although the task employed the two‐interval, two‐alternative forced‐choice (2I2AFC) paradigm thattypically has been used to measure thresholds for offsets in lateral angle, the participant’s task in thecurrent experiments was to indicate whether the second stimulus was offset from the first in a forward
or rearward direction, rather than to discriminate which of the two stimuli was closer to the medianplane (as required for an indication of an offset in lateral angle). It can be argued that indications of
a forward or rearward spatial offset between two successively heard sound sources is a key spatial
hearing skill that is related to functioning successfully in complex and demanding listening situations.
It should be noted that UHL clients have long understood that having only one good ear doesn’tnecessarily mean they will have no hearing problems beyond their difficulty in localizing clearly
audible sounds (13). It is natural to ask, then, for what reasons do people with asymmetric hearing
loss generally seek help? It is certainly not primarily because they have difficulty localizing sound;rather, they complain about reduced environmental awareness (after their hearing loss), and theirinability to distinguish between different talkers in group conversations, noting the ir increased
listening effort and fatigue associated with this demanding task. It was deemed beyond the scope ofthe current research to address these various hearing difficulties beyond the targeted spatial hearing
difficulties. Furthermore, there is no attempt to address the issue of UHL clients’ acceptance of a
spatial auditory training program on offer, or their compliance with such a training program were it
to be recommended by a clinician. Given that a primary motivation for this work was to enableclinicians to offer some rehabilitation that could have a positive impact upon common hearing
difficulties reported by UHL and SSD clients, it is hoped that those clients might have an appetite forsuch rehabilitation. The results of ongoing work in this regard are not reported within this paper, asinterviews with clinicians and their UHL clients are still under way. Suffice it to say here that the
spatial auditory training in which participants in the current study were engaged was a training that
had been selected with awareness that the task should resemble some aspect s of everyday listening.

2.

METHODS

The stimulus generation and psychophysical procedures employed in two experiments aredescribed in this section. The first experiment compared the performance of a single UHL listenerto the performance of 30 NHB listeners. The second experiment compared the performance of justtwo UHL listeners engaged in a spatial auditory training program with a more widely varying set ofstimuli.
2.1

Stimulus generation

As explained in the introduction section of this paper, there was a desire to employ more
ecologically valid stimuli (not filtered noise, but rather, recorded speech sounds). From within a setof 10 anechoically recorded brief consonant-vowel (CV) utterances, just three CV source stimuli(sounding as “ha” “hi” and “hu”) were selected for HRTF-based processing for the pilot study using
30 NHB listeners. Only the ipsilateral-ear channel of those two-channel binaural stimuli wereselected for monotic presentation via the same earspeaker system for a single UHL listener. Aftercompletion of the Pre-training of that one UHL listener, three additional CV stimuli (sounding as “he”
“hae” and “ho”) were added to the set of CV sources to increase the variability of the stimuli used in
the spatial training program for an additional UHL listener, again using the participant’s own HRTFs.
The three anechoically-recorded speech stimuli employed in the first experiment were processedusing either frontward or rearward HRTFs (employing HRTFs that were individually measured foreach of the 30 NHB listeners). These stimuli were further processed for presentation (via AKG
K1000 earspeakers) through convolution with individually-derived earspeaker correction functions(ECFs) using the procedure fully described in a previous report from the same laboratory (18). The
exponentially-swept test signals (27) were reproduced in an anechoic chamber via a pair of small
loudspeakers (Yamaha MSP3s) that were placed at front and rear positions with azimuth angles ofapproximately 40° and 140° degrees, and at a distance of approximately 1.5 meters from the centreof the subject’s head. The mean ipsilateral and contralateral HRTF gain functions for frontward
versus rearward incidence are plotted over frequency (relative to a reference measurement) for the
30 NHB listeners the Left-Panel graph of Figure 1, with the associated ILDs in the Right-Panel.

Figure 1 – Left Panel: Mean ipsilateral and contralateral HRTF gain (relative to a reference
measurement) for the 30 normal-hearing-binaural (NHB) subjects who participated in the listening
experiment. For two horizontal-plane-arrayed loudspeakers located 1.5 m from the center of eachlistener’s head, HRTFs were measured at 40° and 140° in azimuth. The ipsilateral gain overfrequency is plotted using solid or dashed lines, while the contralateral gain over frequency isplotted using dotted or dot-dashed lines (with the frontward response curves for both shown in blue,
and the rearward response curves for both shown in red). Right Panel: Mean Interaural Level
Difference (ILD) plotted over frequency for those same 30 subjects, with the ILD curves for FRONT
versus REAR incidence shown in solid and dashed lines, respectively, just as they were foripsilateral gain plotted in the Left- Panel graph.

2.2

Subjects

The 30 NHB listeners were university students who reported no know hearing loss ( confirmed byan audiological screening that showed no 4-tone-average loss greater than 30 dB HL). The UHL
listener participating in both the first and second experiment had right-ear hearing thresholds within
normal limits (< 30 dB HL); however, that UHL listener’s left-ear hearing thresholds were “off thescale” and consistent with a profound hearing loss (>>70dB HL). The additional UHL listenerparticipating only in the second experiment also had no appreciable loss in the hearing ear (< 30 dB
HL), and showed a profound hearing loss consistent with the SSD designation (>>70 dB HL).
2.3

Procedure

After completing an HRTF and ECF measurement session on a first day, the NHB participants
completed on a second day 3 blocks of 36 spatial offset discrimination trials while seated in a small
anechoic chamber. They completed this first experiment in less than a 30-minute period, and they
were informed of the group results one week later. Always, each trial presented a pair of stimuli within
which each stimulus had been processed with HRTFs measured either for frontward or for rearward
incidence. All combinations of three CVs and two HRTFs were presented within each block of 36
trials, always at a fixed presentation level of 70 dB SPL. The task employed a two‐interval, two‐alternative forced‐choice (2I2AFC) paradigm that required participants to indicate the “spatial offset”
of the second stimulus relative to the first. For the two successive auditory stimuli, participants wereforced to choose between two alternative responses, the allowed “spatial offset” responses being‘FRONTWARD’ or ‘REARWARD’ as follows:
FRONTWARD’: The second source was heard to arrive from a more frontward direction than thefirst, meaning that a forward motion (or offset) was observed; or,
REARWARD: The second source was heard to arrive from a more rearward direction than the first,
meaning that a rearward motion (or offset) was observed.
Although participants were given no feedback with regard to whether their “offset” responses wereconsistent with the angles at which their HRTFs were measured, they were instructed to report theoffset that was perceived by them, and they were not allowed to report that no offset was heard. For
the second experiment, in which only two UHL listeners participated, all combinations of the sameCVs and two HRTFs were presented in each block of 36 trials, but the presentation level was variedin three steps from 50 to 70 dB SPL (a 20-dB “rove”). In contrast to the previous experiment, thestimuli were reproduced via Sennheiser HD-600 circumaural headphones (for which ECFs were
derived). The instructions were the same for both experiments, but ten blocks of trials were distributed
over a 2-week period for the two UHL participants. In addition, the number of trials per block wasincreased to 72 for the second experiment. An expanded set of six CV stimuli were featured in thespatial training that was provided before the post-training tests using the original “spatial offset” task.
3.

RESULTSA “summary Respondent Operating Characteristic (ROC) curve” (28) was generated from the
combined 2I2AFC data of the 30 NHB participants. For analysis from the perspective of Signal
Detection Theory (SDT), this can be done by summing all the response rates across 30 listeners,
allowing for the calculation of a general result that might be said to characterize the sensitivity of a“generic” NHB listener (28), yielding separate d’ values for directional discrimination given all ninestimulus combinations. Instead of plotting percent correct results for those nine different stimulus
combinations, performance changes that were observed as the source was changed between
presentation intervals can be best understood through analysis of both the correct and the incorrect
responses of the participants (using the conventional SDT-defined statistics termed d’ and ). The
correct “Frontward” responses (given when motion was “frontward”) are termed hits, and the incorrect“Frontward” responses (given when motion was not “frontward”) are termed false alarms.

When hit rates and false alarm rates were analyzed for the combined response rates across 30 NHB
listeners for each of the nine stimulus combinations, the calculated d’ values confirmed that sensitivity
was relatively constant across stimulus conditions despite changes in response bias. The calculatedd’ values are plotted for the “generic” NHB listener in the Left Panel of Figure 2. Of course, werelisteners to shift their response criteria to make more “forward” than “rearward” offset judgments, theforward bias can be derived separately (as shown in the Right Panel of Figure 2) for each pairing ofspeech-sound stimuli. Even though stimulus processing for earspeaker presentation was extremely
well-engineered, there were still substantial frontward-rearward confusions for some pairs of speechsound stimuli, as is consistent with the typical confusion rates of around 15% on average observed on
absolute localization tasks (29). The strongest biasing was observed when one of the stimuli (the“hu” sound) was more frequently reported as the more rearward of two stimuli. Note that thediscovery of this bias is a benefit of having adopted the two-stimulus “spatial offset” paradigm andhaving abandoned the single-stimulus absolute localization paradigm, revealing the clear dependence
of offset judgments on HRTF-transformed source spectra.

Figure 2 – Left Panel: Barplot for nine stimulus combinations showing the d’ values that were
calculated from the combined response rates of the 30 NHB participants. Right Panel: Barplot of
the “FORWARD” bias values derived separately for each of the nine pairings of three speech-soundstimuli. The x-axis labels at the bottom of the graphs indicate the vowel sound presented during thefirst interval, and the internal labels indicate the vowel sound presented during the second interval.
In both graphs, the color code of the bars indicates the second-interval vowel sound, with cyan for
the “A” CV, yellow for the “I” CV, and magenta for the “U” CV.
The Left Panel of Figure 3 plots Experiment-1 hit rates as a function of associated false alarm rates
for the nine CV-stimulus combinations and shows the ROC curves fit to two datasets illustrating thelarge difference in sensitivity between the “generic” NHB results and those for the UHL participant,despite the observed shifts in response bias. The average sensitivity value derived from the NHB
response rates was d’ = 2.624, while the average sensitivity value derived from the UHL response
rates was only d’ = 0.580. Although the NHB listeners were given no feedback on their responses
throughout the three blocks of trials that each completed, their FRONTWARD/REARWARD
responses were substantially in agreement with the actual spatial offsets represented by the angles atwhich their individualized HRTFs had been measured. In contrast, the three blocks of trials that the
UHL listener completed prior to training with feedback revealed substantial FRONT/REAR hemifield
confusions. It is apparent that the 108 “Pre-training” responses recorded for the UHL listener also
were somewhat biased by the CV source spectra which had generally influenced responses of the NHB
listeners. Note that those biases did not overwhelm the UHL listener’s responses so greatly that asummary ROC curve could not be fit. For the UHL listener’s z-transformed response rates, the normof the regression-fit ROC curve residuals was 0.81, while that for the 3240 NHB listeners’ ztransformed responses was 0.79 (the smaller value of the norm indicating a slightly better fit for the
NHB data). The Right Panel of Figure 3 plots Experiment-2 results which are the Post-training
results for two UHL trainees (termed here trainee 1 and 2, T1 and T2, respectively).

Figure 3 – Left Panel: Comparison of the summary ROC curves for the UHL participant(square symbols) with the “generic” summary ROC curve that was generated from the combined
2I2AFC data of the 30 NHB participants (plotted as circular symbols using the same color codes as
in Figure 2). The summary ROC curve for the NHB participants is plotted using a solid red line,
while the ROC curve for the UHL participant is plotted using a black dotted line. Right Panel:
Graph showing the summary ROC curves that were generated from Post-training response rates of
two UHL trainees (T1 and T2, with rates plotted respectively using triangular and square symbols).
The graph shown in the Right Panel of Figure 3 summarizes the performance of two UHL
participants after completing ten “spatial training” blocks of 72 trials distributed over a two-weekperiod. During the blocks of “training” trials for the two UHL trainees, immediate feedback wasprovided to inform them whether their FRONTWARD/REARWARD responses were in agreementwith the actual spatial offsets represented by the angles at which their individualized HRTFs had
been measured. While six different CVs were presented in pairs during the training task (sounding
as “ha” “hi” “hu” “he” “hae” and “ho”), the Post-training test presented only the three CVs thatwere included in the Pre-training trials. Note, however, that those three CV stimuli were given a
20-dB rove in level over the Post-training trials. The summary ROC curves that were generated
from Post-training response rates of the two UHL trainees (T1 and T2) showed reasonably goodsensitivity on the “spatial offset” task, with average values of d’ = 1.694 for T1 and d’ = 1.216 forT2 (T2 being the one of two UHL listeners who completed the Pre-training trials).
4.

CONCLUSIONS

Performance on a test of relative sound localization skill was assessed for a group of 30 normal
hearing binaural (NHB) listeners who were given no immediate feedback during testing, in order toestablish a normative measure of sensitivity for comparison with performance of listeners who haveexperienced a sudden unilateral hearing loss (UHL). Performance without immediate feedback on
this so-called “spatial offset” task (7) was assessed for one UHL participant prior to a two-weekperiod of training on the same task, while the training task gave immediate feedback and presented
an expanded set of stimuli that included six different vowel sounds. That UHL participant showed
improved Post-training sensitivity to unilateral HRTF-based cues on a version of the task that
presented the original three (Pre-training-tested) vowel sounds that were given a 20-dB rove in level
over trials. A second UHL participant showed Post-training sensitivity that was comparable to thatof the first UHL participant, despite the strong influence of source spectral variation on “spatialoffset” responses that were observed both for UHL and NHB participants. These preliminary results
suggest that further development and evaluation of the proposed spatial auditory training program iswarranted, noting that its headphone-based deployment is a convenient alternative to loudspeakerbased mobile systems currently in use for testing of monaural sound localization abilities (30).
Further investigations are underway to establish the long-term benefit of the training, which might
be considered a response to a recent review (31) concluding that "longitudinal studies of localization
performance, especially with training, would be useful to provide audiologists information regarding
developing appropriate rehabilitation plans."

ACKNOWLEDGEMENTS
The authors thank their many colleagues at the National Acoustic Laboratories who recently
contributed to this work, most notably P. Kitterick, V. Marnane, J. Meija, J. Monaghan, and V.
Zhang. In addition, it is noted that S. Kellaway aided in running the experiment in which 30
normal-hearing binaural listeners participated.
REFERENCES
1. Arras T, Snapp H, Sangen A, Snels C, Kuntz I, Theunen T, Kheirkhah K, Zarowski A, Wesarg T, vanWieringen A, Agterberg MJ. Instant improvement in monaural spatial hearing abilities throughcognitive feedback. Experimental Brain Research. 2022 May;240(5):1357-69.
2. Firszt JB, Reeder RM, Dwyer NY, Burton H, Holden LK. Localization training results in individuals
with unilateral severe to profound hearing loss. Hearing research. 2015 Jan 1;319:48-55.
3. Slattery III WH, Middlebrooks JC. Monaural sound localization: acute versus chronic unilateral
impairment. Hearing research. 1994 May 1;75(1-2):38-46.
4. Musicant AD, Butler RA. Monaural localization: An analysis of practice effects. Perception &Psychophysics. 1980 May;28(3):236-40.
5. Yost WA. The cocktail party problem: Forty years later. Binaural and spatial hearing in real and virtualenvironments. 1997:329-47.
6. Whitmer WM, McShefferty D, Levy SC, Naylor G, Edwards B. Changes in orientation behavior due toextended high-frequency (5 to 10 kHz) spatial cues. Ear and hearing. 2022 Mar;43(2):545.
7. Martens WL. Discrimination of spatial offsets on the cone of confusion: Closer/farther,
upward/downward, and frontward/rearward. In Proceedings of the 18th Int. Congress on Acoustics(ICA) 2004, Kyoto, Japan.
8. Pastore MT, Natale SJ, Clayton C, Dorman MF, Yost WA, Zhou Y. Effects of head movements onsound-source localization in single-sided deaf patients with their cochlear implant on versus off. Ear andhearing. 2020 Nov 1;41(6):1660-74.
9. Vogt, K., Wasmann, J.-W., Van Opstal, A.J., Snik, A.F.M., Agterberg, M.J.H., Contribution of spectral
pinna cues for sound localization in children with congenital unilateral conductive hearing loss after
hearing rehabilitation, Hearing Research (2019)10. Yost WA, Dye RH, Sheft S. A simulated “cocktail party” with up to three sound sources. Perception &psychophysics. 1996 Jan;58(7):1026-36.
11. Golub JS, Lin FR, Lustig LR, Lalwani AK. Prevalence of adult unilateral hearing loss and hearing aid
use in the United States. The Laryngoscope. 2018 Jul;128(7):1681-6.
12. Galloway, J, Zhang, V, Marnane, V, Hou, S, Stewart, G, Bardy, F. The impact of unilateral hearing loss
on adult life. The Hearing Review. 2019 April.
13. Gatehouse S, Noble W. The speech, spatial and qualities of hearing scale (SSQ). International journalof audiology. 2004 Jan 1;43(2):85-99.
14. Tyler RS, Perreau AE, Ji H. The validation of the spatial hearing questionnaire. Ear and hearing. 2009
Aug;30(4):466.

15. Lucas L, Katiri R, Kitterick PT. The psychological and social consequences of single-sided deafness inadulthood. International journal of audiology. 2018 Jan 2;57(1):21-30.
16. Távora-Vieira D, De Ceulaer G, Govaerts PJ, Rajan GP. Cochlear implantation improves localization
ability in patients with unilateral deafness. Ear and hearing. 2015 May 1;36(3):e93-8.
17. Shiraishi K. Sound localization and lateralization by bilateral bone conduction devices, middle ear
implants, and cartilage conduction hearing aids. Audiology Research. 2021 Sep 30;11(4):508-23.
18. Martens WL, Guru A, Lee D. Effects of individualised headphone response equalization on front/backhemifield discrimination for virtual sources displayed on the horizontal plane. In Proceedings of the20th Int. Congress on Acoustics (ICA) 2010.
19. Letowski T, Letowski S. Localization error: Accuracy and precision of auditory localization. Advancesin sound localization. 2011 Apr 11;55:55-78.
20. Recanzone GH, Makhamra SD, Guard DC. Comparison of relative and absolute sound localization
ability in humans. The journal of the Acoustical Society of America. 1998 Feb;103(2):1085-920.
21. Oldfield SR, Parker SP. Acuity of sound localisation: a topography of auditory space. III. Monauralhearing conditions. Perception. 1986 Feb;15(1):67-81.
22. Wightman FL, Kistler DJ. Monaural sound localization revisited. The Journal of the Acoustical Societyof America. 1997 Feb;101(2):1050-63.
23. Van Wanrooij MM, Van Opstal AJ. Contribution of head shadow and pinna cues to chronic monauralsound localization. Journal of Neuroscience. 2004 Apr 28;24(17):4163-71.
24. Agterberg MJ, Hol MK, Van Wanrooij MM, Van Opstal AJ, Snik AF. Single-sided deafness anddirectional hearing: contribution of spectral cues and high-frequency hearing loss in the hearing ear.
Frontiers in neuroscience. 2014 Jul 4;8:188.
25. Perrott DR, Costantino B, Cisneros J. Auditory and visual localization performance in a sequential
discrimination task. The Journal of the Acoustical Society of America. 1993 Apr;93(4):2134-8.
26. Mills AW. On the minimum audible angle. The Journal of the Acoustical Society of America. 1958
Apr;30(4):237-46.
27. Farina, A. (2000). Simultaneous measurement of impulse response and distortion with a swept-sine
technique. In Audio Engineering Society Convention 108. Audio Engineering Society. Grange JA,
Culling JF. The benefit of head orientation to speech intelligibility in noise. J Acoust Soc Am. 2016;139(2): 703–712.
28. Rosman AS, Korsten MA. Application of summary receiver operating characteristics (sROC) analysis
to diagnostic clinical testing. Advances in Medical Sciences. 2007;52:76-82.
29. Karlsen BL. Spatial localization of speech segments. 1999. Ph.D. thesis, Aalborg University
30. Zheng Y, Swanson J, Koehnke J, Guan J. Sound localization of listeners with normal hearing, impaired
hearing, hearing aids, bone-anchored hearing instruments, and cochlear implants: A review. American
Journal of Audiology. 2022 Aug 2:1-6.
31. Wasmann JA, Janssen AM, Agterberg MJ. A mobile sound localization setup. MethodsX. 2020 Jan
1;7:101131.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0849

A novel ear training game and the training influence on
auditory selective attention
Sungyoung KIM1; Jacob COZZARIN1; Rai SATO2
1
2

Rochester Institute of Technology, U.S.A.

Okinawa Institute of Science and Technology, Japan

ABSTRACTTechnical ear training has been widely used as a tool to train junior audio engineers and improve their timbre
identification. For holistic auditory training, the authors have developed additional three training modules—
auditory localization, consistent judgement, and auditory memory training. Each training module wasdesigned to provide trainees with audiomotor closed-loop feedback so that the person can realize the response
errors and adaptively adjust to learn the intended learning objectives (ILO). For example, the auditory
localization module requires to move the person’s whole body towards a hidden target sound source. One
challenge in the study of such training-induced benefit is the training’s transferability and plasticity. If thebenefit (for example, improved sensitivity) is only limited to the training stimuli, the training cannot beapplied to real-world, unknown stimuli. To investigate general gain associated with this holistic auditory
training, we conducted a CRM test measuring listeners’ speech understanding in a competing, masking
speech. The result shows that the treatment (trained with the proposed training modules) group significantly
improved performance of identifying speech, but not the control group who received traditional audio
engineering training (recording and mixing). The results indicate that a training-induced influence istransferable to non-trained task, speech-in-noise understanding.
Keywords: Ear-training, Speech-in-noise, Gamification

1. INTRODUCTION
Speech understanding in noise (SiN) is influenced by various factors. Researcher have tried todevelop both invasive and non-invasive therapeutic options accustomed to those factors to improve a
listener’s SiN performance. Whitton et al. [1] reported that closed-loop audiomotor training could bean effective, non-invasive option. The authors understood an existing challenge in such a trainingbased treatment: the training’s “plasticity” and/or “transferability” to general gains, which “do nottransfer far beyond the training stimuli.” Musical training, in contrast, is known to be effective toimprove various auditory functions. They analyzed that musical training features a multimodal,interactive activity between listening and self-adjustment, so-called ‘closed-loop audiomotor’ training.
For example, a violinist carefully focuses and controls his or her finger to articulate tuned pitch amidof other competing instrumental sounds. Based on this merit from musical training, they subsequently
designed a new training paradigm similar to a musical practice/training, and conducted an experiment.
The result showed that when participants utilized auditory and motor sensors simultaneously, thetraining improved the non-trained task, SiN performance.
We previously developed various ear-training programs for audio engineers [2 – 4]. The programs
provide trainees with a personalized training sequence, optimized for each trainee after analyzin g theperson’s previous training performances. Recently, we unified and integrated these training programs
to an ear-training game in which users challenge and conquer training tasks to move towards advanced
levels (figure 1). This ear-training game consists of four training modules and follows one simple rule
to move to the next level: train two basic skills (spectral identification (S) and auditory localization

12

sxkiee@rit.edu; jpc4394@g.rit.edu
rai.sato@oist.jp

(L)); and challenge toprove that you canlisten consistently (C)only through internal
memory (M) without
any external reference.
The last part, to listen
consistently in various
listeningsituations
using a built-in andtrainedinternalreference only, is a key,
distinct characteristic
required for expert
audio engineers.
Wesubsequently
observed that all ofthese training activities
are essentially similar
Figure 1. Main screen of ear-training game. A user is guided to move to higher levels by
to the aforementioned
breaking through four training modules in each level. This ear-training game consists of four closed-loop audiomotor
training module with a straightforward rule to move to the next stage: train two basic skills
training.
Audio(spectral identification (S) and auditory localization (L)) and challenge to prove that you
engineers control a
could listen consistently (C) through internal memory (M) without any external reference.
physical device, such asa fader or knob, to adjust timbral and spatial characteristics of target sound objects. In this context,
an audio device is equivalent to a musician’s instrument. As musicians adjust fine tune using theiraudiomotor interaction, the audio engineers control the device to achieve a required sonic quality. For
example, if a given sound source has unwanted +3 dB boost at 1 kHz with Q of 2, you need to findexact amount of variation by adjusting the control devices and listening to sonic variations. Therefore,
this specialized audio-engineering training could be equally effective to improve other auditoryrelated skills (such as SiN performance), as the benefit of a closed-loop audiomotor training [1]. We
investigated this hypothesis through a controlled experiment, the details of which will be presented insubsequent sections.

2. METHODS

Figure 2. The experimental setup. The focal auditory cue “Ready Tiger” is presented through the front loudspeaker, and thebackground auditory cue “Ready Hopper” is presented through one of six spatial positions. A participant responds the colorand number of the instructed auditory cue (red 5 or blue 2) using a tablet.

2.1 Participants
The authors formed two listener groups: the Control group with 4 participants who studied
advanced topics in audio engineering for 14 weeks (without the audiomotor training) and theTreatment group with 4 participants who only received audiomotor training for the same 14 weeks.
These two groups’ SiN performances were compared with the Reference group with 34 participants
who did not receive any training at all.
2.2 SiN performance
We adapted the Coordinate Response Measure (CRM) corpus by Bolia et al. [5] to measure listeners’
SiN performance. The background maskers are competing speeches that interfere with a given focal
target speech. Figure 1 illustrates a task where all listeners are requested to attend to the focal cue of“Ready Tiger” and understand the color and number. In the figure above, the correct answer would be‘red’ and ‘5,’ which the listeners need to respond by selecting the corresponding button in a tablet (bypressing the red-colored 5 square in the screen). As for the other task attending to the background cue,the correct answer would have been ‘blue’ and ‘2.’
The experiment also adapted spatial separation of background from focal. Two recent studies [6][7]showed that the spatial location of background sounds is an important experimental variable for
auditory selective attention. The experiment incorporates this factor by testing six different positions .
We used the correct answer ratio of this CRM as a metric of listeners’ SiN performance.

3. RESULTS
We analyzed the data with two spatial-location conditions: collocated vs. separated condition.. The
Collocated condition was where both focal and background speeches were reproduced from the same,
one location; the Separated condition was where focal and background speeches were reproduced
from different locations (as illustrated in the previous figure 2).
In addition, the experiment adapted two tasks: the ‘Focal’ task asked listeners to identify the focalcued information (color and number) while the ‘Background’ task asked to identify the backgroundcued information.
The results showed that all three listener groups (Reference, Control and Treatment) did not makeany significant difference in the ‘Focal’ task, for both collocated and separated conditions.
Collocated / Focal

Collocated / Background*Figure 3. Result of the experiment (collocated condition only). [Left panel] Correct answer ratios of the ‘Focal’ task show
no significant difference. [Right panel] Correct answer ratios of the ‘Background’ task show that the treatment group witha semester-long training improved significantly. Since the separated conditions are easier for speech identification, no
significant difference was observed for the current focal-to-background level relation. The experiment result revealed that thetraining is effective for the most demanding and thus difficult listening task.

However, in the ‘Background’ task, the collocated condition caused a significant difference
between the reference and treatment group (F(2, 79) = 6.35, p = .003) (the left panel of figure 3). This

result indicates that the given training (long-term and audiomotor training) helped the treatment group
to better perform SiN in the hardest and thus most demanding task (Collocated condition of the‘Background’ task).
The Separated conditions (both ‘Focal’ and ‘Background’) appear easy for the both Treatment
and Control group due to spatial masking release, resulting no significant difference (please refer tothe following section for more details).
This experiment did not account for different masker (background) levels. The authors plan to
conduct follow-up experiments and investigate a potential influence of various masker levels.

4. DISCUSSION
The authors had designed the auditory localization (L) training module to improve a trainee’s
auditory spatial sensitivity. Specifically, this training module aims to assist listeners in identifying
spatial location of a target sound amid of interfering maskers. This accurate identification of spatial
location is a part of features in auditory spatial attention. Therefore, the authors assumed that thetraining would support to enhance auditory spatial attention including SiN performance, especiallyfor the ‘separated’ condition. However, the current result did not show any significant change. It is
probably the training-induced improvement was smaller than spatial release from masking (SRM3) inthis experiment. In other words, ‘separated’ SiN tests were easy (due to SRM) both for the control andtreatment group, and thus incapable to properly evaluate training -induced effect.
We also analyzed the listeners’ response times, the result of which showed a significant change
between the two groups in the separated condition. While their correct answer ratios were similar inthis experiment, the treatment group responded quicker after the training. It may indicate that thetreatment group has become more confident, potentially through auditory localization (L) training
module (which is in line with previous auditory localization training [9]). We would like to investigate
whether this improve listeners’ confidence would lead improvement in SiN performance for morecognitively demanding tasks such as low target-to-masker levels.

ACKNOWLEDGEMENTS
The first author sincerely appreciate the support of Yamaha Corporation and the Department of
Defense (PI, Project Director: Dr. Inyong Choi (The University of Iowa) & Award No. W81XWH-191-0637) for this research project.

REFERENCES
1. Whitton JP, Hancock KE, Shannon JM, Polley DB. Audiomotor Perceptual Training Enhances SpeechIntelligibility in Background Noise. Current Biology. 2017 Nov;27(21):3237-3247.e6.
2. Kim S. An assessment of individualized technical ear training for audio production. J Acoust Soc Am.
2015;138(1):EL110-EL113.
3. Kim S, Imamura H. An assessment of a spatial ear training program for perceived auditory source width.
J Acoust Soc Am. 2017;142(2):EL201–EL204.
4. Kim S. Chon SH, Okumura H, Sakamoto S. Comparing training effects associated with two sets of HRTF
data on auditory localization performance. Proc AES 148th Int Conv; May 2020; Vienna, Austria.
5. Boila RS, Nelson WT, Ericson MA, Simpson BD. A speech corpus for multitalker communications
research. J Acous Soc Am. 2000;107(2):1065-1066.
6. Swaminathan J, Mason CR, Streeter TM, Best V, Kidd G Jr, Patel AD. Musical training, individual
differences and the cocktail party problem. Scientific Reports. 2015; 5, 11628.
7. Carlile S, Corkhill C. Selective spatial attention modulates bottom-up informational masking of speech.
Scientific Reports. 2015; 5, 8662.
8. Dirks DD, Wilson RH. The effect of spatially separated sound sources on speech intelligibility. J Speech
Hear Res. 1969;12:5–38.
9. Chon SH, Kim S. Does Technical ear Training Also Improve Speech-In-Noise Identification? Proc 6th
APSCOM; July 2017; Kyoto, Japan.

3

Please see [8] for more information on SRM.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0405

Using a flipped classroom model to introduce simple harmonic
motion and spectral analysis in an introductory undergraduate
musical acoustics course
Jill A. LINZ
Skidmore College, United States

ABSTRACT
A flipped classroom turns a traditional lecture based class into an active learning environment. Topics areinvestigated using problem solving techniques and hands on activities to gain conceptual insight throughsmall group interactions. Sound & Music is a course in musical acoustics with an emphasis on sound designin a small liberal arts college located in New York State. Students generally have little to no background inphysics, math or music. The pedagogical structure of the course is steeped in the ideas of Physics Education
Research and is closely aligned with those ideas put forth in creating a flipped classroom. The mathematicaltreatment used is a strong element to the overall philosophy of the course. Signal processing and Fourier
Transforms are fundamental concepts to the understanding of musical acoustics. Connections between thesound heard, the graphical representation and its mathematical form are emphasized through many hands on
applications throughout the semester, beginning with a detailed model of a simple mass and a spring tounderstand simple harmonic motion. This paper explores how this model can then be expanded upon to gain
insight into Fourier Transforms and its application of spectral analysis as a tool to understand what is being
heard.
Keywords: acoustics, education, music

1. INTRODUCTION
A flipped classroom turns a traditional lecture based class into an active learning environment. Topics
are investigated using problem solving techniques and hands on activities to gain conceptual insight throughsmall group interactions. Sound & Music is a course in musical acoustics in a small liberal arts college
located in New York State, USA. Students generally have little to no background in physics, math or music.
One of the strongest influences in the design of Sound & Music are ideas that came from Physics Education
Research techniques. The course itself focuses on sound design where an emphasis is placed on recognizing
waveforms and spectra of different sounds and what produced them. Typically, topics such as Fourier’s
Theorem and Signal Processing are out of reach for students in this demographic. By introducing these topics
using a strong hands-on, learner-centered pedagogy, students are able to gain insight into the basis of sounddesign and signal processing.
The learner-centered paradigm of education, or flipped classroom model has become a valuable teachingpedagogy in many disciplines since as early as 2008 (1). It was not until the pandemic that discussions
among many educators turned to considering a flipped classroom model. The core basis of this model issteeped in Physics Education Research (PER) techniques. PER began in earnest in the mid-1970s as a way toaddress issues surrounding the teaching and learning of physics. Initially, researchers, such as Lillian
McDermott at the University of Washington and others, developed methods to teach physics that focused on
student learning and retention. Specific deficits were identified and various approaches were investigatedover the years. The results of these are summarized in an article titled “An Introduction to Physics Education
Research.” (2) The core educational goals that are a centerpiece of PER are those of conceptual
understanding and problem-solving abilities. Investigating what students know and how they learn it plays animportant role in the processes.
Time and care is taken to make sure that students approach the course with the right frame of mind. The
entire first week of classes are dedicated to this. Images and pictures, graphical depictions and symbolic
languages are explored and applied to the 3 primary tools of physics – kinematics, forces and energy. Groupsof 3-4 begin working together on the first day. The approach used to accomplish this are typical problem
solving strategies used in physics education. This integration helps to form the foundation for developing ourmodel of sound. It is essential to begin this model as simply as possible and then expand upon it to create
more realistic model to aid in our understanding of sound. The mathematical relations are not derived fromfirst principles in this course. They are reasoned out through group discussions, observations and hands-on
activities.
All sound perceived by humans is the interpretation of vibrations. We begin with an investigation of thesimplest vibration – that of a simple mass and spring system. A deep understanding of the physics behind thisis key to the understanding of interactions between atoms and molecules in solids, liquids and gases.
Mass/spring systems are studied using the techniques described through pictures, graphical representations
and our symbolic language of mathematics to enable students to gain an understanding of a simple
oscillation. Once this understanding is established, expanding the model to more complex sounds is
relatively straightforward. Investigation into waves on strings as a series of masses and springs lead tovelocity relations and the standing wave patterns that occur. This in turn will aid in the understanding of how
different waves combine.
Signal processing and Fourier Transforms are fundamental concepts to the understanding of musicalacoustics. Connections between the sound heard, the graphical representation and its mathematical form areemphasized through many hands on applications throughout the semester. This paper explores how a simple
model of a mass/spring system can then be expanded upon to gain insight into Fourier Transforms and its
application of spectral analysis as a tool to understand what is being heard.

2. SIMPLE MODEL OF A MASS/SPRING SYSTEM
The mathematical treatment used in this course is a strong element to the overall philosophy of thecourse. Musical acoustics is described by wave theory. Math is the language with which we express theseideas. The mathematical element is, by far, the most difficult aspect to address. Students often come into theclass with the idea that all things musical are the result of magic. It is essential to try to convince them that it
is not magic, but that they can become the magicians. We need to address what the mathematical equations
are telling us. Stressing that the language with which we speak is written in a symbolic form. This coursestresses the language aspect of both the mathematical and musical forms. Students do not “solve” mathproblems. They do learn what the equation,! = #$%&(() + +)(1)

represents when listening to a tone of specific pitch and loudness as well as what is represents graphically.
Beginning with a detailed model of a simple mass and a spring, students first observe different masses
and springs first hand. The motion is then translated into a pictorial representation of the mass oscillating as afunction of time, which can be seen in Figure 1. This is then translated into a graphical representation whichleads into developing a feel for Equation 1. Note that we do not derive this formally, instead, we reason itout. Time is devoted to obtaining a strong intuitive understanding of this system. Students immediately
realize that Figure 1 looks somewhat like a sine graph – or perhaps a cosine graph. Initially, phase is aforeign concept to them. This leads to a discussion about the idea of phase and a review of cosine and sine as
it relates to a circle.

Figure 1 - Time lapse diagram of a simple mass/spring system and its graphical representation.
Emphasis is placed on the physical properties that the variables represent in different types of waves, assummarized in the Table I. To aid students visualize sine waves, the analogy of light waves is used. Both asingle frequency light wave and a single frequency sound wave can be described by Equation 1. How thevariables manifest due to the physical system producing the wave will determine how we perceive each wave
type.
Table I - The similarities and differences between sound waves and light waves are stressed wheneverpossible. Here, the physical manifestations of each variable within Equation 1 is shown for both soundwaves and light waves. Light waves are used as a comparison of these physical manifestations.

Wave Property

SymbolWave Type

PhysicalManifestation

Amplitude

A

Light
Sound

Brightness
Loudness

Frequency

w

Light
Sound

Color
Pitch

Phase

f

Light
Sound

how wavescombineConservation of energy in a mass/spring system is explored in depth through homework and in-classgroup activities using measurements and calculations of the different variables, ultimately determining thesystem’s potential energy, PE and the oscillating mass’ kinetic energy, KE through the Equation 2,,-,.#- = /0 -(2)where k is the spring constant, A is the amplitude of oscillation, m is the oscillating mass and v is the
maximum velocity calculated from measurements of the oscillating mass’ frequency, f using 0 = (# and( = 223 This leads them to derive the resonate frequency of a mass/spring system can be determined by
Equation 3,
(4 =

5
6

where (4 = 2234 is the resonate frequency, k is the spring constant and m is the hanging mass.

3. WAVE MODEL(3)

This simple model is then expanded upon by considering how multiple masses and springs interact. Using
computer simulations, normal modes are introduced and visualized as a transfer of energy. Waves on a string
are initially modeled as a series of masses and springs. Care is taken to understand how the input energy istransferred from one mass to the next and so on. Using a particle wave model such as the one shown in
Figure 2, students can observe, discuss and investigate the velocity, 0 of this energy transfer as a function of
distance travelled over the time needed for a wave pulse to travel from one end to the other.

Figure 2 – Wave Motion Demonstrator from Pasco Scientific
Using the definition of velocity as distance over time, the energy transfer of the wave pulse can be expressedby the relation
0 = 73(4)

Where l is the wavelength and 3 is the frequency of the pulse. The velocity is also modeled by considering
the velocity as related to rate at which one particle moves through a cycle by considering the relation of thestring’s elasticity and density.
0=

8
9

(5)

and t is the tension on the string and m is the linear density of the string. Measurements are taken for avibrating string in order to compare these two methods.
This is followed by a simple exercise on a program called Mathematica where students can directly enterin the equations using frequencies in the audible range. Prior to this activity, students had been using
frequencies that were less than 10 Hz in examples and problems. These equations are then observed
graphically as well as listening to the tones produced. Ultimately they add several sinusoidal waves together.
Figure 3 shows the results of adding two waves with equal amplitude and phase. Both the waveform itself
and the tone produced changes significantly as we alter values within the equations. The graphical results ofa few of these changes can be seen in Figure 3.

Figure 3 - Example of two sine waves added together a) 100 Hz and 200 Hz. b) 100 Hz and 300 Hz.
The difference between these examples can easily be distinguished by listening through the computer’s
speakers. This activity is especially satisfying to the students as they finally get the connection between thesound, the equation and the graph. (3)4. FOURIER’S THEOREM
The introduction of the Fourier Transform to students with little to no mathematical background can be achallenge. Deriving the equations is not an option. Using methods presented in The Science of Sound (4),students add the amplitudes of two waves by hand and connect the dots to produce a complex waveform thatcan be studied in detail. Figure 4 shows the two waves in an overlay plot as well as the combined result.

Figure 4 - Students combine 2 simple sinusoidal waveforms by adding the amplitudes of each together.
Using a program such as Excel allows the students to expand their original graphs to include several
cycles as can be seen in Figure 5. It can be more easily observed that there are multiple frequencies appearing
in the combined waveform that were not a part of the original 2 inputs. Measurement of the periods of thesecan reveal information as to what occurs when the two initial sinusoids are combined.

Figure 5 – Expanded graph to show several cycles with measurements of combined waves in Figure 4.
The measured frequencies are then measured and compared to the input waveforms. As can be seen in
Figure 6, the measured frequencies are mathematical combinations of the original frequencies. The question
presented to the students is how can we keep track of all these different frequencies? Deciding on anorganizational chart seems to be the most logical way to accomplish this. This decision is, of course, reachedwith the guidance of the instructor.

Figure 6 - Organizing all the new frequencies that appear when adding 2 simple sinusoidal waves of single
frequencies.
Students then discuss how important is it to include all of the frequencies, or could they simply include
only the input frequencies? Since we know how the inputs will combine, we can predict the output. Puttingthis together, we create a spectrum graph that will show each input sinusoidal frequency and the amount, or
amplitude, of each one. At this point, we begin to look at recordings of different instruments. Figure 7 shows
the waveform and spectral graph for a bowed violin string in comparison to the waveform and spectral graphof a French horn. It is easy to see that both the waveforms and the spectrum are very different from one
another. The idea that every sound produces a unique waveform and spectrum is then expanded upon.

a)b)

Figure 7 – Waveform and spectrum for a) bowed D string on a violin and b) a G played on a French horn
Fourier’s Theorem states that “Any periodic vibration, however complicated, can be built up from a seriesof simple vibrations, whose frequencies are harmonics of a fundamental frequency, by choosing the proper
amplitudes and phases of these harmonics” (4). Just as with the original model of the mass/spring system,
this statement represents an ideal simplified situation. Each line on the spectral graph represents a single
frequency Sine wave just like the ones students have already seen. The height of the line represents theamplitude of that the sound of each instrument can theoretically be reproduced by adding together all of thedifferent Sine waves in the graph. Fourier’s Theorem is then presented as an analogy to recipes for cooking.
Beginning with creating basic dishes, and then from that, small additions to the dish such as spices andflavorings can result in a unique flavor experience. Here, the recipes that are created are unique for each andevery sound. The ingredients themselves are the sinusoidal frequencies. The basic ingredients themselves canbe found from the harmonic relation,
3: = ;3,(6)

Where 3, is the fundamental frequency and 3: are the higher order harmonics and ; is the integer number ofthe harmonic. Additional ingredients can be added for flavor. The amount of each ingredient is determinedby the amplitudes given using the Fourier Transform. The students do not evaluate this relationship, butinstead they build basic waveforms using the results. This also adds another layer of investigation regarding
the accuracy of using simple models.
Because there is a focus on sound design, students investigate the basic building blocks of soundsynthesis: square wave, triangle wave, sawtooth wave and sine wave. Investigation of these are approachedusing Digital Audio Workstations, as well as through recordings of the waveforms produced by a function
generator and speaker. (3) Digital Audio Workstations also have wave generators that will produce the ideal
form being investigated. These programs do not require a background in coding and are fairly intuitive touse, allowing students to directly apply Fourier’s Theorem. A square wave can be created by adding together
harmonics whose amplitudes are given by the solution to the Fourier Transform. While the math needed toobtain the results of equation 7 is beyond the scope of this course, it is straightforward to evaluate and can befound in many textbooks and web sites. The amplitudes of each harmonic in a square wave is:
<: =

-=
:>

sin

:>
C

)

(7)

Where A is the maximum amplitude, n is the harmonic number and T is the period of the fundamental
harmonic. The program used in class employs a relative percent amplitude rather than a direct calculation of=<: , making each amplitude diminish as <: = . Figure 8 shows a square wave produced using three different:
methods. Figure 8 a) is produced from the built in wave generator provided by the program, b) studentsdirectly add the harmonics and amplitude by following the recipe for a square wave, and c) a square wave
generated through a speaker and recorded. The waveform and spectrum are then observed and compared, as
well as listening to the sound each creates. Questions are raised about the ideal waveforms generated by thecomputer program in comparison to the ones they create.

a)

b)

c)

Figure 8 – Waveforms and spectra for a 500 Hz square wave generated by a) the computer, b) direct
application of the Fourier Transform for 7 harmonics and c) produced by a function generator
through a speaker and recorded.
These results bring up interesting discussions with regards to ideal situations when compared to theapproximated form or the real world situation. Students compare similarities and differences between each of
the three examples. As part of their conclusions, they must try to identify what causes these differences in an
attempt to explain why none of the examples exactly follow our simple model.

5. CONCLUSIONS
By developing a model of a simple mass and spring system, the fundamental principles of sound design,or sound synthesis can be investigated. The basics of wave theory in acoustics is developed using analogies
such as visible light waves. This is then compared to cooking and using recipes to create unique sounds.
Once students understand how the waveforms for the basic building blocks can be created using Fourier’s
Theorem, the understanding of how to recreate sounds from existing acoustic instruments or creating new
and innovative sounds from first principles becomes straight forward. While not part of this paper, the use offilters is also explored Digital Audio Workstations allow combinations of audible waves of different
frequencies, amplitudes and phases. Students realize that subtle changes in these waveforms can create new
and unique sounds. This then leads to an understanding of the different types of synthesis available andopens new avenues for students to explore.

REFERENCES
1. Watson SL, Reigeluth CM. The learner-centered paradigm of education. Educational technology. 2008
Sep 1:42-8.
2. Beichner R, Saul J, Abbott D, Morse J, Deardorff D, Allain R, Bonham S, Dancy M, Risley J. Studentcentered activities for large enrollment undergraduate programs (SCALE-UP) project, in PER-Based
Reform in University Physics, edited by E. F. Redish and P. J. Cooney (American Association of Physics
Teachers, College Park, MD, 2007), Vol. 1
3. Linz JA. Designing an introductory musical acoustics course using physics education research techniques
and the impact COVID-19 pandemic had on the course structure, The Journal of the Acoustical Society ofAmerica 151, 2672 2687 (2022)
4. Rossing T, Moore R, Wheeler P. The Science of Sound. San Francisco, California, USA: Addison
Wesley; 2002. p. 121-170

Filename:
ICA2022 final paper ABS0405.docx
Folder:/Users/jilllinz/Library/Containers/com.microsoft.Word/Data/Documents
Template:/Users/jilllinz/Library/Group Containers/UBF8T346G9.Office/User
Content.localized/Templates.localized/Normal.dotm
Title:
Subject:
Author:Jill Linz
Keywords:
Comments:
Creation Date:
6/28/22 2:45:00 PM
Change Number:
98
Last Saved On:
8/13/22 4:30:00 PM
Last Saved By:
Jill Linz
Total Editing Time:4,567 Minutes
Last Printed On:
8/13/22 4:31:00 PM
As of Last Complete Printing
Number of Pages:
7
Number of Words: 3,232
Number of Characters: 17,595 (approx.)ABS-0464

A Classroom Structure and Practices That Take into Consideration
Experiential Learning
Hiroshi SUDA(1)(1) Chiba Institute of Technology, Japan, hiroshi.suda@p.chibakoudai.jp

ABSTRACT
Acoustics often deals with invisible phenomena. A problem in many classes at institutions of higher educationis the emphasis on acquiring knowledge through mathematical formulas and graphs. I considered it important
to incorporate experiential learning. However, the expensive cost and space limitations of acoustic experimental
equipment make it difficult to fully utilize it for learning. Therefore, I have been developing simulator teaching /learning materials that visualize and make audible the results of calculations on a computer, enabling simulated
experiential learning. In 2020, many universities were closed due to covid-19 and classes were conducted
online. Students were dissatisfied with online classes where they only listened to one-way lectures. Therefore, I
incorporated simulator materials into my online classes to provide hands-on learning on topics such as acoustics.
This paper reports on the structure of the experiential class, the simulator materials used, and their effectiveness.

Keywords: Simulator, Visualization, Audible

1

INTRODUCTION

The educational field of acoustic engineering often deals with invisible phenomena in its content. The learner’s
understanding is hindered by invisible phenomena. Here, the need for an educational method that incorporates
various tools in the field of acoustics was discussed[1][2]. However, the experimental equipment for acoustics is
expensive and cannot be fully utilized for learning due to space limitations. Therefore, the authors have developed simulator materials that enable students to conduct simulated experiments on a computer. The simulator isa software program that immediately responds to the user’s adjustment of parameters and visualizes and makes
them audible[3][4].
In Japan, a state of emergency was declared in April due to the new coronae epidemic (COVID-19) thatbegan in early 2020. In response, many universities were forced to offer online classes. Online classes can bebroadly classified into two types: on-demand classes, which provide pre-recorded lecture videos, and real-timeclasses, which use teleconferencing systems during set class times. Immediately after the start of online classes,
the on-demand type was recommended because both teachers and students were unfamiliar with the system andthe performance of the network and servers was not designed for access by large numbers of students, causing
problems. However, since it is more difficult to obtain learners’ reactions in video-based learning than in faceto-face classes, lectures tend to be long, assuming a variety of situations, and students are pressed for time as
they are forced to increase the amount of assignments to check their understanding.
In order to solve this problem, the author conducted a class in which simple experiments using simulator
materials were incorporated into the online class. The purpose of this paper is to report on the structure of theexperiential class, the simulator materials used, and their effectiveness.

2

OUTLINE OF THE SIMULATOR MATERIALS

2.1 Basic structure of the simulator materials
The simulator materials handled in this study are software that performs numerical calculations in real time
based on parameters set by learners and educators, and visualizes and makes the results audible. The simu-

Figure 1. Websites where simulator materials are publishedlator is designed to be easy to understand (visibility) as a teaching material and easy to operate so that theoperator does not feel confused. In terms of ease of understanding, the software incorporates animation in thevisualization process, a three-dimensional display technique using 3D graphics, and careful use of colors in thegraphs.
Figure 1 shows the website where the simulator teaching materials developed by the authors are available[5].
Figure 2 shows the real-time rendering of the voice waveform and its frequency components input from a
microphone as an example of the simulator materials.
2.2 Outline of a face-to-face class using simulator materials
The author has been conducting face-to-face classes using simulator materials until FY2019. The target classis Calculation Method (specialized / elective, 120 minutes ×13), which is provided in the 5th semester (firstsemester of 3rd year) of the Faculty of Information and Computer Science. The objective of the class is tointroduce algorithms and programs for solving various real-world problems.
Real-world problems are introduced at the beginning of the class, followed by the actual means to solve

Figure 2. Example of simulator teaching material drawing a voice waveform (upper part) and its frequency
components (lower part)them using simulator materials. After the students have grasped the process, the algorithms and programs are
explained to help them understand the flow of the process.
The simulator material shown in Fig. 2 is used to explain the Fourier transform algorithm and program,and to help students understand concepts such as sound height and sound pressure, and how to read a graph offrequency components.

3

Use of simulator materials in on-demand classes

3.1 Concept of simulator materials
Until FY 2019, simulator materials have been used as "materials to be shown for the purpose of explanation,"
but in FY 2020, all Calculation Method classes will be conducted on-demand to prevent the spread of COVID19 infection. Therefore, the simulator materials were also used as "teaching materials for learners to use" inorder to improve the level of understanding in the classes. What is required of a simulator as a teaching material
is to assist understanding of the phenomena and to provide hands-on learning through simulated experiments.
The concept of the simulator materials was to run on as many platforms as possible. For this reason, thesimulator is developed using JavaScript as a program that runs on a Web browser. HTML5 Media Devices API
and Web Audio API are used for sound synthesis and analysis.
3.2 Course Plan
Table 1 shows the lesson plan for the 13-week class, the post-class experimental simulation subjects, and thesimulators used.
In the second week, the students learn the basics of DFT, so they use the Function Generator to generate
sin wave, square wave, and other sounds, and conduct an experiment to determine the frequency components
using Realtime FFT. Figure 3 shows a screenshot of the Function Generator. The four buttons at the top
are for sounding a sin wave, square wave, triangle wave, and sawtooth wave. Below them are buttons thatplay instrumental sounds extracted from the features, which are similar to violin, clarinet, piano, and flute,respectively. The lower part of the screen is used to set the frequency and the duration of the sound. Figure 1
shows the sound of a violin played by the above simulator and analyzed by Realtime FFT.

Table 1. Physical parameters of liquids
Week

Lesson content

Experimental content

Simulator

1

Guidance2

Basics of DFT

Measures the frequency components
of sine waves and voice, and reports the frequencies of the fundamental and overtones

Function Generator, Realtime FFT

3Theory of DFT

4Program of DFT

5

Summary of DFT

6

Numerical representation inside a computer

7

Numerical representation inside a computer

8

Solving the equations

Read the program and compare it totheory9

Numerical integration

Read the program and compare it totheory10

Partial differential equations

Read the program and compare it totheoryString Vibration

11

Methods for creating simulator materials 1

12

Methods for creating simulator materials 2

13

Montecarlo Method

Read the program and compare it totheoryRandom Numbers,Montecarlo method

Phase and Waveform
Read the program and compare it totheoryFigure 3. Simulator material of Function Generator

The third week is for learning about the theory of DFT. The simulator shown in Figure 4 is used to explain
why the cos component is necessary in addition to the sin component to represent the phase difference. The
upper three sliders allow adjustment of the 440 Hz sound, and the lower slider allows adjustment of the 880
Hz sound. The items that can be adjusted are the amplitude of the sin component, the amplitude of the cos
component, and the phase (which affects the amplitude of the sin and cos components). The phase slider canbe set to values from -180° to 360°, which allows the students to observe how the phase changes in the lower
graph.

Figure 4. Simulator material of Phase and Waveform
The tenth week is devoted to the theory and practice of partial differential equations. Using the simulator
shown in Figure 5, the vibration of a string is explained as an example of the practical application of partial
differential equations. Then, the movement of the string is explained when the position where the string isplucked is changed. The students will understand the outline of the above simulator material, and then deepen
their understanding by conducting a set of experimental conditions. The Start/Stop button on the left side of thescreen allows the user to start/pause the animation, and the buttons marked 0.1 to 0.5 below it allow the user
to change the location of the string plucking.

Figure 5. Simulator material of String Vibration
The 13th week is dedicated to learning about random numbers and the Monte Carlo method. The simulator

shown in Figure 6 explains uniform, normal, and exponential random numbers. By varying the amount ofrandom numbers generated, students learn that the accuracy is proportional to the amount of random numbers
generated. The simulator material shown in Figure 7 then calculates the area of a circle/volume of a sphereusing the Monte Carlo method and explains its applications. In this simulator, the learner also learns that theamount of random numbers generated is proportional to the accuracy.
The animation of random numbers being generated by pressing the "start" button at the top of the screen
is displayed. The amount of random numbers generated is determined by the button below it. When the start
button is clicked again after the specified number of calculations, the circle/sphere is overwritten. To erase thedrawing, click the clear button at the top of the screen.

Figure 6. Simulator material of Random Numbers

Figure 7. Simulator material of Monte Carlo Method

4Students’ Responses

A questionnaire survey was conducted on the experiments and evaluation of the simulator materials in the 2022
class. Excerpts from the survey results are explained below.
Figure 8 is a part of the questionnaire survey conducted in the second week, showing the results of the timespent on the experiments. The horizontal axis shows the time spent and the vertical axis shows the number of
respondents. The results show that the students needed more time than the author had expected.

Figure 8. A part of the results of the survey: Time spent on the experiments of second week

The question was asked, "Did the simulator materials help you in understanding the lecture content?" Asshown in Figure 9, the students gave high marks to the simulator materials.

Figure 9. A part of the results of the survey: Did the simulator materials help you in understanding the lecture
content

5

Conclusion

The authors have been developing simulator materials to help students understand the lecture content. The
spread of Covid-19 has made it impossible to hold face-to-face classes. In order to effectively conduct classes
when the face-to-face classes are shifted to on-demand classes, the authors started classes incorporating experiments using the simulator materials. As a result, it became clear that experimental lessons using the simulator
materials were useful for understanding the lecture content.

REFERENCES[1] Masaru KOYASU. Report on symposium “Visual and aural presentation tools for education in acoustics”.
THE JOURNAL OF THE ACOUSTICAL SOCIETY OF JAPAN. 1999, Vol.55, No.3, p. 207-210[2] Yoshiro MIIDA, Masahiro UKIGAI, Hiroshi SUDA. Multimedia-Based Acoustical Engineering Education.
Journal of the educational application of information technologies. 1999, Vol.2, No.1, p.13-18.
[3] Hiroshi SUDA, Yoshiro MIIDA. Acoustic education for college student. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF JAPAN. 2007, Vol.64, No.1, p.41-46.
[4] Hiroshi SUDA, Yasuhiro NAKAGAWA, Naoto NAKAMURA, et al. Deployment of the Simulator BasedMultimedia Teaching Materials to Africa and Middle and Near East. Journal of JSEE. 2009, Vol.57, No.2,
p.40.
[5] Acoustic e-Learning Materials, https://www.sudalab.net/

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0652

A study on audio-visual representation and thinking for designing
acoustic environments
Ayça ŞENTOP DÜMEN1; Mine KOYAZ2
1

Istanbul Bilgi University, Turkey

2

Istanbul Bilgi University, Turkey

ABSTRACTSound conveys information about spatial characteristics, supports development of a multi-sensory cognitivemodel, enforces the identity, and steers users’ actions and emotions. Although building acoustics courses
provide the basic knowledge of acoustics, students encounter difficulties on identifying the design potentialsof the acoustic environment and transferring the obtained knowledge into architectural design. This points tothe necessity of embracing novel aural techniques in design studies and acoustic education.
This study aims to reinterpret conventional design steps from soundscape perspective and apply
representation techniques to deal with sound’s intangibility. A 10-days workshop was conducted in Istanbul
Bilgi University with 25 students, where majority were in the 1st year of architectural education and had
little/no knowledge on acoustics. The theoretical background was initially introduced to the students during a
training soundwalk, followed by seminars and discussions. In the second phase, a selected route was visitedand analysed with soundscape assessment forms, recordings and through developed representation
techniques: ‘soundscape infographic’, ‘sound collage (sound-editing)’, and ‘abstract (physical) sound model’.
Based on the analysis, students proposed design interventions for the selected locations. The results showedthat students not only developed an environmental control attitude but also acknowledged sounds’ potentials
in design.
Keywords: Acoustic education, Design education, Audio-visual thinking, Soundscape, Representation

1. INTRODUCTION
Phenomenological approaches are widely applied in design studies in order to enforce theinteractions between people and spaces and to generate a design identity based on the humanexperience and senses [1]. This requires the architect to develop an awareness of multi-modal
perception of a space and an ability to synthesize the acoustical information with user experience andthe context. Building acoustics education in architectural curriculum, aims to develop the knowledge
on principles of noise control and room acoustics, through theoretical seminars, case studies andvarious exercises. Although it provides the basic understanding of acoustics, students encounter
difficulties on transferring the obtained knowledge to their design and using the potentials of theacoustic environment. Firstly because, it is harder to grasp the sound data during the problem
definition stage due to its intangibility. Secondly, novice designers lack a methodological approach tocommunicate the obtained theoretical knowledge with the multi-dimensional aspects and interrelating
elements of a design problem.
Visual representation is an integral part of design thinking that is used for (1) investigation andtransforming complex data into new solutions, (2) externalizing and making concepts tangible tocommunicate with others, and (3) a way of decreasing the cognitive burden of excessive information[2]. Goldschmidt discriminates the visual representation that is in the form of two- or
three-dimensional descriptions which intend to facilitate communication and evaluation of elements,from visual thinking as ‘the production of ideas, the reasoning that gives rise to ideas and helps bringabout the creation of form in design’ [3]. For the visual thinking to emerge, visual representation
1
2

ayca.sentop@bilgi.edu.tr
mine.koyaz@bilgi.edu.trshould precede the ideas instead of only recording them. A design proposal should find its physicalform only after this interactive abstract thinking process. When designers rush the ‘analysis andproblem definition’ step of a design process and try to develop an embodied image at the early stages,
the creativity of the output tends to weaken [4]. By implementing this approach also into acousticaleducation, it can be asserted that design of the acoustic environments should also benefit fromaudio-visual representation and thinking.
Application of visualization to acoustic environments can be seen in soundscape maps similar tonoise maps. These maps aim to collect objective and subjective data about the acoustic environment asperceived by the people and provide feedback for city development and urban design [5]. Through theuse of colors, such maps can display audibility of specific sound sources [6]–[8]; the normalizedsoundscape index – NDSI which shows the ratio of biophony and antrophony [9], the everyday quiet
areas [10], sonic districts [8], [11], perceived affective quality or ‘emotional components’ [6], [7], [12],and the overall assessment of the sound environment [13]. While being effective communication tools,
these types of maps suffer from being ‘data on silent papers’ [14].
This study aims to explore audio-visual thinking techniques in order to analyse, interpret, represent
and design the acoustic environments. The study further aims to raise students’ awareness of thesoundscape, encourage them to ‘listening’ to a space, and support them in using sound as a design tooland develop sound strategies.

2. METHODOLOGYSoundwalks formed the main methodology of the study. ISO 12913-2:2018 and ISO 12913-3:2019
defines the methodologies for soundscape research [15], [16]. Although these standards were
unavailable at the time of the study (2017) we can see a consistency in the soundwalk methodology andthe chosen terminology. 10 assessment stops were selected on the soundwalk route where students
listened to the sound elements and acoustic features of the environments. Students were further
encouraged to discover the interaction of sound and space through self-generated sounds, Considering
the body as an instrument; footsteps, handclaps and voices become tools for performance andinteraction with the surrounding. For an orchestra, ease of ensemble and stage support are important
acoustical parameters that define the capacity of the orchestra members to play in unison and hearingof oneself. This inspired the idea of additional exercise of ensemble readings at the visited places inorder to develop an understanding of soundscape’s effect on one’s own and/or ensemble performance.
During the soundwalk data were collected with soundscape assessment forms, photos and personal
sound records.
In order to train the students in attentive listening several additional listening exercises wereconducted.
Preparations for the workshop
A soundscape assessment form was prepared in order to guide students in ‘listening’, evaluate
their surroundings and build up keywords to define the acoustic environments. The assessment form
was in the format of A5 cards that consisted of open-ended, 5-point likert scale, and multiple-response
questions (Figure 1). Assessment criteria and scales were determined in the light of other soundscapestudies. The first card was intended for the beginning of the soundwalk and included questions
regarding the familiarity of the route to the participants and their expectances. Cards 2-11 included
rating questions for each of the stops during the soundwalk. Comfort and noisiness ([16] recommends‘loudness’ instead) was evaluated in 5-point scale. The perceived affective quality was evaluated withadjectives (presented as pairs on a radar chart) ‘eventful-uneventful, exciting-monotonuous,pleasant-unpleasant, calm-chaotic’ in accordance with [17]. Sound quality of ensemble readings were
evaluated with the adjective-pairs ‘open-enveloping, unclear-distinct, muffled-shrill’ (e.g.[18], [19]).
Cards 12-21 included verbal description tasks. Participants were asked to describe the soundscape by
selecting keywords from a list of onomatopoeia words and descriptors (e.g. [20]–[22] and by writing atext description. They were also asked to define their evoked feelings by selecting from a list of
adjectives (e.g. [21], [22].
2.1

(a)

(b)
(c)
Figure 1 – Soundscape assessment cards; (a) participants’ familiarity with the route, (b) soundscape rating,(c) verbal description
6 reading texts were selected for ensemble readings (Figure 2). These reading texts were one
paragraph-length narratives related with the soundscape concept written by the authors of this paper,[14], [21], [23], and a stanza from a well-known poem by Orhan Veli Kanık, which gives description ofsounds of Istanbul.

Figure 2 – Reading cards for ensemble readings
Prior to the study, workshop tutors visited the selected soundwalk routes in Istanbul to locate thestops where the soundscape would be evaluated via the assessment forms. In some of the visited places,a sound recording was made while one of the tutors read a pre-written text. These recordings were lateredited to form a video, together with a photo showing the tutor’s standing point and her surroundings.
Resulting video later used as a discussion medium on the sound characteristics of different spaces.
Lastly, workshop participants were asked to make three sound records until the start of theworkshop and send it together with a photo of the recording place. All records were collected andcategorized according to their themes. Sound files and photos were put separately to generate amatching exercise.

2.2 Workshop
A 10-days workshop was conducted in Istanbul Bilgi University with 19 students (15 female, 4male) in 2017 summer. The majority of the students was in the 1st year of architectural education andhad little or no knowledge on acoustics.
Step 1: Training soundwalks and listening exercises
In order to train the students on attentive listening and develop an understanding of the sonicenvironment two training soundwalk routes were determined (Figure 3):• On the first day of the workshop, a soundwalk was conducted inside the Istanbul Bilgi
University campus with several indoor and outdoor stops. This route allowed the students
to experience their every-day environment with a new perspective. During the walk,students were introduced with the basic acoustical principles, and they discussed sound –space – human interactions. The prepared soundscape assessment form was handed out tothe students in order to teach ‘listening’ and familiarizing with the soundscape assessment
method.
• The second route was defined inside a built environment, which offered various
architectural experiences through its closed – open – semi-open spaces and inclusion ofnatural elements. The second route ended with a visit to Dialogue in the Dark: an exhibition
where blind people guide the visitors in complete darkness and the route is discovered
merely via other senses.
Listening exercises included the video showing a compilation of pre-recorded sounds on the firstroute, and the matching exercise where students were asked to match the sound files with correct photo.
Theoretical knowledge was also provided with the presentations.

Figure 3 – Training sound walks and ensemble readings
Step 2: Main soundwalk and data collection
The main soundwalk route for the study was selected in the historical peninsula of Istanbul (Figure4). The selected route covered a variety of urban soundscapes consisting of touristic city centre, hotels
zone, architectural landmarks, natural park and sea side. 10 assessment stops were determined: 5
outdoor (Galata bridge, Hüdavendigar Street, Gülhane Park, Little Hagia Sophia -Küçük AyasofyaStreet, Obelisk of Theodosius Square) and 5 indoor (Central Post Office, Grand Bazaar, Sokullu
Mehmet Pasha Mosque, Arasta Bazaar, Basilica Cistern).
During the soundwalk, students were asked to listen to the elements of acoustic environment, andanalyse the acoustics via self-generated sounds. At each stop, four students performed ensemble
readings while the rest were listening. After attentive listening for 3 minutes, everyone filled theassessment form. Students also made sound recordings, took photos and notes. The first 6 stops werevisited in the morning. After a lunch break the rest of the route was completed.
Step 3: Audio-Visual representation and thinking
On the following day, a discussion session occurred at the studio. Radar charts were prepared foreach stop by averaging perceived affective quality responses (Figure 5-a). Identified sound sourcesand onomatopoeia words formed the keywords. Collected charts and keywords formed a soundscapeinfographic for the sound route.
For the following studies, students were divided into 5 groups and each group further analysed apart of the route via two developed representation methods:

(b)

(c)

(d)

(e)
(a)
Figure 4 – Main soundwalk (a) Map of the route (students’ proposals are marked with blue), (b) Little Hagia
Sophia Street, (c) Grand Bazaar, (d) Central Post Office, (d) Galata Bridge• Sound collage: Traditionally, an architectural collage is a visual representation and design thinking
technique that compiles different elements to convey an idea and/or a multi-layered atmosphere. In
the present study, each group of students were asked to generate a sound collage with the recordingsthat they took during the soundwalk and wordclouds with the keywords that emerged in thediscussion session. The sound collages were appx. 1.5-2 minutes-long mp4 files created in Adobe
Premiere software by compiling significant sound elements as observed by the students.
• Abstract sound-model: Abstraction is ‘a process of revealing the essence of something by removing
irrelevant information and keeping only key elements that can be analyzed according to theirrelevance to the goal’ [24] In architecture abstraction is used to accent e.g. geometrical forms,
selective elements, flow of the circulation etc. as perceived by the architect in order to acquire
knowledge and develop a design idea [25]. During the workshop, students were asked to reflect (1)collected data regarding the soundscape assessment (2) their selection of sound elements thatcharacterizes the space, (3) their perception and senses related with the sound environment, on a
printed map (Fig. 5-b). The works of 5 student groups would converge to create the whole route;therefore they were encouraged to build a common legend for repetitive elements. The sound-map
was later filmed to form a video (Fig. 5-c).

(a)
(b)
(c)
Figure 5 – Representation studies, (a) discussion, (b) preparation of sound-model, (c) filming
Step 4: Proposing a design intervention
Similar to an architectural design study, the research was followed with the identification of adesign problem and concept development. Each student group selected a location on the assigned zone
where the acoustic environment can either be improved or accented with the design interventions.
Each group initially addressed strengths, weaknesses, opportunities and/or threads (SWOT) of thepicked location through brainstorming and later developed a design proposal. Workshop tutors
participated in the brainstorming sessions by listening the students’ ideas, giving critics and providing
assistance when they needed.
Upon concept development, students re-edited the audio files of sound collages to auralise theintended design intervention. Any new ideas or discoveries while working with sound files werewelcomed, so this was not just a representation but also an audio-visual thinking exercise. Finally, thedesign intervention was explained via sketches, texts and diagrams in a presentation board.

3. RESULTS
3.1 Sound representations
The prepared infogram, which contains the perceived affective quality and the keywords, ispresented in Figure 6-a. The hatch area on the radar chart helps to easily differentiate the auralcharacteristics of the selected places.
While the infogram provides a visual representation, the sound collage of the route provides theaural representation of the route. The keywords are also implemented on the sound collage, thus
supporting the discernment of various sound sources (Figure 6-b).
Finally, an abstract sound-model was built to show the essential elements of the soundscape as‘perceived’ by the students. Both visual and aural representation was aimed, by placing interactive
shapes, forms and materials on a map of the route (Figure 6-c). Sounds were mimicked in the model byintroducing sound-making elements. Thus, the participants could experiment with aural representa tionof the frequency and temporal characteristics of the actual sounds similar to a foley artist 3. The final
model was filmed while students moved / pulled / scratched the kinetic elements of the sound -model.

3

Foley artist is a person who recreates the background sounds of a film scene in the postproduction stage.

(b)

(a)

(c)

Figure 6 – Sound representations (a) infogram containing radar charts and keywords, (b) a screenshot fromthe sound collage video, (c) sound model
3.2 Proposed design interventions to the soundscape
In Table 1 the identified design problem and proposal are briefly described for each student group.
Figure 7 shows the final posters (some of them are clipped for brevity). The results showed thatstudents not only developed an environmental control attitude but also saw the potential of sound as adesign tool to create a spatial experience. Through site visits, discussions and application ofaudio-visual representation techniques students developed an understanding of basic acousticalprinciples and applied them on a design problem. The study also provided experimental data on theapplicability of new representation techniques such as sound collages and abstraction through so undmodels and their efficiency in acoustics learning.
Table 1 – Students proposals
Title

Location

SWOT

Design interventiona) Da Solo –By

EminönüThis is a rich acoustic environment including many

A sound pavilion that is formed tomyselfSeasidesoundmarks (sea, gull, ferry horn…) (strength)

focus sounds coming from the sea and(Kendihowever it is unnoticed in the daily rush (weakness)isolates its visitor from the rush of

Kendine)Natural sounds (i.e. sea, gull, wind) require anpedestrians and road-traffic bothemphasize (opportunity) over mechanical sounds

aurally and visually.

(traffic, ferry engine) (threat)b) One step into

GülhaneGülhane Park is a peaceful escapade in theSound installation that provides

peace

Park

city-centre (strength). This experience can beinteraction with the visitor, enforcing

emphasised and visitors can be engaged to the place

the feeling of ‘peacefulness’ however

through new sound elements (opportunity)

do not disturb the natural(Huzura adım)environmentc) Re-listen

Grand

Loud but unique acoustic environment (strength) –Increasing the sound contrast(Yeniden dinle)

Bazaaris it only noise? From cultural heritage perspective

between the old bazaar and the new

the rich soundscape of the Bazaar needs to beannex building to draw attention toprotected and deserves more attention (opportunity)

the sound identity and to distinguishnew from old (restoration approach).
Additional stickers inviting people to‘re-listen’.

d) Song of theArastaTraffic noise disturbs the peaceful environment inNoise control through trafficwater

BazaarKüçük Ayasofya Street where many hotels aremanagement (speed limit, bans on

located. (weakness & threat) Meanwhile Arastra

horns) and water elements that mask

Bazaar is deserted possibly due to lack of attractions

the noise. To attract people, ain the zone. (weakness & threat) It can invite people

performance platform is design (1through sound events short enough to not disturb thehour/day. The sound envelopment issurrounding (opportunity)

provided through water running(Suyun şarkısı)

under the walking route and a bridge
performance platform.
e) Silence of

Yerebatan

The Cistern has a unique sound pattern that enforces

During the constructions, visitors canMedusaCistern

the spatial perception (strength). However there isuse headphones that will filter the(Medusa’nın

an on-going restoration work and construction noise

existing sounds and mix it withsessizliği)blocks perception of the original acoustic

pre-recorded sounds.

environment, affecting the visitor’s experience(threat).

(a)

(b)

(c)

(d)

(e)

Figure 7 – Posters of the proposed interventions (Some parts are cropped to increase the size and legibility)

4. CONCLUSION
Through the developed methods of audio-visual representation, students were able to step by step
rationalize their ideas of the soundscape perception; develop arguments, reshape the sounds,
understand / discover, and reconstruct their arguments; thus, an interactive audio-visual thinking
process occurred. During the 10 days’ workshop, students were educated on attentive listening anddeveloped an understanding of the sonic environment. Results of the study demonstrate that the act of“listening”, enriches the user experience and awareness of soundscape. Understanding the sensory
aspects of sound environment along with its physical aspects, support the implementation of sound as
a design tool and develop sound strategies.

ACKNOWLEDGEMENTS
This study was conducted in Istanbul Bilgi University.

REFERENCES[1] L. Ray, “Chapter 12: Philosophy, phenomenology and the experience of space,” in Research Methods
for Architecture, London: Laurence King Publishing, 2016, pp. 153–163. [Online]. Available:
https://www.ebsco.com/terms-of-use[2]A. Sahli, E. Pei, A. Manohar, and R. Evans, “Knowledge Visualization: A Design centered Framework,”
Procedia CIRP, vol. 109, pp. 629–634, 2022, doi: 10.1016/j.procir.2022.05.305.
[3] G. Goldschmidt, “On visual design thinking: the vis kids of architecture,” in Design Studies, vol. 15,
Butterworth-Heinemann Ltd, 1994, pp. 158–174.
[4] V. Kokotovich, “Problem analysis and thinking tools: an empirical study of non-hierarchical mind
mapping,” Design Studies, vol. 29, no. 1, pp. 49–69, Jan. 2008, doi: 10.1016/j.destud.2007.09.001.
[5] G. Brambilla and F. Pedrielli, “Smartphone-based participatory soundscape mapping for a moresustainable acoustic environment,” Sustainability (Switzerland), vol. 12, no. 19. MDPI, Oct. 01, 2020.
doi: 10.3390/SU12197899.
[6] L. M. Aiello, R. Schifanella, D. Quercia, and F. Aletta, “Chatty maps: Constructing sound maps ofurban areas from social media data,” Royal Society Open Science, vol. 3, no. 3, Mar. 2016, doi:
10.1098/rsos.150690.
[7] E. Margaritis and J. Kang, “Soundscape mapping in environmental noise management and urban
planning: Case studies in two UK cities,” Noise Mapping, vol. 4, no. 1, pp. 87–103, Mar. 2017, doi:
10.1515/noise-2017-0007.
[8] B. Akkaya, “Urban Soundscapes: Evaluating the Sonic Layers of Kadiköy Historical Bazaar and Near
Surroundings,” MSc Thesis, 2014. (in Turkish)[9] Q. Liu, Z. Liu, J. Jiang, and J. Qi, “A new soundscape analysis tool: Soundscape Analysis and Mapping
System (SAMS),” Applied Acoustics, vol. 169, Dec. 2020, doi: 10.1016/j.apacoust.2020.107454.
[10] A. Radicchi, D. Henckel, and M. Memmel, “Citizens as smart, active sensors for a quiet and just city.
The case of the ‘open source soundscapes’ approach to identify, assess and plan ‘everyday quiet areas’in cities,” Noise Mapping, vol. 5, no. 1, pp. 1–20, Mar. 2018, doi: 10.1515/noise-2018-0001.
[11] R. M. Schaefer, Ed., Five Village Soundscapes: No. 4, The Music of the Environment Series. A.R.C.
Publications, 1977.
[12] J. Kang, F. Aletta, E. Margaritis, and M. Yang, “A model for implementing soundscape maps insmart cities,” Noise Mapping, vol. 5, no. 1, pp. 46–59, Mar. 2018, doi: 10.1515/noise-2018-0004.
[13] F. Aletta and J. Kang, “Soundscape approach integrating noise mapping techniques: A case study in
Brighton, UK,” Noise Mapping, vol. 2, no. 1, pp. 1–12, Jan. 2015, doi: 10.1515/noise-2015-0001.
[14] R. M. Schafer, “The Soundscape: Our Sonic Environment and the Tuning of the World,” Rochester,
1993.
[15] ISO, “ISO/TS 12913-3:2019 Acoustics -Soundscape, Part 3: Data analysis.” 2019.
[16] ISO, “ISO/TS 12913-2:2018 Acoustics -Soundscape, Part 2: Data collection and reporting
requirements.” 2018.
[17] Ö. Axelsson, M. E. Nilsson, and B. Berglund, “A principal components model of soundscapeperception,” J Acoust Soc Am, vol. 128, no. 5, pp. 2836–2846, Nov. 2010, doi: 10.1121/1.3493436.
[18] A. Özçevik, “An Approach on the Evaluation of Urban Acoustical Comfort with the SoundscapeConcept” Istanbul, 2012. (in Turkish)[19] A. Özçevik and Z. Yüksel Can, “A Study on the Documentation and Analysis of the Urban
Acoustical Environment in Terms of Soundscape,” Megaron, vol. 7, no. 2, pp. 129–142, 2012.
[20] W. J. Davies et al., “Perception of soundscapes: An interdisciplinary approach,” Applied Acoustics,
vol. 74, no. 2, pp. 224–231, Feb. 2013, doi: 10.1016/j.apacoust.2012.05.010.
[21] S. Bahalı, “Urban Soundscape Research in The Route Of Gezi Park – Tünel Square.” Istanbul
Technical University, İstanbul, 2015. (in Turkish)[22] S. Bahalı and N. Tamer-Bayazıt, “Soundscape research on the Gezi Park – Tunel Square route,”
Applied Acoustics, vol. 116, pp. 260–270, Jan. 2017, doi: 10.1016/j.apacoust.2016.10.002.
[23] E. Thompson, The Soundscape of Modernity: Architectural Acoustics and the Culture of Listening in
America, 1900-1933. The MIT Press, 2004.
[24] M. Palmiero, “The relationships between abstraction and creativity” in Preis, Cosmelli & Kaufman(Eds.). Creativity and the Wandering Mind. Elsevier. 2020.
[25] A. B. Gencosmanoglu and S. Nezor, “Criticizing architectural education through abstraction,” inProcedia - Social and Behavioral Sciences, 2010, vol. 9, pp. 1335–1341. doi:
10.1016/j.sbspro.2010.12.330.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0710

A Training System for Improving the Ability to AdjustMusical Intervals
Qingmin CHANG1; Takayuki ARAI2; Yukari HIRAYAMA2
1

2

Graduate School of Science and Technology, Sophia University, Japan

Dept. of Information and Communication Sciences, Sophia University, Japan

ABSTRACT
Although musicians are required to adjust music intervals accurately, there are not many methods orsystems in place for them to acquire the ability. We have therefore developed a training system to helpmusicians make music interval adjustments, especially the perfect fifth. We implemented the system in theProcessing script language and applied the concept that two complex sounds in the perfect fifth interval havecommon harmonic components and the component pairs produce beats when they are out of tune. In our
experiment based on a set of sawtooth waves, participants were trained to perform the perfect fifth tuningtask for A4 and E5. In addition, we added the harmonic sine wave to the adjustment sound, so that one could
hear the beats clearly when A4 and E5 was not harmonious. Pre- and post-training tests were conductedagainst the perfect fifths for not only A4-E5 but also C5-G5. By comparing the results of the post-training
and pre-training tests, we found that the distribution of the difference from the fundamental frequency of
adjustment sound became smaller in the latter, thus demonstrating the effectiveness of the proposed training
method. Our system can be implemented on people’s individual devices.
Keywords: Training system, Musical intervals, Tuning, Perfect fifth

1. INTRODUCTION
Musicians must tune their instrument before every performance and accurately adjust music
intervals while playing [1][2] to perform well [3]. However, practicing pitch adjustment is difficult
for musicians. Many of them use a tuner to adjust the pitch, though many music teachers say thatmusicians should not always rely on a tuner [4] because visual feedback cannot train the ear. Moreover,
the tuner may provide inaccurate feedback depending on the musical context. Therefore, we need new
methods of practicing musical intervals.
In previous studies, one system evaluated the tuning ability [7], and there was a training method
for tuning the same pitch [8]. However, not many training systems or methods exist for training
intervals with different pitches. Therefore, we have developed a training system that helps musicians
practice interval adjustments of the perfect fifth interval [5][6], especially the system utilizes
harmonics and beats. More explanation about this training system is described in the next section.

2. TRAINING SYSTEM
2.1 System overview
We focused on the perfect fifths interval using an existing system [5][6]. This training system waswritten in Processing, a free, open-source programming language. For the sounds, we used the perfectfifth interval and sawtooth waves. The pitch and waveform of the training system can be changed
freely based on your needs. The equipment we used were loudspeakers, an audio interface, a monitor,
1changqingmin0612@gmail.com

and a computer.
2.2 Training procedures
When participants run the program in this training system, they see a blue background on the screen
and hear the reference sound through the loudspeaker. When they move the mouse, they
simultaneously hear the adjustment sound and the reference sound. Moving the mouse upward raises
the adjustment sound’s level, and moving it downward lowers the adjustment sound’s level. Moving
the mouse rightward raises the adjustment sound’s frequency, and moving it leftward lowers theadjustment sound’s frequency.
The participants are asked to tune the perfect fifth interval. The screen turns green when they clickon the correct perfect fifth interval position and turns red when they click on the incorrect perfect fifthinterval position. Moreover, we both changed the background color and showed the accurate position
as feedback up to the third trial of the training.
2.3 Features
We focused on the concepts of common harmonic components between two complex sounds in theperfect fifth interval and the production of beats from out-of-tune component pairs. We know theperfect fifth has a musical interval frequency ratio of 2:3; therefore, the third harmonic of a lower
sound and the second harmonic of a higher sound have the same frequency. If sounds are adjusted tothe exact ratio of 2:3, the result is smooth and the perfect fifth interval is in tune. If the soundfrequency ratio is close in pitch but not the same, the difference in frequencies causes the beats. We
use the perfect fifth A4-E5 as an example (A4 frequency is 442 Hz, E5 frequency is 663 Hz); theharmonics of A4-E5 are as shown in Fig. 1.
To emphasize the beats, we inserted a sine wave of 1326 Hz, which is the third harmonic componentof the 442-Hz reference sound, into the reference sound. If the perfect fifth is not accurate, theparticipants can hear the beats more clearly. Before the training, we instructed the participants to beaware of the harmonics and beats. We set the level of the sine wave to decrease by 10 dB on threelevels (20 dB, 10 dB, 0 dB). Once the participants got used to listening to the beats, we turned off thesine wave.

Figure 1 – Harmonic frequencies of harmonic components of A4-E5.

3. EXPERIMENTS [5][6]In [5][6], we experimented to determine the effectiveness of our training system. The two groupsof participants were Tokyo College of Music students and Sophia University students. In the pre-test
and post-test, we used musical intervals of A4-E5 and C5-G5; in the training session, we only used
A4-E5. Post-test accuracy improved in both groups. After the training, the participants became awareof the beats, indicating their improved perfect fifth accuracy. The improvement rate of the university
students was higher than that of the music students. The musical intervals accuracy of students who
were not music majors also improved. Moreover, the results with the C5-G5 also improved despite
the training only being conducted with the A4-E5. Our training system was also effective for untrained
music intervals.

ACKNOWLEDGEMENTS
We would like to thank the students of the Tokyo College of Music and Sophia University forparticipating in our experiment.

REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.

Hopkins MT. Eighth-grade violinists' instrument tuning ability: A comparison of pitch perception
and tuning accuracy. Journal of Research in Music Education. 2015;63(3):349–368.
Geringer JM, MacLeod RB, and Sasanfar JK. In tune or out of tune: Are different instruments
and voice heard differently? Journal of Research in Music Education. 2015;63(1):89–101.
Springer DG. Research to resource: Evidence-based strategies for improving wind intonation.
National Association for Music Education. 2020;39(1):4–7.
Schlegel AL and Springer DG. Effects of accurate and inaccurate visual feedback on the tuning
accuracy of high school and college trombonists. International Journal of Music Education.
2018;36 (3):394–406.
Chang QM, Arai T, and Hirayama Y. Proposal of training method and survey on the accuracy ofinterval adjustment: Through perfect fifth tuning. Proc. Spring Meet. Acoust. Soc. Jpn. 2022;3–
4.
Chang QM, Arai T, and Hirayama Y. The effect of the training method for the adjustment task ofperfect fifth: Comparison of music college students and university students who have musicexperience. Proc. Autumn Meet. Acoust. Soc. Jpn. 2022.
Hopkins MT. Pilot-testing of new software for measuring string players’ instrument tuning skills.
Journal of Music, Technology and Education. 2014; 7(1):5–21.
Silvey B, Nápoles J, and Springer DG. Effects of pre-tuning vocalization behaviors on the tuning
accuracy of college instrumentalists. Journal of Research in Music Education. 2019;66(4):392–
407.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0781

A smart outreach activity for educationin acoustics and speech science
Takayuki ARAI 1
1

Sophia University, Japan

ABSTRACT
Outreach activities in education were originally intended to be conducted more or less in-person. However,
the COVID-19 pandemic has resulted in a shift to many online activities since 2020. One such activity inwhich I was involved was a lecture series on speech science in Japan. In an online lecture, I asked participants
to prepare a set of materials for an online workshop. For example, they prepared a cut straw and a piece ofplastic sheet for making a reed-type sound source. They also prepared a pipe and a sliding part with a roll of
cardboard. By combining these materials, we were able to construct a handmade vocal-tract model, even in
the online setting. Another outreach activity we are trying is an online workshop where each participant
controls pieces of equipment in a laboratory remotely. Even if we cannot move our physical vocal-tract model
system to a specific venue, we can remotely connect the laboratory to each participant via a communication
tool. If he/she sends a set of commands to the system, it will move accordingly and finally produce sounds.
This remote system enables us to conduct more outreach activities in a “smart” fashion.
Keywords: Outreach Activity, Education in Acoustics, Education in Speech Science

1. INTRODUCTION

Since 2020, when the COVID-19 pandemic started, we were forced to shift our teaching style fromin-person to online. Because I usually demonstrate how speech sounds are produced with a set ofphysical models including a lung model, vocal-tract models, and sound sources (1), it was difficult
for me to shift into a digital form. One of the things that I have achieved was to release a set of
webpages and video clips of real demonstrations with my explanations not only in Japanese but alsoin English. They are now available through the website and the YouTube channel, called “AcousticPhonetics Demonstrations.” I also conducted a pilot online activity with participants who joined theevent remotely. Furthermore, I am currently planning to prepare an online workshop, whereparticipants can remotely manipulate vocal-tract models in my laboratory.
Thus, we will discuss several approaches for “smart” outreach activities.

2. ACOUSTIC-PHONETICS DEMONSTRATIONS

Since before the COVID-19 pandemic, the demand for online teaching materials (e.g., webpages
and video clips) has increased. Therefore, I made a website called “Acoustic-Phonetics
Demonstrations” or “APD” at https://splab.net/APD/ (2). This website covers a series of topics on
acoustic phonetics and speech science, and each page consists of written explanations, figures, images,
sounds, and videos. The video clips are also uploaded on the “Acoustic-Phonetics Demonstrations”
YouTube channel.
Access to the APD website and the APD videos has also increased. In addition, I added more pages
on the APD website and English versions of the six main video clips on the APD channel. These
English versions are also subtitled in English so that worldwide viewers can learn the contents (Fig.
1).

1

arai@sophia.ac.jp

Figure 1 – Snapshots of three video clips from “Acoustic-Phonetics Demonstrations” YouTube channel.
Left: VTM-T20, middle: VTM-S20, right: lung and head-shaped models.

Figure 2 – Sliding vocal-tract model construction.

3. ONLINE LECTURE WITH MINI WORKSHOP (3)Every year, I lecture at Tokyo Medical and Dental University. In 2020, I lectured online. In addition,
we tried to incorporate hands-on activities. The participants were asked to prepare a set of materials
in advance, such as a cup of water, a straw, balloons, a plastic bottle, a piece of plastic sheet, a paper
towel tube or a plastic wrap roll, long, narrow pieces of cardboard, and a skewer. The cup of waterand the straw were used to visualize the air pressure when we breathe and phonate. The straw wasstuck into the water at about 5-cm deep. Blowing into the straw and causing bubbles to come out fromthe tip of the straw simulates 5-cm H2 O in terms of pressure. The plastic bottle and the balloons wereused to make a lung model. The bottom half of the bottle was cut, and the cut balloon was placed on
the bottom part. Another small balloon was placed inside, and the neck part of the small balloon wasplaced inside out on the neck part of the bottle.
The photos in Fig. 2 show how we made a sliding vocal-tract model. The plastic sheet and thestraw were used to make a reed-type sound source. The paper-towel tube or the plastic wrap roll wasused as an acoustic tube. The long, narrow pieces of cardboard were rolled around the skewer to makean inner slider. This online mini workshop was quite successful in that we were able to remotely
conduct such hands-on activities online with everyday products.

4. ANOTHER ONLINE WORKSHOPWe are currently preparing an online workshop that is interactive between the participants and ourlaboratory. For an in-person workshop, we bring teaching materials to the venue, including vocal-tract
models. For an online workshop, we set up an environment for teaching and telecommunication in thelaboratory. In addition to the regular online workshop, in our new plan, we can remotely change thearticulatory configurations of a vocal-tract model, for instance, so that participants can sendcommands and manipulate their configurations from a distance.

Figure 3 – Setup for smart online workshop.
The diagram of our system is depicted in Fig. 3. In this figure, a robot arm manipulates thearticulatory configurations of a sliding vocal-tract model, or VTM-S24 (4). An advantage of using thismodel is the fewer degrees of freedom. The smallest degree of freedom is one, the left-right positionof the lever. Two commands (e.g., “left” and “right”) are assigned for the left and right movements.
The “go” command is for producing a vowel for a short time period. The commands can be sent fromthe participant’s side to the laboratory in various ways, including visually or verbally. If the verbal
commands are chosen, only a simple automatic speech recognition module could be installed into thesystem in the laboratory. The verbal commands can be sent via a teleconferencing application (e.g.,Zoom). Therefore, we can conduct a workshop with participants anywhere in the world, especially for
those who are poor, in a rural area, or are ill (even from a hospital).

ACKNOWLEDGEMENTS

This work was partially supported by JSPS KAKENHI Grant Number 21K02889.

REFERENCES

1. Arai, T., Education in acoustics and speech science using vocal-tract models. J. Acoust. Soc. Am. 2012;
131: 2444–2454.
2. Arai, T., Learning acoustic phonetics by listening, seeing, and touching. POMA. 2013; 19:025017.
3. Arai, T., For online courses on speech-related acoustics. Proc. of Acoust. Soc. Jpn. Spring Meeting. 2021;
1319–1322.
4. Arai, T., Downsizing of vocal-tract models to line up variations and reduce manufacturing costs. Proc. of
INTERSPEECH. 2021; 960–961.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0847

Teaching Pronunciation: Biofeedback and Laboratory Work
Noriko YAMANE1; Atsushi FUJIMORI2; Brian David TEAMAN3; Ikuyo KANEKO4
1 Hiroshima University, Japan2
3

University of Shizuoka, Japan

Osaka Jogakuin University, Japan
4

Juntendo University, Japan

ABSTRACT
There has been a growing body of research on flipped learning for education in general as well as language
learning. Research has shown positive results for students’ engagement, interaction, creativity, and higherorder thinking; while some challenges have been noted, like teachers’ awareness of student progress, effective
delivery of systematic knowledge, time management in class and homework, and workload for teachers andstudents. Here we present a case study for non-native teachers’ teaching English as a foreign language,
specifically English phonetic aspects to non-English major undergraduate students. Acoustic features as well
as students’ reflections have been recorded, in order to explore how L2 phonetics learning can be enhancedwith in-class activities using various forms of biofeedback as well as laboratory work. The feedback includes
kinesthetic emphasis of stress, visual pitch curves in Praat, and tongue positions using ultrasound, whilelaboratory work includes measurements of pitch range and formants. The research shows how various
activities increase students’ performance in the pronunciation of the target language.
Keywords: Kinesthetic, Praat, Ultrasound

1. INTRODUCTION
There is a growing body of research on flipped learning (1) for education in general as well as
language learning. According to the AALAS, “Flipped Learning is a framework that enables educators
to reach every student. The Flipped Approach inverts the traditional classroom model by introducing
course concepts before class, while allowing educators to use in-class time to guide each student
through active, practical, innovative applications of the course principles.” ( 2). Therefore class timeshould be devoted to student-centered activities, where students can be more engaged in tasks, todevelop ‘higher-order’ skills defined in Bloom’s taxonomy: 1) remember, 2) understand, 3) apply, 4)analyze, 5) evaluate, and 6) create.
Flipped learning implementations in EFL have generally yielded positive results for students’
engagement, interaction, creativity, higher-order thinking, and learning outcomes (3); while somechallenges have been noted, like teachers’ awareness of student progress, effective delivery ofsystematic knowledge, time management in class and homework, increased technical support
workload for teachers and students (4, 5, 6).
The purpose of this study is to share our implementation of phonetic instructions in small flippedclasses of non-English major students in EFL-related classes conducted by non-native speakers ofEnglish. Flipped classes in this case consisted of out-of-class preparation which included reading or
watching videos. Our focus here is on the in-class activities. Student progress as well as students’reflections will be presented.
12
3
4

yamanen@hiroshima-u.ac.jp
fujimori@u-shizuoka-ken.ac.jp
teaman@wilmina.ac.jp
ikaneko@juntendo.ac.jp

2. PHONETICS ACTIVITIES
Since phonetics is a core area of linguistics, it makes strong connections to other areas such as
phonology, morphology, semantics, syntax and pragmatics. It also has many practical applications indiverse areas in human life (7). In general phonetics, students are trained to do phonetic transcription,
rule-writing, spectrography, data analysis, and building analytical skills (8). In addition, there are
performance exercises, hands-on lab activities using Praat include plotting vowels, measuring VOT,
pitch range, speech rate, and pairwise variability indices (9). In order to understand articulatoryphonetics, it is also important to know the anatomy of the vocal tract as in making a clay tongue or apaper larynx (10). Also, instruments such as ultrasound, EPG and EMMA help visualize thearticulation of sounds. A website showing vocal-tract models (11) helps students to understand therelationship between vocal-tract configuration and sound quality, and more abstract concepts such as
the source-filter theory.
L2 pronunciation has a strong teaching-research connection, as shown in the growing number ofconferences, journals, books, researchers working in the area (12), as well as new techniques and tools
used in language classrooms (13, 14). Integration of multisensory-based pronunciation activities caneven be adopted flexibly into reading and writing (15) and learning vocabulary (16). Considering
students’ backgrounds, objectives, levels and age, instructors should customize activities to fit
students’ needs and observe outcomes in order to determine which tools and methods work for aparticular group.
In universities in Japan, first year students, who are rarely exposed to English in their real life,
come to take classes in English as a foreign language. One of the recurrent issues regarding
intelligibility is prosody, especially English rhythm. Another highly important area is pronunciationof challenging sounds that do not exist in Japanese such as /l/ and /ɹ/. Research questions we explore
in this paper are generally shown as: i) What instruction can facilitate better pronunciation targets in
English? ii) How do these instructions affect students’ perception of learning?

3. PROSODY – KINESTHETIC OR PRAAT
Studies have found that enhanced articulatory gestures are often synchronized with prosodic
boundaries (17). It is beneficial for learners to know the link between speech, articulatory gesturesand body gestures, as they can help learners to remember stress positions, and self-regulate the timingof prominent words (18, 19, 20, 21). In doing so, learners develop perceptive and productive skills
using auditory, tactile, kinesthetic, and proprioceptive senses (see 22 for other methods).
This section overviews the implementation of phonetic activities in two university classes in Japan.
Their L1 is Japanese, and the target language is English, both were taught by Japanese instructors inJapanese. Although one is a compulsory communication class (online), and another is a second
language acquisition class (face to face). One of the shared teaching modules is English rhythm. There,
we exposed students to a basic knowledge of English and Japanese rhythm by showing how con tent
and function words, in conjunction with stressed and unstressed syllables work together to create
English rhythm.
Two instructors executed different pedagogical reinforcements to fit the objective and level of theclass. One communication class executed kinesthetic instructions, while the second language
acquisition class executed Praat instructions. In kinesthetic instructions, students watched the
instructional video (Figure 1) where a speaker is air-boxing on every stressed syllable. Students readthe subtitles of the story and imitated the boxing gestures at the same time. Another class used Prattfor self-directed study; students recorded their own speech, measured the duration and F0 of stressed
and unstressed syllables, and compared the result with native speaker models. Intervention activities
of about 10 minutes were implemented for 5 weeks following this initial exposure.
Pretest and posttest oral reading were recorded, and the measurements were done for duration and
F0 between stressed and unstressed syllables in noun phrases and verb phrases. The acoustic results
and students’ reflections will also be reported.

Figure 1 - Air-boxing video for pronunciation practice showing punch gesture aligning withstressed syllables “bought” and “pud-”

4. SEGMENTAL – ULTRASOUND OR PRAAT
Ultrasound technology is an easy-to-use tool to visualize tongue movements of speech (23) to givelearners visual feedback. Studies have found that articulatory training using ultrasound visualfeedback is beneficial for learners acquiring /l/ and /ɹ/ (23, 24, 25). But the training in ultrasoundresearch has mostly occurred in English speaking countries where they were already somewhat
immersed in English and therefore can get continuous phonetic input. It should be worthwhile to test
domestic freshman students who have not been abroad for more than one month.
This section overviews the implementation of phonetic activities in two university classes in Japan.
Their L1 is Japanese, and the target language is English, and they were taught by one Japanese
instructor. Although one is a compulsory communication class (online), and another is English
pronunciation class (face to face). One of the shared teaching modules is con trast of /l/ and /ɹ/ ofEnglish. Thus we taught a basic knowledge of English and Japanese liquid consonants, in terms of
constriction locations, tongue bracing and lip rounding. Both students watched the eNunciate website(26, 27, 28), which shows clickable IPA charts with overlaid ultrasound videos, where a midsagittal
tongue movie is superimposed on a face.
Different pedagogical reinforcements were executed to fit the objective and level of the class. One
communication class received Praat instructions, while the English pronunciation class received
ultrasound instructions. In Praat instruction, each student measured F3 of their own English andJapanese liquid consonants and compared it with a native speaker model. Students of ultrasound cometo a classroom individually, to produce liquid consonants and receive feedback (Figure 2). An
intervention activity of about 15-20 minutes was implemented just within a lesson. The acoustic results
and their reflections will be reported.

Figure 2 –Praat instructions to each individual in classroom. A student puts an ultrasound probe
under the chin on his own.

5. Final Remarks
We have shown some case study of phonetics integration in language related classes. Students ingeneral liked the flipped style and feel they improved the pronunciation.

ACKNOWLEDGEMENTS
This work was supported by Grants-in-Aid for Scientific Research (KAKENHI-C, 22K00621) ofJapan Society for the Promotion of Science.

REFERENCES
1. Bergmann J, Sams A. Flip your classroom: Reach every student in every class every day. Internationalsociety for technology in education; 2012 Jun 21.
2. Academy of Active Learning Arts and Sciences, 2018. https://aalasinternational.org/
3. Lee G, Wallace A. Flipped learning in the English as a foreign language classroom: Outcomes andperceptions. TESOL quarterly. 2018 Mar; 52(1):62-84.
4. Bauer‐Ramazani C, Graney JM, Marshall HW, Sabieh C. Flipped learning in TESOL: Definitions,approaches, and implementation. Tesol Journal. 2016 Jun; 7(2):429-37.
5. Brewer R, Movahedazarhouligh S. Successful stories and conflicts: A literature review on theeffectiveness of flipped learning in higher education. Journal of Computer Assisted Learning. 2018 Aug;34(4):409-16.
6. Akçayır G, Akçayır M. The flipped classroom: A review of its advantages and challenges. Computers &Education. 2018 Nov 1; 126:334-45.
7. Munro, M. Applying Phonetics: Speech Science in Everyday Life. Wiley-Blackwell; 2020 Nov.
8. Ashby P, Comrie B, Corbett G. Understanding Phonetics. Routledge; 2013 Oct 8.
9. Ladefoged P, Johnson K. A course in phonetics. Cengage learning; 2014 Jan 3.
10. Gick B, Wilson I, Derrick D. Articulatory phonetics. John Wiley & Sons; 2013 Jan 22.
11. Arai T. Vocal-tract models and their applications in education for intuitive understanding of speech
production. Acoustical Science and Technology. 2016 Jul 1; 37(4):148-56.
12. Levis JM, Wu A. Pronunciation--Research Into Practice and Practice Into Research. CATESOL Journal.
2018 Jan 1; 30(1).
13. O’Brien MG, Derwing TM, Cucchiarini C, Hardison DM, Mixdorff H, Thomson RI, Strik H, Levis JM,
Munro MJ, Foote JA, Levis GM. Directions for the future of technology in pronunciation research andteaching. Journal of Second Language Pronunciation. 2018 Dec 31; 4(2):182-207.
14. Levis JM, Derwing TM, Sonsaat-Hegelheimer S, editors. Second Language Pronunciation: Bridging the
Gap Between Research and Teaching. John Wiley & Sons; 2022 Mar 29.
15. Levis JM, Echelberger A. Integrating Pronunciation into Language Instruction. Second Language
Pronunciation: Bridging the Gap Between Research and Teaching. 2022 Mar 29:19.
16. Johnson KA, Mellesmoen GM, Lo RY, Gick B. Prior Pronunciation Knowledge Bootstraps Word
Learning. Frontiers in communication. 2018:1.
17. Cho T. Prosodic boundary strengthening in the phonetics–prosody interface. Language and Linguistics
Compass. 2016 Mar; 10(3):120-41.
18. Acton W, Baker AA, Burri M, Teaman B. Preliminaries to haptic-integrated pronunciation instruction. In
Proceedings of the 4th Pronunciation in Second Language Learning and Teaching Conference 2013 (pp.
234-244).
19. Burri M, Baker A. “I never imagined” pronunciation as “such an interesting thing”: Student teacher
perception of innovative practices. International Journal of Applied Linguistics. 2019 Mar; 29(1):95-108.
20. Yamane N, Shinya M, Teaman B, Ogawa M, Akahoshi S. Mirroring beat gestures: Effects on EFL learners.
In Proceedings of the 19th International Congress of Phonetic Sciences 2019 (pp. 3523-3527).
21. Chan MJ. Embodied Pronunciation Learning: Research and Practice. Catesol Journal. 2018; 30(1):47-68.
22. Carrera-Sabaté J. Body to Speech and Back: Considerations on Embodied Pronunciation. In Proc. 3rd
International Symposium on Applied Phonetics (ISAPh 2021) 2021 (pp. 1-8).
23. Gick B, Bernhardt B, Bacsfalvi P, Wilson I, Zampini M. Ultrasound imaging applications in second
language acquisition. Phonology and second language acquisition. 2008 Mar 5; 36(6):309-22.
24. Tsui HM. Ultrasound speech training for Japanese adults learning English as a second language (Doctoral
dissertation, University of British Columbia).
25. Tateishi M, Winters S. Does ultrasound training lead to improved perception of a non-native soundcontrast? Evidence from Japanese learners of English. In Annual Conference of the Canadian Linguistic
Association, Victoria, BC 2013 Jun 1.
26. Abel J, Allen B, Burton S, Kazama M, Kim B, Noguchi M, Tsuda A, Yamane N, Gick B. Ultrasoundenhanced multimodal approaches to pronunciation teaching and learning. Canadian Acoustics. 2015;
43(3):124-5.
27. Bliss H, Johnson K, Burton S, Yamane N, Gick B. Using multimedia resources to integrate ultrasoundvisualization for pronunciation instruction into postsecondary language classes. J. Linguist. Lang. Teach.
2017 Dec 30; 8(2):173-88.
28. Bird S, Gick B. Ultrasound biofeedback in pronunciation teaching and learning. In ISAPh 2018
International Symposium on Applied Phonetics 2018 Sep (No. September, pp. 5-11).

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0266

The International Year of Sound 2020-2021:
the outcomes of the student competition
Sergio LUZZI1; Chiara BARTALUCCI1; Sara DELLE MACCHIE1; Paola PULELLA1
1

Vie en.ro.se Ingegneria s.r.l., Florence

ABSTRACTThe International Year of Sound 2020, which was extended to 2021, is an initiative promoted to highlight
how acoustic is a key component in the balance and comfort of all human beings and in their relationshipwith the world in all its aspects. Several initiatives, including the International Student Competition, wereorganized in order to involve young people in the themes of acoustics. The competition for students wascalled "My world of sounds" and it was linked to the melody "The sound of the world", that was especiallywritten for celebrating the IYS.
The Competition was structured in two categories which involved primary and secondary school students
who were asked to produce a graphic artwork and compose a stanza, respectively. The winning entries fromall over the world were awarded.
Approximately 15.000 students from 16 countries were actively involved and approximately 700 items weresent for the competition, including drawings and stanzas for the song "The sound of the world". Moreover,videos and audio tracks, sent to the Competition Coordinator Office, were assembled in two movies
summarizing the story of the International Year of Sound, focusing on the Student Competition and its
products.
Keywords: IYS, Sounds, Students’ competition, awareness

1. INTRODUCTION
The International Year of Sound (IYS 2020-2021) is an international initiative to emphasize how
the acoustic environment is a key component in the balance and comfort of the daily lives of all humanbeings, in their relationship with each other and with the world, in economic, environmental, social,
medical, industrial, and cultural dimensions. The IYS 2020-2021 is organized by the International
Commission of Acoustics (ICA) and implements the directions of the UNESCO Charter of Sound on
the importance of sound in today's world (ref. Resolution 39 C/59, Sept. 25, 2017).
Several activities have been organized as part of IYS 2020-2021, including the International
Student Competition on the theme "My world of sounds." The contest is closely related to the motto"The importance of sound for the society and the World," the melody "The sound of the world" andits refrain, the musical basis of which has been made available in the section of the IYS 2020 -2021
website dedicated to the Contest [1]. Due to the delicate health situation imposed by the Covid -19
pandemic, the International Year of Sound, which began in 2020, was extended to 2021.
From the publication of the contest rules to the announcement of the winners (see Fig. 1 for the
event’s poster), the Office in charge of the Contest, consisting of the authors of this memoir, provided
for its dissemination through posts on dedicated social networks (Fig. 2), production of commercials,
as well as keeping contacts with National Representatives alive as much as possible and periodically
updating the IYS 2020-2021 steering committee and the ICA.

1sergio.luzzi@vienrose.itFigure 1 – Poster of the initiative.

Figure 2 – Sample post for publicizing the initiative.
The Competition was structured in two categories:- Competition for primary and secondary school students who are asked to produce a graphic entry(drawing, painting, collage, computer art, comic strip, etc.....) related to their world of sound, inspiredby the motto, melody and refrain of the song "The sound of the world."- Competition for secondary school students who are asked to compose a four-verse stanza in theirown language and/or English, inspired by the melody and refrain of the song "The sound of the world,"
as well as by the motto.

2. STUDENTS’ PARTICIPATION AND EVALUATION METHOD
For the first category of the contest (primary and secondary schools) there were 39 drawings
nominated as finalists, with more than 650 submissions received from National Representatives.
For the second category of the competition (secondary schools) there were 33 finalist stanzas, withmore than 50 contributions received from National Representatives.
Overall, about 15,000 students worldwide were reached by the Contest initiative, with the
involvement of about 80 schools distributed in 16 countries, compared with contacts activated with
National Representatives from 39 countries (Fig. 3).

Figure 3 – Countries from which students participated in the Competition (Egypt, Argentina,
Chile, USA, Hong Kong, Thailand, Turkey, Croatia, Greece, Hungary, Italy, Portugal, Russia,
Serbia, Slovenia, Spain).
Finally, as additional contributions, 23 videos and audio recordings were sent, which were later
used to prepare the final videos.
Detailed information about the competition rules is available at the IYS 2020 dedicated page [1]and in [2].
The selection of the winning entries was made through the evaluation of two juries: the expert jury,
consisting of the members of the IYS 2020-2021 steering committee, and the popular jury. As for themembers of the expert jury, they were asked to express their evaluation of the entries received forboth categories by means of a special questionnaire prepared on Google forms and according to thefollowing criteria:a) Is the entry relevant to the theme of the Contest?
b) Is the elaborate original?
c) Does the entry inspire any emotion, feeling, awareness?
Each experienced member of the jury cast a vote for each submitted entry, excluding entries
produced by participants from their own country of origin.
Then the average score of the preferences expressed by each expert was calculated by answering
the three questions (each of which had a score from 1 to 5 associated with it).
As for the popular jury, preferences were expressed by placing likes on the anonymous entries
uploaded to the albums dedicated to them on the IYS Facebook and LinkedIn pages and having each
person correspond to one vote (Fig. 5).

Figure 5 – Publications of materials on IYS Facebook and LinkedIn pages.
For the purpose of overall evaluation, for each category of the Contest and for each jury, the entries
were ordered from the one that had received the most votes to the one that had received the least (39thfor drawings, 33rd for stanzas). At this point, each entry in each category was assigned a decreasing descending sequential score from 39th (33rd) to 1st and weighting of 70% for the experts and 30%
for the popular jury was applied.
The final scores are calculated as a weighted combination of the experts' and popular jury's scores,and on the base of these, the steering committee decided the winners and awarded prizes and special
mentions.

3. RESULTS
As for Category 1, first place went to Çekmeköy Final Middle School, Instanbul-Turkey, second
to Colegio Melipilla, María Pinto-Chile-Chile, and third to Ching Chung Hau Po Woom Primary
School, Tung Chung-Hong Kong.
As for category 2, IC "Puccini," Florence-Italy and Institut Escola Salvador Vilarrasa, BesalúSpain were tied for first place and I.E.S. La Merced, Valladolid -Spain was third.
The winning materials are shown in the Fig. 6 and Fig. 7.

b)

a)

c)
Figure 6 –Winner materials for the first competition category (a = first prize, b = second prize, c= third prize)

a)

b)

c)

Figure 7 –Winner materials for the first competition category (a, b = first prize, c = third prize)

In addition to the official winners, special mentions were also recognized to some participants.
Specifically, for Category 1 a special mention was given to a Croatian school for the interactive
material provided with the drawing, i.e., an App that transforms colors into sounds (Fig. 8); while forCategory 2 special mention was given to all participants who, in addition to the verse, also provided
a video or audio recording as well as to the Spanish school CCEE Padre Apolinar for the emotional
video submitted (not in competition for the Contest).

Figure 8 – Special mention assigned to Lotrščak primary school – Croatia

4. CONCLUSIONSThe International Competition aimed at primary and secondary school students and, more generally,
the International Year of Sound 2020-2021, is an initiative that, despite the spread and continuationof the Covid-19 epidemic, with all the problems this has entailed in educa tional programming inschools around the world, has made it possible to bring about 15,000 children worldwide closer toissues related to the importance of sounds.
As a result of the collection by the Office in charge of the Competition of the numerous
contributions, such as drawings and stanzas required for participation in the competition, andadditional materials, such as audio and video recordings, it was possible to develop two summary
films. In fact, a short film and a longer one summarize the history of the International Year of Sound,its global initiatives, and focus particularly on the Competition aimed at students.
The short film, with a duration between 2 and 3 minutes, constitutes the video clip of the song andassembles the award-winning contributions (drawings and stanzas) and some of the extra contributions.
The longer movie, with a duration of 12 to 15 minutes, assembles many more drawings, videos,etc. submitted by the participants along with short interviews with the organizers, shots of the artists,
statistics, etc.
The dissemination of the short film started during the National Conference of the AcousticalSociety of Italy in May 2022 and is on going through the official channels of the ICA as well as the
Office in charge of the Competition. The longer movie will be officially presented at the 24th
International Congress on Acoustics.

ACKNOWLEDGEMENTS
The International Year of Sound and the Student Competition are supported by the Head -GenuitFoundation to whom the authors address their gratitude on behalf of the IYS Steering Committee andof the Competition Central Office.

REFERENCES
1. https://sound2020.org/society/student-competition/
2. Luzzi S, Bartalucci C, Delle Macchie S, Natale R, Pulella P, The international Year of Sound initiatives forworldwide awareness about the importance of sound. Akustika. 2021; 39: 135-140.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0654

Online acoustic recording education using MARRSHyunkook LEE1; Nikita GODDARD
1

Applied Psychoacoustics Lab (APL), University of Huddersfield, HD1 3DH, Huddersfield, United Kingdom

ABSTRACTThis extended abstract first discusses the importance of teaching and learning spatial audio psychoacoustic
principles in acoustic sound recording education. Then it introduces an interactive Web Audio application
named MARRS for Web, which has been used for the online teaching of a spatial recording module at the
University of Huddersfield during the pandemic period. MARRS (Microphone Array Recording andReproduction Simulator) provides an interactive visualization of the perceived image position of soundsource as the user changes the position and configuration of microphones. The Web version of MARRS also
provides a stereo and binaural playback of a virtual quartet with the user’s chosen sound files and room
impulse responses. The tool has been found highly useful not only for teachers to explain theoreticalprinciples of stereophonic imaging in stereo microphone array recording, but also for students to have a
critical listening training with respect to spatial audio attributes and ultimately design their own microphone
configuration for desired spatial imaging characteristics.
Keywords: Sound Recording, Microphone Technique, MARRS, Online teaching

1. Challenges in Sound Recording Education
Spatial acoustic recording technique using microphone arrays (i.e., stereo to multichannel) is an
important curriculum in sound recording education. There are a number of variables to consider inorder to achieve desired spatial characteristics, e.g., spacing and subtended angle between
microphones, microphone polar pattern, microphones’ height from the floor and distance from thesound sources, etc. Furthermore, optimal settings of such variables would depend on contextual
factors such as the type of musical performance and the acoustic conditions of the recording venue.
Thus, making right decisions in terms of microphone array configuration and placement would require
not only practical experiences and musical understanding but also theoretical knowledge in spatial
audio psychoacoustics.
It is observed that recording engineers often make stereo recordings using famous microphone
configurations such as spaced omnis with 50cm spacing (AB), coincident XY with 90° subtended
angle, ORTF, NOS, Decca Tree, etc. Although such “preset” configurations can be a good starting
point, it is important to understand that all of them were devised in specific contexts and aims, andthey could not be applied in the same ways in all recording situations. Stereo imaging offered by aspecific configuration would primarily depend on the source-to-microphone distance, which should
be determined based on the desired spatial perspective (e.g., conductor or audience) and the acoustic
characteristics of the recording venue (e.g., reverberation time, early reflection energy, etc.).
Especially in education, it is crucial for students to learn fundamental principles behind different
microphone techniques and a systematic process to realize their artistic goal more effectively andefficiently. Blindly using a popular configuration without understanding might lead them to rely onluck for the outcome. Customizing or designing a microphone array to achieve a desired goal byrandom trials and errors might cost a significant amount of time, but the result may not be reproducible
in other conditions. However, best practices built upon systematic approaches based on principles
would make them more versatile engineers who can adopt to various recording scenarios flexibly.

2. Online Teaching using MARRS for Web
Putting theory into practice in sound recording obviously requires students to go through practical1h.lee@hud.ac.uk

experimentations with musical ensembles in an acoustic space using microphones. However, thiswould have been made difficult or impossible due to the COVID-19 restrictions. To overcome thisand carry on with teaching spatial recording techniques in the undergraduate Music Technology course
at the University of Huddersfield during two academic terms in 2020 and 2021, we updated andutilized the MARRS for Web (https://marrsweb.hud.ac.uk/), which is the web application of the
MARRS (Microphone Array Recording and Reproduction Simulator), for online teaching of thespatial recording module. MARRS was originally developed as iOS and Android apps (1), and theinitial Web version of the tool was introduced in (2), where technical details of the psychoacoustic
algorithm used and web audio implementation can be found.
MARRS for Web is an interactive tool that provides both audio (stereo and binaural) and visualpresentations of perceived sound image positions as well as the calculations of interchannel time andlevel differences in real-time depending on the user’s microphone array settings. The tool was used
not only for teaching staff to demonstrate the psychoacoustic principles of various microphone
techniques, but also for the students to virtually experimenting with microphone settings andeventually design their own configurations. MARRS for Web includes the following features.
• The user can change the horizontal and vertical positions of the sound sources and themicrophone array, in the top and side stage views of a virtual quartet.
• Microphone configuration is fully customizable (polar pattern, spacing, angle, height, verticaltilting (pitch), distance).
• The perceived image positions of the sources are predicted depending on the user microphonesetting and interactively displayed in the speaker view.
• Stereo and binaural outputs for auditioning: anechoic string quartet recordings are provided bydefault, but the user can also load other wave files. Each source can be muted or soloed.
• Convolution reverb with a level control is provided for simulating an acoustic space. A set ofstereo room impulse responses (RIRs) are provided, but the user can also load their own RIRs.
• The graph mode displays interchannel time and level differences and localization curve for agiven user microphone configuration.

Figure 1. The Stage and Graph modes in MARRS for Web https://marrsweb.hud.ac.uk/
MARRS for Web was found to be highly useful for students to virtually make recordings using
simulated microphone techniques during the pandemic period. In the post-pandemic era too, it willcontinue to serve as a tool that students can use to consolidate their understanding about underlying
psychoacoustic principles of stereophonic recording in an interactive manner through visualization
and auditioning. Students will go through a series of controlled experimental recording sessions witha virtual musical ensemble in a concert hall, where they try various microphone settings based on thesimulations provided by MARRS for Web, and critically listen to the recordings to discuss pros andcons of each approach. Using such an assistive tool would allow for more effective and efficient
problem solving in sound recording. However, it is strongly encouraged for students to learn thetheoretical principles first rather than blindly relying on the result that the simulation provides.

REFERENCES
1. Lee H, Johnson D, Mironovs M, An interactive and intelligent tool for microphone array design, Proc
143rd AES Int Conv, Oct 2017. e-Brief 390.
2. Goddard N, Lee H, A web-based tool for microphone array design and phantom image prediction using
the web audio API, Proc 146th AES Int Conv, Mar 2019, e-Brief 499.

ABS-0900

Examples and issues in remote class on recording technique
Toru KAMEKAWA(1)(1) Tokyo University of the Arts, Japan, kamekawa@ms.geidai.ac.jp

ABSTRACTIn May — a month behind the usual schedule — the first semester of the 2020 academic year commenced as aremote learning endeavor to prevent the spread of coronavirus disease. In this paper, I report examples of remotelearning on recording and their problems. Tokyo University of the Arts adopted Google Classroom as a learning
management system — a necessary framework for remote learning. It provides the faculty and students with
instructions on using the classroom to prepare for the first semester. We also shared information on conducting
real-time remote lectures with the help of web conferencing systems (Google Meet, Zoom, etc.). In the recordingclass, it was necessary to send sound samples with high quality, so I tried various sound transmission software
but found that it depended greatly on the Internet environment of the students on the receiving end. Therefore,
it was necessary to use both online and on-demand. Finally, it became clear what was not being communicatedonline.

Keywords: remote learning, web conferencing systems, Network communication speed

1

INTRODUCTION

The first semester of the 2020 academic year commenced as a remote learning endeavor to prevent the spreadof Covid 19. My classes, including the recording practice, will be handled online. Although we had previously
used teleconferencing systems such as Skype, none of us had any experience conducting classes using such
online methods. Since Tokyo University of the Arts adopted Google Classroom as a learning management
system, our first choice for web conferencing systems was Google Meet. However, when it came time to teachthe recording class, Google Meet could not handle stereo audio, so it was necessary to look for various otherapplications. Around April 2020, Zoom was the only web conferencing system capable of handling stereo audio.
However, the security vulnerability of Zoom was a problem then, and the university decided not to use Zoom
as an official tool. After a preparation period of about one month starting in April, the distribution of classmaterials by Google Classroom and remote classes using Google meet began on May 11, 2020.

2

QUESTIONNAIRE FOR ALL STUDENTS

To determine the policy for remote teaching, a questionnaire was sent to all students in the university[1]. Including the first questionnaire in April, when all classes became fully remote, the following three questionnaires
were held until July 2020, when the class policy for the second semester was decided.
• 1st time: April 17th to 24th, 2020, percentage of responses received= 54% (1863 persons)• 2nd time: June 2nd to 12th, 2020, percentage of responses received= 24% (825 persons)• 3rd time: July 11th to 13th, 2020, percentage of responses received= 12% (418 persons)
The survey asked various questions, some of which are presented below. Figure 1 shows the speed of data
download on the internet for students. The questionnaire results indicate that 5 to 20 Mbps transfer rates are themost common. Figure 2 shows the students’ responses to what they perceived as problems in remote classes.
This response also indicates problems with the students’ Internet environment. Regarding the satisfaction withthe classes so far and their requests for the next semester, it can be seen that many students were acceptablewith remote lectures but preferred face-to-face practice classes (Figure 3. Nevertheless, it was necessary toconsider how to conduct a recording practice class when it is difficult to achieve a face-to-face class.

s〜
10

0M
bp

80

〜
10

0M
bp

s

s
M
bp

s
60

〜
80

M
bp

s
〜
60
40

〜
40
20

5〜
20

M
bp

s
1〜
5M
bp

M
bp

s
5〜
1M
bp

0.

0k
bp
〜
50

s

500
450
400
350
300
250
200
150
100
50
0
s

Number of answers

the speed of data download of the internet.
(from the results of the 1st questionnaire)

Internet connection speed

Reasons why you could not successfully take a
distance class

Figure 1. The speed of data download of the internet. (from the results of the 1st questionnaire)(From the results of the 2nd questionnaire)

40

Number of answers

35

Difficulty in participating in class
授業に参加しづらい

30

授業の情報が伝わってこない
Class information is notcommunicated.

25
20
15

家の中で声を出したり映像を出し
There is no place in the housefor voices or images.
たりできる場所が無い

10

ネットワークが脆弱
Network is vulnerable

5

機材が対応できていない（カメラ
Equipment such as cameras andmicrophones cannot handle
やマイクが無い）

01-year
1年

2-year
2年

3-year
3年

4-year4年〜
more

Graduate
⼤学院
students

Figure 2. Reasons why students could not successfully take the remote class (from the results of the 2ndquestionnaire)Lectures（on demand）

Practices (remote)dissatisfied
More or lessdissatisfied

Requests for the next semester

satisfied
Remote only
satisfieddissatisfied

Can't sayeither way

More or lesssatisfied
Can't sayeither way

More or lesssatisfied

On demandand remote

More or lessdissatisfied

Comprehensive
In-person

In-personand remote

In-person &real time remote

Figure 3. Evaluation of the classes so far and requests for the next semester (from the results of the 3rdquestionnaire)

3

TRIAL OF THE RECORDING CLASS ON REMOTE

3.1 Problems when using a web conferencing system for the practice classes
The following points need to be considered when conducting the recording class on a remote.
• Is stereo sound available?
• Can dynamic level compressor, and feedback cancellation be deactivated?
• Is there a bottleneck in the student’s internet environment?
• It is unclear what listening environment the students are monitoring with.
As mentioned above, the web conferencing system adopted by the university, Google Meet, did not allow
stereo audio, so various attempts were made to clear up the above problem.
3.2 Stereo playback using on-demand system
To ensure that the stereo audio materials were heard, I uploaded the materials to Google Drive and asked
students to download them. In this case, the most problematic issue is the students’ listening environment. Whenlistening in a typical classroom setting, it is difficult for everyone in the class to experience an accurate stereo
image because only a limited number of students can hear in the middle of the left and right loudspeakers.
Each person can listen in the optimal stereo listening position through Google Drive or YouTube. On the other
hand, many students often listen with headphones or earphones; in the worst-case scenario, they are frequently
monitored by small speakers on their computers. This is very problematic when evaluating sound quality.
In addition, special care should be taken when using content that requires copyright processing on Google
Drive or YouTube. The treatment of copyrighted works in university classes is based on paying a lump sum
for the entire university when used on the Internet. However, these must be handled separately when they areviewed by the general public other than students for purposes other than classes. Therefore, it is also important
to manage security so only students can view the video for purposes other than classes.
3.3 Attempts to use apps for high-quality sound transmission
Audiomover’s Listento was adopted as a way to share high-quality sound in real-time with the class[2]. The
advantage of Listento is that the transmission rate can be set to match the speed of the internet. You can use
videoconferencing systems such as Meet to have a conversation while listening to the sounds you want to hearin high sound quality with Listento. Although there is a fee to use Listento, a license is only required for thesender, and the receiver can listen for free through a web browser such as Chrome, which is also appreciated.
Since Listento can be used as a plug-in for a DAW, it was possible to share the screen of the DAW used forrecordings, such as Reaper or Protools, with Meet and play the sound in stereo. Since the Meet and Listento
were transmitted over separate systems, their timing varied depending on the students’ viewing environment, so
strict synchronization was impossible. However, it was much better than having the students listen to the monoand compressed sounds of the Meet.
Yamaha’s SYNCROOM[3] is also an effective tool when the interactive, high-quality sound exchange isrequired, as in the case of viewing student-created materials. Originally a tool for remote performance, it canbe used with high sound quality and low latency, making it a good choice for viewing recordings. However,
because of the need to install a dedicated application and the need for all participants to create an account, it
is good for small classes but not suitable for classes with a certain number of participants.
3.4 Trial of remote sound production for 3D audio
I studied a method of conducting stereophonic sound production online, which had been conducted in the pastin face-to-face classes. Although 3D audio production tools have been developed, which have become popular
in recent years, it is difficult for students to use the production environment in a home as it is at the studio.
Cockos’ Reaper[4] has a full range of Ambisonics plug-ins, and multi-channel loudspeaker playback can beconverted to binaural for listening on headphones. Plug-ins produced by IEM (Institute of Electronic Musicand Acoustics)[5] and SPARTA[6] developed by AALT University can be used for free-of-charge and are also
a significant attraction.
3.5 Attempts through hybrid classes
As the covid-19 infection situation has settled down and vaccination has reached a large number of population,
the university has resumed face-to-face classes, mainly practical classes such as instrument lessons. In therecording class I teach, we have gradually resumed face-to-face classes using the university studio; however,since not a few students were concerned about infection, we tried to conduct hybrid classes that include bothface-to-face and online classes. Figure 4 shows the systematic image of hybrid classes. Remote teaching tends
to be one-way. This is partly due to the students’ lack of connection speed and partly since many turn off theircameras due to circumstances in their home environment. Therefore, It was fascinating for me to be able togive a face-to-face class for the first time in a while. The reason for this was that I could feel the reactions of
the students. However, on the other side, combining remote teaching with face-to-face meetings is very difficult.
Inevitably, priority was given to the students’ reactions in front of me, and my attention to the students beyond
the remote screen was neglected.

Trial of Hybrid classLAN

Switcher for video

PC for distribution

PC for presentation

Projector loudspeakers

Mixing console
microphone

CameraFigure 4. Systematic image of hybrid classes

4

CONCLUSIONSIn this paper, I introduced the recording class initiatives at Tokyo University of the Arts in the Covid19 situation.
For practical classes dealing with sound, online implementation using web conferencing systems such as Zoom
or Meet is still not a realistic environment regarding sound quality and latency. For sound quality, on-demandusing YouTube or Google Drive is better. When dealing with DAWs, Audiomover’s Listento is another option.
When dealing with 3D audio, binarization using Ambisonics plug-ins such as the IEM plug-in or SPARTA isalso effective.
The recorded classes, which had been conducted as a matter of course, will now be operated remotely, andthe importance of working them face-to-face was reaffirmed. In a class dealing with sound, I was reminded of"how important it is to listen to the same sound simultaneously and in the same space."

REFERENCES[1] Kamekawa, T. “Current Status and Issues of Distance Learning in the Faculty of Music,” Bulletin, theFaculty of Music of Tokyo University of the Art, No.46, 2022 (available in Japanese).
[2] Auduiomover, Listento, https://audiomovers.com/wp/ (Viewed August 15, 2022)[3] Yamaha, syncroom, https://syncroom.yamaha.com/ (Viewed August 15, 2022)[4] https://www.reaper.fm/ (Viewed August 15, 2022)[5] IEM plug-in, https://plugins.iem.at/ (Viewed August 15, 2022)[6] SPARTA, https://leomccormack.github.io/sparta-site/ (Viewed August 15, 2022)

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0975

Development and management of training system for Online
Technical Listening Training
Kazuhiko KAWAHARA 1; Masayuki TAKADA 2; Ken’ichi MORIO3, Ken FUJIWARA3

3

1

Kyushu University, Japan

2

Kyushu University, Japan

Nihon Onkyo Engineering Co. Ltd., Japan

ABSTRACTThe COVID-19 pandemic prevents us from enjoying live in-person classes. Authors developed online
technical listening training (ear training) system using zoom and audiomovers’ listento plugin on the protools.
Also, the Authors employed the Shinji-online service. The Aim and concepts of the system development aredescribed in this paper. The authors did not have enough time to develop the new online technical listening
training system. Then authors provide the solution by combining the existing services. The quality of sounddelivery was good enough for our ear training. But students required some time to get used to it.
Keywords: Ear Training, Technical Listening Training, Online Education

1. INTRODUCTION

The social context of the coronavirus outbreak in 2020 (the so-called "coronavirus disaster") has
forced many educational institutions to respond to online classes. The same is likely to be true formany universities. At Kyushu University, only online classes will be offered in the spring semester ofthe 2020 academic year. Against this social background, the Department of Acoustic Design in the
Faculty of Art and Design, Kyushu University, has started using an online conferencing service forthe second year of Technical Listening Training II [1], which is an education in sound sensitivity and
it is a kind of “Critical Listening.” [2] The class was taught using an online conferencing service.
Kyushu University has established a policy that only online classes will be offered during thespring semester of Academic year of 2020 (May 7 to June 24, 2020).
In addition, in view of the social situation of the spread of coronavirus infection, during the summer
term (June 25 to August 11, 2020), some in-person classes, such as experiments and practical training,were conducted on the premise that adequate measures could be taken to prevent infection.
At Kyushu University, students are required to have a PC at the time of admission to undergraduate
school, and each department and course purchases a PC with specifications specified by thedepartment or course. Although students are supposed to have a PC at hand, we did not investigate
the network environment at home.
In the Department of Acoustic Design, Apple’s Macintoshes are recommended mainly due toconstraints in computer programming classes. The Department of Acoustic Design recommends
Apple's Macintosh for online classes. However, since we did not expect to receive online course
materials, we set up the recommended headphones and earphones. This was one of the obstacles in
the implementation of the distance classes for Technical Listening Training II.
This paper shows the report the trials of the distance learning classes of Technical Listening
Training on the year of 2020 and 2021.

2. SYSTEM DESIGN

Figure. 1 shows the system configuration for our training. The system is a combination of Shinji

12kawahara@design.kyushu-u.ac.jp
takada@design.kyushu-u.ac.jp

3 kenichi_morio@hibino.co.jp

Online, Zoom, and Audiomovers' Listento. Lectures related to the training were given from a PC for
lectures via Zoom. Lecture materials were PowerPoint presentations, etc.
During Technical Listening Training, the output from the host PC of Shinji Online was delivered
using the Listento plug-in on the delivery PC. Participants accessed the Listento URL specified by the
moolde with a browser and listened to the training sound signals. It was sufficient for the browser tab
to be running in the background. The participants were asked to log in to the system via anotherbrowser tab or their smartphones, and to respond to the system's instructions from the system.

Figure 1 System configuration for online Technical Listening Training using Shinji Online,
Zoom and Aduiomovers’ Listento.

3. CONCLUSION

This report describes an attempt to conduct Technical Listening Training II online and its operation
due to the spread of coronavirus infection in 2020 and 2021.
Technical Listening Training II was taught online in 2020 and 2021. But it was able to be taught
in person in 2022.
It is important to note that we were able to acquire the know-how to conduct Technical Listening
Training remotely. For example, the possibility of conducting online Technical Listening Training forworking people was created.

REFERENCES

1. Shin-ichiro Iwamiya,Yoshitaka Nakajima,Kazuo Ueda, Kazuhiko Kawahara and Masayuki Takada,"Technical Listening Training: Improvement of sound sensitivity for acoustic engineers and sounddesigners,'' Acoustical Science and Technology, Vol.24，No.1，pp.27-31, 2003.01.
2. Jason Corey, ``Audio Production and Critical Listening - Technical Ear Training -,'' Focal Press (2010).

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0998

A Brief History of Noise Legislation
Part I: Roman Antiquity
Wim van KEULEN1; Milan van KEULEN212

VANKEULEN advies, the Netherlands

Radboud University Nijmegen, the Netherlands

ABSTRACT
Urban noise is a common problem for many modern cities. Yet, noise abatement in an urban environment is
not just a modern initiative. Several historic laws regarding the suppression of noise sources are also known
to us. This paper concerns the first part of the study with respect to the history of legislation: legislation in
the Roman Empire which is often considered to be the first in history.
Examining the Roman attitudes to noise pollution and the actions they have taken to combat noise tells a lot
about how (noise) pollution is perceived and treated by policy makers when. This paper shows that, despite
identifying sources of noise and acknowledging their negative health effects, reducing these sources proved
difficult as they persisted to be a problem throughout the Julio-Claudian dynasty’s reign.
In addition, this paper shows how those who made the laws to suppress noise did so in a way that benefited
themselves in the first place.
Keywords: Noise, Legislation, History

1

NOISE POLLUTION RECOGNIZED AS A LONG-TERM PROBLEM

The negative effects of noise pollution on health have been recognized since the 6th century BC[1]. In modern times, it has become increasingly targeted as both population and industry continue togrow exponentially with noise levels increasing accordingly.
In 2020, the European Environment Agency reported [2] that more than 100 million people inEurope were exposed to harmful levels of environmental noise. In Europe, long-term exposure toenvironmental noise contributes to 48,000 new cases of heart disease and 12,000 premature deaths
each year. Additionally, 22 million people suffer from chronic high annoyance, and 6.5 million sufferfrom chronic high sleep-disturbance.
Annoyance and sleep disturbance are responsible for most of the burden of disease associated withnoise. Road traffic noise particularly is a public health problem in many urban areas. Noise is clearlya contributing factor in a large-scale health epidemic and as such legislation is increasingly being
introduced and adapted to combat noise pollution.
However, this acknowledgement is not a new development. The effects of noise pollution, andespecially road traffic noise, on health have been recognized for very long.
Julius Caesar himself is often said to have laid the foundation for many modern laws regarding thegoverning of noise pollution as early as 49 BC. It is a convenient image connecting the beginnings ofmodern legislation with such a famous person. However, it was not just the actions of a single man,but a much more complex process that led to the enactment of his municipal law.

2

SOME ASPECTS OF ROMAN LAW MAKING

Roman law was both extensive and complicated sometimes even being described as “a living andadaptable set of legal and political norms” [3]. Jurists collected laws from long bygone days into newlawbooks. As such, it could give the impression that it is static, remaining absolute and unchanged for1
2

info@vankeulenadvies.nl
keulen@scarlet.nl

millennia.
Roman Law, however, was largely fluid and responsive to the changing needs of Rome and its
citizens. In addition to the continued presence of old legislation, there were several institutions andpolitical positions, such as the praetor, that had the power to enact new laws. This was carried out
when the old laws were deemed insufficient to resolve disputes or issues facing contemporary Romans.
It was only after the reign of Augustus that the position of Emperor had the power to makepronouncements in which their word literally became law by issuing edicts in response to claims madebefore them.
Although Julius Caesar was a powerful dictator, he was no Emperor and did not have the power toissue laws as he saw fit, but he did so in accordance with the late republican constitution: laws were
presented to the Senate and ratified by the people’s assembly.
The law took the name of the presiding consul(s). This was probably the case with the law now
often cited as the first noise-focused law in Western history. The law in question related to theregulation of urban traffic in Roman streets and was preserved in the Lex Iulia Municipalis, acollection of laws regarding the arrangements of the urban public space [4].
Such laws were often engraved on bronze tablets, which could serve as city charters in conquered
territories throughout the Empire. Thus, Caesar brought both Roman rule and law to the Mediterranean
and much of Europe.

3

ROMAN LAW ON NOISE

Caesar’s law regarding traffic is the fourteenth law on the Tablets of Herculaneum, themselves partof a broader and more diverse set of laws in effect throughout the Empire, and its translation reads asfollows [5]:After January 1 next, no one shall drive a wagon along the streets of Rome or along those streetsin the suburbs where there is continuous housing after sunrise or before the tenth hour of the day,except whatever will be proper for the transportation and the importation of material for building
temples of the immortal gods, or for public works, or for removing from the city rubbish from thosebuildings for whose demolition public contracts have been let. For these purposes permission shallbe granted by this law to specified persons to drive wagons for the reasons stated.
Although often regarded as the beginning of modern noise abatement legislation, the wording of
the law makes it apparent that it is not such a clear-cut case. Traffic was restricted to nighttime only,with certain exceptions allowed if their aim was the common good of the city, whether tangible suchas clearing rubble or more spiritual, such as rituals to ensure the favor of the gods. Also, long lastinglarge construction works had an exemption [6] proving that exemptions are not a modern invention.
While noise levels dropped during the day, they increased during the night! This led to many
Romans developing sleeping issues, which has been documented in the works of contemporary writers
like Juvenalis [7]. While these writers may not always have been objective in the issues they addressedor the severity they attributed to these issues, they are still indicative of how such issues wereperceived by the population.

4

ROMAN ATTITUDES TOWARDS NOISE

4.1 Traffic control versus Noise
Now the question arises: why did Caesar and his fellow magistrates enact this seemingly counterintuitive law restricting traffic to nighttime hours, thus negatively impacting the health of theircitizens? This paper proposes that the answer lies in how we perceive this law. Instead of seeing it asa law concerning noise, it must be seen as a law concerning specific traffic flows. The drawbacks ofincreased noise levels at nighttime were considered by Caesar and the other members of the Senate asa small price to pay for the benefit of improved urban traffic during the day. One possible explanation
for their reasoning lies in how the Roman view of a city center differs from our current perspective.
To the Romans, the center of a city was not just a convenient place for traffic flows and markets
as it is often today. Instead, it was seen as a symbol for the city’s prestige. A city center had to bemonumental, both in scope and decorations. It was the Roman belief that a city “needed to be equippedwith a monumental center just as much as laws, a constitution, local magistrates, and a local senate.
Any city worthy of its name had to have a built environment for the proper functioning of its political,
religious, and social life” [8].

Much of Roman life took place in the public spaces, meaning that during the day, many citizens
would walk the streets while they were going about their business. This was true even for the elite.
While their homes might have been far more prestigious than the average citizen’s houses and flats,
the elite moved in the same public space and used the same streets. Often, they rode in palanquins,seated literally above the crowd. For them, life on the streets was a way showing their power andparticipating in political exchanges, while around them the common people engaged in economic
exchanges [9]. As such, the streets of Rome were often filled with crowds during the day. Allowing
traffic in the form of wagons during these peak hours would undoubtedly lead to congestion.
4.2 Elite versus Common People
Such traffic jams could have been considered a blot on Rome’s idealized city center, in addition to
being a general nuisance. In addition, it would have negatively affected the mobility of the elite. It is
far easier to manipulate a crowd into breaking up than it is to turn a wagon around in a narrow street.
Thus, removing wagons entirely from daily traffic would improve the flow of the city population andallow the elite to go about their business freely. So great positives, but that does not diminish theserious negative effects that nighttime traffic had on the general population.
Why was the law still enacted? It is common believe that the reason was that Caesar could not bear
the noise of the wagons and had them completely removed from his environment. This desire for quiet
is not unique to Caesar. Primary literary sources tell us that tranquility was highly valued by the
Roman elite. Pliny the Younger mentions it in one of his letters about the advantages of Caesar’s new
mansion in the countryside: “This profound tranquillity and seclusion are occasioned by a passage
separating the wall of this room from that of the garden, and thus, by means of this intervening space,every noise is drowned” [10].
Due to the location and construction his house, the nighttime noise did not annoy Caesar, even if
it kept many fellow Romans awake. This explanation has some merit, as it is certainly true that the
Roman elite did not share the same sufferings as the normal population, largely due to the geographicaldistribution of their mansions.
This is especially evident in the way many Roman elites were not affected by the terrible floods of
the Tiber due to living in elevated areas. While many Roman citizens drowned and had to navigate
the city streets in boats, the elite sat high and dry [11]. When the Senate raised a plan to dam thesources of the Tiber the proposal was almost unanimously rejected because flooding was notrecognized as a problem severe enough problem to anger the god of the river, Tiberinus Pater [12]. Aswith the floods, excessive noise might have not been registered as a severe enough problem among
the elite compared to improving traffic flows in Rome.

5

CONCLUSIONS

Wagons should be heard not seen. They were only allowed to enter the city during the day if it was
clear that they served the common good. By otherwise removing them from the city streets during theday, the traffic flows of Rome’s population increased, as much of Roman life occurred on the streets.
Thus, the removal of wagons ensured that Romans could engage in political and economic exchanges
between all layers of society while moving together in the public space.
The resulting increase in noise at nighttime was not seen as a big enough issue to outweigh thebenefits mentioned, especially since these negative consequences affected the elite far less than the
average Roman citizen. Remaining largely unaffected themselves, the Roman elite, and thus its
policymakers, saw no reason to give up the advantage gained, even at the cost of the health of theircitizens.
So, one might conclude that the common opinion of many modern acousticians that Julius Caesar
was the founder of the first noise control act is debatable. In fact, his traffic control law induced
extensive noise pollution for a large part of the inhabitants of Rome. Unlucky for them, they were notconsidered important enough to include their serious health risks in adapted noise legislation.
However, luckily this law did raise awareness for the risks of noise pollution which has eventually
led to more and better legislation on noise for all inhabitants of urban agglomerations.

ACKNOWLEDGEMENTS
Authors like to thank dr. Mirjam Groen-Vallinga of the Radboud University Nijmegen for her
constructive comments and additions.

REFERENCES
1. Goldsmith, M. https://mikegoldsmith.weebly.com/history-of-noise.html.
2. EEA Report No. 22, E.U., Environmental noise in Europe — 2020. 2019.
3. Domingo, R., Roman Law: an Introduction 2018: Routlegde.
4. Hardy, E.G., The Table of Heraclea and the Lex Iulia Municipalis The Journal of Roman Studies 1914.
4:1: p. 65-110.
5. https://avalon.law.yale.edu. Avalon Project.
6. Favro, D., Construction Traffic in Imperial Rome, in Rome, Ostia, Pompeii: Movement and Space,
R.L.a.D.J. Newsome, Editor. March 2015, Oxford Scholarship Online: Oxford.
7. Juvenal, P., Juvenal and Persius, in Loeb Classical Library S.M. Braund, Editor. 2004, Harvard University
Press: Cambridge MA.
8. Edmondson, J., Cities and Urban Life in the Western Provinces of the Roman Empire, 30 BCE - 250 CE,
in A Companion to the Roman Empire, D.S. Potter, Editor. 2009, Wiley-Blackwell: Oxford. p. 250.
9. Hartnett, J., The Roman Street: urban life and society in Pompeii, Herculaneum and Rome. 2017,
Cambridge.
10. Younger, P.t., Letters, Volume I: Books 1-7, in Loeb Classical Library B. Radice, Editor. 1969, Harvard
University Press: Cambridge MA.
11. Aldrete, G.S., Floods of the Tiber in ancient Rome 2007, Baltimore
12. Tacitus, Annals: Books 4-6, 11-12, in Loeb Classical Library J. Jackson, Editor. 1937, Harvard University
Press: Cambridge MA.