A02:
BIO-ACOUSTICS

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0283

Influence of Maternal Separation on Ultrasonic Vocalization in ICR
Mice: Acoustic Features and Syllable Patterns

Ryo NISHIBORI1 ; Yuki ITO2 ; Jun MOTOYAMA3 and Kohta I KOBAYASI4
1, 4 Graduate School of Life and Medical Sciences, Doshisha University, Japan
2 Organization for Research Initiatives and Development, Doshisha University, Japan
3 Graduate School of Brain Sciences, Doshisha University, Japan

ABSTRACT
Abuse and neglect during childhood can affect brain development, and maternal separation (MS) is frequently
used as an early life stress model in rodents to investigate behavioral responses under stressful conditions.
Ultrasonic vocalizations (USV) of pups trigger nurturing behavior of parents and are a quantifiable indicatorof urge for maternal care. This study measured effects of MS on their USV to evaluate how pups adaptively
respond to neglect. We subjected ICR mice to MS for 24 hours from postnatal days (PD) 4 to 5 and recorded
USVs of pups on PD4 to 30. Control group was able to live with and receive care from their mothers throughsame period. Their USV was classified into 12 patterns based on their acoustic characteristics such aspresence of frequency modulation (FM). Compared to control group, MS group significantly vocalized USV
more often. USV in MS group was longer and louder than that in control group. In addition, the share of FM
syllables increased in MS group. Frequency modulation might help mothers to locate the sound source. Our
results show that MS alters various acoustic features of USV of pups and these changes may serve to increase
the mother's care.
Keywords: Ultrasonic Vocalizations, Maternal Separation, Syllable Patterns

1. INTRODUCTION
Stress from childhood, such as neglect or abuse, has been associated with depression, anxiety
disorder, Post Traumatic Stress Disorder (PTSD) and other neurological and psychiatric disorders 1 ) .
To elucidate the impact of childhood stress on development and mechanisms, animal models of early
postnatal stress have been developed. Maternal separation (MS) is frequently used as an early life
stress model in rodents to investigate behavioral and neurological responses under stressful conditions,and neglect have been shown to increase the risk of later mental disorders2).
Vocal communication in animals has been observed many species. Rodents can produce andperceive ultrasonic vocalizations (USV) at frequencies above human hearing threshold of 20 kHz. In
particular, pups produce isolation-induced USV to trigger maternal behaviors such as retrieval andsuckling of pups by the parents3). It has also been shown that the USV of pup mice is affected byenvironmental stimuli such as stress4). This study measured effects of MS on their USV to evaluate
how pups adaptively respond to neglect.

1
2

ctug1025@mail4.doshisha.ac.jp
yuito@mail.doshisha.ac.jp

3jmotoyam@mail.doshisha.ac.jp

4kkobayas@mail.doshisha.ac.jp

2. METHOD
2.1 Subject
A total of 20 ICR mice, between P0 and P30 (Postnatal day), were used in this experiment. (Tenpups: each born same day to two mothers) 20 pups were randomly divided into 10 MS group and 10
Control group. (5 pups in MS group and 5 pups in Control group from one parent)
2.2 Maternal Separation
In MS group, pups were separated from their mothers at 24 hours between P4 and P5. MS was
carried out on P4 starting at 10 a.m. During the separation period, each pup was individually placedin a disposable white paper cup (5 cm in diameter, 7.5 cm in height: KOKUYO co., Ltd, Osaka, Japan)
with fresh bedding (Clean Chip SP, Clea Japan, Osaka, Japan). A heating blanket (iKuma, Guangdong,
China) set at 35–37°C was placed underneath the paper cups to assist thermoregulation during the MS.
No food or water was available during the MS. All separated animals were placed in a separate roomwith the same temperature, humidity and lighting conditions as the home stable. After 24 hours of MS,
MS group were moved back to live with their mothers. On the other hand, Control group was able tolive with and receive care from their mothers during the separation.
2.3 Sound recording
Pups were placed one by one in a beaker. A recording microphone (Ultrasound Microphone CMPA P48／CM16 SN34, Avisoft Bioacoustics, Berlin, Germany) was placed on 30 cm above the top of
beaker, and vocalizations were recorded by a sound card (UltraSoundGate, 116Hb, Avisoft
Bioacoustics, Berlin, Germany) with a sample rate of 300 kHz at 16 bits／sample. To remove broadspectrum background sound caused by the movements of pups, we used MATLAB scripts (USVSEG)
were applied to convert audio files to the spectrogram, and USV syllables were automaticallysegmented5). Each segmented syllable was manually inspected to exclude noise. Finally, the syllables
were analyzed for duration, frequency, frequency modulation, and amplitude.
2.4 Classification of ultrasound vocalization
A typical sonogram for each syllable pattern were shown in Fig.1 Classified according to Duration,
Start frequency, End frequency, Max frequency, Min frequency and frequency modulation. The
classification of syllable patterns is based on Scattoni (2008) 6). The following was a classificati on
method.
(1) Flat calls had a constant beginning and end of the frequency (Max freq. - Min freq. ≤ 5kHz).
(2) Chevron calls included inverse U-shaped frequency modulation (Max freq. - Start freq. ≥ 5kHz and Max
freq. - End freq. ≥ 5kHz).
(3) Reverse chevron calls included U-shaped frequency modulation (Start freq. - Min freq. ≥ 5kHz and End
freq. - Min freq. ≥ 5kHz).
(4) Short calls were shorter less than 5ms.
(5) Upward calls continuously increased in frequency (End freq. - Start Freq. ≥ 5kHz).
(6) Downward calls continuously decreased in frequency (Start Freq. - End freq. ≥ 5kHz).
(7) 1 frequency step calls were a frequency change that appeared as a 'step' in the vertical direction (Onefrequency step above 10 kHz).
(8) 2 frequency step calls were two frequency change that appeared as a 'step' in the vertical direction (Twofrequency step above 10 kHz).
(9) More frequency step calls were more than three frequency change that appeared as a 'step' in the verticaldirection (Three or more frequency step above 10 kHz).
(10) Complex calls exhibited two or more sounds in a single syllable (Frequency pitch changes ≥ 5kHz).
(11) Arch calls included corrugated shaped frequency modulation (Max freq. - Start freq. ≥ 3kHz and Endfreq. - Start freq. ≥ 7kHz).
(12) Other Not listed above.

Fig. 1. Sonograms of typical patterns of USV

3. RESULTS
3.1 Effect of maternal separation on pup vocalizations
Measurements of ultrasonic vocalizations in pups detected prominent differences between MS
group and Control group. Fig.2A signified USV number in 5 minutes. On P8 and P9, MS group emitted
significantly more calls (P8:t(18)=-2.51, p<0.05; P9:t(18)=-3.71, p<0.001). Although there was a gap
in the two groups' peaks, the temporal change in USV number showed a similar trend. In P4, we
observed a very low number of USVs in both groups. Control group peaked at P6 and then decreased,nearing 0 on P15. On the other hand, MS group peaked at P8 and then decreased, nearing 0 on P15.
Fig.2B signified duration of vocalizations. MS group emitted significantly longer calls than Control
group between P6 and P9 (P6:t(18)=-3.46, p<0.001，P8:t(18)=-2.79, p<0.05，P9:t(18)=-2.24, p<0.05).
Although there was a gap in the two groups' peaks, the temporal change in duration of vocalization
showed a similar trend. At P4, duration of vocalization was short in both groups despite increased
with growth. Control group peaked at P10 and then decreased, while MS group peaked at P8 and thendecreased. Fig.2C signified peak amplitude. MS group emitted significantly louder calls than Control
group at P6 (t(18)=-2.60, p<0.05). The temporal change in peak amplitude showed similar trend inboth groups. In both groups, vocalizations at P4 were quiet, but became louder with growth. After
that, both groups peaked at P10 and then decreased.
3.2 Comparison of syllable pattern
We classified vocalization based on 2.4 classification of speech patterns. Fig.3 signified occurrence
rate by each syllable pattern. In Chevron, MS group emitted significantly more calls than Control
group at P9 and P10 (P9:χ2(1)=13.16, p<0.001, P10:χ2(1)=4.68, p<0.05). Similarly, In 2 frequency
step, MS group emitted significantly more calls at P9 (χ2(1)=7.38, p<0.05). There was a marked
difference between two groups within syllable patterns.

Fig. 2 MS effect on USVs.
Significant in A) Call Count B) Duration C) Peak Amplitude. *p<.05, **p<.01 paired t test.

Fig.3 Occurrence rate by each syllable pattern.
Significant in A) 2 Frequency Step B) Chevron, *p<.05, **p<.01 χ 2 test.

4. DISCUSSION
There were significant differences in USV number, duration of vocalizations and peak amplitude
between MS group and Control group. In the study of Xiaowen (2016) 7), pups of C57BL／6J wereseparated from their mothers for 180 minutes per day at P1 to P14. As their results, USV number andduration of MS group increased more than Control group with a peak at P8. Our experiments on ICR
pups showed similar temporal changes to previous studies, with a significant increase in MS groupover Control group. Therefore, it has been shown that the changes in USV number and duration caused
by maternal separation are similar in ICR pups as in other strains. The fact that USV depends on thematernal behavior 8) suggests that these changs may be the result of a desire for more maternal care.
In addition, the percentage of vocalizations that included frequency modulation increased. Thissuggests that voice complexity may be important for care.

REFERENCES
1. M. H. Teicher, A. Tomoda and S. L. Andersen, "Neurobiological Consequences of Early Stress andChildhood Maltreatment: Are Results from Human and Animal Studies Comparable?", The New York
Academy Sciences, vol.1071, no.1, p313-323, July 2006.
2. N. Freund, B. S. Thompson, J. DeNormandie, K. Vaccarro and S. L. Andersen, "Windows ofVulnerability: Maternal Separation, Age, and Fluoxetine on Adolescent Depressive-Like Behavior inRats", Neuroscience, vol.249, no.26, p88-97, September 2013.

3. M. Wöhr and R. Schwarting, “Affective Communication in Rodents: Ultrasonic Vocalizations as a Toolfor Research on Emotion and Motivation”, Cell and Tissue Research, vol.354, p81-97, April 2013.
4. J. T. Allin and E. M. Banks, "Functional aspects of ultrasound production by infant albino rats (Rattus
norvegicus)", Animal Behavior, vol.20, no.1,175-185, February 1972.
5. R. O. Tachibana, K. Kanno, S. Okabe, K. I. Kobayasi and K. Okanoya, "USVSEG: A Robust Method for
Segmentation of Ultrasonic Vocalizations in Rodents", PLoS ONE, vol.15, p1-16, February 2020.
6. M. L. Scattoni, S. U. Gandhy, L. Ricceri and J. N. Crawley, "Unusual Repertoire of Vocalizations in the
BTBR T+tf/J Mouse Model of Autism", PLoS ONE, vol.3, no.8, p48-52, August 2008.
7. Y. Xiaowen, C. Ling, X. Yong, C, Qunkang, Y. Jiabei, Y. Yan, W. Zhaoxin, W. Haojie, D. Jianshu, D.
Yuqiang and Z. Xudong, "Maternal Deprivation Influences Pup Ultrasonic Vocalizations of C57BL/6J
Mice", PLoS ONE, vol.11, no.8, p1-20, August 2016.
8. H. N. Shair "Parental Potentiation of Vocalization as a Marker for Filial Bonds in Infant Animals",
Developmental Psychobiology, vol.56, no.8, p1689-1697, June 2016.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0386

Echo recovery in bats using acoustic simulation
Yu TESHIMA1,a; Takao TSUCHIYA 2; Yuta HASEGAWA1; Shoko GENDA1; Ryota MORIYAMA1;
Taku KAWAMURA1; Shizuko HIRYU1,b
1

Faculty of Life and Medical Sciences, Doshisha University, Japan
2

Faculty of Science and Engineering, Doshisha University, Japan

ABSTRACT
In order to study echolocation behavior in bats, it is essential to measure the echoes bats hear. However, it
is difficult to measure the echoes that reach both ears of bats in flight. In this study, we propose an approachthat combines advanced behavioral experiments using a telemetry microphone recording with acoustic
simulations to reproduce echoes reaching both ears of bats during flight. We examined how the echo spacechanged before and after the bat became familiar with their environment using data from a behavioral
experiment of multiple consecutive flights of bats in the same obstacle space. In addition, although therelationship between the direction of flight and the pulse direction, which is considered to be the direction of
attention of bats, has been investigated (e.g. Fujioka, E. et al., (2016). PNAS; Ghose, K. et al., (2006). J.
Neurosci.), we revealed that the echo direction, which is the input information to bats, is important for thedecision-making of the flight trajectory of bats. The results of simulations of the effects of moving soundsources and bat head shape on echoes as a way to recover echoes in more detail will also be presented.

Keywords: Bats, Echolocation, Acoustic simulation, FDTD

1. INTRODUCTION
Bats are known to have advanced ultrasound sensing capabilities, such as instantly avoiding collisions
with obstacles in the dark and catching small prey one after another by analyzing echoes of ultrasound pulses
emitted by themselves. Elucidation of the ideas and mechanisms involved in extracting information fromechoes and determining behavior, which enable bats' advanced sensing, is expected not only to clarify theactual behavior of bats, but also to expand into the field of information and engineering based on sensing
technology. Until now, behavioral measurements such as obstacle avoidance behavior of bats have been
examined mainly by measuring pulse and flight paths[1–3]. In addition, recent reports have describedbehaviors such as colliding against walls with smooth surfaces[4] and misperceiving objects of equal size
but different materials as objects of different sizes[5]. These findings suggest that the space that bats perceive
through echoes is different from the space that we humans perceive primarily visually. Therefore,
understanding the perceived space and information based on echoes and interpreting the behaviors
represented are necessary to elucidate the sensing strategies of bats. At present, however, no technology exists
to measure all the faint echoes that reach the binaural ears of bats in flight. Therefore, sensing strategies basedon the echoes heard by bats and the information obtained from them have not been explored to date. To solve
this problem, this paper proposed a method to calculate echoes with high accuracy by combining behavioral
measurement experiments and acoustic simulations. The proposed method reveals that there is a difference
between the obstacle space where the bats flew, and the space composed by the echoes from the bat pulses("echo space")[6]. In addition, it was possible to reproduce the echoes, including the Doppler effect, by
making the source and receiver of the acoustic simulation movable. This simulation was adapted to the
Doppler shift compensation behavior of bats, and the gazing transition of bats in flight was clarified[7]. The
proposed method enables the acquisition of bat input information (echo information) as well as bat behavioral
output (flight paths and pulse information), which can be measured by conventional behavioral measurement.
abyuteshima18@gmail.com
shiryu@mail.doshisha.ac.jp

Therefore, the input-output relationship of bats can be used to model their behavioral decisions. In this study,
we present a new research method and its results that can further advance echolocation research.

2. Research Summary
2.1 Visualization of “echo space”
In this research, to address the problem of echomeasurement, which has been difficult due to technicalproblems, we proposed a method to calculate echo
sequences reaching both ears of bats in flight by
combining behavioral experiments to measure pulse
and flight paths and the Finite Difference Time
Domain (FDTD) method[8]. Specifically, the positionand direction of the bat's pulse emissions measured in
the behavioral experiment are introduced into thesimulation space, and the echoes from thesurroundings are calculated by setting a receiving
point at the position of the bilateral ears of the bat. Thismethod was adapted to the respective data in unknown
space (first flight) and known space (12th flight) for
Fig. 1 Comparison of obstacle space flown
bats by measuring the behavior of bats during their Sby bats and “echo space” composed by echo
shaped flight through an obstacle space composed ofsources calculated from echoes.
three boards. The results showed that “echo space” (theobstacle space composed of echo sources calculated from the binaural echoes) differed from the actual
obstacle space in that it was composed only of important areas for avoidance behavior (Fig. 1). In thetwelfth flight, the echo sources were more concentrated at the inner edge of the board, which is an
obstacle, than in the first flight, suggesting that the obstacle space may be composed of only the
information necessary for obstacle avoidance. In addition, although it has been clear that there is acorrelation between the direction of pulse emission and the turn rate, it was revealed for the first timein this study that the direction of arrival of the echoes correlates with the turn rate of the bats.
2.2

Auditory sensitive bandwidth
Gaze transition of bats in flight
The knowledge of what an animal ispaying attention to during an activity is an
Echo frequency
from each wall
important piece of information that allows
Pulse point
us to infer the "thoughts" of the animal
Pulse frequency
during its behavior. Echoes reflected fromthe surrounding environment by the bat's
emittedpulsesare
duplicatedsimultaneously
from
each
object.
Fig. 2 Pulse frequencies of bats and echo frequencies
Furthermore, the frequency of each echo isfrom each wall (top). Transition of gazing wall during bat
modulated by the Doppler effect according
flight (bottom). The color of the pulse points plots theto the relative velocity of each object. The
color of the wall of the echoes calculated in the twoRhinolophus
ferrumequinum
nippondimensional FDTD that were in the band of auditory
adjusts the frequency of its pulses so thatsensitivity.
only the frequencies of the echoes from thetarget it is gazing at always match the frequency band of its own auditory sensitivity. This is called
Doppler shift compensation (DSC) behavior[9,10]. Accordingly, if an echo within the band of highauditory sensitivity is revealed, the bat is found to be paying attention to the object from which theecho comes. However, until now, it has been difficult to measure the echoes to be analyzed. In thisresearch, a moving source and moving receiver point were implemented in the FDTD to enable precise
calculation of echoes including the effect of Doppler effect[11]. We measured the behavior data ofbats as they flew in circles in a space surrounded by four walls and calculated the echoes from eachwall independently by the FDTD in two-dimensional space at top view(x-y). As a result, we found forthe first time that the wall whose echo frequency matched the band of high auditory sensitivity
changed as the bats circled, and that the bats shifted the target of their attention as if they were"anticipating" where they were going to turn during flight (Fig. 2).

ACKNOWLEDGEMENTS
This work was supported by JSPS KAKENHI Grant Numbers JP 18H03786, 21H05295, 22H01503.

REFERENCES
1.

Yovel Y, Falk B, Moss CF, Ulanovsky N. Optimal localization by pointing off axis. Science. 2010;327:
701–704.

2.

Ghose K, Moss CF. Steering by hearing: a bat’s acoustic gaze is linked to its flight motor output by adelayed, adaptive linear law. J Neurosci. 2006;26: 1704–1710.

3.

Yamada Y, Mibe Y, Yamamoto Y, Ito K, Heim O, Hiryu S. Modulation of acoustic navigation behaviourby spatial learning in the echolocating bat Rhinolophus ferrumequinum nippon. Sci Rep. 2020;10:10751.

4.

Greif S, Zsebők S, Schmieder D, Siemers BM. Acoustic mirrors as sensory traps for bats. Science.
2017;357: 1045–1047.

5.

Danilovich S, Shalev G, Boonman A, Goldshtein A, Yovel Y. Echolocating bats detect but misperceivea multidimensional incongruent acoustic stimulus. Proc Natl Acad Sci U S A. 2020;117: 28475–28484.

6.

Teshima Y, Yamada Y, Tsuchiya T, Heim O, Hiryu S. Analysis of echolocation behavior of bats in “echospace” using acoustic simulation. BMC Biol. 2022;20: 59.

7.

Teshima Y, Hasegawa Y, Tsuchiya T, Moriyama R, Genda S, Kawamura T, et al. Reconstruction ofechoes reaching bats in flight from arbitrary targets by acoustic simulation. J Acoust Soc Am. 2022;151:
2127–2134.

8.

Yee K. Numerical solution of initial boundary value problems involving maxwell’s equations inisotropic media. IEEE Trans Antennas Propag. 1966;14: 302–307.

9.

Schnitzler H-U. Die Ultraschall-Ortungslaute der Hufeisen-Fledermäuse (Chiroptera-Rhinolophidae) inverschiedenen Orientierungssituationen. J Comp Physiol A Neuroethol Sens Neural Behav Physiol.
1968;57: 376–408.

10. Hiryu S, Shiori Y, Hosokawa T, Riquimaroux H, Watanabe Y. On-board telemetry of emitted sounds
from free-flying bats: compensation for velocity and distance stabilizes echo frequency and amplitude.
J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 2008;194: 841–851.
11. Tsuchiya T, Teshima Y, Hiryu S. Two-dimensional finite difference-time domain simulation of moving
sound source and receiver. Acoust Sci Technol. 2022;43: 57–65.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022, in Gyeongju, Korea
Extended abstractABS-0440

Build a training interface to install the bat's echolocation skills
in humans
Miyoko TSUMAKI1,a, Yu TESHIMA1, Takao TSUCHIYA2, Kaoru ASHIHARA3,
Kohta I. KOBAYASI1, Shizuko HIRYU1,b
1

Graduate School of Life and Medical Sciences, Doshisha University, Japan
2

3

Faculty of Science and Engineering, Doshisha University, Japan

Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science andTechnology (AIST), Japan

ABSTRACT
Bats use a sophisticated ultrasonic sensing method called echolocation to recognize the environment.
Recently, it has been reported that normal sight participants with no prior experience in echolocation can
improve their ability to perceive the spatial layout of various environments through training to hear echoes(Norman, et al., 2021). In this study, we developed a new training system for human echolocation using theeye tracker. Binaural echoes of consecutive downward linear FM pulses that were inspired by feeding
strategies of echolocating bats were simulated using the wave equation finite difference time domain method.
In the experiment, the sighted participants were asked to identify the shapes of the hidden target, and thevirtual echoes were presented in response to their eye movements on the monitor. Our analyses suggest thatthe training shortened the sensing time required for shape identification, and the accuracy of shapeidentification was improved by the strategy to focusing on the edges of the target. The proposed system isconsidered useful for quantitative verification of human echolocation.
Keywords: Eye-tracking, Human Echolocation, Virtual echo

1. INTRODUCTION
Blind echolocation experts (EEs) use self-generated mouth clicks to perceive their surroundings
via reflected sound waves(echoes) [1]. Meanwhile, it has been reported that even sighted participants
with no prior experience in echolocation can improve their spatial recognition skills through t raining[2]. However, the actual echo measurement requires time and effort for experimentation [3], and there
are many unclear points about effective training methods and learning processes of echolocation skills .
In this study, we propose a system that trains participants to learn echolocation skills withgenerated echoes from virtual targets of various shapes using 3D acoustic simulation. The participantsecholocate the target by hearing the virtual echoes that are generated based on their eye movement on
the monitor. Using that system, we analyzed the sensing timing, direction, and shape-identification
process of the participants based on the gaze position obtained by the eye-tracker, and examined thesensing strategy of the echolocation inexperienced sighted participants.

2. METHODS
2.1 Estimated Echoes based on the Impulse Responses
Using a three-dimensional acoustic simulation based on a wave equation finite difference time
domain (WE-FDTD) method [4], the shape of a dummy head (HATS; Head and Torso Simulator Brüelamtsumaki2021@gmail.com
shiryu@mail.doshisha.ac.jpb& Kjær TYPE 4128-C) was digitized and saved as the stereolithography (STL) file. The STL modelsof the HATS and a virtual target were placed in the simulation space (1.5 × 1.5 × 1.5 m) (Fig. 1) . The
spatial resolutions of the simulations were 1.5 mm, Courant-Friedrichs-Lewy (CFL) =0.5, and thesound velocity was set to 344 m/s which satisfied the condition for 3D simulation stability. A
transmitted signal (impulse) was emitted from the mouth of the HATS, and the echo from the virtual
target at the entrance of the left and right ear canals of the HATS as the receiving point was calculated
based on numerical analyses. This calculated echo was defined as the echo impulse response. At thistime, the target was divided into 5 x 5 cm meshes, and the x and z coordinates of the position of thesound source (mouth of the HATS) were set to the x and z coordinates of the center position of eachmesh to calculate the virtual echo. There were 8 types of virtual targets for the training task, andtogether with 5 untrained versions, the test task was performed on 13 different targets (Fig. 2).

Fig. 1 An example of geometricalconfiguration of the HATS and the virtual
target in a 3D simulation space. The
transmitted signal was incident at any positionon the target by the HATS tilted to change thedirection of the mouth.

Fig. 2 Virtual targets used in training sessions
and test sessions.

2.2 Psychoacoustic Experiment
The psychoacoustic experiment was conducted in a sound-attenuated chamber (2.3×1.6×1.4 m).
The participants were 1 male and 4 females (22 - 25 years old) with normal sight. It was confirmedthat the hearing levels of all participants were within 30 dB in the 0.25 to 3 kHz range based onstandard pure-tone audiometry using an audiometer (AA-77A, RION, Tokyo, Japan). The
psychoacoustic experiment using humans was approved by the ethics committee of Doshisha
University. After obtaining informed consent from the participants and conducting a he aring test, webegan the psychoacoustic experiment. One sound stimulus consisted of, a 10 ms downward linear FM
sound (7 kHz - 1 kHz) that was convoluted with the echo impulse response described 2.1, and 30
repeated sounds were produced at 30 ms intervals to simulate the buzz sound inspired by feeding
strategies of echolocating bats. The presentation of the sound stimuli to the participants was controlledby PsychoPy3 (Standalone 2021.2.3 for 64bit Windows (using Python3.6)) [5], and the presentationof the sound stimuli to the participants was controlled by PsychoPy3, and the sound stimuli were
presented via headphones (MDR-CD900ST; SONY) through the audio interface (OCTA-CAPTURE
UA–1010; Roland) from the personal computer.
The participants' gaze position (point of gaze, POG) was recorded using a 150Hz desktop eyetracker (GP3HD Eye tracker, Gazepoint Vancouver, Canada). The mesh-like guideline was displayedon the monitor, and virtual echoes from the mesh corresponding to the POG were presented fromheadphones. Participants freely sensed the shape of a virtual target hidden on the monitor with theirgaze voluntarily. When the participants judged that they could draw the shape, they ended the session
by themselves and then drew the shape of the target they recognized on the paper with the mesh on it.
There were 8 training trials (with correct feedback) and 7 testing trials (without correct feedback).
After completing the trial, a written questionnaire was conducted.

3. RESULTS and DISSCUTION
3.1 Participant’s sensing strategy in the virtual echolocation training
Figure 3 shows a result of the shape of the target (shown in red dashed line) drawn by one
participant after sensing. The degree of similarity is calculated as follows. To quantitatively evaluate
the difference between the shape drawn by the participant and the outline of the virtual target presented(hereinafter called "difference"), we calculate it by using matchShapes function of the OpenCV library
based on the Hu moment [6]. If the threshold was less than 0.02, it was judged to be the same shape(e.g., The lower the result, the better the match, and the closer to zero the returned value, the higher
the similarity). The difference in the trained targets of all participants was 0.11 on ave rage, and theaverage of the untrained (i.e., try for the first time) targets was 0.17, which suggests that shapediscrimination to some extent even for the untrained target. In addition, the average sensing time forthe training session until shape identification was 179.6 seconds. The average sensing time for the test
session (trained target) was 155.0 seconds and the average sensing time for the test session (untrainedtarget) was 167.0 seconds, which was shorter than the training session. From this result, it isconsidered that there is a learning effect of echolocation by training.
The POG data were analyzed for differences in the distribution among trained targets and untrainedtargets. Comparing the training session and the test session (trained target), the test session (trainedtarget) tended to gaze more at the edges and outside of the target , especially in participants who
showed results with low differences in shapes. In addition, more attention to the edge and shorter
sensing times were observed for trained targets than for untrained targets in the test session. These
results suggest that the participants modified their sensing strategies during the training process tofixate their gaze on the edges or outside of the target. The POG distribution of each participant andthe responses to the questionnaire after all sessions indicated that the virtual echoes from the target
edges were distinctive.

Fig. 3 Shapes drawn in questionnaires(top), heatmaps in the test sessions (middle) and difference
values between the target and the shape drawn by the subject (bottom). The red dashed line indicatesthe shapes drawn by the subject.

4. CONCLUSION
In this study, we propose a system that participants listen to virtual echoes while freely sensing by
the gaze of sight and investigated the learning process of human echolocation. The results show thatthe accuracy of shape identification using echolocation is improved by participants changing tosensing strategies that focus on the edge of the virtual targets. This echolocation training system is aneffective quantitative approach to the echolocation learning process.

REFERENCES
1.

Thaler L, Goodale MA. Echolocation in humans: an overview. WIREs Cogn Sci. 2016;7: 382–393.
pmid:27538733.

2.

Norman L, Dodsworth C, Foresteire D, Thaler L (2021) Human click-based echolocation: effects ofblindness and age, and real-life implications in a 10-week training program. PLoS ONE 16(6): e0252330

3.

Sumiya M, Ashihara K, Yoshino K, Gogami M, Nagatani Y, Kobayasi KI, et al. Bat-inspired signal
design for target discrimination in human echolocation. J Acoust Soc Am. 2019;145(4):2221–2236.
pmid:31046316

4.

Y. Miyazaki, and T. Tsuchiya, “Perfectly Matched Layer for the Wave Equation Finite Difference Time
Domain Method”, Jpn. J. Appl. Phys. 51, 07GB02-1–5 (2012).

5.

Peirce, J. W., Gray, J. R., Simpson, S., MacAskill, M. R., Höchenberger, R., Sogo, H., Kastman, E.,
Lindeløv, J. (2019). PsychoPy2: experiments in behavior made easy. Behavior Research Methods.
10.3758/s13428-018-01193-y

6.

OpenCV, OpenCV modules, https://docs,opencv.org/4.6.0/ (2022/7/15)

ABS-0532

Augmenting training dataset with real noise environment for classification of
exotic and endangered animal soundKyungdeuk KO(1) , Chulwon CHOI(2) , Hanseok KO(3)(1) Korea University, South Korea, kdko@korea.ac.kr(2) Korea University, South Korea, cchoi1022korea.ac.kr(3) Korea University, South Korea, hsko@korea.ac.kr

ABSTRACTThe interest in ecosystem conservation is growing with many remedial policies sought and implemented. In
particular, the public recently has perceived awareness that it is essential to eradicate exotic invasive species andprotect endangered species because they destroy the food chain and the ecosystem. With the recent advancementof deep learning-based technologies, several studies on acoustic signal classification have taken place, but thestudies on animal sound classification considering actual noise environment are still scarce. In this paper, we
propose a data augmentation scheme for training by setting up an experimental environment that includes thesounds of exotic invasive species, endangered species, and realistic environmental noise. With the proposed augmentation scheme incorporating more real environment under various SNR conditions, we construct the training
set, validation set, and test set to classify the animal sounds. For validation, we compare the performances ofvarious deep learning-based methods used in signal classification tasks by employing the constructed dataset.

Keywords: Animal sound classification, Bio-acoustic signal classification, Deep learning

1

INTRODUCTION

Recently, the interest in ecosystem conservation has been growing. In particular, the public recently has perceived awareness that it is essential to eradicate exotic invasive species and protect endangered species. Figure
1 shows the spectrograms of various exotic invasive species and endangered species. The exotic species can destroy the food chain and the ecosystem. Meanwhile, some species such as Kaloula borealis and Hyla suweonensis became endangered by hunting, pollution, abnormal climate, and exotic species. As a result, the endangered
species may no longer exist. Therefore, many efforts have been made to eradicate exotic invasive species andprotect endangered species. However, most of these efforts are ineffective because they require people to act
physically. Recent advancement of deep learning can be a solution to these problems.
For the past ten years, studies have taken place to classify signals using deep learning. AlexNet (1) iscomposed of a deep Convolutional Neural Network (CNN) to improve image signal classification performancecompared to traditional pattern recognition methods. VGG (2) is deeply structured with small-sized kernels towiden the receptive field while reducing the number of parameters. ResNet (3) uses skip connections to prevent
vanishing gradient. DenseNet (4) is constructed such that all feature maps of previous layers are concatenatedto have strong gradient flow and information flow while reducing the number of parameters. The self-attention
mechanism is applied to SENet (5) and CBAM (6) to emphasize important features. Moreover, studies haveverified that deep learning produces high performance in image classification tasks. Studies to classify signals
using deep learning have been actively conducted not only on image signals but also on the acoustic signal.
TCNet (7) and TENet (8) are representative neural networks for acoustic signal classification. Both TCNet
and TENet architectures use temporal convolution to reduce the number of parameters and FLOPs. Temporalconvolution operates like 1D convolution on the time axis. In addition, DSL (9) employs a dynamic filter toimprove performance and robustness in unseen noise environments.

Figure 1. The animal sound examples. First row is the endangered species, and second row is the exotic
species. (a) Pelophylax chosenicus, (b) Kaloula borealis, (c) Hyla suweonens , (d) Lithobates catesbeianus,(e)Xenopus laevis, and (f) Rhinella marina
Deep learning for signal classification has recently migrated to the animal sound classification task. Xie
et al. (10), Hidayat et al. (11), and Nasirahmadi et al. (12) proposed a CNN-based method for bird soundclassification, scope owl sound classification, and pecking activity detection of turkeys, respectively. Their paper
shows that the method has a limitation in that the target species restrict to birds only. Ko et al. (13) proposeda diverse animal sound classification method based on CNN and Support Vector Machine (SVM). Three CNN
networks are pretrained and consolidated into one to classify amphibians, insects, and birds. As a result, efficient
re-training is possible when species are added or changed. CFAM (14) employs a self-attention mechanism (5,6) along the frequency and channel axis. Zhong et al. (15) use transfer learning with pseudo-labeling by
employing unlabeled data. However, these methods do not consider a real noisy environment in which theanimals make sounds.
This paper proposes a new data augmentation method for animal sound classification by considering the realenvironment. First, we suppose various natural noise environments such as wind and rain. Second, we consider
the environment in which other species make sounds at the same time. Lastly, we assume that a species otherthan the species to be trained can cry. Subsequent sections describe the database used for the experiment, theexperimental model, and experimental results.

2

PROPOSED METHOD

Our proposed data augmentation method is displayed and shown in Figure 2. The proposed data augmentation
method is composed of four steps. Fist, an animal sound s0 , another animal sound s1 , and a background noise
sb are selected. Second, we apply random cropping to each sound, so a new 1-sec clip gets created from the
2-sec clip at every epoch. Third, the MixUp [17] based data augmentation method is employed to incorporatea real environment wherein various animal sounds can be mixed and background noise can exist. Then, a value
w1 between 0 and 1 is randomly selected, and another value w2 is defined as w2 = 1 − w1 . As a result, training

Figure 2. The example of the proposed data augmentation considering the real environment
data s f is formulated as follows:s f = s0 + w1 ∗ s1 + w2 ∗ sb .

(1)

Lastly, in order to focus on learning about s0 , the label for each signal is defined as follows:
l0 = 1 − w1 /5 − w2 /5, l1 = w1 /5, l2 = w2 /5.

(2)

At every epoch, new training data gets created by random cropping and MixUp-based data augmentation. Additionally, to proceed with the training of the background noise, training is carried out by concatenating thebackground noise as much as the batch size into the training batch.
In the real environment, a new animal specie may emerge or those species not included in training may
make sounds. It would be difficult to predict which animal other than the target species will make a soundin the real environment. Considering such a real environmental scenario, we select the specific target species
for the classification task and group all other animal sounds into the Unknown class. In our case, out of 120
classes, 23 target classes are selected, including the exotic invasive species, the endangered species, and thebackground noises. The other 97 species get labeled as Unknown classes.

3

EXPERIMENTS

3.1 Model
Two representative models, ResNet (3) and DenseNet (4), frequently used in image and acoustic signal classification are selected, and the experiments are conducted. ResNet is composed of 12 residual blocks. Batch

normalization (16) and an identity block, which is composed of 1 × 1 convolution, are applied to each residual
block. In the last layer, the dimension is reduced using global average pooling and then made to the numberof classes using a fully-connected layer. DenseNet consists of three dense blocks. In dense blocks, the resultof the convolution operation gets concatenated with the previous features. Like ResNet, DenseNet uses global
average pooling after the last layer.
In addition, TCNet14 (7) and TENet12 (8) show high performance in acoustic signal classification, so both
TCNet14 and TENet12 architectures are used for animal sound classification experiments. TCNet14 is composed of six residual blocks, and each residual block has temporal convolutional layers and a skip connection.
TENet12 contains 12 residual blocks that consist of depth-wise convolution and point-wise temporal convolutional layers. The global average pooling layer is applied to both TCNet14 and TENet12 after the last layer.
3.2 Database
For the experiments, the database is composed of 120 classes that include the sounds of 118 animal species,
various microphone noises, and various background noises. The animal sounds are divided into 15 speciesof amphibians, 37 species of birds, 58 species of insects, and 8 species of mammals. The animal sounds arecollected by the animal experts setting up recorders and employing web crawling. Amphibian sounds include theendangered species such as Kaloula borealis and Hyla suweonensis, and the exotic species such as Lithobates
catesbeianus and Xenopus laevis. The insect and the bird sounds include cicadas and Hypsipetes amaurotis
common in forests and parks. Mammal sounds include dog, cat, and human sounds that can be commonly heard
in everyday life. The noise is composed of various microphone noises and background noises, in considerationof the recording environment. The microphone noises consist of white noise and pink noise. In addition, thebackground noises such as rain, wind, forest, and river are collected by web crawling. The recorded data isconverted to 22050 Hz and is divided into 2-sec clips. The classes and the number of data are summarized inTable 1.
Table 1. The scientific name and the number of dataClass

The number of dataClass

The number of data

Lithobates catesbeianus

48471

Lanius bucephalus

566

Rhinella marina

110

Hypsipetes amaurotis

1155

Xenopus laevis

4158

Pitta nympha1205

Pelophylax chosenicus

606

Zoothera dauma

1123

Kaloula borealis

25502

Hyalessa maculaticollis

979

Rana dybowskii

1605

Meloimorpha japonica

554

Hyla suweonens

3505

Homo sapiens

857

Glandirana emeljanovi

1704

Canis lupus familiaris

3507

Pelophylax nigromaculatus

2705

Felis catus

563

Hyla japonica

1911

Unknown

13689

Bombina orientalis

5370

Background noise

455271

Terpsiphone atrocaudata

397

Microphone noise

6000

3.3 Implementation details
Mel-spectrogram is used for the input feature. The window length is set to 512, hop length is set to 256, and thenumber of mel filters is set to 80. In this case, min-max normalization is applied to the input features. Binarycross entropy with sigmoid function is used as the loss function. SGDR (18) is employed as the optimizer.
The max learning rate is set at 0.001 while the min learning rate is set at 0.00001. Also we set T0 = 5 andTmult = 2.
3.4 Evaluation metrics
Various evaluation metrics are used for the performance evaluation. First, Top1 and Top5 accuracies are evaluated at various Signal-to-Noise (SNR) levels. Second, the number of parameters and FLoating point OPerations(FLOPs) of the model is measured to compare the efficiency of the model. Lastly, the ultimate goal is toclassify a particular species better, such as an exotic species. Therefore, the target species is set as Lithobates
catesbeianus, and how well it discriminates from background noise is measured. In this case, the accuracy of
Lithobates catesbeianus is specified as binary accuracy.
3.5 Experimental results
The experimental results are summarized in Table 2. Table 2 shows the Top1 accuracy of each SNR according
to the data augmentation method. When mixup-based data augmentation is used alone without random cropping, slight improvement outcome (0.9%) is achieved compared to achieving relatively significant improvement
outcome (3.15%) when no data augmentation is employed. When random cropping is applied to mixup-based
data augmentation, the table shows that the performance is improved by about 5% at SNR 20dB and by 13%
at SNR 0dB.
Table 2. The experimental results according to data augmentation
Model

TENet

Data augmentation

20

15

10

5

0

w/o data augmentation

79.18%

78.95%

77.59%

70.83%

51.30%

w/o random cropping

81.31%

81.07%

78.84%

71.73%

54.45%

proposed method

86.71%

86.32%

84.43%

79.12%

67.20%The experimental results according to various CNNs are summarized in Table 3. ResNet shows the highest
Top5 accuracy at SNR 20dB and 15dB, and the highest binary accuracy at SNR 20dB, 15dB, and 10dB. In
addition, ResNet has the highest number of parameters and FLOPs. DenseNet shows the highest Top5 accuracy
at SNR 10dB and 5dB but has the lowest Top1 accuracy at all SNRs. TCNet shows the lowest reduction inbinary accuracy depending on the noise environment. Lastly, TENet not only has the fewest parameters and
FLOPs but also shows the highest Top1 accuracy at all SNRs.

4

CONCLUSIONSWe proposed to inject a more realistic environment for training when conducting animal sound classification, in
which various noise and animal sounds are mixed. In addition, we added an unknown class in considerationof the case where the species not used for training makes a sound. For the experiment, we used ResNet and
DenseNet, frequently used in image classification, and TCNet and TENet, often used in sound signal classification. TENet not only has the fewest parameters and FLOPs but also shows the highest Top1 accuracy at all
SNRs. Consequently, when comprehensively considering performance, memory, and computational speed, TENet
is found as the most efficient in animal sound classification. Additionally, when comparing the performance bydata augmentation, when using the mixup-based data augmentation with random cropping, it improved by about
7% at SNR 20dB and about 18% at SNR 0dB. It confirms that the more noisy the situation, the more helpful
the proposed data augmentation is when conducting animal sound classification.

Table 3. The experimental results according to classification model
SNR

20

15

10

5

0

Model

Top1 accuracy

Top5 accuracy

Binary accuracy

Parameters

FLOPs

ResNet84.68%

96.36%

91.80%

0.54G

11.18M

DenseNet

77.67%

96.25%

84.82%

0.12G

791.64K

TCNet

83.66%

94.86%

80.55%

8.26M

305K

TENet

86.71%

94.61%

83.72%

6.26M

100K

ResNet84.13%

96.25%

89.57%

0.54G

11.18M

DenseNet

77.78%

96.18%

82.50%

0.12G

791.64K

TCNet

82.93%

94.70%

80.45%

8.26M

305K

TENet

86.32%

94.33%

83.58%

6.26M

100K

ResNet82.90%

95.94%

85.52%

0.54G

11.18M

DenseNet

74.53%

96.00%

78.97%

0.12G

791.64K

TCNet

80.93%

94.58%

79.83%

8.26M

305K

TENet

84.43%

93.81%

83.17%

6.26M

100K

ResNet78.48%

95.16%

77.98%

0.54G

11.18M

DenseNet

66.92%

95.25%

74.28%

0.12G

791.64K

TCNet

75.78%

93.62%

77.23%

8.26M

305K

TENet

79.12%

93.62%

79.85%

6.26M

100K

ResNet66.02%

93.09%

72.48%

0.54G

11.18M

DenseNet

53.77%

92.47%

67.13%

0.12G

791.64K

TCNet

62.75%

89.67%

77.23%

8.26M

305K

TENet

67.20%

93.49%

75.13%

6.26M

100K

ACKNOWLEDGEMENTS
This work was supported by Korea Environment Industry & Technology Institute(KEITI) through Exotic Invasive Species Management Program, funded by Korea Ministry of Environment(MOE)(2021002280004)

REFERENCES[1] Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks.
Advances in neural information processing systems, 2012, 25.
[2] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv
preprint arXiv:1409.1556, 2014.
[3] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE
conference on computer vision and pattern recognition. 2016. p. 770-778.
[4] Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 4700-4708.

[5] Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. p. 7132-7141.
[6] Woo S, Park J, Lee JY, Kweon IS. Cbam: Convolutional block attention module. In: Proceedings of theEuropean conference on computer vision (ECCV). 2018. p. 3-19.
[7] Choi S, Seo S, Shin B, Byun H, Kersner M, Kim B, Kim D, Ha S. Temporal convolution for real-time
keyword spotting on mobile devices. arXiv preprint arXiv:1904.03814, 2019.
[8] Li X, Wei X, Qin X. Small-footprint keyword spotting with multi-scale temporal convolution. arXiv preprint
arXiv:2010.09960, 2020.
[9] Kim D, Park J, Han DK, Ko H. Dual Stage Learning Based Dynamic Time-Frequency Mask Generation for
Audio Event Classification. In: INTERSPEECH. 2020. p. 836-840.
[10] Xie J, Hu K, Zhu M, Yu J, Zhu Q. Investigation of different CNN-based models for improved bird soundclassification. IEEE Access, 2019, 7: 175353-175361.
[11] Hidayat AA, Cenggoro TW, Pardamean B. Convolutional neural networks for scops owl sound classification. Procedia Computer Science, 2021, 179: 81-87.
[12] Nasirahmadi A, Gonzalez J, Sturm B, Hensel O, Knierim U. Pecking activity detection in group-housed
turkeys using acoustic data and a deep learning technique. Biosystems engineering, 2020, 194: 40-48.
[13] Ko K, Park S, Ko H. Convolutional feature vectors and support vector machine for animal sound classification. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology
Society (EMBC). IEEE, 2018. p. 376-379.
[14] Ko K, Park J, Han DK, Ko H. Channel and frequency attention module for diverse animal sound classification. IEICE Transactions on Information and Systems, 2019, 102.12: 2615-2618.
[15] Zhong M, LeBien J, Campos-Cerqueira M, Dodhia R, Ferres JL, Velev JP, Aide TM. Multispecies bioacoustic classification using transfer learning of deep convolutional neural networks with pseudo-labeling.
Applied Acoustics, 2020, 166: 107375.
[16] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate
shift. In: International conference on machine learning. PMLR, 2015. p. 448-456.
[17] Zhang H, Cisse M, Dauphin YN, Lopez-Paz D. mixup: Beyond empirical risk minimization. arXiv preprint
arXiv:1710.09412, 2017.
[18] Loshchilov I, Hutter F. Sgdr:
arXiv:1608.03983, 2016.

Stochastic gradient descent with warm restarts. arXiv preprint

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0667

Disease-Detection Based on AI Ear Tag
Using Body-Conducted Sounds in Pigs
Kazunari TSUCHIYA1;Shunsuke ISHIMITSU1, Natsumi OKADA1; Seiji YUMOTO1,
Osamu MIKAMI2,Hiroaki INOUE3,Mitsuyoshi ISHIDA3 and Daisuke TANAKA4
1
2
3

Graduate School of Information Sciences, Hiroshima City University, Japan

Division of Animal Health, National Agriculture and Food Research Organization, Japan

Division of Animal Metabolism and Nutrition, National Agriculture and Food Research Organization, Japan
4

Nihama College, National Institute of Technology, Japan

ABSTRACTWith the increase in farm size in Japan’s livestock industry, managing individual animals has become quitedifficult. Under such circumstances, disease detection becomes complicated, and a delay in detection
increases disease spread and severity. This study aimed to establish an early detection AI ear-tag system andidentify respiratory diseases in pigs using body-conducted sounds. An AI ear tag was used for physicalrobustness and classification algorithms in the pigpen. Additionally, using the porcine reproductive andrespiratory syndrome virus inoculation test, we confirmed the feasibility of using an AI ear-tag system fordetection. Finally, it was suggested that early detection of respiratory diseases could be achieved by assessing
acoustic features and classification methods.
Keywords: Body-conducted sound，Piezo sensor，Respiratory disease, Heart sound

1. INTRODUCTION
Economic loss due to respiratory diseases is a major problem in the livestock industry. The early
detection of symptoms is important to prevent the spread of livestock diseases. However, as farms
increase in size and the number of workers per livestock decreases, the early detection of diseases inindividual livestock has become difficult. To solve this problem, a technology that monitors individual
animals for various vital signs of livestock health and automatically detects and diagnoses
abnormalities is necessary to help avoid economic losses. This study focused on the early detectionof health anomalies, which is an important issue in managing group-housed livestock, such as
domestic pigs.
To increase the early disease detection in livestock, a method of diagnosing respiratory disease fromthe coughing sounds of pigs, using a microphone and camera installed in the barn, has been attempted[1]. However, when the number of pigs is large, the environmental noise caused by the pig itself has
a significant impact on the system’s diagnostic value. Therefore, there is a need for a system that issensitive enough to detect early onset signals of disease in noisy environments. Hence, a bodyconducted sound system that is robust to noise was employed. Figure 1 shows the outline of the
proposed method. The body-conducted sounds were recorded using an ear tag, and based on theserecorded sounds, biological information, such as respiratory sounds and heartbeat, was extracted andanalyzed using acoustic features to understand the pig’s health condition. Previous studies haveconfirmed that respiration and heartbeat can be extracted from body-conducted sounds collected using
a piezo sensor [2]. Then, respiration and heartbeat information were extracted even with signals
containing noise by preprocessing these signals using independent component analysis and adaptive
signal processing [3]. Additionally, using the porcine reproductive and respiratory syndrome virus
inoculation test, we confirmed the feasibility of acoustic analysis for obtaining the characteristics ofthe disease to make a diagnosis.
Here, the recording method, ear-tag sensor robustness, and data communication stability are1ishimitu@hiroshima-cu.ac.jp

reviewed. We also attempted to classify the time-series data recorded by the system into two classes:
normal (before inoculation) and diseased (after inoculation), using long short-term memory (LSTM).

Figure 1 – Outline of proposed method

2. DISEASE IDENTIFICATION SYSTEM
The workflow used to create the disease identification system is shown in Figure 2. First, an ear
tag sensor was used to collect the body-conducted sounds of livestock. Preprocessing was thenperformed on the recorded data to create an identification model. Subsequently, machine learning andother algorithms are employed to construct a model for disease judgment. In this study, we used an
ear-tag sensor to record the body-conducted sounds of pigs. During preprocessing, outliers andmissing data were removed, and the data was split for the next stage. To create the identification model,
LSTM was used to classify the time-series data obtained from the recording system into two classes:
pre- and post-affected.

Figure 2 – Workflow for disease discrimination model

3. RECORDING OF BODY=CONDUCTED SPEECH
3.1 Recording system configuration
Biological signals, such as respiration, change with infection. Therefore, it is necessary to collectbiological signals from each individual to identify morbidity. Additionally, the free movement of
livestock and background noise must be considered when recording the desired sound. Therefore, wedeveloped a wireless system to record body-conducted sounds that are robust against noise. A system
overview for recording body-conducted sounds of livestock is shown in Figure 1. The sound wasrecorded using an ear-tag-type sensor attached to the ear, and the recorded data was transmitted via
Wi-Fi3.2 Problems and Countermeasures in Recording system
There are 4 problems with this recording system. Table 1 lists each problem and countermeasuresused in this study. The measurements in Table 1 (a) increased the recording time by a factor of 3 to 8.
The measures in (b) and (c) confirmed a packet loss reduction, and the amplitude of the recorded
signal increased by a factor of 1.5. The old sensor [1] was damaged during the recording experimentbecause of pigs biting each other. However, with the improvement in (d), the sensor was no longer
damaged during the recording experiment.
Table 1 – Problems and countermeasures
ProblemsaShort recording time
(3 hours of continuous operation)

Countermeasures
Change of measurement method(addition of intermittent measurement
and low power consumption mode)b

Lots of packet loss

Wired use between PC and routerc

Weak recorded signal

Change sensor componentd

Low durability

Change sensor mechanism

4. DISEASE RECORDING OF BODY-CONDUCTED SOUND
4.1 Pasteurella Disease
In this study, we attempted to classify two classes of Pasteurella disease-infected pigs, before andafter the disease. Pasteurella disease is commonly referred to as various diseases caused by Pasteurella
species. Symptoms begin with the loss of energy and appetite, and in severe cases, the patient is prone
to coughing and heavy abdominal breathing. Acute cases present with a fever of 40 to 42 °C, andmany of them die within a few days after illness onset [5]. Animal experiments were conducted incompliance with Notification No. 71 of the Ministry of Education, Culture, Sports, Science, andTechnology and were properly conducted based on the 3Rs (replacement, reduction, and refinement)
concept of using alternative methods, reducing the number of animals used and reducing suffering,after review by the National Institute of Agricultural Science Animal Experiment Committee.
4.2 Recording environment of Body-Conducted Sound
In this experiment, 5-week-old SPF pigs (at the time of inoculation) were individually inoculatedwith 2 mL of Pasteurella sp. to record the body-conducted sounds of Pasteurella disease-infected pigs.
Individuals who did not have a fever of 40 °C or higher on the following day were re-inoculated. The
conducted body sounds were recorded at a sampling frequency of 20,000 Hz using an ear-tag sensor.
The ear-tag sensor integrates a piezo, battery, transmitter, amplifier, and housing. Figure 3 shows an
ear-tag sensor attached to a pig.

Figure 3 – Recording sensor

5. DISEASE IDENTIFICATION BY LSTM
5.1 Creation of Identification Model
Data from three 5-week-old pigs infected with Pasteurella disease were used to develop a modelthat classified the pigs into two classes, pre- and post-affected, using LSTM. The data for training,validation, and testing of the model was obtained from the pre- and post-affected data recorded by thesystem, and approximately 100 to 400 data pieces were extracted for each pig during 5 s of restingstate. Table 2 lists the classification error rates for training, validation, and testing during model
creation. However, data on the day of symptoms were used for the post-affected days based on theclinical symptoms and the results of diagnosis by veterinarians of the National Agricultural ResearchInstitute.
5.2 Disease Identification of another individual by the model
Using the model created from the data of the three individuals, we attempted to classify the dataof another individual into two classes: pre- and post-affected. The evaluation data were recorded
before, 3 days after, and 6 days after inoculation of a 5-week-old Pasteurella disease-infected pig.
Table 3 presents the results for each classification.
Table 2 – Classification accuracy
Training

Validation

Test(5s x 705)(5s x 201)

(5s x 103)

94.9 %

93.0 %

93.2 %

Table 3 – Classification result
3 days after

6 days after

inoculation

inoculation(5s x 65)

(5s x 177)

99.0 %

7.7 %

0.6 %

1.0%

92.3 %

99.4 %Classification

Pre-affected

Result(5s x 96)Pre-affected
After-affected

6. CONCLUSION
In this study, we improved the performance of a recording system using an ear tag sensor to record
body-conducted sounds in pigs and classified the time-series data recorded by the system into twoclasses, pre- and post-affected by the disease, using LSTM. First, the data collection efficiency
increased by a factor of approximately five, owing to the improved performance of the system. The
disease classification by LSTM was confirmed to be over 90% for training, validation, and testing inthree pigs. We also confirmed that the model is valid for other individuals.
In the future, we plan to conduct further clinical experiments to increase the amount of data forverification and to construct an actual system that can be provided to livestock farmers.

ACKNOWLEDGEMENTS
This study was supported by the Japan Racing and Livestock Promotion Foundation.

REFERENCES
1. Chung, Y., Oh, S., Lee, J., Park, D., Chang, H. and Kim, S., Automatic detection and recognition of pig
wasting diseases using sound data in audio surveil-lance systems, Sensors, Vol.13, No.10, pp.1292912942, 2013.
2. Cheng, Y., Narusawa, K., Iijima, S., Nakayama, M., Ishimitsu, S., Ishida, A., Mikami, O., “Developmentof an Early Detection System for Respiratory Diseases in Pigs” Biomedical Soft Computing and HumanSciences, Vol.24, No.2, 2019. p. 65-73,
3. Tsuchiya, K., Ishimitsu, S., Morinaga, Y., Cheng, Y., Narusawa, K., Nakayama, M., Mikami, O., Takagi,
M. and Inoue, H., Study on disease-detection methods using body-conducted sounds in pigs International
Journal of Biomedical Soft Computing and Human Sciences, Vol.25, No.2, 2021, p.49-56.
4. Takagi, M., What is Porcine Reproductive and Respiratory Reaction Syndrome (PRRS)?, , Japan Livestock
Industry Association, 2011.
5. Kashiwazaki, M., Kubo, M., Ogue, E., Shimizu, S., Degushi, E., FUruya, O., Yamaguchi, T., Hyology
and Hyoiatrics, 1999; KIndai Pub., Japan.

ABS-0694

Animal sound separation with unknown number of sources
Chulwon CHOI(1) , Kyungdeuk KO(2) , Hanseok KO(3) ,...
(1) Korea University, KOR, cchoi1022@korea.ac.kr(2) Korea University, KOR, kdko@korea.ac.kr(3) Korea University, KOR, hsko@korea.ac.kr

ABSTRACTAs environmental concerns have increased over the recent years, animal sound classification has been receiving
more attention. However, collecting a good animal sound database for this task is difficult. This is because
recorded animal audio files tend to be impure, as they contain other sounds such as environmental noise andaudio from other species. Such a dataset would not be suitable for testing animal sound classification models as
the mixed sounds may cause misclassifications. Source separation is a remedy that addresses this problem. If
source separation is successfully applied to separate these mixtures, a proper sound dataset can be constructed.
Over the years, several separation methods developed. Few source separation methods have been created toseparate animal sound mixtures. However, most source separation methods depend on knowing the number ofsources in the mixture audio file to work, even though this is rarely the case in the real world. While there
were studies on source separation for an unknown number of speakers, none applied to animal sounds. For thatreason, we propose to combine a source number estimator with a source separation method for an unknown
number of speakers. In this paper, we attempted to find the optimal method of animal sound source separation
for an unknown number of sources by training the SuDORM-RF, an efficient general purpose neural network
for audio source separation, on animal audio files with 2, 3, and 4 sources, and applying it in numerous ways.
We use the Gerschgorin disk estimator to estimate the number of sources in mixtures and use that knowledge
to determine the model that is best suited to separate that number of sources. We also assessed how recurrent
source separation works with animal sounds, as well as the performance of models trained for a higher numberof sources used on mixtures with a lower number of sources. Results show that the models trained for a higher
number of sources output false estimates alongside the acceptable estimates when applied to mixtures with alower number of sources due to the structure of the model. Experiments with the recurrent source separation
showed that ordinary methods applied in the speech separation for recurrent source separation cannot be applied
to animal sound separation. Finally, the usage of the Gerschgorin disk estimator to estimate the number ofsources in mixtures and determine the model to separate that number of sources proved to be the most stable
method to separate animal sound sources.

Keywords: Animal sound, Source separation, Source number estimation

1

INTRODUCTION

In recent years, environmental concerns have started attracting more attention. The usage of audio to detect andclassify animals is now something researchers are looking into, as auditory signals cover a much wider range
than vision in areas with little visibility such as forests. People have developed classification networks developed
specifically for that purpose [1] [2]. However, it is difficult to obtain good test data for these classification
networks, as most sounds in the wild are mixed. It is often the case that different sounds are in the mixture,
be it sounds from distinct species of animals, environmental noise, or the like. This prevents researchers fromgetting clean audio sounds for the testing of their classification models. The lack of clean testing data resultsin the likelihood of inaccurate classification models being passed as accurate. To solve this issue, there havebeen studies that generate animal sounds with Generative Adversarial Networks (GAN) to augment the data

[3]. However, even this requires a minimum amount of good data to start. To overcome this, we must performsource separation on the existing mixtures. Various methods for sound source separation exist, including ones
that use networks such as ConvTasNet [4], Dual-path Residual Neural Network (DPRNN) [5], and various otherneural networks. Unfortunately, these methods mostly focus on separating specific types of sounds, such as
separation of human speeches, separation of music, or separation of noise. None of these quite apply to animal
sound separation, as each of these tasks relies on specific features that make those separation tasks easier.
However, there are source separation methods that do not require prior knowledge about the sources, called
Blind Source Separation (BSS). These methods have little to no information about the sources of the mixture
and the mixing process, and therefore we can apply them to all types of sound mixtures. These methods
include Independent Component Analysis (ICA) [6], Principal Component Analysis(PCA) [7], and Non-negative
Matrix Factorization (NMF) [8]. Hassan et al. have tried out these different methods to separate frog sounds toovercome the problems of animal sound classification [9]. However even these are not ideal for animal soundseparation. While variations of BSS may perform as well as the state-of-the-art source separators, PCA andNMF are not known for their superior performance in the task. Variations of ICA are among the state-ofthe-art source separation methods, but ICA requires the number of sources to be known beforehand and stillcannot match a neural network trained to separate specific types of sounds. Therefore, it is ideal to createa source separation method specialized in separating animal sounds. Most source separation methods need toknow the number of sources beforehand to perform accurately. There have been attempts to tackle this problem
by separating an unknown number of speakers by Takahashi et al. [10]. and Nachmani et al. [11]. ]. in theirrespective works. Takahashi attempted this by using a binary classifier to determine if there is speech left in
the residual mixture after the separation of the strongest sound. Nachmani attempted this by running an activity
detector at the output of their model so that if there is no output on the Nth channel, there are fewer than Nsources. Unlike Takahashi’s model is only applicable to human speech as it uses a speech classifier, Nachmani’s
method appeared to apply to non-human sounds as well. We decided to research a source separation method anda source number estimator applicable to animal sounds and apply it to the existing source separation method to
produce a novel solution for animal sound separation with an unknown number of sources.

2

RELATED STUDIES

There are different separation methods when separating an unknown number of sources. Naively, there are two
methods for this task. The first is to create a source separation model for every number of sources and use asource number estimator to determine which model to use. The second is to recursively separate the mixture
for as long as the source number estimator indicates that there is more than one source in the residual mixture.
In speech separation, researchers have evaluated both methods.
Nachmani et al. created a model for each number of sources and used source number estimation to determine the model to use. They first processed the mixture with the model trained for the maximum number of
speakers and used an activity detector at the outputs until the activity detector indicated that there are activities
for every channel.
On the other hand, Takahashi et al. went for the opposite approach and created a singular model and abinary speech classifier to recursively separate speech mixtures until the speech classifier determined that there
was no speech in the residual mixture. Unlike Nachmani’s method, this method required only one model to be
trained.
Aside from the source separation methods, source estimation methods also exist. Well-known methods include Akaike’s information criterion (AIC) and minimum description length (MDL) [12]. ]. Another popularmethod is the Gerschgorin Disk Estimator, which uses the Gerschgorin radii and Gerschgorin criterion to estimate the number of sources in a signal [13].

3

METHODIn our experiments, we trained a source separation model for each number of speakers, ranging from 2 to 4. We
then used a source number estimator to determine the number of sources in the mixture and used the estimatedsource number to determine the model to use for source separation. We use the Gerschgorin Disk Estimator toestimate the number of sources and use the estimated number of sources to determine the model to separate
the mixture. The separation model is a model made by Tznis et al., using Successive Down-sampling andResampling of Multi-Resolution Features (SuDORM-RF), which is an efficient low-cost source separation modelthat still performs at state-of-the-art approaches [14]. We chose this model for its efficiency and performance,
as it has fewer parameters than most models, but still performs excellently. In the original work, Tznis etal. proved SuDORM-RF’s performance in speech and non-speech. However, they assessed the non-speech parton ESC-50 [15], a dataset for environmental sound classification, rather than specifically animal sounds. Thismeans that while this model has proven itself on distinct sounds, it has not yet done so with similar animals
that make similar sounds, such as varied species of frogs. This model is composed of an adaptive encoder anddecoder and uses a mask-based architecture to separate the mixtures.
3.1 Gerschgorin Disk Estimation
The GDE criterion works on the assumption that the Gerschgorin disks are into two distinct sets, where one set
is associated with the signal sources, and the other is associated with noise. The source separation method usedin this study is an improvement to the basic GDE.
We assume that the measured array data vector x(k) is composed of M sources corrupted by noise such thatM

x(k) = ∑ si (k)a(ωi ) + n(k) = A(ω)s(k) + n(k)

(1)

i=1

where A(ω) = [a(ω1 )a(ω2 )...(ωL )] is the direction matrix, and s(k) is the source vector and sm (k) is the amplitude of the mth source. We take the covariance matrix of x(k) and get
C = E[x(k)x(k)H ] = A(ω)Cs AH (ω) + σn2 I(2)where Cs is the covariance matrix of s(k), and σn2 is the variance of Gaussian noise.
To apply this to a single channel mixture, we must first rearrange the single channel mixture into a multichannel one. To achieve this, we use a delay process like that of Dong et al. [16], such that y)i = y(n + (i − 1)d)where y(n) is the single channel data, and d is a delay factor. We receive N channels such that


y(n)






y(n + d)




Y =
(3)

.






.


y(n + (N − 1)d)We perform convolution to this Y matrix with impulse response functions of FIR filters for each channel,
such that the final data matrix is as below
y(n)$\*h_1$ y(n + d)$\*h_2$ 




Y =
(4)

.






.


y(n + (N − 1)d)$*h_N$
With this process, we transform the single channel signal with the dimensions 1 x L into a multi-channel
signal with the dimensions N × (d + l − 1). Once we finish with this process, we can finally apply the GDE
method. We use the eigenvalue decomposition such that:

Figure 1. Architecture of the SuDORM-RF model, taken from [14]R = U∆U H(5)where U is a p-order unitary matrix constructed by the eigenvectors of covariance matrix R, and ∆ is a porder diagonal matrix with diagonal elements composed of the eigenvalues of R. and sort the eigenvalues in thedescending order. We reconstruct this as following

λ1′
0 ... ρ1




′

0
λ
...
ρ
2 
1

(6)
S = UdH RUd = 


 0
0 ...
. 


ρ1 ∗ ρ2 ∗ ... ρ p p
We use the criterion:p−1

GDE(k) = Ck − D(N)/(p − 1) ∑ Ci , k = 1, ..., p − 1

(7)

i=1

where D(N) is between 0 and 1. When the GDE becomes a negative value for the first time, we estimate thenumber of sources to be k-1.
3.2 SuDORM-RF
SuDORM-RF is primarily composed of three parts: the encoder, the separator, and the decoder. We show theoverarching structure of this model combined with the Gerschgorin Disk Estimation in Figure 1. The SuDORMRF model is a reproduction of the one in Tzinis’s article [14].
3.2.1 Encoder
In the encoder, the mixture goes through a one-dimensional convolution with kernel size Kε stride of Kε /2.
This transforms each convolved input audio segment of Kε samples a Cε )-dimensional vector representation
where Cε ) is the number of output channels of the 1-D convolution. We apply ReLU to force the output of theencoder to be non-negative. The output of the encoder is the encoded mixture representation vx , wherevx = E(x) = ReLU(Conv1DCε,Kε,Kε/2 (x))
The vx is passed to the separator after this step.

(8)

Figure 2. Architecture of U-ConvBlock, taken from [14]3.2.2 Separator
When the vx , the mixture representation gets to the separator module, it goes through another 1D convolution
and a Layer Normalization[17] layer that uses the moments to extract the temporal dimension for each channel
separately. The resulting output isy0 = Conv1DC1,1 (LN(vx ))(9)y0 then goes into the U-ConvBlocks, which resembles the depth-wise separable convolution with a skip connection like in ConvTasNet, but instead of performing a regular depth-wise convolution, U-ConvBlocks extract
information from multiple resolutions using Q successive temporal down-sampling and Q up-sampling operations
like a U-Net architecture [18]. Figure 2, from Tzini’s article, provides an abstract view of the U-ConvBlocks.
There can be a variable number of U-ConvBlocks in the model, but in the model used for this study, weused 16 U-ConvBlocks. The U-ConvBlocks extract information from multiple resolutions. We aggregated theextracted information by applying a 1-D convolution for each source on the transposed feature representation
yTByT B. These becomezi = Conv1DC1,ε,1 (yTB )T(10)

where z is the latent representation. This step has proven to make the training process more stable than whenusing the activation from yB to estimate the masks. . The SoftMax operation combines the latent representations
zi to estimate the mask estimates mi . We apply the masks to the mixture representation to estimate the latent
representations v̂i for each source by multiplying the encoded mixture representation element-wise with themasks.
v̂i = vx ⊙ m̂i(11)

where a ⊙ b is element-wise multiplication. The resulting v̂i proceed to the decoder section.
3.2.3 Decoder
In the decoder section, we transform the latent space representation v̂i back into the time domain by applying
a 1-D convolution decoder to get the estimated sources:sˆi = ConvTr1DCε,Kε,Kε/2 (v̂i )

(12)

4

EXPERIMENT

4.1 Dataset and Evaluation method
The dataset used for these experiments was animal sound files collected through recording and web crawling.
It consists of fifty animal classes, recorded at 44100Hz, cropped to a two-second duration, with one hundred
samples per class. We did not add any noise to the training dataset. We mixed the audio files randomly
with no overlapping classes. We generated 5000, 2000, and 1000 samples for training, validation, and testing,respectively.
We used Si-SDR, SAR, SDR, and SIR for evaluation [19], but the primary concern for this project was the
Si-SDR, the scale-invariant signal-to-distortion ratio.
4.2 Model Configuration
For the encoder and decoder modules, we used the same configuration as Tzinis et al. did, meaning kernel size
Kε = 21, the number of basis equal to Cε = 512, and the input number of channels for each U-ConvBlock is
C = 128, and the number of successive resampling operations Q = 4, and the expanded number of channels
CU = 512. In each subsampling operation, the temporal dimension reduces by half and the depth-wise separable
convolutions have the kernel length of KU = 5 and stride of SU = 2. We used 16 U-ConvBLocks, and the Adam
optimizer for training[20], with an initial learning rate of 0.001 and set to decrease by a factor of five every
fifty epochs. In total, we trained three models for five hundred epochs each. A model for two sources, a modelfor three sources, and a model for four sources. We further trained the two sources model to one thousand
epochs to determine if the amount of training after five hundred epochs would be effective.
Another experiment conducted was to recurrently separate mixtures using the model trained for two sources
until the model separates the estimated number of sources. Unlike Takahashi et al., it was impossible to use
a classifier and recurrently separate, as the number of animal sounds proved to be impossible to get a proper
binary animal sound classification. The attempt failed as the classifier continued to tell the model to separateeven when fed the separated output of non-mixtures. Therefore, we used the source number estimation to set
the number of recurrent source separations.

5

RESULTS

5.0.1 Source Separation with different number of sources
The experiment results from Table 1 showed that the overall score gets lower with more sources. However, it
was However, Table 2 showed that when we use a model trained for a higher number of sources to separate
mixtures with a smaller number of sources, the Si-SDR does not go down, and instead went up. This meansthat we can use the models trained for a higher number of sources to separate mixtures with a lower numberof sources, just as Nachmani et al. did, using activity detectors. However, due to the structure of the model,
the model outputs a higher number of sources than necessary. Among the outputs, two of them were proper,
and the other was not.
Source-Number

SAR

SDR

Si-SDR

SIR

2

19.51406

17.09192

16.02764

23.61402

3

14.52067

11.41481

10.20743

16.93783

412.50392

9.16902

7.88399

14.69545

Table 1. Result of mixed audio source separators from models with two sources to four sources.

5.0.2 Recurrent Source Separation
We discovered that, unlike speech separation, source separation for animal sounds does not separate singularsources individually. They separate them as mixtures, so separation of a 4-source mixture would require a two-# of sources model was trained for# sources in Mixture

SAR

SDR

Si-SDR

SIR

3

2

12.75536

12.30959

10.24097

26.03803

3314.52067

11.41481

10.20743

16.93783

Table 2. Result of using models trained for more sources to separate mixtures with fewer sources.

source separator to separate the mixture, and then separate both resulting mixtures, instead of separating thesame mixture over and over. If we do not do this, at least two sources will remain mixed, even if you recurrently perform source separation on the same mixture repeatedly. We performed the recurrent source separation
experiment for both 3 and 4 sources. As a result, we confirmed that separating a 4-source mixture into two
mixtures and separating each of the two mixtures gave results superior to that of the sources separated by amodel trained to separate four sources. When separating mixtures with three sources, this task became moredifficult. Initially, we tried separating mixtures with average power, but this showed varied results.
5.1 Conclusion
In this paper, we proposed to use the Gerschgorin Disk Estimation method to estimate the number of sources inan animal sound mixture and optimize the performances of source separators for animal sounds. We found that
this is important, as we cannot apply the majority of the techniques used for speaker separation to animal soundseparation, as can be seen with the recursive separation method. Without the right estimation of the number ofsources, the separation models such as the SuDORM-RF generate masks for an incorrect number of sources.
This leads to unwanted scenarios where models trained for different numbers of sources are

ACKNOWLEDGEMENTS
This work was supported by Korea Environment Industry & Technology Institute(KEITI) through Exotic Invasive Species Management Program, funded by Korea Ministry of Environment(MOE)(2021002280004)

REFERENCES[1] Donghyeon Kim, Younglo Lee, and Hanseok Ko. Multi-task learning for animal species and group categoryclassification. In Proceedings of the 2019 7th International Conference on Information Technology: IoT and
Smart City, pages 435–438, 2019.
[2] Kyungdeuk Ko, Sangwook Park, and Hanseok Ko. Convolutional feature vectors and support vector machine for animal sound classification. In 2018 40th Annual International Conference of the IEEE Engineeringin Medicine and Biology Society (EMBC), pages 376–379. IEEE, 2018.
[3] Sangwook Park, Mounya Elhilali, David K. Han, and Hanseok Ko. Amphibian sounds generating network
based on adversarial learning. IEEE Signal Processing Letters, 27:640–644, 2020.
[4] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech
separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8):1256–1266, 2019.
[5] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for timedomain single-channel speech separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 46–50. IEEE, 2020.
[6] Seungjin Choi, Andrzej Cichocki, Hyung-Min Park, and Soo-Young Lee. Blind source separation andindependent component analysis: A review. Neural Inf. Process. Lett. Rev, 6, 11 2004.

[7] Juha Karhunen, Liuyue Wang, and Ricardo Vigário. Nonlinear pca type approaches for source separation and independent component analysis. IEEE International Conference on Neural Networks - Conference
Proceedings, 2, 05 1996.
[8] Cédric Févotte, Emmanuel Vincent, and Alexey Ozerov. Single-Channel Audio Source Separation with NMF:
Divergences, Constraints and Algorithms, pages 1–24. 03 2018.
[9] Norsalina Hassan and Dzati Ramli. A comparative study of blind source separation for bioacoustics sounds
based on fastica, pca and nmf. Procedia Computer Science, 126:363–372, 01 2018.
[10] Naoya Takahashi, Sudarsanam Parthasaarathy, Nabarun Goswami, and Yuki Mitsufuji. Recursive speech
separation for unknown number of speakers. CoRR, abs/1904.03065, 2019.
[11] Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers, 2020.
[12] M. Wax and T. Kailath. Detection of signals by information theoretic criteria. IEEE Transactions on
Acoustics, Speech, and Signal Processing, 33(2):387–392, 1985.
[13] Olivier Caspary, P. Nus, and T. Cecchin. The source number estimation based on gerschgorin radii. volume 4, pages 1993 – 1996 vol.4, 06 1998.
[14] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo RM -RF: Efficient networks for universal audio
source separation. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing(MLSP). IEEE, sep 2020.
[15] Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual
ACM Conference on Multimedia, pages 1015–1018. ACM Press.
[16] Zhi Dong, Junpeng Hu, Bolun Du, and Yunze He. Improvement of source number estimation method forsingle channel signal. PLOS ONE, 11(10), OCT 13 2016.
[17] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[18] Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-toend audio source separation. arXiv preprint arXiv:1806.03185, 2018.
[19] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source
separation. IEEE transactions on audio, speech, and language processing, 14(4):1462–1469, 2006.
[20] Diederik P Kingma and Jimmy Ba.
arXiv:1412.6980, 2014.

Adam: A method for stochastic optimization.

arXiv preprintABS-0701

Animal Sound Separation using Dual-Path RNN and Classifier Loss
Yongmin KIM(1) , Chulwon CHOI(2) , Yuanming LI(3) , Hanseok KO(4)(1) Korea University, KOR, dydals3819@korea.ac.kr(2) Korea University, KOR, cchoi1022@korea.ac.kr(3) Korea University, KOR, lym7499500@korea.ac.kr(4) Korea University, KOR, hsko@korea.ac.kr

ABSTRACT
Source separation is a task that aims to separate multiple sounds from mixed audio. Recent source separation
studies suggest that they process the separation primarily through a time-domain approach and that most are
carried out mainly on speech separation in a clean environment. A few studies of animal sound separation havemade advances that would contain various environmental noises. To address this issue, we propose a novel
method to separate the animal sounds among background noise and two overlapping sources. The proposed
method focuses on taking into account the real-world environment by adding background noise. The proposedmodel structure adds a classification network to the dual-path recurrent neural network (DPRNN). In particular,
the mixed audio becomes separated through a mask of the single source estimated within the DPRNN. The
separated source is then converted into a mel-spectrogram for feature representation. We use the resulting
feature as input to a classification network for classification for verification of the separation performance. The
experimental results confirm that the proposed method achieves better separation performance than when using
DPRNN alone.

Keywords: Animal sound separation, deep learning, time domain

1

INTRODUCTION

Recent studies on deep learning-based source separation use a time-domain approach [1] [2] [3]. Unlike thefrequency-domain method, the time-domain method is helpful for learning long-term dependency. In addition,
the phase and magnitude of the signal are not decoupled, preventing an upper bound to the accuracy during
the phase reconstruction of the clean signals. Therefore, it is better to configure a recurrent neural network(RNN) [4] than a general convolution neural network (CNN). Since research on speech or instrument separationis active and research on animal sound is scarce, we research sound separation for animal sounds.
The current time-domain method is mainly divided into the adaptive front-end approach and the direct regression approach. The adaptive front-end approach replaces the Short-Time Fourier Transform (STFT) andinverse STFT (ISTFT) with a differentiable transform. The adaptive front-end approach consists of an encoderand decoder, and it gets updated by learning along with the separation network. It aims to create an optimized
audio representation. On the other hand, the direct regression approach uses one dimensional convolution (1-D
Conv) to pass input signals [5] [6]. The reconstruct part also uses 1-D Conv.
There are challenges in trying to separate animal sounds from the real-world environment. We focus on two
main challenges: diversity of animal classes and variation of background noises. The number of the targetedanimal species is considerably large, and some species sound similar. Also, since background noise varies, it isessential to separate the animal sounds as accurately as possible.
In this paper, we propose DPRNN-based animal sound separation [7]. The proposed method is mainlycomposed of DPRNN and classification networks. We select DPRNN because the model size is smaller thanother models, and the performance is satisfactory. When used in a real-time-based real-world environment, theprocess should not be computationally high-loaded, so a small model size is preferable. This network uses

Figure 1. The overall network structure including separation network and classification network.
a front-end approach and the RNN for a separate network. The entire network divides into three functional
processes: Segmentation, DPRNN block, and Overlap-Add. In Segmentation, the input divides into N chunks
and feeds to the DPRNN block. DPRNN block divides into intra-chunk and inter-chunk for local and global
modeling. After passing through the block, the signals separated through the overlap process get reconstructed.
Next, we construct a classification network behind DPRNN. There are many classification networks [8] [9][10], but among them we use temporal temporal efficient neural network (TENet) [11], as it is an efficientmodel, unlike other models. This network constructs an efficient neural network using depth-wise temporalconvolutions. While the model is lightweight, it precisely captures temporally informative features. We also
use the focal loss as the classification loss. Balanced learning gets achieved by adding more weight to thedifficult or easily misclassified samples through focal loss. In Section 4, we can see that SDRi gets better when
the classification network gets added, proving that adding a classification network improves source separation
performance.

2

THE MODELThe proposed model divides into a separation network and a classification network. The separation network
consists of an encoder, a decoder, and a DPRNN block. After passing through the network, mel-spectrogram
representation is created. We insert the input represented by the mel-spectrogram into the classification network.
The classification network constructs an inverted bottleneck block (IBB) through convolution.
2.1 Encoder and Decoder
Time-domain 3-mixed audio E ∈ R1×T is input as the encoder. It passes through convolution layer and ReLU′
function to represent it as a learnable feature E ∈ RW ×H .
E ′ = ReLU(conv1d(x)).

(1)′

An element-wise multiplication between estimation mask mk of the source k and the encoder output E isentered as the decoder input. Then it passes through the transposed convolution layer.
D′ = conv1d-transpose(mk ∗ E ′ )).

(2)

Figure 2. Detailed structure of DPRNN block.
′′

The form of D represents D ∈ R1×T and has the same shape as E.
2.2 DPRNN
The DPRNN divides into segmentation, DPRNN block, and overlap-add. In the segmentation, the encoder
output is used to create N chunks of size C that overlap each other by 50 %. We denote the output ofsegmentation as h ∈ RW ×K×Nk . W is the channel, K is the chunk size, and Nk is the number of chunks.
After dividing the input into chunks, then the output of the segmentation enters the DPRNN block. In Figure
2, the DPRNN block has the same internal structure as intra-chunk and inter-chunk. The output of segmentationenters as the input and passes through in the order of bi-directional RNN, FC layer, and LayerNorm. Then,a residual connection is performed between the input h and the output of LayerNorm [12]. The inter-chunk′process that follows is the same as before. We denote the output of the DPRNN block as h ∈ RW ×K×Nk . In′′′
overlap-add, the form of h is converted back to the form of h ∈ RW ×H .
2.3 TENet′′
The output of the decoder gets converted into a mel-spectrogram. This output will be F ∈ RM×T . M is the′frequency and T is the time. After conversion, it enters the input of TENet. This network is inspired by
the concept of inverted residual structure in MobileNetV2 [13]. The TENet key point is an inverted bottleneck
block (IBB) containing the bottleneck temporal depth-separable convolution with residuals. IBB consists of 9×1
depthwise convolution and 1 × 1 convolution, and strides are divided into 1 and 2. Batch normalization (BN)and ReLU function are included between convolutions. After IBB block average pooling layer, FC layer, andsoftmax function are performed. The output result is Y ∈ R1×N . N is the total number of classes.
2.4 Loss
We use the pair-wise loss of the estimated mask, the true mask, and the classification loss. Pair-wise loss isa value that minimizes the SI-SDR [14] value by considering the number of all possible separation cases. In
Equation 3, i and j get mapped according to the number of sources. If the number of sources is 2, the numberof possible cases is 2. Therefore, we only need to find the minimum value of Ci j . yi (t) and x̂ are true target
source. si (t) and x are estimated target source.
LSI−SDR = Ci j = −SI-SDR(yi (t), s j (t))
α=

x̂T x
∥x∥2

SI-SDR(x̂, x) = 10log10= argmin∥αx − x̂∥2∥αx∥2∥αx − x̂∥2

(3)

(4)

α

The classification loss uses a focal loss. The formula for focal loss is the same as equation 5, and α and γ
are 1 and 5. pt is the true label of passing the softmax.
LCLS = −αt (1 − pt )γ log(pt )(5)Therefore, the total loss can be expressed as equation 5, where β > 0 is the hyper-parameter for balancing
the losses.
Ltotal = min(LSI−SDR ) − β LCLS
(6)

3

EXPERIMENT SETUP

3.1 Dataset
We are tasked with separating and recognizing animal sounds. Since our focus is on the real-world environment,
we proceed 3-mix separation with added background noise. We collect the dataset for animal sounds andbackground noise through recording and web crawling. The collected animal sounds are cropped into 2-second
samples with 50 % overlap to construct data. The number of classes of animals is 52, and background noise
includes natural phenomena such as wind, rain, and lightning. Each class has 100 samples, and the backgroundnoise has 3000 samples. The mixed audio is randomly mixed without overlapping. The hours of data in the
training, validation, and test datasets are about 45, 11, and 11 hours, respectively. The training and validation
datasets are randomly sampled between 0dB and 20dB relative levels of animal sound and background noise.
The test dataset are sampled at relative levels of 0, 5, 10, 15, and 20dB, respectively. The sampling rate of thedata is 16 kHz.
3.2 Model Configuration
The encoder uses a 64 convolution filter with a kernel size of 16 and a stride of 8. The decoder takes the same
kernel and stride factor as the encoder. In the segmentation part of the separation network, we set the chunksize to 100 and overlap to 50 %. Two DPRNN blocks are stacked. Each DPRNN block is composed of Intraand inter chunks, which contain bi-directional LSTM in them. The bi-directional LSTM [15] has a hidden unitof 128. In the classification network, IBB blocks of 12 are used. When converting to mel-spectrogram, thewindow length is 400, the hop length is 256, and the number of mel is set to 80.
3.3 Experiment Configuration
We train up to 100 epochs, and the batch size is 16. Each epoch takes about 0.5 hours, and a single RTX 3080ti
GPU is used. The learning rate is set to 1e−3 , and adam [16] is used as the optimization. The gradient clipping
is applied with a maximum L2-norm of 5. The DPRNN network is trained with utterance-level permutation
invariant training (uPIT) [17] [18] to maximize scale-invariant SNR (SI-SNR). If the validation error continues
to increase for 10 epochs, the learning gets stopped. The performance of the network is measured by signal-todistortion ratio improvement (SDRi) [19] [14].
Table 1. Comparison 3-mixed audio results between using only DPRNN and adding a classification network.
CLS stands for Classification. SDRi is used for evaluation. s1 and s2 are animal sounds and N is backgroundnoise.
Method4

Training

SDRi - s1, s2, N (dB)Model size
0dB

5dB

10dB

15dB

20dB

DPRNN

end-to-end

1.3M

12.41314.70616.00715.34813.456DPRNN + CLS

end-to-end

1.4M

12.455

14.73

16.01

15.36

13.486DPRNN + CLS

pre-training

1.4M

12.48314.75

16.04115.402

13.586DPRNN + CLS

fine-tuning

1.4M

12.39414.69916.01415.39

13.525RESULTSIn the overall experiment, CLS is trained in various ways and compared to each other. The training methods
employed consist of end-to-end, pre-training, and fine-tuning. From the results in Tables 1, 2, and 3, it can beseen that DPRNN+CLS performs better overall compared to when DPRNN is used alone. It is also observedthat the smaller the SNR the greater the performance improvement. This means that even with some noise, itis well separated. It appears to be the best under pre-training CLS. In the model size, DPRNN+CLS is 0.1 M

Table 2. Comparison 3-mixed audio results between using only DPRNN and adding a classification network.
SDRi evaluation is made for two sources.
MethodTraining

SDRi - s1, s2 (dB)Model size
0dB

5dB

10dB

15dB

20dB

DPRNN

end-to-end

1.3M

13.516

13.61314.65415.53315.883DPRNN + CLS

end-to-end

1.4M

13.64513.76414.81715.68316.045DPRNN + CLS

pre-training

1.4M

13.75513.85

14.89215.76816.143DPRNN + CLS

fine-tuning

1.4M

13.58113.71814.801

15.69716.068

Table 3. Comparison 2-mixed audio results between using only DPRNN and adding a classification network.
MethodTraining

SDRi - s1, s2 (dB)Model size
0dB

5dB

10dB

15dB

20dB

DPRNN

end-to-end

1.3M

12.82513.30514.45115.38815.684DPRNN + CLS

end-to-end

1.4M

12.923

13.42

14.57915.48715.774DPRNN + CLS

pre-training

1.4M

13.33113.75114.80215.52316.012DPRNN + CLS

fine-tuning

1.4M

13.261

13.61214.79815.49

15.952larger than just DPRNN alone.
Tables 2 and 3 present performance comparisons between 2-mixed and 3-mixed audio cases. In particular,
Table 2 shows the SDRi evaluation for animal sounds in 3-mixed audio. Table 3 shows the SDRi evaluation
for animal sounds in 2-mixed audio. By comparing the results of Table 2 from Table 3, it is observed that the
animal sounds are separated better when separating 3-mixed compared to 2-mixed. We find that there is little
difference in model size, and the performance is satisfactory under various SNRs. Moreover, it is clear that theoverall performance improved when the classification gets added.

5

CONCLUSIONSIn this paper, we proposed a network to better separate animal sounds and background noise from mixed audio.
This network adds a TENet to DPRNN. By using the depth-wise convolution of TENet and bi-directional LSTM
of DPRNN blocks, it is possible to capture the temporal feature information between the contiguous data, whichis advantageous for learning. With this, we were able to estimate the mask of individual sources well in mixed
audio. Experimental results confirmed that the proposed model separates better than using DPRNN alone. This
result shows the efficiency of the proposed approach in animal sound datasets.

ACKNOWLEDGEMENTS
This work was supported by Korea Environment Industry & Technology Institute(KEITI) through Exotic Invasive Species Management Program, funded by Korea Ministry of Environment(MOE)(2021002280004)

REFERENCES[1] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech
separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8):1256–1266, 2019.

[2] Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single-channel
speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 696–700. IEEE, 2018.
[3] Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-toend audio source separation. arXiv preprint arXiv:1806.03185, 2018.
[4] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks.
arXiv preprint arXiv:1609.01704, 2016.
[5] Szu-Wei Fu, Tao-Wei Wang, Yu Tsao, Xugang Lu, and Hisashi Kawai. End-to-end waveform utterance
enhancement for direct evaluation metrics optimization by fully convolutional neural networks. IEEE/ACM
Transactions on Audio, Speech, and Language Processing, 26(9):1570–1584, 2018.
[6] Francesc Lluís, Jordi Pons, and Xavier Serra. End-to-end music source separation: is it possible in the
waveform domain? arXiv preprint arXiv:1810.12187, 2018.
[7] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for timedomain single-channel speech separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 46–50. IEEE, 2020.
[8] Donghyeon Kim, Younglo Lee, and Hanseok Ko. Multi-task learning for animal species and group categoryclassification. In Proceedings of the 2019 7th International Conference on Information Technology: IoT and
Smart City, pages 435–438, 2019.
[9] Kyungdeuk Ko, Sangwook Park, and Hanseok Ko. Convolutional feature vectors and support vector machine for animal sound classification. In 2018 40th Annual International Conference of the IEEE Engineeringin Medicine and Biology Society (EMBC), pages 376–379. IEEE, 2018.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[11] Ximin Li, Xiaodong Wei, and Xiaowei Qin. Small-footprint keyword spotting with multi-scale temporalconvolution. arXiv preprint arXiv:2010.09960, 2020.
[12] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
arXiv:1607.06450, 2016.

Layer normalization.

arXiv preprint[13] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2:
Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 4510–4520, 2018.
[14] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr–half-baked or well done?
In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
pages 626–630. IEEE, 2019.
[15] Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages
37–45, 2012.
[16] Diederik P Kingma and Jimmy Ba.
arXiv:1412.6980, 2014.

Adam: A method for stochastic optimization.

arXiv preprint[17] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deepmodels for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), pages 241–245. IEEE, 2017.

[18] Morten Kolbæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen. Multitalker speech separation withutterance-level permutation invariant training of deep recurrent neural networks. IEEE/ACM Transactions
on Audio, Speech, and Language Processing, 25(10):1901–1913, 2017.
[19] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source
separation. IEEE transactions on audio, speech, and language processing, 14(4):1462–1469, 2006.

PROCEEDINGS of the24th International Congress on Acoustics
October 24 to 28, 2022 in Gyeongju, Korea

ABS-0754

Class-conditional WaveGAN-based data augmentation scheme foranimal sound classification
Eunbeen KIM1; Jaeuk MOON2; Jonghwa SHIM3; Eenjun HWANG4*
1, 2, 3, 4

School of Electrical Engineering, Korea University, South Korea

ABSTRACT
Even though recent deep learning-based models have shown outstanding performance, they require a large
amount of high-quality data for training. Recently, a generative adversarial network (GAN) has demonstrated
the potential to generate virtual data based on small amounts of real data. However, in a multi-classenvironment, the original GAN is expensive as it requires constructing a separate generative model for eachclass. In this paper, we propose a class-conditional WaveGAN-based data augmentation scheme. In particular,
we want to generate sounds of animals belonging to the same order in biological classification using a single
model. To do that, we first learn common features from animal sound data of all target classes. Then, byembedding class label information into the WaveGAN, we generate virtual animal sound data that have theunique characteristics of each of the target classes. To effectively learn the characteristics of real animal
sounds, we use multi-resolution short-time Fourier transform loss. To evaluate the performance of the
proposed scheme, we compare it with traditional signal augmentation methods. Experimental results showthat our scheme effectively generates realistic animal sounds, and achieves better performance in animal
sound classification than other signal augmentation methods in terms of various classification metrics.
Keywords: Data augmentation, GAN, Animal sound classification, Multi-class environment

1. INTRODUCTION
Over the past decades, various industrial activities by humans in the natural environment have had
negative impact on wildlife animals. For example, excessive urban development has fragmented
animal habitats and obstructed their migration (1). In order to reduce such damage and determine
conservation policies for endangered animals, various studies have been conducted to monitor the
status of wildlife (e.g., occurrence, migration, and geographical distribution) . However, manual
wildlife monitoring has required lots of time and labor. As an alternative, automatic wildlife
monitoring based on trap cameras and acoustic sensors is attracting much attention (2).
Animal sound classification (ASC) is one of the essential tasks in automatic wildlife monitoring
because it can identify animals by sound. ASC can also be used effectively even in cases where
identification is difficult by appearance, such as small, nocturnal, or camouflaged animals. As in othersignal processing fields, deep learning-based models such as Convolutional Neural Network (CNN)
have been mainly used for ASC (3). Even though they can achieve outstanding performances byextracting feature representation suitable for sound classification, their performance highly relies on
the quality and quantity of the animal sound data. Collecting a sufficient amount of high-quality
animal sound data is expensive and time-consuming. If the amount of animal sound data is insufficient(i.e., data shortage problem) or the data contains significant noise, their classification performance
will degrade seriously due to improper learning. This data shortage problem becomes morechallenging especially when the target animal is difficult to observe, like rare species.
One representative approach to solve the problem of data shortage is data augmentation. Data
gichanac@korea.ac.kr
jaewookmo@korea.ac.kr
3
indexlibrorum3822@korea.ac.kr
4* ehwang04@korea.ac.kr, corresponding author
12

augmentation applies various transformations to existing data to increase the number of data andutilize them as training data. It can prevent overfitting that causes classification performancedegradation and enhance the generalization ability. However, their use in animal sounds may belimited. For instance, sounds of various anuran species have different pitches or speeds, but traditional
methods such as pitch scaling and time stretching dismiss the unique characteristics of each species,
which can even degrade the ASC performance. In other words, in order to properly transform the data,
traditional augmentation methods require prior knowledge of the sound of the target animals.
A more recent approach to data augmentation is to use a Generative Adversarial Network (GAN)(4) that generates realistic virtual data by learning real data distribution. So far, it has shown
outstanding generative performance in various domains such as image, time series, and music (5, 6).
A typical GAN consists of two networks, a generator and a discriminator, and utilizes the adversarial
learning process of the two networks to generate virtual data representing the unique characteristics
of real data. Owing to this ability, GAN-based augmentation schemes have been used in various signal
processing fields (7, 8).
Despite of their potential in signal augmentation, GAN-based augmentation schemes still facesome challenges when it comes to multi-class classification task such as ASC. For effective
augmentation, virtual data must be generated for each animal sound class. As this requires a separate
GAN model for each animal sound class, we need to train the GAN mod el using data from that class.
As in rare animals, when the sound data of a class is not sufficient to train the GAN, the quality of thegenerated data is very low, and in our experiment, it was more like noise than animal sound. With
such low-quality virtual data, we cannot expect good ASC performance.
One possible solution to alleviate this problem is to use multiple animal sound classes together intraining so that they learn their common features. At the same time, by embedding information onclass labels to the GAN, we make the GAN learn distinct feature of each animal sound class . Hence,in this study, we propose a new animal sound augmentation scheme for ASC that generates virtual
animal sound data based on class labels. To do this, we use WaveGAN, a popular GAN-based modelfor generating waveform data (6) and generate data of a specific class based on all animal soundclasses by adding class conditions to the WaveGAN. Here, we use multi-resolution short-time Fouriertransform (STFT) loss to imitate the principal characteristics of real animal sounds effectively. To
evaluate the effectiveness of the proposed scheme, we performed various experiments to comparewith other data augmentation methods and measured ASC performance in terms of popularclassification metrics.
The rest of this paper is organized as follows. In Section 2, we introduce the related works of ASC
and data augmentation. In Section 3, we concisely describe the proposed scheme used in animal soundgeneration. We present experimental configurations and some of the results in Section 4. Lastly, we
discuss the conclusion in Section 5.

2. RELATED WORKS
Recently, with the remarkable advances in deep learning techniques, CNNs have been dominantly
used in audio classification tasks due to their advantages such as trainable feature extraction, complex
pattern recognition capability, and robustness to noise. For instance, Ko et al. (9) achieved excellent
classification performance by extracting convolutional feature vectors containing the perceptual
representation of audio data through pre-trained CNN. Salamon et al. (10) used CNNs to classify
large-scale bird sounds using spectrograms, which are frequency-time characteristics (image-like
representation). They showed that spectrogram-based CNN outperformed the support vector machineclassifier with Mel-frequency cepstral coefficients feature in identifying large-scale bird species.
However, deep learning-based approaches require sufficient data for training, otherwise they sufferfrom severe performance degradation.
Data augmentation is a widely used method to address this data shortage problem. For instance,
pitch shifting, time stretching and noise addition are popular augmentation operations that expand thefrequency and temporal diversity of signal data. Time stretching changes the speed of a signal without
affecting its pitch. On the contrary, pitch shifting changes the pitch without affecting the speed. Noise
addition transforms target signals by adding various types of noise. For example, white noise, which
is a random signal with uniform intensity at different frequencies, can be used to improvegeneralization capacity. Lasseck et al. (11) showed that even simple data augmentation like noise
addition could improve the classification performance of CNN in bird sound classification. Similarly,
Nanni et al. (12) performed extensive experiments using bird and cat sound datasets with diverse dataaugmentation methods including pitch shifting, time stretching and noise addition for CNN. Although
they found augmentation techniques applicable to waveforms or frequency-time features, they warn
against the naive use of data augmentation protocols. According to their experiments, someaugmentation techniques were useless or even worsened the classification ability. This indicates thatthe characteristics of the data domain should be considered when choosing augmentation techniquesand determining their transformation strength.
Due to the domain-dependence of existing augmentation methods, GAN-based data augmentation
schemes have been suggested in several sound classification tasks. For example, Madhu et al. (13)suggested EnvGAN, which produces a waveform with four seconds length suitable for environmental
sound classification and demonstrated excellent performance on three environmental sound datasetsby combining EnvGAN and CNN classifiers. Bouallegue et al. (14) proposed a data augmentation
scheme based on Wasserstein GAN (WGAN) for electrical signal data of the brain. They proved that
artificial electrical signal data of WGAN have high similarity with real data in both time and frequency
domains and significantly improved the training effectiveness of classification model. Recently,
WaveGAN has been proposed for generating audio waveforms in various data domains such as
birdsongs and piano sounds (6). As WaveGAN generates realistic virtual data by learning real data
distribution without delicate consideration for the data domain, the aforementioned domain
dependence problem of the existing augmentation methods can be solved.

3. METHOD
In this section, we first describe in detail the structure of the proposed scheme. Then, we present
the loss functions used for training.
3.1

Architecture of Proposed Model: Fully-Connected Layer

Real Data: Embedding Layer: Channel-Wise Concatenation
Virtual Data

Latent Vector

Generator

DiscriminatorLabel Information

Figure 1 – Overall architecture of class-conditional WaveGAN
For animal sound generation, we propose a class-conditional WaveGAN. WaveGAN can effectively
generate a variety of perceivable sounds with a length of one second (16 kHz), such as drum sounds,
birdsongs, and piano sounds (6). However, Although general GAN models including WaveGAN use
multi-class data for training, they cannot selectively generate data of the desired class. To generate
virtual data for each animal sound class individually, we might construct a GAN model for each class.
In this case, we need to train each model using data from that class. This approach faces a data shortage
problem when there is not enough data belonging to each class, such as rare animal sounds, and suffersfrom the cost and hassle of constructing multiple models. In particular, we want to generate sounds ofanimals belonging to the same order in biological taxonomy (e.g., anura or passeriformes) using asingle GAN model. As those animals make sounds in a similar way, their sounds will have bothcommon and unique features. By training a GAN by adding a class label representing the animal ofthat sound to the sound data of various animals, the GAN can generate the desired animal sound.
Figure 1 shows the overall architecture of the proposed class-conditional WaveGAN for animal
sound generation. The model consists of a generator that generates a specific waveform of a givenlabel and a discriminator that determines the authenticity of the input waveform with the given label.

The generator receives a random latent vector 𝑧 from a normal distribution between 0 and 1 and takeslabel condition information 𝑐 indicating the particular class it wants to generate. To allow thegenerator to receive the extra condition, we added an embedding layer that transforms the input label
into an embedding vector. Since this embedding vector is trainable, it can embed a richer
representation than a constant label value by learning. Then, the input latent vector and the embedding
vector are combined into feature vectors through fully connected layer and channel -wiseconcatenation. The generator converts these feature vectors from low to high-resolution using multiple
transposed convolutions. As a result, the generator produces a raw waveform of 16384 samples
corresponding to the given label. Meanwhile, the discriminator receives the generated or real
waveform and its label information. Similar to the generator, the input label is converted into anembedding vector and combined with the input waveform through the fully connected layer andchannel-wise concatenation. Then, the discriminator extracts features of the input waveform throughmultiple convolution layers to determine the authenticity of the input waveform.
3.2 Loss of Class-Conditional WaveGAN
In this section, we describe the adversarial loss and multi-resolution STFT loss used to train the
proposed class-conditional WaveGAN. The adversarial loss of the generation model utilizes the loss
function of the WGAN-GP (15), as it can improve the training stability of the GAN. The loss functionof WGAN-GP consists of the loss function of the WGAN (first two terms) and the gradient-penalty
loss (last term). Unlike vanilla GAN’s loss function, which uses a log function (4), WGAN uses a
simple difference between discriminator outputs of real and generated data. WGAN’s loss function
guarantees more stable training convergence compared to the vanilla GAN ’s loss. On the other hand,
gradient-penalty loss induces the discriminator to become a 1-Lipschitz function by adjusting thegradient norm of the discriminator for random data x̂ ~ px̂ , where px̂ is the generated virtual data
distribution. 1-Lipschitz discriminator satisfies the condition of using WGAN loss function andimproves training stability. Equation (2) describes the loss function of the WGAN-GP.
Ladv (D, G) = Ez~pz [D(G(z))] - Ex~pr [D(x)] + λEx̂~px̂ [(‖∇x̂ D(x̂)‖2 - 1)2 ](2)Here, pr and pz are the distribution of real data and latent vector, respectively, and λ represents
a coefficient of the gradient-penalty.
Another loss used for our generator is the multi-resolution STFT loss, which was proposed toeffectively enhance the perceptual quality of synthesis sounds for the text-to-speech task (16).
Because this loss helps the generator to learn the detailed characteristics of the real data by comparing
the real and generated data, we use this for high-quality animal sound generation. The multi-resolution
STFT loss is the sum of STFT losses, and the STFT loss consists of a spectral convergence loss and alog STFT magnitude loss. Equations (3) and (4) define the spectral convergence loss Lsc and log
STFT magnitude loss Lm , respectively. Here, ‖ ∙ ‖F and ‖ ∙ ‖1 stand for the Frobenius and L1 norms,
respectively. |STFT(∙)| indicates the STFT magnitude which is the frequency-time feature
representation of the waveform, and M indicates the number of elements in the magnitude. Equation(5) defines the single STFT loss, which refers to LS (G) . Finally, Equation (6) defines the multiresolution STFT loss LMR , where N is the number of STFT losses.
‖|STFT(x)| - |STFT(G(z))|‖F‖|STFT(x)|‖F(3)

1‖log|STFT(x)| - log|STFT(G(z))|‖1
M(4)

Lsc (x, G(z)) =Lm (x, G(z)) =LS (G) = Ez~pz, x~pr [Lsc (x, G(z)) + Lm (x, G(z))]

(5)

N

1
(n)
LMR (G) = ∑ LS (G)
N
n=1

(6)

The spectral convergence loss allows the model to focus the difference between real and virtual
data on large spectral components, while the log STFT magnitude loss allows the model to adjust on
small components in detail. The two loss functions, due to their complementary roles, contribute to
the generator imitating the frequency-time features of a real waveform. Furthermore, since the multi-

resolution STFT loss is calculated using STFT losses with N sets of STFT hyperparameters (e.g., FFT
size, window size, hop size), the generator can learn from feature representations of various temporaland frequency resolution of real data, producing realistic virtual data more stably.
Therefore, the final loss function used in our generation model can be defined by Equation (7). All
loss functions in the equation are calculated using real and virtual data with the same class for theconditional generation.
L(D, G) = Ladv (D, G) + LMR (G)(7)

4. EXPERIMENT
4.1 Dataset
In this study, we use five anurans living in South Korea as target classes: Kaloula borealis,
Bombina orientalis, Hyla suweonensis, Pelophylax nigromaculatus and Hyla japonica. Anurans are
sensitive to temperature and exhibit distinct behaviors depending on the season. Their spawningseason is highly dependent on weather and temperature, and early spawning implies a change inclimate. In addition, since anurans are both predators and prey in the middle of the food-chain, theirabundance affects the survival of many other species. Because of their role in ecosystems, they areoften considered environmental indicator species that captures changes in climate or ecosystem.
Therefore, in order to effectively monitor them, various studies have been conducted to identify eachspecies based on its unique sound (9).
Anuran sounds were recorded in their habitats at a sample rate of 44.1 kHz, single channel, and 16
bit-resolution. Most of the recorded sounds contain natural sounds that differ de pending on the timeand place of collection, and some recording intervals overlapped with non-subject animal sounds.
Therefore, in order to obtain the clearest possible sounds, we divided the cry section into multiple
segments with 1-second length by using the end-point detection method in (9) and removed distortions
where natural sounds were too strong or overlapped with sounds of other species. Lastly, all segments
were resampled at a sample rate of 16 kHz.
4.2 Data Augmentation
For the qualitative evaluation of the proposed augmentation scheme, we visualized the waveforms
generated by the scheme. Figure 2 shows the visualization results of real and virtual waveforms foreach anuran class. It can be seen that the virtual waveforms of all classes in Fig. 2(b) are similar to
the real waveforms in Fig. 2(a). This indicates that our scheme has learned the unique characteristics
of each real animal sound. As the proposed scheme can learn time-frequency characteristics of various
resolutions from real waveforms through the multi-resolution STFT loss, the generated virtual
waveforms mimic the sound waveforms of real animals well.
Bombina orientalis

Hyla suweonensis

Pelophylax nigromaclulatus

Hyla suweonensis

Pelophylax nigromaclulatus

Hyla suweonensis

Amplitude

Kaloula borealis

Time (s)(a) Real data
Bombina orientalis

Hyla suweonensis

Amplitude

Kaloula borealis

Time (s)(b) Virtual data
Figure 2 – Visualization of real and virtual waveforms in each anuran class4.3 ASCTo evaluate the effectiveness of our data augmentation scheme, we performed ASC using theaugmented data. In general, sound classification models use spectrogram as input to represent
abundant features from the waveform. Therefore, we used ResNet-18, a popular CNN-based image
classification model, as a baseline. ResNet can learn high-level audio features by stacking layers
deeply with residual connections, which reduces the vanishing gradient problem. To preprocess the
input waveform of the ResNet model, we performed STFT (256 FFT size, 128 window size, 128 hopsize) and scaled amplitude logarithmically to get a log -scaled spectrogram. Then, we normalized thespectrogram with the mean and standard deviation of each frequency bin. Therefore, this preprocess
converts the input waveform with 16384 lengths into the log-scaled spectrogram with 129 × 129 size.
The experiments were conducted with an Intel Core i7-9700 CPU, 32GB RAM, and NVIDIA
GeForce GTX 1080ti GPU under Python environments. For training our generation model, we used
the Adam optimizer with a learning rate of 0.0001, β 1 of 0.5, and β2 of 0.9 for both generator anddiscriminator. We also updated the discriminator five times per generator update , as suggested in (15).
The training iteration of our scheme was set to a maximum of 10000 iterations. Meanwhile, the ResNetmodel was trained a maximum of 100 epochs on the train set and stopped early when the loss ofvalidation set had converged. The validation set was used to adjust the hyperparameters of the ResNetmodel.
As a performance evaluation metric for the ASC model, we used the traditional measure of
accuracy in classification tasks. Accuracy can be defined by Equation (8), which represents the ratioof correct predictions among all predictions for the entire class. However, the accuracy does notconsider each class individually, which can lead to overestimation of model performance for
imbalanced datasets. Therefore, we measured precision, recall and F1-score for each class andcalculated the average for the entire class. Precision indicates the fraction of correctly predicted
samples among all positive predictions to a specific class, and recall indicates the fraction of correctlyclassified samples among total samples within a class. Precision and recall are defined by Equations(9) and (10), where Tp , Fp and Fn indicate true positive, false positive and false negative in the
confusion matrix, respectively. F1-score is a harmonic mean of precision and recall and is calculated
by Equation (11).
Accuracy =Number of correct predictions
Number of all predictions(8)

Tp
Tp + Fp(9)

Tp
Tp + Fn(10)

2 × Precision × Recall
Precision + Recall(11)

Precision =Recall =F1-score =4.4

Experimental Results
Table 1 – Summary of the anuran dataset for ASC evaluation
Target Class

Train set

Validation set

Test set

Total

Kaloula borealis

1962

218

975

3155

Bombina orientalis

3479

386

445

4310

Hyla suweonensis

3143

349

118

3610

Pelophylax nigromaculatus

2398

266

350

3014

Hyla japonica

1641

182

333

2156

Table 2 – Comparison of augmentation methods
MethodAccuracy

Avg. Precision

Avg. Recall

Avg. F1-score

ResNet

0.658

0.770

0.738

0.695

Time stretching

0.628 (-0.030)

0.785 (+0.015)

0.699 (-0.039)

0.657 (-0.038)

Pitch shifting

0.538 (-0.120)

0.662 (-0.108)

0.612 (-0.126)

0.561 (-0.134)

Background noise addition

0.689 (+0.031)

0.762 (-0.008)

0.758 (+0.020)

0.698 (-0.003)

White noise addition

0.707 (+0.049)

0.784 (+0.014)

0.791 (+0.053)

0.751 (+0.056)

WaveGAN0.703 (+0.045)

0.743 (-0.027)

0.669 (-0.069)

0.679 (-0.016)

Proposed scheme

0.825 (+0.167)

0.810 (+0.040)

0.759 (+0.021)

0.762 (+0.067)

For the ASC evaluation, we split the collected data into a training set and a test set from based onwhere they were collected. For the validation set, we randomly selected 10% from the training set.
Table 1 shows a summary of the established anuran sound dataset.
To compare the proposed scheme with other traditional augmentation methods, we chose two signal
deformation methods, time stretching and pitch shifting. For each input waveform in the training loopof the ASC model, we used random deformation strength between 0.8 to 1.2 for time stretching and-2 to 2 random deformation strength for pitch shifting. In addition, we used the noise addition method,which adds two types of noise that can occur in real-world recording situations: background noise andwhite noise. For the background noise, we used various environmental sounds (e.g., wind, rain, bird s
chirping and stream sounds) collected from Freesound web database. In the case of white noise, weused random signals with uniform intensity. Both noise addition methods mix noise with input
waveforms as one signal-to-noise ratio in the training loop. Lastly, we constructed a general
unconditional GAN-based augmentation scheme. To this end, we built a WaveGAN model thataugments the data for each class and used it for comparative experiments. WaveGAN and our scheme
generated 5000 samples for each class and used them as a training data for the ASC model.
Table 2 shows the classification performance of the ResNet-based ASC model according to theaugmentation method. The table shows that time stretching and pitch shifting deteriorated theperformance of the ResNet model in most metrics compared to when no augmentation was applied(First row in Tab 2.). Considering that cry of each anuran has a distinct speed and pitch height, time
stretching and pitch shifting was not effective because they transformed the discriminative featuresof anurans. Background noise addition resulted in a slight performance improvement in terms ofaccuracy, but there was no significant change in terms of F1-score. On the contrary, white noise
addition has improved performance in all metrics, achieving the best score in terms of recall.
Background noise addition gave little advantage due to the uneven quality of background noisecollected from the web, whereas white noise addition has helped identify a regular pattern of eachanimal sound. For the rest of the metrics except recall, our scheme achieved the highest performancecompared to other methods and improved accuracy and F1-score by a large margin. Furthermore, theproposed scheme enhanced the ResNet model in all metrics, whereas WaveGAN only improved theaccuracy. Overall, even in the case of insufficient training data, our proposed scheme can generate
high-quality data, which can greatly contribute to the effective training of ASC models.

5. CONCLUSIONSIn this study, we proposed a GAN-based animal sound data augmentation scheme that can solve
the common data shortage problem in ASC. To generate virtual animal sound data, we proposed aclass-conditional WaveGAN that embeds class label information into WaveGAN. We also used multiresolution STFT loss to learn the detailed characteristics of real animal sounds effectively. By learning
both the common and individual characteristics of the target classes, the proposed scheme caneffectively generate multiple animal sounds even using a single model. We verified the effectivenessof the proposed scheme through comparison with ResNet-based ASC model and other traditionalsignal augmentation methods. The experimental results demonstrated that our proposed scheme cangenerate realistic animal sound data and improve the performance of ASC tasks. In future work, wewill find a more suitable model architecture for augmenting animal sounds and a data selection module
to accurately filter high-quality virtual data.

ACKNOWLEDGEMENTS
This work was supported by Korea Environment Industry & Technology Institute (KEITI) throughExotic Invasive Species Management Program, funded by Korea Ministry of Environment (MOE)(2021002280004).

REFERENCES
1. Dugan PJ, Barlow C, Agostinho AA, Baran E, Cada GF, Chen D, et al. Fish migration, dams, and loss of
ecosystem services in the Mekong basin. Ambio. 2010;39(4):344–8.
2. Zwerts JA, Stephenson PJ, Maisels F, Rowcliffe M, Astaras C, Jansen PA, et al. Methods for wildlife
monitoring in tropical forests: Comparing human observations, camera traps, and passive acoustic sensors.
Conservat Sci and Prac. 2021;3(12).
3. Nanni L, Costa YMG, Aguiar RL, Mangolin RB, Brahnam S, Silla CN Jr. Ensemble of convolutional
neural networks to improve animal audio classification. EURASIP J Audio Speech Music Process.
2020;2020(1):1–14.
4. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative Adversarial
Networks. arXiv. 2014.
5. Moon J, Jung S, Park S, Hwang E. Conditional tabular GAN-based two-stage data generation scheme forshort-term load forecasting. IEEE Access. 2020;8:205327–39.
6. Donahue C, McAuley J, Puckette M. Adversarial Audio Synthesis. arXiv. 2018.
7. Aggelina C., Georgios P., Dimitris S., Georgios P., Malvina N., Theodoros G., et al. Data Augmentation
Using GANs for Speech Emotion Recognition. Proc. Interspeech 2019; p. 171–175.
8. Wang X, Huang H, Hu Y, Yang Y. Partial discharge pattern recognition with data augmentation based on
generative adversarial networks. In: 2018 Condition Monitoring and Diagnosis (CMD). IEEE; 2018. p.
1–4.
9. Ko K, Park S, Ko H. Convolutional feature vectors and support vector machine for animal soundclassification. Annu Int Conf IEEE Eng Med Biol Soc. 2018; p. 376–9.
10. Salamon J, Bello JP, Farnsworth A, Kelling S. Fusing shallow and deep learning for bioacoustic bird
species classification. In: 2017 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP). IEEE; 2017. p. 141–5.
11. Lasseck M. Audio-based bird species identification with deep Convolutional Neural Networks. CLEF.
2018.
12. Nanni L, Maguolo G, Paci M. Data augmentation approaches for improving animal audio classification.
Ecol Inform. 2020;57(101084):101084.
13. Madhu A, Suresh. EnvGAN: a GAN-based augmentation to improve environmental sound classification.
Artif Intell Rev.;2022:1–20.
14. Bouallegue G, Djemal R. EEG data augmentation using Wasserstein GAN. In: 2020 20th International
Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA). IEEE;
2020. p. 40–5.
15. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A. Improved training of wasserstein GANs.
In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Red
Hook, NY, USA: Curran Associates Inc.; 2017. p. 5769–79.
16.Yamamoto R, Song E, Kim J-M. Parallel WaveGAN: A fast waveform generation model based on
generative adversarial networks with multi-resolution spectrogram. arXiv 2019.