SoundCount: Sound Counting from Raw Audio with Dyadic Decomposition Neural Network
Authors: Yuhang He
Zhuangzhuang Dai
Niki Trigoni
Andrew Markham
Venue: Submitted to ICLR 2023 (2023)
Keywords: Sound Crowd Count
Dyadic Decomposition Network
Learnable Filters
Acoustic Crowd Counting

Abstract: In this paper, we study an underexplored, yet important and challenging problem: counting the number of distinct sound events in data characterized by a high degree of polyphonicity and spectral overlap. A key example is counting individual bird calls in bioacoustic data, from which biodiversity can be estimated. We do so by systematically proposing a novel end-to-end trainable neural network, designing new evaluation protocols, quantifying the difficulty of counting depending on sound polyphonicity, and creating a new dataset tailored for concurrent sound event counting. Unlike existing methods that all apply frequency-selective filters on the raw waveform in a one-stage manner, our neural network progressively decomposes the raw waveform dyadically in frequency domain. Taking inspiration from wavelet decomposition, intermediate waveforms convolved by a parent filter are successively processed by a pair of children filters that evenly split the parent filter's carried frequency response. An energy gain normalization module is introduced to normalize received sound events' loudness variance and spectrum overlap. The network is fully convolutional and parameter-frugal so it is light-weight and computationally efficient. We further design a set of polyphony-aware metrics to quantify sound counting difficulty level from different perspectives. To show the efficiency and generalization of our method (we call DyDecNet), we do experiments on both bioacoustic bird sound (both synthetic and real-world sound), telephone-ring sound and music sound data. Comprehensive experiment results show our method outperforms existing sound event detection (SED) methods significantly. The dyadic decomposition front-end network can be used by existing methods to improve their performance accordingly.

Under review as a conference paper at ICLR 2023
SOUNDCOUNT: SOUND COUNTING FROM RAW AUDIO WITH DYADIC DECOMPOSITION NEURAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we study an underexplored. yet important and challenging problem: counting the number of distinct sound in raw audio characterized by high degree of polyphonicity. We do so by systematically proposing a novel end-to-end train- able neural network (we call DyDecNet, comprising of dyadic decomposition front-end and backbone network), and quantifying the difficulty level of counting depending sound polyphonicity. Unlike existing audio-processing methods that uniformly apply a set of frequency-selectte one-stage manner to get time- freeffrey (TF) representation, sition front-end progressively decomposes the raw waveform dyadically along the frequency axis to obtain TF representation in multi-stage, coarse-to to-eet manner. Each intermediate waveform convolved by parent filter further processed by a pair of child filters that evenly split the parent filter's s carried frequency response, with the higher-half child filter encoding the detail and lower-half child filter en- coding the approximation. We further introduce an energy gain normalization to normalize sound loudness variance and spectrum overlap, and apply it each intermediate parent waveform before feeding the two child filters. We argue that such dyadic decomposition front-end better characterizes sound polyphonicity and concurrency that commonly exist in sound counting task, while introducing negligible extra computational cost. To better quantify sound counting difficulty level, we further design three polyphony-awane metries: polyphony ratio, max polyphony andmean polyphony. We test DyDecNet three main sound datasets from different domains: bioacoustic sound (both synthetic and real-world sound), telephone-ring sound and music sound. Comprehensive experiment results show our method outperforms existing sound event detection (SED) methods signifi- cantly. The dyadic decomposition front-end can be used as a general front-end by existing methods to improve their performance accordingly.
filters on the raw waveform in a
dyadic decompo-
INTRODUCTION
Suppose you went to the seaside and heard a cacophony of seagulls, squawking and squabbling. An interesting question that naturally arises is whether you tell the number of seagulls flocking around you from the sound you heard? Although a trivial example, this sound "crowd counting" problem has number of important applications. For example, passive acoustic monitoring (PAM) widely used to record sounds in natural habitats, which provides measures of ecosystem diversity and density [2. 15, 12]. Sound counting helps quantify and map sound pollution by counting the number of individual polluting events [4]. It can also be used in music content analysis [24]. Despite its importance, research on sound counting has far lagged behind than its estabissall crowd counting counterparts from either images [49, 46), video [29] or joint audio-visual [22].
We conjecture that the lack of exploration stems from three main factors. First, sound counting has long been thought of as an over-solved problem by sound event detection (SED) methods [35, 9, 1, 19]. in which SED goes further to identify each sound event's (e.g. bird call) start time, end time and semantic identity. Sound counting number then becomes easily accessible by simply adding up all detected events. Secondly, current SED only tags whether a class of sound event is present within window, regardless of the number of concurrent sound sources of the same class like series of baby crying or multiple bird calls [41]. Thirdly, labelling acoustic data is technically- harder and more time-consuming than labelling images, due to the overlap of concurrent and diverse sources. The lack

---

Under review as a conference paper at ICLR 2023
of well-labelled sound data in crowded sound scenes naturally hampers research progress. Existing SED sound datasets [1, 20] capture simple acoustic scenarios with low polyphony and where the event variance is small. The simplified acoustic scenario in turn makes sound counting task by SED methods tackleable. But when the sound scene becomes much more complex with highly concurrent sound events, SED methods soon lose their capability in discriminating different sound events [38, 9]. Therefore, a study specific for sound counting problem is desirable and overdue.
In this paper, we study the general sound counting problem under highly polyphonic, cluttered and concurrent situation. Whilst the challenges of image-based crowd counting mainly lie in spatial density, occlusion and view perspective distortion, the sound counting challenges are two-fold. Firstly, acoustic scenes are additive mixtures of sound along both time and frequency axes, making counting overlapping sounds difficult (temporal concurrence and spectrum-overlap), Secondly, there is large variance in event loudness due to spherical signal attenuation with distance (loudness variance). Tackling these challenges require a more elegant method to process sound raw waveform so as to better localize sound in time-frequency domain.
In this paper, we propose a novel dyadic decomposition neural network to learn sound density representation capable of estimating cardinality directly from raw sound waveform. Unlike existing sound waveform processing methods that all apply frequency-selectn seleccive filters on the raw waveform in single stage [19, 10, 48, 18, 14], our network progressively decomposes raw sound waveform in dyadic manner, where the intermediate waveform convolved by each parent filter is further processed by its two child filters. The two child filters evenly split the parent filter's frequency response, with one child filter encoding the waveform approximation (the one with the lower-half frequency response) and the other one encoding the waveform details (the one with the higher-half frequency response). To accommodate sound loudness variance, spectrum-ovvrlap and time-concurrence we further propose an energy gain normalization module to regularize each intermediate parent waveform before feeding it to two child filters for further processing. This hierarchical dyadic decomposition front-end enables the neural network to learn a robust TF representation in multi-stage coarse-to-fing manner, while introducing negligible extra computation cost. By setting each filter's 's frequency cutoff parameters to be learnable and sclf-adjustable during optimization in data-driven way, the final learned TF representation can better characterize sound existence in time and frequency domain. Following the front-end, we add a backbone network to continue to learn a time framewise representation. Such representation can be used to derive the final sound count number by either directly regressing the count number, regressing density map (the one we choose) or following SED pipeline.
Apart from the network, we further propose three polyphony-aware metrics to quantify sound counting task difficulty level: polyphony ratio, maximum polyphony and mean polyphony. We will give detailed discussion to show the feasibility of three metrics.
We run experiments on four cross-domain sound datasets: a bird sound set (both real-world and synthetic), telephone-ring sound set (synthetic), and music sound [24] (real-world), Experimental results show our method (DyDecNet) outperforms exiting SED-based methods significantly on both real-world and synthetic dataset. Replacing existing methods' one-stage sound raw waveform pro- cessing front-end with our dyadic decomposition front-end dramatically improves their performance accordingly. Since the real-world datasets contain relatively small polyphony level, we specially synthesize a bird sound dataset that contain much higher sound polyphonic level and spectral overlap. The synthesized sound dataset has two sub-sets: one involves four kinds of bird sound (exhibits het- erophony); ); the other has just one kind of sound (this encapsulates homophonic scenario). Experiment on such synthetic dataset helps to test performance under highly polyphonic situation.
In summary, we make three main contributions: First, propose dyadic decomposition front-end to decompose the raw waveform in a stutii-ttgee coarse-to-fine manner, which better handles loudness variance, spectrum-overlap and tme-ccnnureence. Second, propose new set of polyphony-aware evaluation metrics to comprehensively and objectively quantify sound counting difficulty level. Third, Show the efficiency and generalization of DyDecNet on sound datasets across different domains.
2 RELATED WORK
Crowd counting from images or visuall has been thoroughly studied in recent years [49, 22], the target of which is to estimate the instance number from very crowded scenes (e.g. pedestrian in train station) that cannot be efficiently handled by object detection methods. The methods approaching image crowd counting chronically evolve from the early detection-based [26] the later regression-
2

---

Under review as a conference paper at ICLR 2023
Dyadic Frequuncy Dexunpositin Module
Backhone Module
SoundCounnt Flat.
Flat. Dim
Apprux.
Detai)
h
Pee-chaenel Pool
Time-Frog, Map
Inter claunel Couv.
Soumd Warrefoom
Time
Depth
Depth
Depti. d
Fremewion Regiess
Figure 1: DyDecNet pipeline. We first feed the input raw sound waveform the dyadic decomposition front-end to learn a time-freyuency representation, which is further fed to a backhone neural network to continue to learn framewise representation. Such representation retains time information, it is general enough to get count number by either regression or SED method. The dyadic decomposition front-end consists of set of parameterized learnable band-pass filters. Each intermediate waveform processed by parent filter is further processed by two child filters, with lower-half filter (red color) encoding approximation and higher-half filter (light-blue) encoding details.
based [11] and density map estimation [27] methods. Accompanying these methods, various neural network architectures have been designed to achieve higher performance.
The counterpart task purely in sound, however, has been nearly ignored. Existing research mainly focus on sound event detection, including spatio-temporal sound event detection (SELD) [19, 18, 1, 10] from microphone array and temporal sound event detection [7, 39] and high-frequency time series analysis [36]. They often combine convolutional neural networks (CNN) [8] and reccurent neural network [1] to separate sound sources. The datasets they work on are relatively simple, in which the sound scenes are relatively simple and contain few overlapping sound events.
The common way to process raw sound waveform is to first convert the 1D waveform into 2D time-freet freeuerey representation so that sound events' frequency property and their variation along time axis are explicitly split out. Most existing methods [10, 1, 7, 39] adopt Fourier transform [14] or Wavelet transform [33] to obtain such 2D representation, in which the whole conversion process is fixed. Some recent work [19, 48, 42, 45, 37] re- parameterize the conversion frequency-seleetie filters to be learnable that the whole neural network is able to directly learn from raw sound waveform. Experimental results show enabling the neural network to learn from the raw waveform often achieve better performance than traditional fixed conversion. These methods, however, convert the raw waveform in a one-stage manner. Our proposed dyadic decomposition neural network instead processes the raw waveform in dyadic multi-stage manner.
Dyadic Network Dyadic representation idea has been initially proposed to represent signal hierarchi- cally [6. 3], in multi-scale manner. Its core idea is construct bank of filters (either learnable or fixed) so that different filter extracts different feature at a certain scale or resolution. Summarizing them together leads to more comprehensive and complete analysis. Similar idea has been widely used in computer vision community, including pyramid feature representation for object detection [30] and semantic segmentation [43, 28].
DYADIC DECOMPOSITION NEURAL NETWORK
Different sound classes typically exhibit different spectral properties. A canonical way to process raw sound waveform is to apply a ceqquency-selectiin filter bank F, (falk to project the raw sound waveform onto different frequency bins. Traditional Fourier transform [14] or Wavelet transform [33] construct fixed filter banks which all filter-construction relevant hyperparameters are empirically chosen and thus may not be optimal for particular task. Recent methods [19, 48] relax some hyperparameters to be trainable so that the filter bank can be optimized a data-driven way. A learnable filter bank often leads to better performance than fixed filters. However, all existing methods apply all filters, either learnable or fixed, on the raw waveform in a one-stage manner. Such shallow and one-stage processing may fail to learn powerful and robust representation for sound counting task where large loudness variance and heavy spectrum overlap exist. In our dyadic decomposition framework, instead adopt progressive pairwise decomposition strategy to obtain the time-frequency (TF) representation. It learns TF representation from coarse-grained to fine-grained granularity, Particularly, it consists of a dyadic frontend and backbone.
3

---

Under review as a conference paper at ICLR 2023
3.1 DYADIC FREQUENCY DECOMPOSITION FRONTEND
In dyadic frequency decomposition frontend, we construct a set of D hierarchical filter banks F dyadie = 1, F2 F J. The d-th filter bank has 2 filters, each filter is parameterized by a learnable high freuqney-cutoff parameter and a low frequency-cutoff parameter. By cascading these filter banks, we consecutively decompose the raw waveform in frequency domain dyadically, leading to coarse- saarse-griiedd to fine-gr tie-grained TF representation. Specifically, we denote the dyadic filter banks depth by D. in the depth d filter bank F2g. we have 2 filters evenly divide the waveform sampling frequency F.. Therefore, each single filter's frequency response length is & 2 the i-th filter f high
frequency cutoff F2. and low frequency cutoff Ft are initialized as,
F. = 2d 1),
F. 2d
FR.
F(U)))
From Eqn. (1) we can see that dyadic decomposition frontend forms complete binary-tree-like structure, in which the filter number doubles and each filter's 's frequency response length halves as the tree's depth increases by one. The intermediate waveform processed by a "parent" filter is just further processed by its two "childre" filters. The frequency responses of the two children filters evenly split their parent filter's frequency response. The child filter carrying the higher half frequency response encode the parent's 's processed intermediate waveform's 's detail while the other one carrying the lower half frequency response instead encodes the approximation. For example, for the filter f in the d-th filter bank, its frequency response lies in 15 -i, E. (i 1)], its two children filters ed+1 and pd+11 in the depth d + evenly divide its frequency range, fa+ carries (i + 2), 2 (i+ 1)].
2
F. F. i zd (i + })]. f4+1 carries
With the pre-constructed dyadic decomposition filter banks, we cascade them together to process the raw sound waveform, progressively learning the final TF representation. our implementation. each filter in dyadic filter banks is learnable band-pass filter. We adopt rectangular band-pass in frequency domain filter which comprises of learnable high frequency cutoff parameter F, and a learnable low frequency cutoff parameter Fi. Converting it to time domain through inverse Fourier transform, we get sinc(-) function like filter that is used to convolve with the waveform. For example, the filter f in Eqn. (1) can be represented as,
f Fy, Fi = 2F, 2f) 1 2F(sinc(2mFitt)
where sinc(z) sin(x))z (z/ t indicates the filter's ' representation at time t. Fy and Fi are initialized according to Eqn. (1), but they can be further adjusted during training process. sinc(-) filters have been successfully used in speech recognition [42] and sound event detection and localization [19]. In our dyadic decomposition frontend, each filter from different depth has separate and independent constructed learnable parameters (high frequency cutoff and low frequency cutoff). Morcover,
filter is much longer ( (1025 in our case) than traditional ID/2D Conv filters (3 or 5). Its wide length characteristic enables the filter to have wide feee on the raw waveform. Cascading them together allows the filters in later layers (larger depth) to have even wider field-of-vied on the input raw waveform. With this advantage, explicitly with RNN network. As a result, the whole dyadic frequency decomposition frontend is fully convolutional and parametrically learnable, it is parameter-frugal and computationally efficient. In practice, the dyadic decomposition frontend depth is 8, so the output TF representation has 256 frequency bins. At the same time, we downsample the intermediate waveform by before feeding to its two children filters in the initial 5 dyadic filter banks to reduce the memory cost.
do not have model sound event temporal dependency
3.2 ENERGY GAIN NORMALIZATION
We further design an energy gain normalization module to regularize each intermediate waveform before feeding them to the next dyadic filter bank. The motivation of introducing energy gain normalization is two-fold: first, to reduce sound event loudness variance led by sound events' different spatial locations; Second, to reinforce the frontend to learn to better tackle spectrum overlap challenge led by intra-class sound events in the sound scene. Specifically, for the intermediate waveform W,, processed by a dyadic filter fi', we first smooth with a learnable 1D Gaussian kernel of parameterized by learnable width get the corresponding smoothed waveform Wa which just
4

---

Under review as a conference paper at ICLR 2023
50x 256
V/s+
1/s
1/
Vos
FC
FC
Feature A3ds
20
26
Density Map
Count
Add
1/1x
1/23- 1/2
MSFLos
MSF
CB Loss
Evenn::
[2.0x,4.5g
1/21
1/
GT Count: 2
1/21
0
2.66s
Prod Count Sum density
Pred Count: aggregate events
Time
A. Backhone I camed Fenture
B. Ground Truth Density Map
C. Count from Density Map
D. Direct Prediction
E. SED Method
Figure 2: Three counting methods illustration. For density map (sub-fig, C), the sum (or integral) of the density map equals to the count number. We can also direct regress the final count number (sub- fig. D), use SED method (sub-fig. E). Detailed illustration is in Appendix Fig. V.
contains loudness. We then introduce a learnable automatic gain control parameter a to mitigate sound loudness impact. Furthermore, another two learnable compression parameters 5 and introduced to further compress we The overall energy gain normalization can be represented as,
W (W
W
+ +87 57
where 8 and y are learnable parameters. As result, the energy gain normalization eg- eg-Norm is fully learnable and parametersized by four learnable parameters eg-Norm(o, ar, 6) Practically, each filter in dyadic filter banks is associated with an independent eg-Norm module. Similar energy normalization has been successfully used in tasks like keyword spotting [47, 31]. The difference lies in the fact that they apply exponential moving average operation to get smoothed waveform representation, so the computation is very slow because it iterates along the time axis to compute the averaged value step by step. Our proposed energy gain normalized strategy instead adopts Gaussian kernel to get the smoothed waveform, in which it can be easily implemented as ID convolution. The dyadic filter visualization and energy normalization module is shown Fig.
3.3 BACKBONE NEURAL NETWORK
We add a lightweight backbone neural network the frontend neural network to further learn representation useful for call counting. The backbone network consists of two parts: per-channel pooling and inter-channel ID convolution. Unlike existing methods [9, 1] that first convert ID sound waveform into 2D map with fixed FFT-like transform, then learns from the 2D map with 2D Conv. operations, our method directly learn from sound raw waveform with learnable 1D Conv.. Specifically, we downsample each channel separately by assigning each channel with an independent frequency-sensitive learnable filter. We call such learnable downsampling per-channel pooling. helps to learn sound event's 's frequency variance along time axis individually. Moreover, we add normal ID Conv. achieve inter-channel communication, which enhances the neural network to learn concurrent sound events interaction. Detailed illustration is given in Appendix Table IV, The backbone serves as as the backend to learn framewise representation for counting.
3.4 DENSITY MAP AND LOSS FUNCTION
The backbone network discussed above learns framewise representation [Tb, Fo). where T. indicates the time steps and F, indicates feature size. There are three potential ways to derive final sound count number fromthe learned representation: directly regress the count number; 2.SED method: detect sound events first and then aggregate results to get final count; 3. predict the density map. For sound event with time location [2), its density map a 1D vector with value t 12-t, during its occurrence time, otherwise is O. So the count number equals the vector integral. We show regressing density map produces the best result (see Table 6). We thus adopt the mean squared error (MSE) loss during training directly regress the density map. The comparison of three methods shown in Fig. 2.
4 EVALUATION METRIC DISCUSSION
Mean absolute error (MAE) and mean squared error (MSE) are two widely used metrics in crowd counting [32, 49]. Specifically, denote the ground truth count and predicted count by yi and ý, respectively, for the i-th sound clip. MAE is defined as MAF A E |y 5.l. MSE is defined
5

---

Under review as a conference paper at ICLR 2023
Ourgot Ware
ff
t- 57
filters
Soand Event Num.
rati-polyp= (cyan]
4
Time Stops
4ono--ooyp-4iii mean-plyp= (blbee
Freq.
Giter ft
W
Wavee
A. Dyadic Filter Visualization
Dyndie Filter Conv. Block
Time
Figure Dyadic filter illustration. Left: In time domain, dyadic filter is sinc function curve. In frequency domain, dyadic filter is a rectangular band-pass filter with learnable high frequency /2 and low frequency cutoff fi. The filter's two child filters (left-top) evenly splits the parent filter's frequency response. Right: dyadic filter convolution block. The input wave- form fed to an energy normalization module. Then skip-connection is added.
Figure The max polyphony level in this sound clip is 4 (show in red, time step 5), so max-polyp=4. The mean-polyp indicates the purple area, averaging them over time gets R. ralio-polyp measures polyphony (no fewer than two concurrent sound events) existence ra- tio along the time axis (cyan), so it 5 6'
MSE E 2. We also involve accuracy rate (AccuRate) show the ratio of accurately predicted count. We introduce tolerance term p, where p = 0 means predicted count number has to be exactly the same with ground truth number in order to be treated as an accurate counting: p = relaxes the constraint so there can one count mismatch for an accurate counting.
4.1 POLYPHONY-AWARE DIFFICULTY QUANTIFICATION
The aforementioned three general metrics do not reflect the impact of sound scene nature on algo- rithms. We introduce three polyphony-aware metrics quantify sound counting difficulty level reflected by sound scene nature. The three metrics are time-window invariant so they can be used as general metrics quantify difficulty level of sound scene of various lengths.
Polyphony Ratio (ratio-polyp) describes the ratio of polyphony (at least two sound events happen the same time) over a period of time. It binarizes each time step either polyphonic non- polyphonic (monophoinc or silent) so the value lies between [0, 1].
Maximum Polyphony (max-polyp) focuses on the maximum polyphony level over time period, It is motivated by the fact that human's 's capability in discriminating different sound events reduces seriously when the number of temporal-over -verlaapie sound event number increases. It is positive integer and helps to understand an algorithm's 's capability in tackling polyphony peak.
Mean Polyphony (mean--plyy instead focuses on the averaging level polyphony involved within time period. It is designed to reflect algorithm's 's capability in tackling the average polyphony level over an arbitrary time window.
Given T, time steps sound vector P2, .T., where p; 2 0 is the sound event number happening at time step T,. The three metrics are defined as,
Di I2(p.) max max-polyp max Pe; mean-polyp i=1.-
Σ
- max(p; 1,
ratio-polyp
n
Tn
(4)
where I2(p;) is indicator function, it is 1 if P. > 2, otherwise O. With the three quantifying metrics, we can report the general metrics (MAE, MSE) against various difficulty levels.
EXPERIMENT
We run experiments on six sound datasets derive from four commonly seen domains.

---

Under review as a conference paper at ICLR 2023
1. Bioacoustic Sound. We focus on bird sound bird sound is ubiquitous is most terrestrial environment with distinctive vocal acoustic properties. Specifically, we test three datasets: one real- world NorthEastUS Bird [12] dataset and other two synthesized datasets: Polyphony4Birds (for heterophony test) and Polyphony Bird (for homophony test). NorthEastUS. [aad Bird data is recorded in nature reserve in northeastern of United States. It encompasses 385 minutes of dawn chorus recordings collected in July 2018, with a total of 48 bird species. The average bird sound temporal length is very short (less than 1s) and the polyphony level (max-polyp and mean-polyp) is small.
To test performance under highly polyphonic situation, we synthesize two bird sound datasets. Specifically, The first dataset contains four sounds: junco, American redhead, cagle, and rooster from copyright-free website f findsounds.. com. We call it Polyphony4Birds (heterophony test). The second dataset contains one sound: rooster. We call it Polyphony Bird (homophony test).
2. Indoor Sound. We count telephone ring sound, the telephone ring seed sound comes from the same copyright-free website. We follow Polyphony B Bird synthesis procedure except synthesize in a much smaller room (10m 10m 3m) to reflect room reverberation effect.
3. Outdoor Sound. We count car engine sound, as it is widely heard in outdoor urban scenario.
The car engine seed sound also comes from the same copyright-free website. We also follow Polyphony/ Bird synthesis procedure to create car engine dataset.
4. Music Sound. We use OpenMic2018 8 dataset [24]. The target is to count the musical instrument class number being played in the audio clip, regardless of the number of times each single musical instrument class being played in the audio clip. This dataset contains 20 musical instrument categories, but we do not know each instrument's 's playing start time and end time, but instead the total sound count number within each clip. Therefore, we directly regress the number.
Table 1: The comparison on the six datasets, in terms of data size, sound event class number and polyphony level.
Data
Size
Class
mean-poly
max-poly
NorthEastUS [12]
6.41
48
0.1
4
Polyphony4Birds
55,56
4
1.244
9
Polyphony I Bird
55.56 h
1
1.975
9
Telephone Ring
55.6 h
1
1.975
9
Car Engine
55.6 h
1.975
9
OpenMic [24]
55.6
20
n/a
n/a
The direct comparison between the six datasets is given in Table 1. We highly refer to Appendix Sec. B for more discussion about the data synthesis process.
We compare our framework with two main method categories: 1) traditional deterministic signal processing methods, including Librosa-onset and Aubio-onset; 2) based Methods. Librosa-onset [34] provides an onset/offset detection method for music note detection. It measures the uplift or shift of spectral energy to decide the starting time of a note. We use its onset/offset detection ability to count sound event number. Aubio-onset [5] achieves pitch tracking by aligning period and phase of the Mel spectrogram. We use its pitch tracking to count.
Comparing Methods:
SED-based methods build on traditional fixed TF representation, such short time Fourier trans- form (STFT) and LogMel. The TF representation is treated as a 2D image to be processed by sequence of 2D Conv. operators. GRU [13] and LSTM [21] are often adopted model temporal dependency. we compare three typical SED methods: 1) CRNNNet [9] consists of 2D Conv. to learn multiple compressed TF representations from the input TF map. Then it concatenates them together along the frequency dimension and further feed it to LSTM [21] to learn framewise representation. 2) DND-SED [16] instead adopts depthwise 2D convolution and dilated convolution avoid using RNN. 3) SELDNet [1] is originally used for joint sound event detection and localization. It adopts 2D Conv. to convolve the 2D TF map, and bidirectional GRU to model temporal dependency. The three comparing methods' network architectures are slightly adjusted to fit our dataset. We call our method Dyadic Decomposition Network (DyDecNet).
Implementation Detail and Experiment Configuration For all the six datasets, all input audios are segmented into second long clips, with sampling rate 24 k Hz. So the input waveform has 120,000 data points and is normalized into [-1, 1]. We train the models with Pytorch [40] on TITAN RTX
7

---

Under review as a conference paper at ICLR 2023
Table 2: MSE and MAE results on the six datasets. We leave the Accuracy Rate metric in the Appendix due to space limitation.
Dataset
OpenMIC
NorthEasUUS
Polyphony-BBirs
PolyphonyIBir
Telephonc Ring
CarEngine
Method
MSEL
MAF
MSK)
MAE.
MSEL
MAE.,
MSFJ
MAE.
MSEJ
MAF
MSF,
MAFT
Libresa-unset
25.3
4,00
2.31
1.65
28.3
4,09
37.65
5.5
30.03
4.50
33.13
4.51
Aubin-onset
8.43
35.33
5.27
33.20
SELDNet
0.90
1.37
1.35
1.79
0.92
1.41
0.89
1.19
0.97
1.30
0.92
1.23
CRNNNet [9]
0.71
1.00
1.33
1.77
0.74
1.10
0.87
1.16
0.92
1.31
0.86
1.15
DND-SED [16]
0.93
1.27
1.19
1.64
0.95
1.34
1.04
1.27
1.23
1.34
1.00
1.21
DyDoeNet
0.32
0.72
0.85
1.19
0.46
0.92
0.54
0.85
0.58
0.89
0.54
0.87
GPU, Network architecture of DyDecNet is shown in Appendix Table IV. To train the neural network, adopt Adam optimizer [25] with initial learning rate 0.001 which decays every 20 epochs with a decaying rate 0.5. Overall, we train 60 epochs. We train each method 10 times independently and report the mean value and standard deviation. We do not report the standard deviation explicitly in the table because we find them very small (about 0.03). We first train the comparing SED methods with both their suggested training strategy and our training strategy, then choose the with the better performance as the final result. For the energy gain normalization we initialize them as = 0.96, 2., 7 0.5, o 0.5. The batchsize is 128.
5.1 EXPERIMENTAL RESULT
The quantitative result on MSE MAE and is given Table 2, and the accuracy rate result given in Table II in Appendix Material. From the two tables we can learn that proposed DyDecNet outperforms both classic signal processing deterministic methods and existing SED methods by a large margin. Our framework is better than the baselines discussed in this paper in both real-wor world and synthesized sound scenes. is capable of learning powerful representation from both weak sound signals (NorthFastUS. Bird dataset), highly polyphonic (Our synthesized four datasets) and heavy spectrum-overlapping, loudness- loudness-varying varying sound events.
At the same time, also observe that the two signal processing deterministic methods (Librosa-onset and Aubio-onset) generate the worst result over both SED based methods and DyDecNet. The higher of the polyphony level of the dataset, the worse performance the two deterministic methods lead to. For example, in NorthEastUS Bird dataset with relatively smaller polyphony level, Librosa-one and Aubio-onset generate relatively good performance with accuracy rate (p = 1) reaching 0.58. In our synthesized two datasets with much higher polyphony levels, however, their accuracy drops significantly near zeros. It thus shows traditional signal processing methods do not fit for sound event counting from crowded acoustic scenes.
Among the three datasets, SED-based methods and DyDecNet produce decreas- ing performance on Polyphony-Birdss Polyphony Bird and NorthEas- IUS Birds dataset, respectively. The largest performance drop observed real-world North EastUS FastUS_Birdds dataset, which shows counting from real-world dataset is a tough task that desires more future attention. Spectrum-overlap led by intra-class sound events is another po- tential challenge (better performance on Polyphony4Birt than Polyphony Bird) that may need more work to tackle it.
20
10.0
15
7.5
15
uSE
5.0
MSE
10
2.5-
2
0.50
0.75
Max polyp
Retio - polyp
Mean - polyg
3-
MAE
MAE
MAE
0.75 polyp
Mean polyo
CRNNNet
DND.SED
SELDNet
DyDecNet
The MSE and MAE variation against max-polyp, ratio-polyp and mean- polyp difficulty level on NorthEas- tUS Bird are shown in Fig. 5. We can observe that our proposed three metrics mar-polyp. ratio-polyp and mean- mean-pplyy are effective ways to accurately quantify sound counting tasks difficulty level. The three metrics has observed dramatic performance drop their the difficulty
Figure 5: MSE and MAE variation against maz-polyp. ratio-polyp and mean-polyp on NorthFastUSS_P _ird dataset. More results are in Appendix.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study on dyadic decomposition ef- ficiency discussion: we com- pare existing methods with and without dyadic decompo- sition frontend.
Table 4: Ablation study on tra- ditional T-F feature for count- ing task: DyDecNet's dyadic decompostion frontend is re- placed by various classic T-F features extractors, such STFT, LogMel, MFCC and Gabor.
Table 5: Ablation study on var- ious DyDecNet variants.
Method
MSRL
MAR]
DeDeeNet SingSeale
1.22
DyDDeNNe, BN
1,07
1.25
1.15
1.37
DyDocNet
0.85
1.19
Method
MSEL
MAEL
Table 6: Ablation study on var- ious counting method.
SELDNet [17]
1.35
1.79
Method
MSE
MAEL
SELDNe Dydec
1.43
DyD STFF
1.51
CRNNNet 19]
1.33
1.77
DeDeNNe LogMel
1.33
1,50
Method
MSFL
MAEL
CRNNNet_ Dydec
1.20
1.51
DeDecNet MFCC
DyDeNet Gabor
1.33
1.48
1.49
DyDorNer SED
DyDecNNet, RegConus
2,09
1.33
3.06
1.39
DND-SED [16]
1.19
1.64
DND-SED Sydoe
0.89
1.40
DeDecNet
1.19
DyDecNot
0.85
1.19
level increases. Nevertheless, DyDecNet remains as the best one among all the three difficulty levels, showing DyDecNet outperforms the comparing methods under difficult levels discussed in this paper.
5.2 ABLATION STUDY
We do ablation study on NorthEastUS Bird data.
First, disentangling our proposed framework's dyadic decomposition frontend and backbone network so as to figure out their individual contribution. To this end, on the one hand, we concatenate dyadic decomposition frontend to the three SED methods backbone networks so that they can learn TF representation from raw waveform. We call them SELDNet dydec, CRNNNet _ddde and DND-SED dydec respectively. On the other hand, we feed our backbone neural network with fixed pre-extracted TF features, including short time Fourier transform (STFT), Log MFCC and Gabor Wavelet filter. We call them DyDecNet_ DeDeeNetSSTTT, DyDecNet_Laa LogMel and DyDecNet_M _FCC, DeDDeNeeGGaabr,
DyDecNet_ Gabor, respectively. The results are in Table 3 and 4. We can observe that: 1) replacing traditional fixed TF feature with dyadic decomposition frontend significantly improves the perfor- mance (Table 3). The gain stems from two-fold: our dyadic decomposition frontend enables the network to directly learn from the raw waveform so that all seaquuncc--sllectie filters are adjustable during training process. Second, the dyadic progressive decomposition enables the neural network to learn robust representation for sound counting. Similarly, huge performance drop is observed if we let our proposed backbone neural network to learn from traditional fixed TF features (Table 4). Therefore, it shows that both the dyadic decomposition frontend and backbone neural networks are important for sound counting task.
Second, we want to figure out if the dyadic decomposition is essential for sound counting, and the importance of energy normalization block. We test three variants: our network with simply single scale decomposition which means applying all filters the raw waveform (DyDecNet ( which helps validate necessity of hicrarchical dyadically decomposition framework; replacing Energy- normalization module with traditional batch normalization [23] (DyDecNet BN); without any nor- malization (DyDecNet (eteeNee___Noomm The result in Table 5, from which we can clearly observe that either removing energy normalization or replacing it with batch normalization significantly reduces the performance. It thus shows the importance of energy normalization.
Lastly To show the effectiveness of density map, run two ablation studies to directly regress the final count number or to firstly detect the sound events. From the result in Table 6, we can conclude that directly regressing sound event count number leads to inferior performance than estimating density map. Treating it as sound event detection problem leads to the worst performance.
Another ablation study on the impact of energy gain normalization on traditional TF feature is presented in Appendix Sec. D.5. We refer reader to this section for more details.
Limitation Discussion and Conclusion We do not discuss using microphone-aree array for enhanced counting, nor test our dyadic decompasition front-end for other acoustic tasks (c.g. source separation). Another limitation is that we just used one instance for each sound category in our synthetic dataset, which does not reflect the real scenario. A more convincing dataset is to involve as many diverse instances for each sound as possible, it also remains as future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Sharath Adavanne, Pasi Pertilä, and Tuomas Virtanen. Sound event detection using spatial features and convolutional recurrent neural network. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
[2] Rafael Aguiar, Gianluca Maguolo, Loris Nanni, Yandre Costa, and Carlos Silla. On the impor- tance of passive acoustic monitoring filters. Journal of Marine Science and Engineering (JMSE), 2021.
[3] Joakim Andén, Vincent Lostanlen, and Stéphane Mallat. Joint time-iimeeeeeee frequeecy scattering. In IEEE Transaction on Signal Processing. 2019.
[4] Juan Pablo Bello, Charlie Mydlarz, and Justin Salamon. Sound Analysis in Smart Cities. Springer International Publishing, 2018.
[5] Paul Brossier. Automatic annotation of musical audio for interactive system. PhD thesis, Queen Mary University of London, 2006.
[6] Joan Bruna and Stéphane Mallat. Scattering invariant deep networks for classification. In IEEE Transaction on Pattern Anlysis and Machine Intelligence (T-PAMI), 2012.
[7] Rui Cai, Alan Hanjalic, Hong-Jiang Zhang, and Lian-Hong Cai. A flexible framework for key audio effects detection and auditory context inference. IEEE Transaction on Audio, Speech and Language Processing (TASLP), 2006.
[8] Emre Cakir, Ezgi C. Ozan, and Tuomas Virtanen. Filterbank learning for deep neural net- work based polyphonic sound event detection. In International Joint Conference on Neural Networks (IJCNN) 2016.
[9] Emre Cakir, Giambattista Parascandolo, Toni Heittola, Heikki Huttunen, and Tuomas Virtanen. Convolutional recurrent neural networks for polyphonic sound event detection. In IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 2017.
[10] Yin Cao, Turab lqbal, Qiuqiang Kong, An Fengyan, Wenwu Wang, and Mark Plumbley. An improved event-independent network for polyphonic sound event localization and detection. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[11] Antoni B. Chan, Zhang-Sheng John Liang, and Nuno Vasconcelos. Privacy preserving crowd monitoring:
Counting people without people models or tracking. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
[12] Lauren M. Chronister, Tessa A. Rhinehart, Aidan Place, and Justin Kitzes. An annotated set of audio recordings of eastern north american birds containing frequency, time, and species information, 2021.
[13] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural ntworks on sequence modelling. In Advances Neural Information Processing System (NeurIPS), 2014.
[14] Steven Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recogmition in continuously spoken sentences. In IEEE Transactions on Acoustics, Speech, and Signal Processing (TASSP), 1980.
[15] Kota Dohi, Takashi Endo, Harsh Purohit, Ryo Tanabe, and Yohei Kawaguchi. Flow-based self- supervised density estimation for anomalous sound detection. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[16] Konstantinos Drossos, Stylianos Mimilakis, Shayan Gharib, Yanxiong Li, and Tuomas Virta- nen. Sound event detection with depthwise separable and dilated convolutions. International Joint Conference on Neural Networks (IJCNN), 2020.
[17] Francois Grondin, James Glass, Iwona Sobieraj, and Plumbley Mark D. A study of the complexity and accuracy of direction of arrival estimation methods based on gee-phat for a pair of close microphones. In Proceedings of the Delection and Classification of Acoustic Scenes and Events Workshop, 2019.
[18] Yuhang He and Andrew Markham. SoundDoA: Learn sound source direction of arrival and semantics from sound raw waveforms. In Interspeech, 2022.
10

---

Under review as a conference paper at ICLR 2023
[19] Yuhang He, Niki Trigoni, and Andrew Markham. SoundDet: Polyphonic moving sound event detection and localization from raw waveform. In International Conference on Machine Learning (ICML), 2021.
[20] Toni Heittola, Annamaria Mesaros, Antti Eronen, and Tuomas Virtanen. Audio context recog- nition using audio event histograms. In European Signal Processing Conference Conference (EUSIPCO), 2010.
[21] Sepp Hochreiter and Schmidhuber Juğen. Long short-term memory. In Neural Computation, 1997.
[22] Di Hu, Lichao Mou, Qingzhong Wang, Junyu Gao, Yuansheng Hua, Dejing Dou, and Xiao Xiang Zhu. Ambient sound helps: Audiovisual crowd counting in extreme conditions. arXiv preprint, 2020.
[23] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
[24] Eric J. Humphrey, Simon Durand, and Brian McFee. Openmic-2018: An open dataset for multiple instrument recognition. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), 2018.
[25] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representation (ICLR), 2015.
[26] B. Leibe, E. Seemann, and B. Schiele. Pedestrian detection in crowded scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005.
[27] Victor Lempitsky and Andrew Zisserman. Learning count objects in images. In Advances in Neural Information Processing Systems (NeurIPS), 2010.
[28] Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang. Pyramid attention network for semantic segmentation. In British Machine Vision Conference (BMVC), 2018.
[29] Yuhong Li, Xiaofan Zhang, and Deming Chen. Csmet: Dilated convolutional neural networks for understanding the highly congested scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[30] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, He Kaiming, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[31] Vincent Lostanlen, Justin Salamon, Mark Cartwright, Brian McFee, Andrew Farnsworth, Steve Kelling, and Juan Pablo Bello. Per-channel energy normalization: Why and how. IEEE Signal Processing Letters (SPL), 2019.
[32] Chen Change Loy, Ke Chen, Shaogang Gong, and Tao Xiang. Crowd counting and profiling: Methodology and evaluation. Modeling, Simulation and Visual Analysis of Crowds, 2013.
[33] Stéphane Mallat. A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way. Academic Press, Inc., USA, 3rd edition, 2008.
[34] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt Mc Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, 2015.
[35] Annamaria Mesaros, Toni Heittola, Tuomas Virtanen, and Mark D. Plumbley. Sound event detection: A tutorial. IEEE Signal Processing Magazine, 2021.
[36] Gabriel Michau, Gaetan Frusque, and Olga Fink. Fully learnable deep wavelet transform for unsupervised monitoring of high-frequency time series. Proceedings of the National Academy of Sciences, 2022.
[37] Dimitri Palaz, Ronan Collobert, and Mathew Magimai.-Doss. Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks. In Interspeech, 2013.
[38] Arjun Pankajakshan, Helen L. Bear, and Emmanouil Benetos. Polyphonic sound event and sound activity detection: A multi-task approach. 2019 IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pp. 323-327, 2019.
11

---

Under review as a conference paper at ICLR 2023
[39]
Giambattista Parascandolo. Heikki Huttunen, and Tuomas Virtanen. Acoustic event detection in real life recordings. In 18th European Signal Processing Conference CeeieenceeeUSIPPP (EUSIPCO), 2010.
[40]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Sys- tems (NewrIPS). 2019.
[41] H. Phan, T. N. Tho Nguyen, P. Koch, and A. Mertins. Polyphonic audio event detection: Multi-label or multi-class multi-task classification problem? IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[42] Mirco Ravanelli and Yoshua Bengio. Speaker recognition from raw waveform with sincnet. In In IEEE Workshop on Spoken Language Technology (SLT), 2018
[43] Selim Seferbekov, Vladimir Iglovikov, Alexander Buslaev, and Alexey Shvets. Feature pyramid network for multi-class land segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.
[44] Dan Stowell, Michael D. Wood, Hanna Pamuta, Annis Stylianou, and Herve Glotin. Automatic acoustic detection of birds through deep learning: The first bird audio detection challenge. Methods in Ecology and Evolution, 2019.
[45] Zoltán Tüske, Pavel Golik, Ralf Schlüter, and Hermann Hey. Acoustic modeling with deep neural networks using raw time signal for Ivesr. In Interspeech, 2014.
[46] Qi Wang. Junyu Gao, Wei Lin, and Yuan Yuan. Learning from synthetic data for crowd counting in the wild. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[47] Yuxuan Wang, Pascal Getreuer, Thad Hughes, Richard Lyon, and Rif Saurous. Trainable frontend for robust and far-field keyword spotting. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
[48] Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry, and Marco Tagliasacchi. Leaf: A learnable frontend for audio classification. International Conference on Learning Representa- tions (ICLR), 2021.
[49] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network, In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A SOUND COUNTING PROBLEM DEFINITION
Given a mono-channel seconds raw sound waveform zxt) sampled at a fixed sampling rate F.. the sound recording has recorded N independent sound events E = (E, sound event freely undergoes either stationary or moving motion in the open area. The target is to design neural network // parameterized by to predict sound event number N from raw sound waveform == N (())))))- In our formulation, the counting process is class-agnostic, so all sound events treated as instances to count, regardless of their elasses.
(e)) each single
Three challenges make it a challenging task: 1) Large Datasize: microphone usually records sound high frequency rate (i.e. 24 kHZ), resulting in large data size in the raw waveform. It thus requires more accessible filters with few parameters and computation cost to process the raw sound waveform. Concurrent Sound Events (polyphony): sound events freely overlap both spatially and temporally, resulting highly polyphonic sound recording. It is a tough task to separate them apart from compressed ID waveform. 3) Loudness Variance and Spectrum Overlap: sound events of the same class but different spatial location have large variance in their received loudness. They also have heavy spectrum overlap the frequency domain. The above issues make counting tough task.
B MORE DISCUSSION ON DATASET CREATION
B. MOTIVATION OF POLYPHONYY4BRRDS AND POLYPHONY IBIRD DATASET CREATION
Our motivation of synthesizing Polyphony4Birds and Polyphony Birds are three-fold:
1. NorthEastUS Bird dataset has as many as 48 different kind of bird categories. helps to test various methods' capability in tackling high bird diversity challenge.
2. Polyphony4Birds dataset contains kinds of bird sounds, but in much higher polyphony level (in terms of ratio-polyp, mar ppotppppppppppooo and mean-polyp). It helps us to test various methods' capabil- ity in tackling limited bird categories but high polyphony level (heterophony test).
3. Polyphony Bird dataset contains 1 bird sound class in much higher polyphony level. This dataset involves heavy spectrum-overlap (due to the temporal inter-category bird sounds overlap), so it helps test various methods' capability in tackling high spectrum-overlap and high-polyphony challenge (homophony test).
In Polyphony4Birds dataset, 4 is an arbitrary number. We experimentally find involving 4 bird sounds representative enough for heterophony test. We note that there are some other relevant public bird sound dataset [44, 20, 9], but we find they not suitable for our study. For example, in TUT-SED 2009 data [20], the polyphony-level is small and the involved bird sound usually lasts too long (not temporally separable and countable). Similarly, the Bird Audio Detection challenge (BAD challenge) [44] contains highly-sparse bird chirps (very small polyphony-lveel sound). Moreover, the two real-world bird sound datasets [20, 44] do not provide bird sound start time and end time label, so they are suitable for our study. The other synthesized dataset TUT-SED Synthetic 2016 [9] also contains very limited samples of high polyphony. The direct comparison between these datasets is given in Table I, from which we can see our created two datasets enjoy much higher polyphony-level, making them more suitable for our sound counting task.
Table I: Comparison between various sound dataset, where "n/a" means not available.
Dataset
Data Source
Size
Event Classes
mean-polyp n/a n/a
max mlax-polyp
BAD Challenge [44]
Natural
23 h
2
TUT-SED 2009 [20]
Natural
18.9 h
61
TUT-SED Synthetic 2016 [9]
Synthetic
9.3
16
0.659
NorthEastUS Bird [12]
Natural
6.41
48
0.
Ours Polyphony4Birds
Synthetic
55,56 h
4
1.244
Ours Polyphony IBird
Synthetic
55.56 h
1
1.975
13

---

Under review as a conference paper at ICLR 2023
Table II: Accuracy Rate results on the six datasets.
OpenMIC
NorthEastUS
Polyphony-BBirds Accu, Rate
Polyphony BBird Accu.
TelephencRing
CarEngine
Dataset
Method
Accu, Rate
Accu. Rate
Rate
Accu. Rate
Accu. Rate
p=1
p=0
p=0
p=1
0
Librosa-onset
0.05
0.03
0.20
0.58
0.01
0.02
0.01
0.02
0.01
0.01
0.01
0.01
Aubio-onset
0.22
0.12
0.32
0.08
0.09
0.02
0.01
SELDNer [17]
0.49
0.75
0.25
0.62
0.48
0.71
0.33
0.81
0.28
0.69
0.31
0.78
CRNNNet [9]
0.49
0.85
0.26
0.64
0.47
0.82
0.35
0.81
0.27
0.76
0.30
0.80
DND-SED [16]
0.70
0.28
0.24
0.71
0.63
0.20
0.68
DyDecNet
0.68
0.92
0.40
0.82
0.70
0.88
0.55
0.92
0.53
0.89
0.57
0.94
Table III: Comparison of Various Methods. The network block column labels are: 1. ID Conv, 2. 2D Conv, GRU, 4. LSTM, 5. Depthwise Conv, 6. Dilated Conv. 7. FC, 8. Bi-LSTM, 9. Bi-GRU. The inference time in tested on Intel(R) Core(TM) 19-7920X CPU, we report the average time of 100 independent tests with one 5s waveform.
Method Name
Input
Param Size
Network Block
Inf. Time
end2end trainable?
Librosa-onset
Raw Waveform
0.1s
Aubio-onset
Raw Waveform
0.1s
x
DND-SED [16]
LellL
6.9 M
6,
s,0
x
CRNNNet [9]
STFT
4.1 M
3,
3.3
x
SELDNet [17]
LogMel
0.8 M
3. 7.9
1.2s
X
DyDecNet
Raw Waveform
3.9 M
1.7
2.7
B.2 HOW TO SIMULATE OPEN AREA ENVIRONMENT
We collect seed sounds from copyright-free website 1: junco, American redhead, eagle, and rooster. To maximally reflect outdoor scenario, simulate large openarea environment [100m, 100m, 100m] with one microphone at [50m, 50m, 1m]. The wall is associated with high sound absorption coefficient, the reverberation is negligible scenario. We introduce a random SNR (Signal-to-Noise Ratio) at two Gaussian means (-33 decibels and -20 decibels) at the microphone receiver. We put each seed sound at random 3D spatial location and a random start time to imitate natural bird sounds that emit sound from random location and random start time. A post-processing step is added keep dataset balance between various polyphony-level metrics.
resemble outdoor open
C MORE DISCUSSION ON COMPARING METHODS
More detailed comparison between various methods is given in table III. We can see that our proposed DyDecNet is lightweight and directly learns from sound raw waveform (so it is end-to-end trainable). thus strikes a good balance between model performance and model efficiency (inference time).
D MORE EXPERIMENT RESULT DISCUSSION
D.I EXPERIMENT ON NORTHEASTUS BIRD DATASET AND TELEPHONE RING DATASET
More detailed experimental result (MAE variation) on NorthEastUS Bird is given in Fig. 5, from which we can observe that with the increasing of maz-polyp. ratio-polyp and mean-polyp. all methods (including our DyDecNet) reduces their performances. The three comparing meth- ods (CRNNNet [9J.DND-SED [16], and SELDNet [17]) have observed sharp performance drop when the our proposed three sound counting difficulty levels increases, whereas our proposed DyDecNet largely mitigates the challenge caused by higher counting difficulty level (the blue line increases slightly as the counting difficulty level increases). thus shows 1) our proposed maz-polyp, ratio- polyp and mean-polyp are capable of accurately measuring sound counting task difficulty level from different perspectives; 2) our proposed DeDee is capable of mitigating these sound counting difficulties.
httpttt///// www. findsounds.dnn.. com/
14

---

Under review as a conference paper at ICLR 2023
2.5
5
4-
2.0
MAF
3-
MAF
1.5
MAE
3
2-
1.0
2
1.
0.5
1
0.0
0
2
4
0.4
0.6
0.8
1.0
0
2
Max polyp
Ratio polyp
Mean - polyp
CRNNNet
DND-SED
SELDNet
DyDecNet
Figure I: MAE variation against max-polyp, ralio-polyp and mean-polyp on NorthEastUS_Binn
Dataset.
D.2 MORE RESULT ON POLYPHONY I BIRD AND POLYPHONY4BIRDS DATASETS
We also provide the detailed results for Polyphony Bird and Polyphony-Birds in Fig. II and Fig. III, respectively. They contain the accuracy rate, MSE and MAE variation against maz-polyp, ratio- polyp and mean-polyp. From the two figures, we can get similar conclusion as of NorthEastUS, Bird dataset (Fig. I): with the increasing of maz-polyp. ratio-polyp and mean-polyp. all methods' performance gradually reduces. Our proposed DyDecNet stays as the best-performing one under all sound counting difficulty level metries. Specifically, we can see that:
All methods give the hest performance on Polyphony4Birds dataset, second best performance Polyphonyl Birrd Bird dataset, and the worst performance on NorthEastUS Bird dataset. It thus shows 1) spectrum-overlap due to high inter-class sound overlap temporally (represented by Polyphony Bird dataset) remains in open area where noise pollution, high sound diversity (in our case, diversity means bird categories, we have 48 bird classes in NorthEastUS Bird dataset), and small labelled data availability exist remains as another challenge for sound counting task. We hope to attract more researchers to consider sound counting task more challenging scenario.
challenge for sound counting task. 2) sound counting
We do not observe such sharp performance drop (as we observed on NorthEastUS Bird dataset) on our two synthetic datasets, which is in contrast with the real-world dataset NorthFastUS. Bird. thus shows real-world sound counting task becomes increasingly challenging when our proposed three sound counting difficulty level metrics increase. We guess the large model and large training dataset are needed to achieve better performance, which can be treated future research direction.
D.3 COUNTING ON MORE BIRDS CLASSES
In the main paper, our two synthetic datasets Polyphony4Birds and Polyphony Bird have just involved limited bird classes (up to 4). We naturally want to figure out the performance of all methods (including DyDecNet and the other three comparing methods) under more bird classes situation. We thus follow more the same data creation procedure synthesize four extra datasets. They contain 2/6/8/10 bird classes, respectively. The extra bird seed sound classes are collected from findsound. com too. The quantitative result is given in table V, from which we can learn that all comparing methods have observed performance increasing when the bird classes reach to 6 (values in bold font), then performance decline when bird classes increase to 8 or 10; DeCece reaches the best performance around 8 bird classes, then begin to decline. It thus shows: 1) all methods can successfully handle a reasonable amount of bird classes (in our case, maximum bird classes are 8), given the model parameter size budget (less than 10 M) discussed in this paper. When we have to handle much larger bird diverse classes, we might need much larger model (which remains as a future research topic figure out the relationship between model size and sound counting class diversity); 2) Our proposed DyDecNet exhibits strong capability sound counting in diverse bird classes than the three comparing methods (it reaches the best performance at higher bird classes (8 bird clas))..
15

---

Under review as a conference paper at ICLR 2023
2.5
2.00
0.6
1.75
2.0
0.5
1.50
Accuracy
1.5
1.25
0.4
MSE
MAE
1.00
1.0
0.25
0.2
0.00
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Ratio polyp
Ratio polyp
Ratio
polyp
2.5
2.00
0.6
1.75
0.5
Accuracy
1.5
1.25
0.4
MSB
MAE
1.00
1.0
0.75
0.3
0.50
0.5
0.25
0.2
0.0
0.00
2.0
2.5
3.0
3.0
3.5
3.0
3.5
Mean
polyp
Mean polyp
Mean polyp
2.5
2.00
0.6
1.75
2.0
0.5
1.50
Accuracy
1.5
1.25
MAE
0.4
1.00
0.75
0.50
0.25
0.00
8
5
8
Max polyp
Max polyp
Max polyp
CRNNNet
DND-SED
SELDNet
DyDecNet
Figure II: AccuRatc/SE//MAE variation against max-polyp, ratio-polyp and mean-polyp on
Polyphony IBird Dataset.
D.4 NOISE DISCUSSION
In order to show DyDecNet and other comparing methods' performance under different noise inter- ference level, we show the performance Polyphony-Birds dataset under various noise level (SNR, Signal-to-Notee Noiso Ratio) Fig. IV). We can clearly see from this figure that DyDecNet exhibits better robustness to noise than the three comparing methods. Its dyadic decomposition strategy (differentiat- ing between approximation and detail explicitly) and the proposed energy normalization help reduce noise interference.
16

---

Under review as a conference paper at ICLR 2023
2.5
2.00
0.75-
1.75
0.70
2.0
1.50
Accuracy
0.65
1.5
1.25
0.60
MSE
MAE
1.00
0.55
1.0
0.75
0.50
0.50
0.5
0.45-
0.25
0.0
0.00
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Ratio
polyp
Ratio polyp
Ratio polyp
2.5
2.00
0.70
1.75
0.65
2.0
1.50
Accuracy
0.60
1.5
1.25
MSE
0.55
1.00
1.0
0.75
0.50
0.50
0.45
0.5
0.25
0.40
0.0
0.00
2.0
2.5
3.0
3.5
2.0
2.5
3.0
3.5
2.0
2.5
3.5
Mean
polyp
Mean polyp
Mean polyp
2.5
2.00
0.7
1.75
2.0
0.6
1.50
Accuracy
1.25
0.5
MAE
0.4
0.50
0.3
0.25
0.2
0.0
0.00
5
6
8
9
8
9
Max - polyp
Max polyp
Max polyp
CRNNNet
DND-SED
SELDNet
DyDecNet
Figure III: AccuRate/MSE/MAE variation against max--poyp, ralio-polyp and mean-polyp Polyphony4Birds Dataset.
D.5 LEARNABLE ENERGY NORMALIZATION WITH TRADITIONAL T-F FEATURE
To test the efficiency of our proposed energy normalization module, especially combining them with traditional one-stage T-F features, we explicitly add one learnable energy normalization module just after the T-F feature extracted by traditional time frequency feature extractors. The comparison is given in Table VI and Table VII, from which can observe that performance of traditional T-F feature slightly increases after introducing the energy normalization module. thus shows the necessity of energy normalization for sound counting task high-polyphonic situation. However, they still lead to inferior performance than DyDeNNet, which shows hierarchically dyadic decomposition with energy normalization is essential for sound counting task.
17

---

Under review as a conference paper at ICLR 2023
Table IV: Dyadic Decomposition Neural Network Architecture Illustration. Input audio is 5 long, sampling rate is 24 k Hz. The input waveform and intermediate features are in [channelnum, height, width] format. In the dyadic decomposition front-end, the constructed filters' initialized trainable frequency cutoff's (high frequency cutoff and low frequency cutoff) evenly divide the frequency range the input sound waveform (half of sampling rate). In our design, the filter number doubles as the "Depth" increases by 1. So each filter in the preceding depth associates with two child filters in the next depth, in which the two child filters carried frequency cutoffs evenly divide the frequency cutoff of their parent filter (see Fig. I in the main paper). In implementation. we just connect a filter in the preceding depth with its two child filters in the next depth. We organize filters in channel dimension. In the dyadic decomposition front-end, each filter is instantiated as Sinc(-) filter, which comprises of a learnable high frequency cutoff' and low learnable frequency cutoff.
Layer Name
Filter Num
Output Size
Input Size: 5 s audio waveform: [1. I, 24000 x 5]
Dyadic Decomposition Front-End
Dyadic Decomp. Depth
filter num 2 2x downsampple
[2. 1, 12000
Dyadic Decomp. Depth 2
filter num = 22, 2x downsampple
[4, 1, 6000 5]
Dyadic Decomp. Depth
filter num = 23, 2x downsampple
[8, 1, 3000 x 5]
Dyadic Decomp. Depth
filter num 2 2x downsampple
[16. 1, 1500 5]
Dyadic Decomp. Depth 5
filter num = 2 2x downsampple
[32, 1, 7,0 5]
Dyadic Decomp. Depth 6
filter num = 2
[64, 1, x 5]
Dyadic Decomp. Depth
filter num 2
[128, 750
Dyadic Decomp. Depth 8
filter num 2
[256, 750 x 5]
Backbone Network
Per-channel Pool
LoneLow Filters, stride = 5
[256, 150
Cross-channel Conv.
1D Conv., filter num = 512
[512, 150 x
Per-channel Pool
SincLowPass Filters, stride =
[512, 150 x 1]
Cross-chamnn Conv.
1D Conv., filter num 1024
[1024, 1, 150 x 1]
Per-channel Pool
SincLowPass Filters, stride = 3
[1024. 50 x 1]
Cross-chandn Conv.
ID Conv., filter num 512
[512, 1, 50 1]
Cross-channel Conv.
ID Conv., filter num 256
[256. 1, 50 1]
FC
FC, output feat =
[50, 1]
Table V: MSE/MAE/AccuRR. results Multiple Bird Classes. The three values split by ' in each entry indicate MSE, MAE and AccuRate, respectively. For the AccuRate, we just report the accuracy each model 10 times
rate under p O. Following the experiment setting in the main paper, independently. We do not report the standard deviation, they are all within 0.003.
Bird Classes
SELDNet [17]
CRNNNet [9]
DND-SED [16]
DyDecNet
1 Bird
0.89/ 1.19 0.33
0.87 187/1111/00 0..35
1.04 1.27 / 0.24
0.54 /..4 / 0.55
4 Birds
0.92/ 1.41 0.48
0.74 0.47
0.95 1.34 / 0.43
0.46 / 0.92 / 0.70
6 Birds
0.88 1.40 0.49
0.09/1009/0 0.49
0.93 1.26 0.48
0.43 / 0.90 / 0.74
8 Birds
0.93 1.45 0.40
0.95 1.37 0.42
0.41 / 0.88 / 0.77
10 Birds
0.95 / 1,49 0.37
0.78 / 0.17
0.97 / 1.30
0.44 / 0.82 / 0.72
E NETWORK ARCHITECTURE
The DyDecNet architecture is shown in Table IV.
F DENSITY MAP EXPLANATION
We provide more detailed illustration about density map in Fig. V. Specifically, the dyadic decomposi- tion front-end and backbone network learn 2D time-frequency feature representation (sub-figure A), in which the time dimension size 50 (equals to s audio length, each feature has a time resolution of 100 milliseconds). Such representation can be used to either regress the density map (as
18

---

Under review as a conference paper at ICLR 2023
2.0
1.5
MSE
1.0-
0.5
-40
-35
-30
-25
-20
-15
SNR (dB)
Figure IV: MAE variation under different signal-to co-raioo Different color indicates different methods, which aligns with the color in other figures the paper. Light Green: CRNNNet, Black: SELDNet, Magenta: DND-SED, Blue: DeDecNe.. The horizontal axis goes (from left to right) with reduced noise interference.
Table VI: DyDecNet with Traditional T-F feature and learnable energy normalization module
Table VII: DyDecNet with Traditional T-F fea- ture
Method
MSE,
MAEI
Method
MSEJ
MAEI
DyDecNet STFT
1.30
1.46
DyDecNet STFT
1.35
1.51
DyDecNet LogMel
1.27
1.47
DyDecNet LogMel
1.33
1.50
DyDecNet_MPCC
1.26
1.44
DyDecNet_MFCC
1.32
1.49
ByDecNet_Gabob
1.30
1.43
DyDecNet_Gaab
1.33
1.48
DyDecNet
0.85
1.19
DyDecNet
0.85
1.19
do, sub-figure C) or classify the event class for each time frame (as sound event detection methods do, sub-figure E), by adding full-connection layer (FC) to reduce the feature dimension size (can also be treated as frrquency dimension) to 1. At the same time, can use two full-connection layers to reduce the 2D representation to a scalar value, so as to directly regress the sound count number (sub-figure D).
The process of constructing density map is shown in sub- sub-figure B. Please note that since we use supervised learning, know each sound event start time and end time. So the density map be easily constructed by setting the same value to the range between the start time and end time so that they are added up sound count number. Adopting density map for counting problem is widely used vision-based crowd counting tasks [49, 26]. they show predicting density map usually give superior performance than object detection methods (in our case, SED method), and direct regressing count number. Their conclusion in aision-based crowd counting tasks matches our experimental result in sound-base counting tasks.
The reason why density map based method outperforms SED and direct regression lie two fold (ac- cording to our understanding): 1) unlike SED methods that try to discriminate different sound event class from temporally overlapping sound input, density map based method ignores sound event class but instead treat all sound events as an instance spanning their active time range. The reduced difficulty enables the neural network (DyDecNet) to learn expressive representations. 2) One characteristic of sound events is that each sound event has a certain start time and end time in the time dimension. The 2D time-frequens dyadic decomposition front-end naturally maintain such characteristic. Using the 2D time- tmeefrequeecy representation regress density map internally exploits the temporal location of various sound events, which helps sound counting task.
representation learned by the backbone network and
19

---

Under review as a conference paper at ICLR 2023
50x
1/27
1/2
1/13
1/2s n
E
FC
FC
Dansily Map
1/23 1/2s
Los
Loss
Event:: [2.0s, 4.5g]
Count:
1//2
26
Ci event class Prod Count: aggregate
Event:: [0.3s,
Prod Count
Sum density map
Time AXI5
eevents
A. Backbone Learned Feature
B. Ground Truth Demsity Map
Count from Density Map D. Direet Prediction
SED Method
Figure V: Density map illustration. Each single sound event corresponds to individual density map, which vector spans from the input audio length. The density value covers the whole event (connects its start time and end time) and sums to 1. For example, for the "Eventl" with start time 0.3 s and end time 2.6 s in sub-figure B, its density map is vector of 50, the values lie in [3, 26] 23 Finally, the overall density map for input audio is obtained by summarizing all events' density map together. The sum (or integral) of the overall density map equals to the sound count number involved in the audio (in this case, 2). The comparison between three count methods shown in sub-figure C, D and E, respectively. Given the feature representation learned by dyadic decomposition front-end and backbone network, further use a full-connection layers (FC) to reduce the channel dimension to 1 but keep the time dimension, we can obtain a vector of the same density map size. We can either regress the density map (sub-figure C) or classify the event for each time step (SED method, it is multi-label classification, sub-figure E): 2) two full-coneection layers (FC) consecutively reduce both channel dimension and time dimension 1, so that can directly predict the count number (sub-figure D).
20


Reviews:
[{"review_id": "nqDV0GoAZSN", "reviewer": "Reviewer_6EcX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose DyDecNet, a new network architecture for counting sounds in highly polyphonic environments. The method contains a series of T-F band-pass filters which double in each layer to provide a learnable alternative to other frequency representation methods. It then contains a backbone and regresses the final count of sound sources. They also propose 3 new metrics for measuring polyphonic sounds.", "summary_of_the_review": "Interesting approach, good results and experiments. Paper needs to be more clearly presented and qualitative results/examples would strengthen the work a lot by giving the necessary context for the problem. ", "strength_and_weaknesses": "The paper overall is good and interesting. The problem is one that has not been explored as thoroughly, and they are tackling highly polyphonic sounds in a way that hasn't been done before. The idea of learnable TF filters that grow exponentially with depth is interesting, and the ablation studies show that it is better than using other features like Mel or STFT features. The results seem pretty good, able to achieve solid counting accuracy with high numbers of concurrent sounds.\n\nThe clarity and organization of the paper could be improved significantly. For example, the section with Table 1-4 are very confusing. It should be possible to look at the tables and interpret them without searching for the description in the text. Table 1 is a dataset description, 2, and 3 are results, and 4 is an ablation study. But they are all presented as one unit, making it confusing. The caption for table 3 needs to say what dataset it was on, and table 4 should say \"ablation study\" and probably be included further down when the text is talking about the result. All the tables in general could do with a slightly more descriptive caption describe what result is being presented. I had to search through the text and back at the table to understand them.\n\nI would also like to have some supplementary results to listen and compare  qualitatively . Hearing some of these datasets would be great (both for reviewers and listeners). It would also help us understand how difficult this problem is for human listeners. The ideal demo would be a video of highly polyphonic sounds with the prediction from your network being overlaid as the video plays.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lCkLMuItA22", "reviewer": "Reviewer_xW8d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new task, sound counting and a novel neural network architecture to tackle it based on a dual decomposition of the raw waveform. The proposed Dyadic model is first presented, along with new metrics for the task. The model is then evaluated on four partially synthezised datasets and compared with baseline methods. The proposed model is shown to outperforms the baselines on every dataset. Ablation studies are also presented, validating the different aspect of the architecture.", "summary_of_the_review": "The paper introduces a new audio task and proposed a novel approach to solve it, which outperform the baselines. The architecture is novel as far as I can tell and the findings are significant, so I am keen to accept the paper if all of my concerns about clarity are addressed. But in the current form I recommend rejection.", "strength_and_weaknesses": "Strengths:\n - The paper introduces a new task, proposes a novel model to tackle it and provides thorough comparison with baseline models.\n - The paper introduces new metrics for the task.\n - The proposed approach is significant as it outperforms the baselines\n\nWeakness:\n - The paper is not very clear:\n      - It sometime makes some very bold and unsupported statements (see below)\n      - Some references are missing and the proposed approach needs to be situate better with the existing works.\n      - The experiments are a bit confusing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XSWiZr2OH9", "reviewer": "Reviewer_5ud3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduce a soundcount method. Their method is about how to conunt the number of distinct sound event. ", "summary_of_the_review": "I spend about 1 hour to review this paper. I just feel that the authors needs to provide more details about how do they solve the domain generalization problems. ", "strength_and_weaknesses": "I am not a person in acoustic data. Everything in this area is interesting to me. My comments for this work are listed as follows:\n1. Can the author better claify the correlation bettwen this work and blind signal speration? It seems that this work has some close correlation with that problem. \n2. I am a person in image separation. I know that such kinds of problems are very difficult.Even with different priors or assumption, we still cannot infer some highly mixed image contents. I think it should also be a problem for the acoustic issue. How do they address this problem in your work?\n3. Do you need to consider the domain adaption issues? Or do you need to rely on some specific priors to achieve this goal about sound count?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R9-CoIBKh9y", "reviewer": "Reviewer_6cJE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes:\n1) A novel model architecture and training method for the task of sound source counting.\n2) Several novel measures to assess the difficulty of the sound source counting task.\n\nThe model architecture comprises 1) a feature extractor that operates hierarchically as a binary tree of cascaded filters, 2) a mechanism to pool across filters, and 3) 3 different methods to compute the output count from the pooled features.  Counting can be done both per-frame and per-session.\n\n3 measures are proposed to assess the difficulty of the sound source counting task, all related to the amount of polyphony present in the audio.\n\nMultiple datasets are used to assess the performance of the proposal, but public release of the created synthetic datasets is not reported in the paper for experimental reproduction.\n\nQuestions:\n- What aspect of the model's performance is the chosen depth expected to impact?  Does the depth affect the effective maximum number of sources that the model can count?  Does the depth affect the frequency resolution that the model can rely on?  Does the depth affect the high frequency limit that the model can use?  How does the exponential scaling of the number of filters with depth affect the computational cost of training and inference?\n- What are the sizes of the train and test splits of each dataset?  Such information is important to allow a reader to judge the applicability of the measured results.\n- No reference is given to justify the motivating assumption of \"human’s capability in discriminating different sound events reduces\nseriously when the number of temporal-overlapping sound event number increases\".\n- In the statement \"The child filter carrying the higher half frequency response\nencode the parent’s processed intermediate waveform’s detail while the other one carrying the lower\nhalf frequency response instead encodes the approximation\", why does the higher half encode detail and the lower half encode approximation?  What do you mean by \"approximation\" and \"detail\"?\n\nIthenticate similarity score is 1%, which is good.\n", "summary_of_the_review": "The claimed novelty of the task and model are somewhat diminished by the existence of prior works in diarisation and sound source separation.  However, the proposed task difficulty measures still appear fairly novel.  Taken together, the novelty in this paper should suffice.\n\nSufficient details are given in the paper for adequate experimental reproduction.\n\nThe proposed approaches seem well justified.\n\nThere is a need to further proofread the paper, to minimise spelling and grammatical inaccuracies.\n", "strength_and_weaknesses": "Strengths:\n- The design of the proposed model and training, and the spectrum of datasets experimented upon, seem quite well thought through, to overcome many potential questions that may have otherwise arisen.  Specific justifying examples are:\n-- The use of the hierarchical filtering binary tree seems to be well justified, based on an understanding of the expected acoustic properties of the sounds involved.\n-- The separation of the model into feature extractor and backbone provides a natural taxonomisation of the model parts to different functions, while still being able to be jointly trained.\n-- The model is able to count the sound sources both per-frame and per-session.\n-- The use of energy gain normalisation shows an example of the authors carefully considering potential issues that may go wrong, for this specific task.\n- The experiments are repeated on multiple diverse datasets, to demonstrate generalisable trends of the proposal.\n\nWeaknesses:\n- There is a lack of literature review to the highly related fields of speaker diarisation and sound source separation.  Sound source separation is only mentioned briefly in the conclusion, and no reference is given to related papers.  Speaker diarisation consists of two subtasks; 1) counting the number of speakers in an audio session, and 2) clustering the time segments of audio that belong to the same speaker together.  Sound source separation consists of two subtasks; 1) Counting the number of sources in an audio session, and 2) computing the audio of each separated source.  In both speaker diarisation and sound source separation, the two subtasks can either be done jointly within a single model or separately in a cascaded manner.  If done separately, then the first subtask is exactly the same task that this current paper is trying to address.  If done jointly, then the speaker count can be computed from the joint output, which indirectly also does the task that this current paper is trying to address.  Both speaker diarisation and sound source separation have several publications that consider neural network methods to either do these subtasks separately or jointly.  Thus, the existence of these works that have not been described in this current paper may somewhat reduce the novelty that this current paper claims.\n\n- There are many spelling and grammatical inaccuracies in the paper.  Please proofread the paper.\n\n- The created synthetic datasets do not seem to be made publicly available for experimental reproduction.  However, the paper also uses existing public dataset, which should allow the results to be experimentally reproducible.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DQSBHE7X7tf", "reviewer": "Reviewer_3j8s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses a new network architecture for counting the number of sound events in an audio signal, primarily focusing on bird calls but also evaluating with other sound classes. The main contribution of the work are two-fold:\n1. A novel frontend for raw audio feature learning called dyadic frequency decomposition. This involves progressively increasing number of band-pass filters focusing on extracted information from high frequency and low frequency content of the parent filter's outputs. Additionally, a learned energy normalization layer is applied at each layer for improved loudness invariance. This approach converts raw waveform to a frequency representation using a multi-stage approach as opposed to other approaches which use a single-stage filterbank. \n2. The authors propose new polyphony-based difficulty metrics for tasks involving polyphonic audio. They rate the datasets they use for evaluation using these metrics. They find that the results are consistent with the difficulty of the datasets.\nComparison against baselines from sound-event detection literature show that the so-called DyDecNet outperforms these baselines on most metrics. These comparisons are carried out using various modalities of audio data such as bird sounds, everyday sounds like phone calls, car engine noise, as well as music.\nVarious ablation studies show the benefits of the different techniques used in the network.", "summary_of_the_review": "My overall impression is that the authors identified an issue in existing learnable audio frontends and proposed a new network architecture to address those issues. However, they demonstrate its effectiveness using the sound-counting task which is not a very popular area of research and could be considered a novel task. The issue lies in the fact that it is not clear how the new architecture is specialized for counting. The paper would be stronger if the evaluations included additional tasks like audio classification, source separation, etc. \nAdditionally, while I commend the authors including a music dataset since music is often highly polyphonic, based on my knowledge it would seem that there is a mismatch between the dataset chosen and the task the network is trained for as I have described in the weakness section. Given these points I am choosing a rating of 3.", "strength_and_weaknesses": "Strengths:\n\n- The front-end network is novel and involving a multi-stage filterbank design for audio representation learning. The authors make a strong case for this frontend compared to traditional approaches for fixed or learnable filters for audio.\n- The results obtained are strong compared to the baseline approaches.\n- The ablation studies account for most if not all moving parts of the proposed model DyDecNet. \n- The authors evaluate on a combination of real and synthetic datasets. The synthetic dataset is constructed using non-naive approaches involving room acoustics simulation and stochasticity.\n\nWeaknesses:\n\n- The Dydecnet architecture is presented as a generally more useful frontend for processing audio in neural networks, however it is only evaluated using sound counting as the task. It is not clear how this architecture is specifically useful for the counting problem. This paper has the potential to be a really strong paper had the authors not focused only on sound counting but also evaluated for other audio-related tasks as well, such as SED, source separation, etc.\n- I am very familiar with the OpenMIC dataset for musical instrument classification. The authors mention that it contains the number of instrument events present in each clip. This is not true. The OpenMIC dataset is a partially labeled dataset where each audio clip is labeled with whether a subset from 20 instruments is present or not. As an example, there may be a clip which is labeled with 'vocals', 'guitar' and 'bass' present, and 'flute' absent. This does not mean that the clip has only 3 counts of sound events. It could have any number of notes being played by each instrument. Additionally, this also does not mean that the remaining 16 instruments are absent. It could very much be the case that there are 'drums' and 'violin' playing in the clip. They are just not labeled during the crowdsourcing campaign. This makes me question the validity of any experiments involving this dataset since the regression target is neither the number of sound events bring played, nor the number of unique instruments present. Perhaps a different music dataset should have been using such as MusicNet (https://zenodo.org/record/5120004).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject