## 1 Introduction  
The audio-based automatic recognition of bird species has become an increasingly common and effective method in the context of bird species monitoring, studying the behavior of birds, and understanding their communication patterns [1, 2]. Notwithstanding the advantages of using bird vocalizations to infer ecologically relevant information, there are certain challenges associated with processing field recordings to produce robust results. Unattended field recordings can be quite noisy. Depending on the distance from the recording device, sound clips can be faded or distorted, and recordings can include overlapping sounds from the same or different bird species. Several authors [3, 4] have addressed the influences of noise by adding artificial noise to recordings.  
On surveying the literature available on automatic audio-based recognition of bird species [5, 6, 7, 8, 9], it has been found that most analysis has been performed using less noisy recordings and relatively small datasets, as has already also been indicated in the review paper [2]. The measures for classification success that different authors use to report their results vary, which makes it difficult to compare the performance of different classifiers [8, 10, 11, 12, 13, 14].  
However, the way in which a set of species is selected for the classification experiment can influence the classification success. It is clearly easier to distinguish bird vocalizations that are qualitatively very different types of sounds, such as sounds produced by crows or song-birds.  
One of the main focuses of this contribution is to look into the robustness of the classification results and the reliability of accuracy measures when the bird species are selected in a randomized way. Additionally, the influence of the number of selected species (classes) on different measures for multi-class classification success is investigated (see Section 3.4). For this work, we use the dataset curated by the organizers of the BirdClef 2019 challenge [15]. This dataset contains recordings of 659 bird species from South and North America and was originally drawn from the Xeno-Canto repository for bird sounds [16].  
In this contribution, out of 659 available species, \(n\) species were randomly drawn with varying \(n\) incrementally between 10 and 300. For each \(n\), we generated 20 randomly composed lists of species, and, for each species, we chose 200 recordings of comparable length to generate balanced subsets (as will be explained in more detail in Section 2.1). The robustness of classification results is then accessed by training a feed-forward neural network on each subset (see Section 2.3).  
Our motivations to choose a shallow feed-forward neural network over very deep networks lie mainly in the model simplicity, the lower computational costs, and the relatively small amount of data required to train such networks. We wanted to analyze the classification performance using a simple model that can be trained with handcrafted sound features. The performance of classification is accessed using several measures for classification success, such as accuracy, precision, recall, area under the receiver-operator-characteristics curve, and the mean average precision (see Section 2.4).  
As a consequence of these repeated randomized classification experiments, we can, hence, also provide box-and-whisker plots for each performance metric (see Section 3). Additionally, it is possible to infer functional relations between the number of classes and the classification success (see Section 3.2). Furthermore, a discussion is included on how confidently the probabilistic classifier classifies different species. In more detail, we discuss whether the predictions made by the model are proportional to the probabilistic confidence that the model assigns to the predictions and how this can be used as a measure to evaluate the performance of the classifier (see Section 3.3).