dataesr/scientific-paragraphs-categorization huggingface dataset:
The reported results are the harmonic mean over all the datasets, which is appropriate for combining percentage results, and ensures that a system should perform well across all datasets to achieve a strong score. We thus use an averaged F-score as our main summary statistic for each submitted system. To explore system performance in more detail, we also inspect the F-scores per dataset, and per class in each dataset, in particular to examine whether differences in acoustic characteristics correlate with differences in performance. The F-score metric is designed to summarise how well a system's outputs correspond to the desired outputs. However, there are many factors that affect the usefulness of such outputs, meaning that it is difficult to estimate a technology readiness level from only numerical scores.
Results We report here the results of our public challenge. We have conducted three editions to date  (2021, 2022, 2023) , and each year the evaluation dataset has been extended to cover a wider range of bioacoustic sources. After the first edition, it was agreed that evaluation datasets should be refined and expanded to give a more robust estimation of system performance. We thus report here on the systems submitted to the second and third editions, evaluated on the datasets of the second edition for comparability. For completeness, a summary of the 2021 outcomes is given in the Supplementary Information. In the 2022 edition, 15 teams participated submitting a total of 46 systems and in 2023 there were 6 teams with a total of 22 systems. We present in Table  2  the overall scores of the best system submitted by each team in these two editions of the challenge.
Analysis of dataset dependencies The submitted systems exhibit variations in their performance across our datasets (Figure  5 ). The same is true even when we look at the target class level, Figure  6  presents the Fscore results for the different target classes in each dataset. The easiest classes to be detected are CHE_chaffinches, CT_chirpgrunts and DC_robins, where several systems reach above 75% F-score. On the other side, CT_Chitters, DC_Cuckoo and the QU_Quacks seem to be the classes where systems struggled the most to make correct predictions. The disparity in score between systems is also evident. The performance on MGE_Chick_Pleasure_calls is a good example where Du_NERCSLIP's systems show a significant advantage over the others. To determine which data characteristics might be the strongest factors in these performance variations, we investigated five data attributes, three commonly considered in soundscape analysis: SNR, event sparsity, and event length, plus similarity between events and the 5 shots and stereotypy (both defined in section 2.
The Fscore results are presented in Table  4. Because systems are developed based on the development and validation sets alone, the design decisions might not be what works best for the evaluation set, specifically in the case where these datasets vary substantially. This might explain why the original submitted architecture (first row of table 4) did not result in the best performance across all the variations tested here. In fact, instead of applying PCEN and Delta MFCCs as input features, the system modified to use Log Mel spectrograms resulted in the top performance on F-score. Similarly, here we see that including Negative Contrastive learning does not work well on this evaluation set and indeed the performance of the system decreases. Experimenting with different number of ways, confirms the expected that as the number of ways in the support set increases from 10 to 30, so does the performance.
Expert use analysis We are interested in understanding how far away the best scoring systems are from being incorporated into the annotation practice and how helpful or misleading their predictions can be. The expert annotators of QU, CT and MS datasets were given the predictions resulting from the 3 top scoring systems of the 2022 edition of this task, (Du_NERCSLIP, Liu_Surrey and Martins-son_RISE) and asked to analyse them in terms of a) Usability and b) Types of errors. Here are the main topics and highlights received. The full feedback can be read in Appendix  A. 3  All consider that at least one of the systems results in useful predictions that can be used as a starting point for manual editing. The best ranking system overall (Du_NERCSLIP) is not always the one selected by the experts as the best predictor of events in the different datasets and it also changes for different classes within the same dataset.
However the experts' selection almost always agree with the F-score results by class shown in Figures  5  and 6. As to the type of errors, the experts identified several instances of missed detections, misclassifications either on non-target calls or noise events, and in general imprecise detection of the duration of calls. Another aspect highlighted for both QU and CT datasets were situations where the capability of the systems to produce correct predictions decreased over time, meaning that events happening further away from the beginning (where the 5 shots examples happen) were less well predicted. The reason for some missed detections might be due to the selection of the 5 examples from which the systems need to learn the pattern of the target class. For both MS and CT datasets the experts commented that the range of variation of the target calls was not well captured within the 5 initial examples.
Discussion In this work we have formulated few-shot bioacoustic event detection as a machine learning task. We have evaluated many approaches to the task, and demonstrated that both the meta-learning and the transfer learning methodologies can successfully generalise to novel sub-tasks in bioacoustic FSED-thus, transcribing animal sounds with a precision unobtainable with other automated methods, in the absence of huge training datasets. Our sub-tasks were chosen to be diverse and non-trivial: they differed in taxon, target sound characteristics, background noise, stereotypy, stationarity, duration, and more. We believe that we have shown that the many related recognition tasks in computational bioacoustics can be unified within a generalised approach to machine learning. Leading systems achieve over 30% F-score on all 6 tasks.
It is not trivially solved by prior art in few-shot learning, nor by pretrained networks; yet we report very strong progress through the public challenges. We also consider that our chosen evaluation measure-an event-based F-score-has good external validity, since it aligns well with expert evaluations of automatic transcripts. Our aim to generalise over a range of loosely-related datasets/tasks is of current interest in machine learning. There are some comparable initiatives in wildlife monitoring. The 'BEANS' project collects together animal sound datasets and aims to provide a general evaluation benchmark  (Hagiwara et al. , 2022). Their work focuses on classification rather than temporal detection, and it does not consider few-shot learning or meta-learning-however it may be possible to re-use their data for such things.
Similarly, in image recognition the NeWT benchmark provides a suite of tasks for wildlife images, using a classification framework to ask a very wide range of ecology questions from a single data representation  (Van Horn et al. , 2021). Key differences between these and our work are our few-shot setting, and the explicit inclusion of temporal structure in the input and output of our formulation. Based on the results presented here, are few-shot bioacoustic SED systems good enough to use? Yes, as determined by feedback from our panel of experts. Although the outputs from such systems are far from perfect, they were judged to be of sufficient quality for active use, in place of fully-manual annotation. The quantitative results demonstrate that, when presented with a new dataset with no large training corpus, few-shot bioacoustic SED outperforms common methods such as template-matching.
Aspects of bioacoustic datasets that affect performance Many aspects of bioacoustic datasets make them complex to analyse: noise, highly sparse or dense events, varied levels of stereotypy, and non-stationarity (drift) in conditions. This is further illustrated when we show the variation in performance even across classes of the same dataset (Figure  6 ). We selected datasets which varied across many of these characteristics, and we sought to evaluate which of them were key factors influencing the difficulty of the task. A quantitative analysis (multivariate regression) was unable to identify any factors that consistently affected the F-score results across these datasets. However, qualitative feedback from expert users indicated that non-stationarity exerted an effect: this was shown by a reduction in performance for time-regions distant from the annotated examples.
A separate issue picked up by our annotators was that the support set was not always a good representation of the class to be detected, either because it did not include examples of every call type or due to the low stereotypy characteristics of certain classes. A trivial response is that we may include more than 5 examples, or curate the examples so they span the desired range of calls. In such cases we would expect stronger performance, at the slight cost of reintroducing some manual intervention. Our design decision to use the first few examples in a dataset is reflective of initialising a system before deploying it in a new recording situation. With offline analysis, and more flexibility in selecting examples, higher performance can be achieved. Since bioacoustic targets may include multiple call types or non-stereotyped sounds, it is worth noting one aspect of prototypical networks.
Datasets QU and MGE contained very short sounds at around this scale; Du_NERCSLIP(23)'s framewise CNN excelled on these datasets. Secondly, CNNs (used by all submitted systems) have relatively small "receptive fields" and do not consider the whole spectrogram but local feature patterns. Many of the strongest systems adapted themselves at query-time to the expected event length inferred from the 5 examples, in particular Martinsson_RISE who trained a set of embedding functions, each designed for a different duration. Some notable systems augmented their core CNN with architectures that are able to integrate information over long durations and directly infer onset and offset locations, such as a CRNN event filter (Du_NERCSLIP), perceiver and/or region proposal network  (Wolters et al. , 2021).
A single embedding for bioacoustic SED? Contrary to query-time adaptation, in machine learning there is current interest in learning good feature representations (good embeddings) from data. If an embedding can be re-used unmodified, this has an appeal of providing a general, reusable, and potentially low-complexity analysis tool, a component to be used in many systems. For audio data, some of the most widely-used deep embeddings are those derived from pretraining with the large-scale AudioSet dataset, originally designed for classifying many different (human-centric) acoustic categories  (Gemmeke et al. , 2017). More recent work evaluates this and many more ways to create an embedding  (Turian et al. , 2022). Our evaluation shows that improved prototypical network methods create powerful embeddings, useful even with no test-time adaptation.
Conclusion In this study, we have considered how best to automate the task of sound event detection in bioacoustics, and highlighted its potential as a few-shot learning problem. Unifying a set of loosely-related detection tasks makes possible the training of systems that do not need to be designed afresh, or trained afresh, for each new bioacoustic dataset. We have curated a rich and diverse set of data representative of many sub-tasks in this scenario, varying across many of the aspects a bioacoustician might consider (such as SNR or stereotypy). We framed few shot bioacoustic event detection as a public challenge. Over the three editions of the challenge, the evaluation dataset has been extended with more bioacoustic sources, pushing participating teams to create systems with impressive generalisation capabilities.
Our paradigm is not at all limited to just 5 arbitrary examples per sound event, and can be used in any situation where training datasets are not large. Manually curating a small but high-quality set of examples falls outside the present study, but can easily be expected to boost performance beyond the fully hands-off results reported here. Our formulation, and the diverse stronglyperforming systems analysed in our public evaluation, thus move us towards a post-template matching era for bioacoustic sound event detection. Appendix A. 2. Measuring similarity between events A metric to evaluate similarity between sound events is needed in order to analyse two aspects of the datasets: 1) How well do the initial 5 POS events represent the remaining POS events and 2) How stereotyped are the vocalisations in each dataset.
CT dataset (Coati) Feedback by EG: General answers to question 1) and 2) above: 1. The usability of the predictions is dependent on the call type under detection. The chitter predictions from H1 were best performing because they found at least one chitter in most of the chitter bouts, so manual labelling for the other chitters in these bouts would be necessary. A 1-hour wave file with many chitters can take 8 hours of manual labelling -so having a tool to pinpoint the areas to focus labelling effort would save time, it would likely save 2-4 hours of manual labelling time which would facilitate our work. D1 and M1 missed most chitters so I would not use these for labelling. In general, I prefer over-predicting calls to under-predicting, as deleting incorrect labels is faster than listening to whole wave files.
The system ranked in second place also improved over the prototypical baseline by using additional data from Audioset to train a ResNet for the feature extraction part. They have also adopted embedding propagation  (Rodríguez et al. , 2020) , with the objective of smoothing the decision boundaries as a way of increasing the generalisation capabilities of the few-shot system. Also of note, the work in  Cheng et al. (2021)  uses i-vectors as input features; both submissions in  Zhang et al. (2021)  and  Johannsmeier and Stober (2021) , explicitly create a negative class to model background noise and construct a negative prototype; and in  Bielecki (2021) , the team opted for combining the prototypical loss, with knowledge distillation and attention transfer loss. For most high-performing systems, there was a drop in F-score from validation to the evaluation set  (Table A.
6 ). This suggests that the systems are generally dataset sensitive, and our datasets vary in difficulty. To highlight this aspect further, we report the F-score results per dataset in the evaluation set  (Table A. 6),. Most systems have a low performance on the DC set, comprised of dawn chorus recordings, while perform better on ME and ML that include mainly mammal vocalisations. Complex acoustic environments such as dawn chorus may yet need further techniques to be employed for robust SED.
Acknowledgments The authors would like to thank the participants of the successive editions of the Few-shot Bioacoustic Event Detection task at the DCASE Challenge, without who this work would not have been possible. IN is supported by the  Engineering and Physical Sciences Research Council  (grant number  EP/R513106/1 ). SS is a research student at the  UKRI Centre for Doctoral Training in Artificial Intelligence  and Music, supported jointly by  UK Research and Innovation  [grant number  EP/S022694/1 ] and  Queen Mary University of London. VL acknowledges funding from  CNRS   MITI award CAPTEO , and from  WeAMEC  award  PETREL. ASP acknowledges funding from  Human Frontier Science Program  award  RGP0051/2019. The work was also funded by the  Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)  under Germany's  Excellence Strategy -EXC 2117-422037984.
presents an overview of all the datasets in the development and evaluation sets used in the 2022 edition of the challenge. duced as part of the BirdVox project WebsiteoftheBirdVoxproject:\url{https: //wp. nyu. edu/birdvox}, whose goal is to monitor bird migration with au- tonomous recording units (Lostanlen et al. , 2018). The recordings were obtained in four locations of Tompkins County, NY, US, during the 2015 fall migration season. An expert ornithologist, Andrew Farnsworth, has annotated 2,662 flight calls from 11 species of passerines, e. g. , Swainson's thrush (Catharus ustulatus) and White-throated sparrow (Zonotrichia albicollis. These flight calls have a duration in the range 50-150 milliseconds and a fundamental frequency in the range 2-10 kHz. Hyenas (HT-Training set): The HT dataset contains five recordings from hyenas.
Spotted hyena vocalisation data were recorded on custom-developed audio tags (DTAG) designed by Mark Johnson and integrated into combined GPS/acoustic collars (Followit Sweden AB) by Frants Jensen and Mark John- son, # Total # # Mean event 2022 Dataset Taxon Mic type Files duration Labels Events Density length (sec) Training BV HT MT JD Birds Mammals Mammals Birds fixed on-body on-body on-body 5 5 2 1 10 hours 5 hours 70 mins 10 mins 11 5 4 1 9026 611 1294 357 0. 038 0. 047 0. 042 0. 062 0. 15 1. 42 0. 15 0. 11 WMW Birds various 161 5 hours 26 2941 0. 25 1. 54 Val. HB PB Insects Birds handheld fixed 10 6 2. 4 hours 3 hours 1 2 712 292 0. 67 0. 003 11. 67 0. 11 ME Mammals handheld 2 20 mins 2 73 0. 011 0. 19 Evaluation CHE DC CT MS QU MGE Birds Birds Mammals Birds Mammals (marine) on-body fixed fixed on-body fixed Birds fixed 18 10 3 4 8 3 3 hours 95 mins 48 mins 40 mins 74 mins 32 mins 3 3 3 1 1 2 2550 967 365 1087 3441 1195 0.
263 0. 350 0. 017 0. 084 0. 045 0. 194 1. 94 1. 66 0. 16 0. 18 0. 06 0. 27 Table 1: Information on each dataset used in the 2022 challenge. "Density" is calculated as in signal processing: (total duration of events) / (total duration of audio), thus values close to 0 are sparse, and close to 1 are dense. BirdVox-DCASE-10h (BV-Training set): The BV dataset was pro-.
In particular, fast bouts of begging are in some cases missed entirely. The five shots from the beginning of the file do not come from fast begging bouts, and so are not representative of the range of possible chick begging vocalisations. Additionally, adult vocalisations and background noise are sometimes misclassified as chick begging vocalisations. The error profile of Martinsson_RISE differs from the error profiles of Du_NERCSLIP and Liu_Surrey. 2. • Martisson_RISE 1. Excellent. MS dataset (Manx Shearwaters) Feedback by JM: (answers to the questions above for each of the systems independently) • Du_NERCSLIP 1. Good. The predictions successfully classify the target class of chick begging vocalisations. Additionally, they rarely misclassify adult grunting vocalisations or other background sounds as chick begging vocalisations.
This paper introduces a new sparse unmixing technique using archetypal analysis (SUnAA). First, we design a new model based on archetypal analysis. We assume that the endmembers of interest are a convex combination of endmembers provided by a spectral library and that the number of endmembers of interest is known. Then, we propose a minimization problem. Unlike most conventional sparse unmixing methods, here the minimization problem is non-convex. We minimize the optimization objective iteratively using an active set algorithm. Our method is robust to the initialization and only requires the number of endmembers of interest. SUnAA is evaluated using two simulated datasets for which results confirm its better performance over other conventional and advanced techniques in terms of signal-to-reconstruction error.
Instead of softmax, a sparse variation of softmax is used to avoid the full support of softmax while enforcing ASC. In conventional sparse unmixing, the endmembers library is fixed, and the abundances estimation is of interest. However, even a pruned and well-selected spectral library cannot flaw-lessly represent the endmembers of materials in a real-world dataset. There are several factors, such as noise, atmospheric effects, illumination variations, and the intrinsic variation of materials which may affect the endmembers and induce scaling factors for the endmembers present in the scene compared to the ones from the library. To address this issue, we assume that endmembers of interest can be modeled by a convex combination of the library endmembers. This corresponds to the formulation of archetypal analysis (AA)  [13].
Recently, archetypal analysis has been successfully harnessed for blind unmixing in  [14]. In  [15] , 1 sparsity-constrained archetypal analysis was proposed for blind unmixing where the sparsity was enforced on the abundances. In this paper, we propose solving sparse unmixing using archetypal analysis (SUnAA). In the proposed model, an additional matrix is introduced, which defines the contributions of the endmembers from the library to the estimated spectra of endmembers present in the scene. Here the ASC can be enforced but the resulting proposed minimization is jointly non-convex. The optimization problem is solved using an active set algorithm, leading to a parameter-free technique, besides the number of endmembers of interest that is required. The experimental results confirm that SUnAA outperforms conventional and deep learningbased sparse unmixing techniques in terms of signal-toreconstruction error (SRE) for two simulated datasets and visually for the Cuprite dataset.
III. EXPERIMENTAL RESULTS We compare SUnAA with seven sparse unmixing techniques: SUnSAL  [16] , SUnSAL-S (SUnSAL with ASC and without sparsity penalty, i. e. λ = 0), SUnSAL-TV  [3] , S 2 WSU [6], MUA (using BPT segmentation)  [7] , MUSIC-CSR  [9] , and SUnCNN  [11]  applied to two simulated datasets and Cuprite. All the parameters are set as default for the competing methods. The results are mean values over ten experiments.
A. Simulated Datasets Two synthetic datacubes (DC1 and DC2) were used for simulated experiments. For DC1, a synthetic library composed of 240 spectral signatures selected from the USGS library with a minimum pair-spectra angle of 4. 44°is used. DC1 was simulated using a linear mixing model with 5 endmembers selected from the library and 75×75 pixels. The abundance maps are composed of five rows of square regions uniformly distributed over the spatial dimension. DC2 is a challenging simulated dataset with no pure pixels and two mixed pixels on the facet of the data simplex (more details in  [18] ). It contains 105×105 pixels, simulated by the linear combination of six endmembers from the USGS library. Two endmembers show no absorption features throughout the wavelength and are similar but scaled versions of each other, making the scenario even more challenging.
All the other techniques give low SRE. This could be attributed to the complexity of the dataset. Since the library may have several scaled versions of one endmember, the conventional sparse regression might not lead to a sparse solution. A remedy to this problem is to prune the library by removing the spectra with small spectral angle distances. However, in the case of complex datasets such as DC2, the endmembers may be removed from the library, which also leads to poor estimation. Overall, comparing the SRE reveals that SUnAA outperforms the other techniques. SUnAA also show consistent performances with respect to the noise level and datasets. All the other methods fail for the complex dataset. For the simple datasets, SUnCNN gives the second-best performance. MUA outperforms the other techniques for very low SNR, i.