dataesr/scientific-paragraphs-categorization huggingface dataset:
Random image generation We used images of D train , D valid and D test as patches to generate 1536x1536 images and created the set of artificial images X l train , X l valid and X l test with their associated pixel-wise one-hot vector maps S l train , S l valid and S l test. In order to generate images that are comparable with historical documents, we selected a total of 97 images from the RECITAL dataset without any handwriting, and used them as background. We also manually segmented 50 handwritten strokes (accolades, separators,. ) from the RECITAL dataset for adding various structures, distinct from the word/number patches. The procedure to generate an image is defined by Algorithm 1. The first part of the procedure randomly create a grid G in a 1536x1536 area using a randomly selected background image.
NEURAL NETWORK MODELING AND TRAINING The model is based on a fully-convolutional neural network (FCNN) that produces a pixel-wise classification of every pixels of the input image in three classes; background, number and word. A graphical representation of the architecture of the trained FCNN if presented on Fig. 2  and detailed in TA-BLE 1. Since it is fully convolutional, this model is adaptable to all input image sizes so that it can be applied on the mails from the RIMES database and on the unlabeled documents from the RECITAL database X u. A 5-level pyramid representation of the input image was used as input to enlarge the reception field of each feature maps and ensure that the network can handle recognition of various sizes of structures. Roughly, the network is composed of two parts: Features extraction and Structures map construction.
, 2012). The softmax function performed a exponential normalization of each one-hot vector of the produced map, thus ∀i, y 0 i + y 1 i + y 2 i = 1, defined by Softmax(x) j = e x j ∑ k e x k with k = 0, 1, 2. The model was built and trained using the Python library Theano  (Bergstra et al. , 2010)  and the deep learning wrapper Lasagne  (Dieleman et al. , 2015). We trained the model by minimizing the objective function C (equation 2) using the Adam stochastic gradient-based algorithm described in  (Kingma and Ba, 2014). Symmetrically to the set of inputs images X l train , X l valid and X l test , the set of output maps of our network are denoted Y l train , Y l valid and Y l test. Note that the direct output of the network is Y = {y 1 , y 2 ,. , y N } with y i = (y 0 i y 1 i y 2 i ) T and y k i ∈ [0, 1].
Page level qualitative evaluation We qualitatively analyze the transferability of our model by considering RIMES paragraphs and RECITAL complete pages. Note that none of these documents were seen during the training. Fig. 5  shows six examples of structures spotting on real handwritten documents; three mails of the RIMES database and three RECITAL documents. Concerning the RIMES mails, we can observe that the totality of the word structures were well retrieved. However, in Fig. 5 (b), we can note that some numbers structures (especially the "20 and "2007" within the first sentence) were partially missed by the network. This could be due to the cursively-writing style of these structures, rarely present within the patches containing number used during training. The RECITAL documents are more challenging because of the natural noise included in the background, the different sizes of characters within the same page, and mainly the writing style which is completely different from modern IROnOff dataset.
Despite that most of the numbers were retrieved, we can observe that calligraphic letters, present at the top of the are often detected as digits. This can be explained by the fact that the IROnOff database does not contains calligraphic writing. Thus, our model did not learn to distinguish such large structures from digits. Also, we can observe that, at the pixel level, some number structures starting with "10" are missed by the network and classified as words. We think that the model cannot differentiate "10" structures from "lo", "la" and "le" structures, often present in IROnOff word patches.
CONCLUSION In this paper, we proposed a method for word/number structures spotting in handwritten documents based on a fully-convolutional neural network trained on artificial documents. Since the targeted database of documents were not annotated, we tackled this as a transfer learning problem. We proposed to train the model on randomly generated labeled images, built with number/word patches of the IROnOff database and page backgrounds from the RECITAL database. The performance of our model was assessed as a pixel-wise classification of the structures within a set of artificial documents. On artificial data, the system was able to correctly classify 96. 9% of the pixels. This shows that our model performed robust pixelwise classification on artificial data. We also evaluated the model as a digit detector by using the word/number patches of the RIMES database.
Three examples of generated images with randomly selected background, word and digit patches from the IROnOff database. (a),(b) and (c) present three examples of 1536x1536 artificial document generated using the Algorithm 1. (d), (e) and (f) present the target classification map (bounding boxes of the numbers and words) superposed on the corresponding artificial document. (g), (h) and (i) are the corresponding outputs of the detection system.
performance of the structuresspotting pixel-wise localization carried out by our model in three phases: firstly, by considering the set of artificial generated images X l test and the corresponding set of target classification map S l test ; secondly, by evaluating the detection of numbers among the N R patches of the RIMES database, R = {R 1 , R 2 ,. , R N R }, and their associated states B = {b 1 , b 2 ,. , b N R }, b i ∈ {0, 1}, indicating the presence or not of a number.
Recent decades have witnessed extremely strong growth in the abilities of machine learning. The advances are due to increased dataset sizes and computational power but also due to deep learning methods that can learn to make predictions in extremely nonlinear problem settings, such as speech recognition or visual object recognition  (LeCun et al. , 2015). These methods have indeed been applied to bioacoustic audio tasks  (Goëau et al. , 2016; Salamon and Bello, 2017; Knight et al. , 2017) , and it is clear that their use could enable many organisations to work more cheaply and efficiently  (Joppa, 2017). However, even with the strong performance of modern machine learning there remain important questions about generalisability  (Knight et al. , 2017). Machine learning workflows use a "training set" of data from which the algorithm "learns", optionally a "validation set" used to determine when the learning has achieved a satisfactory level, and then a "testing set" which is used for the actual evaluation, to estimate the algorithm's typical performance on unseen data.
Such evaluation is typically performed in matched conditions, meaning the training and testing sets are drawn from the same pool of data, and thus general properties of the datasets-such as the number of positive versus negative cases-are expected to be similar. This enables users to test that the algorithm can generalise to new items drawn from the same distribution. However, in practical deployments of machine learning the new items are rarely drawn from the same distribution: conditions drift, or the tool is applied to new data for which no training data are available  (Sugiyama and Kawanabe, 2012; Knight et al. , 2017). This is one reason that accuracy results obtained in research papers might not translate to the field. In order to address such problems, we designed a public evaluation campaign focused on a highly-general version of the bird detection task, intended specifically to encourage detection methods which are able to generalise well: agnostic to species, and able to work in unseen acoustic environments.
In this work we present the new acoustic datasets we collated and annotated for the purpose, the design of the challenge, and its outcomes, with new deep learning methods able to achieve strong results despite the difficult task. We analyse the submitted system outputs for their detection ability as well as their robust calibration, we perform a detailed error analysis to inspect the sound types that remain difficult for machine learning detectors, and apply the leading system to a separate held-out dataset of night flight calls. We conclude by discussing the new state of the art represented by the deep learning methods that excelled in our challenge, the quality of their outputs and the feasibility of deployment in remote monitoring projects.
| MATERIALS AND METHODS To conduct the evaluation campaign we designed a detection task to be solved-specific but illustrative of generalpurpose detection issues-gathered multiple datasets and annotated them, and then led a public campaign evaluating the results submitted by various teams. After the campaign, we performed detailed analysis of the system outputs, inspecting questions of accuracy, generality and calibration. Our aim to facilitate general-purpose robust bird detection, agnostic to any specific application, was key to how we designed the challenge specification. The task of 'detecting' birds in audio can be operationalised in multiple ways: for example, a system that emits a trigger signal in continuous time representing the onset of each bird call, a system that identifies regions of pixels in a spectrogram representation (time-frequency 'boxes'), or a system that estimates the number of calling individuals in a given time region  (Benetos et al.
| Datasets We gathered and annotated datasets from multiple sources. The purpose of this was twofold: firstly to provide better evaluation of the generality of algorithms, and secondly to provide challenge participants with development data (e. g. to perform trial runs, or to train machine learning algorithms) in addition to testing data. We used audio data from remote monitoring projects and also from crowdsourced audio recordings. These two dataset types differ from each other in many ways, for example: remote monitoring audio was passively gathered, while crowdsourced audio recordings were actively captured; the ratio of positive and negative items was different; remote monitoring used fixed and known recording equipment, while crowdsourcing used uncontrolled equipment. These differences were deliberately introduced for their use in ensuring that the challenge would be a strong test of generalisation.
Chernobyl dataset: Our primary remote-monitoring dataset was collected in the Chernobyl Exclusion Zone (CEZ) for a project to investigate the long-term effects of the Chernobyl accident on local ecology  (Wood and Beresford, 2016; Gashchak et al. , 2017). The project had captured over 10,000 hours of audio since June 2015, across various CEZ environments, using Wildlife Acoustics SM2 units. For the present work we selected six of recording locations representing different environments (Table  1 ), and from those selected a deterministic subsample: continuous 5-minute audio segments at hourly intervals, across multiple days. Annotators manually labelled all time intervals in which birds were heard (using Raven Pro software), and then we split recordings and metadata automatically into ten-second segments.
Codename Warblr dataset: Our first crowdsourced dataset came from a UK-wide project Warblr. Warblr is a software app available for Android and Apple smartphones, which offers automatic bird species classification (using the method of  Stowell and Plumbley (2014a) ) for members of the public via the submission of ten-second audio recordings. We extracted a dataset of 10,000 audio files gathered in 2015-2016. The audio files were thus actively collected, recorded on diverse mobile phone devices, and likely to contain various human noise such as speech and handling noise. No assumptions can be made that the data were a representative sample of geographic locations, weather conditions, or bird species. Metadata for the files indicated that they covered all the UK seasons, many times of day (with a bias towards weekends and mornings) and geographically spread all around the UK, with a bias toward population centres.
All recordings were selected that fell within the time window of available data, limited to a maximum of 10,000. No selection or filtering of the data were performed beyond the self-selection inherent in crowdsourcing. Although the data included automatic estimates of which bird species were present, these were not precise enough to be converted to ground-truth data for the detection challenge. We thus performed manual annotation, with each item being labelled as positive or negative according to the challenge specification. Most items were singleannotated, although we were able to obtain double-annotation for a small number of items, which allowed us to estimate inter-rater reliability. Annotation was performed by experienced listeners using headphone listening and a simple web interface.
freefield1010 dataset: Our second crowdsourced dataset was an existing public dataset called freefield1010  Stowell and Plumbley (2014b). This consists of 7690 audio clips selected from the Freesound online audio archive. To create this dataset the audio clips had been selected such that they were labelled with the 'field-recording' tag in the database, and trimmed to ten seconds duration. The data were of different origin than Warblr: they covered a global geographic range, and the recording devices used were almost never documented, but likely to include hand-held audio recorders as used by pro-amateur sound recordists, as well as some mobile phones and some higher-end recording devices. The Freesound database is crowdsourced and thus largely uncontrolled. These data did not come with labels suitable for our challenge; instead, each item came with a set of freely-chosen tags to indicate the content generally.
We investigated the 'birdsong' tag, one of the most commonly used (2. 6% of items), but found this insufficiently accurate. We therefore had these audio annotated through the same process as the Warblr data. range), so it may happen that some low pitched bird species (e. g. resident owl calls) obtained no-bird label. However, such calls were extremely rare in described dataset, so this did not bias results significantly. The selected recordings were then split into 1 s clips. The chosen clip length, different from other datasets, was chosen due to nocturnal bird calls typical duration (10-300 ms). More details about the dataset structure (and analysis of the effect of audio clip duration) may be found in  Pamuła et al. (2017). All sound files used in the public challenge were normalised in amplitude and saved as 16-bit single-channel PCM at 44.
To make good use of annotator time we used mismatch between automatically inferred decisions and manual annotations to search for mislabelled items in the dataset. For this we used the mean decision from the strongest three submissions to the challenge. All items in the testing set with a negative groundtruth label but a mean decision greater than 0. 2, and all items with a positive groundtruth label but a mean decision less than 0. 3, were examined and relabeled if needed. One might expect the threshold for re-validation to be 0. 5: the asymmetry is because systems generally exhibited a bias towards low confidence, as will be seen later (Section 3. 1). This re-validation process refined the testing set, but also allowed us to calculate a value for the inter-rater agreement for manual annotation, which we will express as an AUC for comparison against the results of automatic detection.
Both of these components are known to work well with difficult classification scenarios, such as multi-modal classes, unbalanced datasets, and outliers. We thus selected this second baseline as a representative of modern and flexible machine learning, designed for bird sounds. In principle it could be more vulnerable to overfitting than the first baseline. However, because of the inherent difficulty of the task, we expected skfl to perform more strongly than smacpy, and to provide a high performing baseline. Python code for each of the baseline classifiers has previously been published  (Stowell and Plumbley, 2014a; Stowell et al. , 2015). Initial results using the baseline classifiers were published online as a guide to the challenge participants. 2.
| The public challenge Conduct of the public challenge followed the design of previous successful contests on related topics  (Stowell et al. , 2015; Goëau et al. , 2016). Having already determined the need for the challenge from remote monitoring literature and practitioners  (Stowell et al. , 2016) , we announced intentions and led community discussion on the task design via a dedicated mailing list. We collated our audio datasets into three portions: development data to be publicly shared (7,690 items of freefield1010 plus 8,000 items from Warblr), testing data whose true labels were to be kept private (10,000 items from Chernobyl plus 2,000 items from Warblr), and a separate set not used for the challenge itself but for further study of algorithm generalisation (PolandNFC). Providing two distinct development data sets allowed participants to test generalisation from one to the other, as part of their own algorithm development process, while keeping some datasets fully private allowed us to evaluate generalisation without concern that algorithms might have been configured to the specifics of a given dataset.
The public development datasets were distributed in September 2016, both audio and groundtruth annotations. Teams could then begin to develop methods and train their systems. In December 2016 we released the testing data (audio only), with a one month deadline for the submission of inferred detection labels. The short time horizon of one month was intended to minimise the opportunity for overly adapting the methods to the characteristics of the testing data. During this period, teams could make multiple submissions, but limited to a maximum of one per day. "Preview" results, calculated from 15% of the testing data, were provided in an interactive online plot, in order to give approximate feedback on performance (Figure  2 ). Participants were allowed to run their software on their own machines and then to submit merely the outputs (as opposed to the software code), which our online system would then score without revealing the groundtruth labels for the testing data.
The AUC measure has numerous qualities that make it well-suited to evaluation such classification tasks: it generalises over all the possible thresholds that one might apply to real-valued detector outputs; unlike raw accuracy, it is not affected by "unbalanced" datasets having an uneven mixture of positive and negative items; chance performance for AUC is always 50% irrespective of dataset; and it has a probabilistic interpretation, as the probability that a given algorithm will rank a randomly-selected positive instance more highly than a randomly-selected negative instance  (Fawcett, 2006). The ranking interpretation just mentioned highlights another aspect of the AUC statistic: it treats detector outputs essentially as ranked values and is thus invariant to any monotonic mapping of the outputs, in particular to whether the outputs are well-calibrated probabilities or not.
| RESULTS In re-validating the testing set we examined those items with the strongest mismatch between manual and automatic detection, to determine which was in error: 500 presumed negative and 1243 presumed positive items. This showed inter-rater disagreement in 16. 6% of such cases-predominantly, the most ambiguous cases with barely-audible bird sounds with amplitude close to the noise threshold. Note that this percentage is not representative of disagreement across the whole dataset, but only on the 'controversial' cases. We also observed that a strong mismatch according to the automatic detectors did not necessarily imply human mislabelling: some perceptually obvious data items could be consistently misjudged by algorithms. We will discuss algorithm errors further below. Through re-validation the inter-rater reliability, measured via the AUC, was measured as 96.
This implies a spread of 'difficulty' for the test items: there were some positive items that were easy to detect without incurring extra false-positives as a side-effect, while many remained difficult to detect. The most balanced of the ROC plots inspected was that of 'cakir' , implying that this system had a more balanced distribution of its discriminative power across the easy and difficult cases. Since the testing data consisted of items from multiple 'sites'-i. e. known sites in the Chernobyl Exclusion Zone, plus the Warblr (UK) data considered as a separate single site-we were able to calculate the AUC scores on a per-site basis (Figure  7 , left). These showed a strong site-dependency of algorithm performance. Whereas Warblr data could be detected with an AUC of over 95%, Chernobyl sites showed varying difficulty for the detectors overall, some as low as 80% even for the leading algorithms.
The very strongest scoring systems did not cluster tightly together, indicating that there remains some diversity in the strategies implicit in these high-performing detectors. We measured calibration curves separately for the Warblr and Chernobyl testing data (Figure  9 ). Calibration was generally better for Warblr, as one might expect given the availability of Warblr training data. Notably, the highestscoring submission 'bulbul' had by far the worst calibration on the Chernobyl data: around 80% of cases it assigned a prediction value of 0. 25 were indeed positive (versus around 30% on the Warblr data). The second-highest-scoring submission 'cakir' exhibited quite different behaviour, remaining relatively well calibrated even when assessing the unseen Chernobyl data.
Low SNR was also a factor in the third most common presumed cause, noise masking. This category included general broadband 'pink' noise sources including wind and rivers. There were other more specific categories of sound that appeared to act as masker or distractor causing systems to overlook the bird sound: insect noise was common in the CEZ data, while human sounds such as speech, whistling, or TVs were present in the Warblr data. The second most common presumed cause of false negatives was however the presence of extremely short calls: often a single "chink" sound, which might perhaps be overlooked or confused with rain-drop sounds. Some sounds were presumed to be missed because they were unusual for the dataset (e. g. goose honking), although this was not seen as a major factor. False positives occurred at a much lower rate in the top 500 most mismatched items.
| Further analysis via PolandNFC dataset results We then applied the highest-scoring method ('bulbul') to the separate and unseen acoustic monitoring dataset Poland-NFC. The bulbul algorithm performed its inference in two stages: the first stage applied the pre-trained neural network to make initial predictions; the second stage allowed the neural network to adapt to the observed data conditions, by feeding back the most confident predictions as new training data  (Grill and Schlüter, 2017). We evaluated the outputs from each stage (Table  3 ). AUC results were of high quality and were most strongly affected by the choice of training data, the matched-conditions training yielding much more accurate predictions. The second stage adaptation offered some improvement in the case where the training set came from mismatched conditions.