dataesr/scientific-paragraphs-categorization huggingface dataset:
IDRiD Image-and Pixel-Level Evaluation Datasets Performance was also evaluated on the Indian Diabetic Retinopathy Image Dataset (IDRiD) 2  , which originates from an eye clinic located in Nanded, India. This dataset was collected for the purpose of an image analysis challenge  (Porwal et al. , 2019). For image-level evaluation, we used the 103 CFPs released for the onsite evaluation of algorithms competing in sub-challenge 2 (disease grading). Let IDRiD image denote this dataset. For pixel-level evaluation, we used the 143 CFPs released for training the algorithms competing in sub-challenge 1 (lesion segmentation). Let IDRiD pixel denote this dataset. Experts manually segmented four types of abnormalities related to DR in those images: microaneurysms, hemorrhages, soft exudates and hard exudates  (Porwal et al.
DeepDR Image-Level Evaluation Dataset Finally, image-level performance was evaluated on images from the DeepDR challenge 3. We used the 400 CFPs from 200 eyes of 100 patients (2 CFPs per eye) released for validating the algorithms competing in sub-challenge 1 (disease grading). Part of these images originate from screening programs in China: the Shanghai Diabetic Complication Screening Project (SDCSP), the Nicheng Diabetes Screening Project (NDSP), and Nationwide Screening for Complications of Diabetes (NSCD). Some images originate from an eye clinic located in the Department of Ophthalmology, Shanghai Jiao Tong University, China. Let DeepDR valid denote this dataset. The onsite evaluation dataset, denoted by DeepDR onsite , is not publicly-available yet. No training or fine-tuning was performed on DeepDR data, to ensure independent evaluation.
Hyperparameter Optimization Hyperparameters of each algorithm were optimized at the image level using the validation dataset. A ROC analysis was performed: hyperparameters maximizing the average per-label AUC on the validation subset were selected. For ExplAIn, the following hyperparameters had to be set: • the number M of pixel-level labels (see section 3. 1), • the CNN backbone (see section 3. 2), • training weights α, β and γ (see section 3. 6). Training weight γ, which trades off image-level and pixel-level classification quality (see section 3. 6. 5), was empirically set to 0. 1. Then, M, α and β were chosen to maximize the average per-backbone and per-label AUC. Finally, the best CNN backbone, given the optimal M, α and β, was selected. A similar procedure was followed for each algorithm (γ or equivalent set to 0.
Results The hyperparameters of ExplAIn maximizing image-level classification performance on the validation dataset are as follows: • M = 6 pixel-level labels (see section 3. 1), • EfficientNet-B5 backbone (see section 3. 2), • training weights α = β = 0. 1 (see section 3. 6). ROC curves obtained with these hyperparameters in the OPHDIAT eval dataset are reported in Fig. 2. For comparison purposes, image-level performance achieved with different Ef-ficientNet backbones and with the ExplAIn-CAM and "blackbox" AI baselines is summarized in Table  1. The combination of multiple ExplAIn models was also investigated: eight models with different backbones (EfficientNet-B0 to -B7) were combined through logistic regression on the validation dataset. Results on OPHDIAT eval are reported in Table  1.
Combining multiple models improves image-level classification performance. However, the improvement is limited and it comes at the expense of reduced explainability. Therefore, a simple Ex-plAIn model (with the optimal hyperparameters) is used in the remainder of the study. Next, the multiclass classification performance of this solution, on multiple evaluation datasets, is reported in Table  2 : Ex-plAIn is compared with competing solutions of the IDRiD and DeepDR challenges.
Results Classification performance at the pixel level in the IDRiD pixel evaluation dataset is summarized in Fig. 3  (PR curves) and 4 (ROC curves) for all architectures. Typical pixel-level classification maps, obtained with the optimal hyperparameters, are reported in Fig. 5. From a visual inspection, it appears that hemorrhages are targeted by map P 3 , exudates are targeted by map P 4 and microaneurysms are targeted by map P 5. This is confirmed by the lesion-specific quantitative evaluation described in section 4. 7. 1, the results of which are summarized in Fig. 4  (e). Map P 6 seems to target signs of advanced DR, such as neovascularization. In a few images from the IDRiD dataset, we observe that P 6 also detects laser scars, a possible sign that neovascularization is being treated (IDRiD is not a screening dataset).
ExplAIn models were trained to classify DR at the image level, using data from French and American DR screening programs (OPHDIAT and EyePACS): the goal was to grade DR severity in one eye according to the ICDR scale. On an Indian dataset (IDRiD), we found that manually-segmented DR lesions (microaneurysms, hemorrhages, soft exudates and hard exudates) could be detected well (see Fig. 4 ). Overall, these lesions could also be differentiated correctly (see Fig. 5 ). Using the optimal number of pixel-level labels (M = 6) and the optimal CNN backbone (EfficientNet-B5), unsupervised pixellevel classification can be schematized as follows. By design, label m = 1 was assigned to the background (see section 3. 4). One pixel-level label was assigned to microaneurysms (m = 5), another one was assigned to hemorrhages (m = 3), another one to exudates (m = 4) and another one to advanced DR signs (m = 6).
Fortunately, for the same task on the same evaluation dataset, image-level classification performance of ExplAIn (see Fig. 2 ) is similar to previously reported results for a "black-box" AI solution from our group  (Quellec et al. , 2019). Clearly, this "black-box" solution was designed without explainability constraints: in particular, it consisted of an ensemble of multiple CNNs while, for explainability purposes, ExplAIn necessarily consists of a single CNN. In details, two classification criteria have improved in ExplAIn (see Table  1 ): • "moderate NPDR or more", the most important criterion in screening applications: AUC increases from 0. 9859 to 0. 9939. Besides, for a 100% sensitivity, specificity increases from 12. 5% to 88. 1%, curves in Fig. 3  indicate that pixel-level precision is lower: over-segmentation is more problematic.
The latter solution was retained in the ExplAIn-CAM baseline, as illustrated in Fig. 7  (j) and (t). In ExplAIn, however, none of these solutions was retained at the end of training. Each type of abnormalities is assigned to a single pixel-level label and most of the abnormalities are detected (see Fig. 5 ): pixel-level probabilities simply tend to increase with the number of abnormalities and with their spread across the image. Although explainability is a desirable feature, it should not come at the expense of decreased classification performance. Fortunately, for the same task on the same evaluation dataset (OPHDIAT eval ), image-level classification performance of Ex-plAIn (see Fig. 2 ) is similar to previously reported results for a "black-box" AI solution from our group  (Quellec et al.
3% ("black-box" AI) or 94. 1% (ExplAIn), • "PDR": sensitivity = 100%, specificity = 98. 9% (in both solutions). We note that ExplAIn is also more efficient than ExplAIn-CAM for classification at the image level (see Table  1 ), al-though the difference is more subtle than at the pixel level. ExplAIn was also evaluated on two independent datasets: the Indian IRDiD dataset and the Chinese DeepDR dataset. The proposed algorithm seems to compare favorably with baseline algorithms, even though these baseline algorithms were trained or fine-tuned on non-independent training datasets (see Table  2 ). Note that the comparison is not direct on DeepDR: the proposed and baseline algorithms were evaluated on slightly different datasets (validation and onsite evaluation datasets). As shown in Table  2 , the accuracy of ExplAIn varies across evaluation datasets.
These variations can be explained by several factors: a different distribution of DR severity levels, different annotators (the OPHDIAT evaluation images were read by at least two graders), different population characteristics, different acquisition devices, etc. Variations in terms of quadratic weighted Kappa (κ) are more limited. Importantly, a large κ value was measured in all evaluation datasets considered in this paper (0. 8673 ≤ κ ≤ 0. 9243), indicating that disagreements between the AI and experts generally have a limited amplitude. The very good classification performance of ExplAIn at the image level was somehow unexpected. One possible explanation for this success is that the proposed generalized occlusion method acts as a regularization operator. Enforcing foreground sparsity encourages the AI to look for local patterns in images rather than to analyze the image globally.
Given an image, this framework 1) localizes and categorizes relevant patterns in the image, 2) classifies the image and 3) explains how exactly each pattern contributes to the classification. If experts can assign keywords to these patterns, then a textual report can also be generated. For the task of DR diagnosis using CFP, the new explainability feature comes without loss of classification performance. Thanks to this new feature, we expect healthcare AI systems to gain the trust of clinicians and patients more easily, which would facilitate their deployment. (q) P 6 (r) P 6 Fig. 5 : Pixel-level predictions for images from the IDRiD evaluation dataset, using EfficientNet-B5 and M = 6. For each pre-processed image (I), manual segmentations (hemorrhages, exudates and microaneurysms) are listed below, and the M -1 foreground label maps (P 2 , P 3 , P 5 , P 5 and P 6 ) are listed on the right.
Figure5: Pixel-level predictions for images from the IDRiD evaluation dataset, using the EfficientNet-B5 backbone and M = 6 pixel-level labels. For each preprocessed image (I), manual segmentations (hemorrhages, exudates and microaneurysms) are listed below, and the M -1 foreground label maps (P 2 , P 3 , P 4 , P 5 and P 6 ) are listed on the right.
Image-level classification performance of various architectures, in terms of AUC, in the OPHDIAT evaluation dataset. The "/Bx" suffix in the architecture names indicates the EfficientNet-"Bx" model used as backbone. The selected ExplAIn model, which maximizes the mean AUC (both in the validation subset and in the OPHDIAT evaluation dataset), is in bold. EfficientNet-B1 is the best backbone for ExplAIn-CAM in terms of mean AUC (both in the validation subset and in the OPHDIAT evaluation dataset). 'Combination' denotes the optimal combination of multiple ExplAIn models.
Porwal et al. (2019)cation performance, in terms of accuracy and quadratic weighted Kappa (κ), in various datasets. The "proposed" approach is ExplAIn with the EfficientNet-B5 backbone and M = 6 pixel-level labels. Results of the IDRiD challenge originate fromPorwal et al. (2019). Results of the DeepDR challenge are from the challenge website: the 4 best solutions (out of 8) are reported (SJTU: Shanghai Jiao Tong University; CUHK: Chinese University of Hong Kong) -the onsite evaluation dataset is not public yet.
This paper reports on progress integrating the speech recognition toolkit ESPnet into Elpis, a web front-end originally designed to provide access to the Kaldi automatic speech recognition toolkit. The goal of this work is to make end-to-end speech recognition models available to language workers via a user-friendly graphical interface. Encouraging results are reported on (i) development of an ESPnet recipe for use in Elpis, with preliminary results on data sets previously used for training acoustic models with the Persephone toolkit along with a new data set that had not previously been used in speech recognition, and (ii) incorporating ESPnet into Elpis along with UI enhancements and a CUDA-supported Dockerfile.
Thus, the tools are largely inaccessible by many people who would benefit from their use. Elpis 1  is a tool created to allow language workers with minimal computational experience to build their own speech recognition models and automatically transcribe audio  (Foley et al. , 2018 (Foley et al. , , 2019)). Elpis uses the Kaldi 2  automatic speech recognition (ASR) toolkit  (Povey et al. , 2011)  as its backend. Kaldi is a mature, widely used and well-supported speech recognition toolkit which supports a range of hidden Markov model based speech recognition models. In this paper we report on the ongoing integration of ESPnet 3 into Elpis as an alternative to the current Kaldi system. We opted to integrate ESPnet  (Watanabe et al. , 2018)  as it is a widely used and actively developed tool with state-of-the-art end-to-end neural network models.
By supporting ESPnet in Elpis, we aim to bring a wider range of advances in speech recognition to a broad group of users, and provide alternative model options that may better suit some data circumstances, such as an absence of a pronunciation lexicon. In the rest of this paper, we describe changes to the Elpis toolkit to support the new backend, and preliminary experiments applying our ESPnet recipe to several datasets from a language documentation context. Finally, we discuss plans going forward with this project.
Related Work Automatic phonetic/phonemic transcription in language documentation As a subset of speech recognition research, work has been done in applying speech recognition systems to the very low-resource phonemic data scenarios typical in the language documentation context. Encouraging results capitalizing on the advances in speech recognition technology for automatic phonemic transcription in a language documentation context were reported by  Adams et al. (2018). Their work used a neural network architecture with connectionist temporal classification  (Graves et al. , 2006)  for phonemic (including tonal) transcription. A command line toolkit was released called Persephone. To assess the reproducibility of the results on other languages, experiments were extended beyond the Chatino, Na and Tsuut'ina data sets, to a sample of languages from the Pangloss Collection, an online archive of under-resourced languages  (Michailovsky et al.
, 2014). The results confirmed that end-to-end models for automatic phonemic transcription deliver promising performance, and also suggested that preprocessing tasks can to a large extent be automated, thereby increasing the attractiveness of the tool for language documentation workflows  (Wisniewski et al. , 2020). Another effort in this space is Allosaurus  (Li et al. , 2020) , which leverages multilingual models for phonetic transcription and jointly models language independent phones and language-dependent phonemes. This stands as a 3 https://github. com/espnet/espnet promising step towards effective universal phonetic recognition, which would be of great value in the language documentation process.
User-friendly speech recognition interfaces Since such research tools do not have user friendly interfaces, efforts have been put into making these tools accessible to wider audience of users. The authors of Allosaurus provide a web interface online. 4 To integrate Persephone into the language documentation workflow, a plugin, Persephone-ELAN, 5  was developed for ELAN, 6  a piece of software that is widely used for annotation in language documentation  (Cox, 2019). Meanwhile, Elpis is a toolkit that provides a user-friendly front-end to the Kaldi speech recognition system. The interface steps the user through the process of preparing language recordings using existing ELAN transcription files, training a model and applying the model to obtain a hypothesis orthographic transcription for untranscribed speech recordings.
Bringing ESPnet to Elpis ESPnet is an end-to-end neural network-based speech recognition toolkit. Developed with Pytorch  (Paszke et al. , 2019)  in a research context, the tool satisfies three desiderata for our purposes: (a) it is easy to modify training recipes, which consist of collections of scripts and configuration files that make it easy to perform training and decoding by calling a wrapper script. These recipes describe a wide range of the hyperparameters and architecture choices of the model; (b) it is actively developed, with frequent integration of the latest advances in end-to-end speech recognition; and (c) it supports Kaldi-style data formatting, which makes it a natural end-to-end counterpart to Kaldi backend that was already supported in Elpis. These points make it a more appealing candidate backend than Persephone, primarily due to ESPnet's larger developer base.
Development of an ESPnet recipe for Elpis One goal of the integration is to create a default ESPnet recipe for Elpis to use, that performs well across a variety of languages and with the small amount and type of data typically available in a language documentation context. To get a sense of how easy it would be using ESPnet to get similar performance as previously attained we applied it to the single-speaker Na and Chatino datasets as used in  Adams et al. (2018)  (see Table  1 , which includes other details of the datasets used, including the amount of training data). We report character error rate (CER) rather than phoneme error rate (PER) because it is general, does not require a subsequent languagespecific post-processing step, and also captures characters that a linguist might want transcribed that aren't strictly phonemic.
Because of minor differences in the training sets, their preprocessing, and metrics used, these numbers are not intended to be directly comparable with previous work. While these results are not directly comparable to the results they reported, the performance was good enough to confirm that integrating ES-Pnet was preferable to Persephone. We do no language-specific preprocessing, though the Elpis interface allows the user to define a character set for which instances of those characters will be removed from the text. For the Na data and the Japhug data in §4, the Pangloss XML format is converted to ELAN XML using a XSLT-based tool, Pangloss-Elpis. 7 hile we did not aggressively tune hyperparameters and architecture details, they do have a substantial impact on performance and computational requirements.
Owing to the small datasets and limited computational resources of many of the machines that Elpis may run on, we used a relatively small neural network. In the future we aim to grow a representative suite of evaluation languages from a language documentation setting for further tuning to determine what hyperparameters and architecture best suit different scenarios. Though we aim for a recipe that does well across a range of possible language documentation data circumstances, the best architecture and hyperparameters will vary depending on the characteristics of the input dataset. Rather than have the user fiddle with such hyperparameters directly, which would undermine the user-friendliness of the tool, there is potential to automatically adjust the hyperparameters of the model on the basis of the data supplied to the model.
The logs can also provide useful contextual information when debugging an experiment in collaborations between language workers and software engineers. CUDA-supported Docker image The type of Kaldi model which Elpis trains (using a Gaussian mixture model as the acoustic model) was originally selected to be computationally efficient, and able to run on the type of computers commonly used by language researchers, which often don't have a GPU (graphics processing unit). With the addition of ESPnet, the benefit of using a GPU will be felt through vastly reduced training times for the neural network. To this end, Elpis has been 8 https://github. com/ persephone-tools/espnet/commit/ 1c529eab738cc8e68617aebbae520f7c9c919081 adapted to include Compute Unified Device Architecture (CUDA) support, which is essential in order to leverage a GPU when training ESPnet on a machine that has one available.
Application to a new data set: Japhug The point of this work is to provide a tool that can be used by linguists in their limited-data scenarios. To this end we aim to experiment with diverse datasets that reflect the breadth of language documentation contexts. Going forward, this will be useful in getting a sense of what sort of model performance users can expect given the characteristics of dataset. In this section we report on further application of the model underpinning the Elpis-ESPnet integration to another data set. Japhug is a Sino-Tibetan language with a rich system of consonant clusters, as well as flamboyant morphology. In Japhug, syllables can have initial clusters containing at most three consonants, and at most one coda  (Jacques, 2019). Japhug does not have lexical tones. The language's phonological profile is thus very different from Na (about which see  Michaud, 2017)  and Chatino  (Cruz, 2011; Cruz and Woodbury, 2014; Cavar et al.
, 2016). The data set comprises a total of about 30 hours of transcribed recordings of narratives, timealigned at the level of the sentence  (Macaire, 2020) , which is a huge amount in a language documentation context. The recordings were made in the course of field trips from the first years of the century until now, in a quiet environment, and almost all of a single speaker. Our tests on various data sets so far suggest that these settings (one speaker -hence no speaker overlap -and clean audio) are those in which performance is most likely to be good when one happens to be training an acoustic model from scratch. The full data set is openly accessible online from the Pangloss Collection, under a Creative Commons license, allowing visitors to browse the texts, and computer scientists to try their hand at the data set.
9 The data collector's generous approach to data sharing sets an impressive example, putting into practice some principles which gather increasing support, but which are not yet systematically translated into institutional and editorial policies  (Garellek et al. , 2020). The dataset can be downloaded by sending a request to the Cocoon data repository, which hosts the Pangloss Collection. A script, retriever. py, 10 retrieves resources with a certain language name. Data sets can then be created in various ways, such as sorting by speaker (tests suggest that singlespeaker models are a good way to start) and by genre, e. g. excluding materials such as songs, which are a very different kettle of fish from ordinary speech and complicate model training. Figure  2  shows how the phoneme error rate decreases as the amount of training data increases up to 170 minutes.
Is automatic speech recognition software too complex for language workers? A first concern is that automatic speech recognition software is simply too complex for language workers. But it should be recalled that new technologies that seem inaccessible to language workers can be game-changers in linguistics. For instance, the L A T E X software is the typesetting backend used by the journal Glossa  (Rooryck, 2016)  and by the publishing house Language Science Press  (Nordhoff, 2018) , which publish research in linguistics, offering high-quality open-access venues with no author fees or reader fees. Thus, L A T E X, a piece of software which is notorious for its complexity, is used on a large scale in linguistics publishing: Glossa publishes more than 100 articles a year, and Language Science Press about 30 books a year.