dataesr/scientific-paragraphs-categorization huggingface dataset:
The act of choosing an appropriate loss is typically quite an involved one, as it can influence model robustness and computational cost (and sometimes even the feasibility of the training). A simple example is shown in Figure  1. 4, where we fit two linear models, one with a least-squares loss and one with a Huber loss  [46]  to a simple linear dataset with outliers. Clearly in this case the effect of the Huber loss is to render the linear model more robust to the outliers. In testing, the key performance measure is the evaluation metric applied to the test-set: since we are most interested in the ability of the learning algorithm to generalize to unseen data, a portion of the available data is reserved for the testing procedure. The trained algorithm is evaluated on this 'unseen' separate dataset, which was not involved in the training procedure.
The evaluation metric is usually, but not necessarily, the same as the loss function. Finally the experience E provided to the learning algorithm allows us to draw one of the primary distinctions between types of learning algorithms: supervised learning algorithms are provided with a dataset containing input features where each example is associated with a target ( the y-axis in Figures 1. 3 and 1. 4). The term 'supervised' originates from the idea that by providing the learning algorithm with a target we are essentially instructing it to find patterns related to this target  [41]. On the other hand, in an unsupervised learning setting the learning algorithm is provided with the input features, but no label or target. The goal of the learning procedures is then to learn useful patterns from the dataset such as dividing the dataset into similar groups (clustering), or learning the generating probability distribution behind the dataset (density estimation, denoising, synthesis)  [41].
Learning Capacity We noted in section 1. 2. 1 that the central aim of ML is to learn a model which can generalize effectively to unseen data (i. e. hypothesis testing). During the learning process, an ML algorithm has access to some training data, on which it must attempt to reduce some error measure, known as the training error. What separates ML from fitting, is the requirement that not only must this training error be minimized, but the generalization error on an unseen test dataset must also be reduced. The values of the training and test error and their relationship are linked to two central issues in ML: underfitting, occurring when the model is incapable of minimizing the training error to an satisfactory level and overfitting, when the training error has been reduced but the model cannot reduce the gap between the training error and test error (e.
g. , the model fits the training dataset but exhibits a poor fit with the training dataset). These issues are governed by the ML model's representational capacity, or complexity. Models with low capacity may struggle to capture the full spectrum of patterns in a complex training set, and thus underfit to them. Likewise models with high capacity can memorize properties of the training set which are irrelavant to the model's ability to generalize to the test set, thus overfitting. Figure  1. 5 illustrates this principle: we take the simple task of fitting a dataset generated by a quadratic function: the linear model (green frame) does not have the capacity to fit the data, whereas the high-degree polynomial model (blue frame), with more parameters than training examples, overfits due to the fact is has the representational capacity to pick many potential solutions to the problem, reducing the chances of selecting the correct solution.
6 d, with the results of these different models shown in Figure  1. 7 b. It is stated in  [64]  that the data from the points located above the velocity weakening regions allowed for the learning of a model with 0. 7 < < 0. 8, even when using shorter training windows of 5 cycles. Analysing the spatio-temporal distribution of the time to failure predictions allowed for the prediction of not only the timing and location of upcoming ruptures but also the size by examining how many adjacent points were expected to rupture simultaneously  [64]. The most predictive features were determined to be measures related to cumulative displacement at specific points, indicating the learning algorithm is heavily informed by the relative loading A further study was conducted by Corbi et al. on the same dataset in  [66].
Rather than framing the problem as a regression task  [64] , the authors instead chose to train an ML algorithm to determine whether the deformation observed was characteristic of a time window immediately preceding slip onset or not: a binary classification problem  [66]. The authors also modified the 'positions' of the synthetic GPS network in the experiment, excluding data originating from the equivalent of the 'offshore' seismogenic zone in an attempt to mimic limitations in geodetic coverage in real subduction zones. The features used in this study consisted of the displacement measured at a given synthetic GPS station. The only features used for this study were thus displacements parallel and orthogonal to the 'trench' ( or by association, the coastline), as these were determined to be the variables most predictive of slip  [64].
The target variable was derived from the same 9 locations as in  [64] , where the "slip" was identified at times where the displacement at these locations exceeded a threshold. For each event, the category of 'alarm' was assigned to the Δ seconds prior to the onset of this displacement rate. The output thus consists of 9 response vectors, with each element of each vector indicating whether a given time step is part of the 'alarm' category or not. The authors use an ML algorithm known as Random Undersampling Boosting (RUSBoost)  [67] , an algorithm which combines random undersampling of the most prominent class with adaptive boosting  [68]. This is an effective method in dealing with the imbalanced nature of the dataset: there are simply more 'negative' examples in the dataset than positive 'alarm' examples ( depending on the Δ the alarms represent 3-27% of the total observations).
All the studies mentioned above relied on shallow, feed-forward neural networks (with the exception of  [88] , which utilized slightly more complex architectures in the form of simple recurrent networks). They relied on hand-built features (either waveform spectra, STA/LTA functions, or both), and were trained on a small number of examples (from a few dozen to several hundred). Most of the following studies have relied on the same tools, and were also based on relatively small datasets, with examples in the 100-1000 range  [89, 90, 91, 92]. However the majority of these studies also incorporated ML approaches other than feed-forward networks, such as SVMs  [89, 90] , Random Forests  [91] , or Logistic Regressions  [91] , which were found to systematically outperform the simple perceptron  [93]  models.
A parallel algorithmic avenue explored to detect earthquakes has relied on probabilistic graphical models, using either Hidden Markov Models  [94]  or Dynamic Bayesian Networks  [95]. These approaches provide further benefits in associating a true Bayesian uncertainty measure to each detection, which can be used to help filter out false positives. In the two studies, the authors comment that their algorithms produced a number of false detections, but the uncertainties associated to those false detections were much higher than those associated to real earthquakes. Unsupervised algorithms based upon regrouping event waveforms by similarity have also been explored  [96, 97, 98]. Recently, researchers have begun to train models using larger datasets [99, 100, 101, 102], enabling the use of more complex neural network architectures and automatic feature extraction.
In particular, Mousavi et al. [101]  used a database of 250000 earthquakes and 250000 noise samples in California to train a residual network based upon convolutional and recurrent units, and reported that their algorithm outperforms other approaches. Interestingly, when attempting to generalize their model from California to Arkansas, the model produced a higher rate of false positives, suggesting that complex models built upon large databases can have difficulties generalizing to other areas. Overall, the analysis of the existing literature shows that earthquake detection with ML algorithms clearly outperform a standard Short Time Average over Long Time Average (STA/LTA) approach, in particular due to a lower rate of false positives as shown in Figure  1. 11. The comparison with template matching algorithms is less clear-cut.
Some definitive advantages of ML earthquake detection over template matching are i) reduced computation time, and ii) the fact that events not included in the template list can be detected. However, ML algorithms may generate more false positives than template matching. Further developments of Bayesian deep networks trained on large datasets to associate clear uncertainties to each detection, such as that proposed by  [103] , would be a straight-forward improvement of the state-of-the-art. The comparison between template matching and neural network detection using convolutional networks is of particular interest. Because convolutional layers perform a cross-correlation operation, template matching corresponds effectively to a single 1D convolutional layer using one kernel of known weights for each template, and as many channels as the number of stations.
A similar analysis by the same authors  [108]  extended the approach to arrival detections from a single station component. In both studies, a model trained on a very small number of examples (9 pairs of seismic arrivals and noise) was able to reach a relatively high picking accuracy on hundreds of earthquakes. Shallow perceptron architectures were also applied for the same purpose by  [109, 110]. Gentili and Michelini  [111]  found that picking from a neural tree was robust to errors from noise data and outperforms a standard STA/LTA algorithm. Chen  [112]  also found that unsupervised algorithms outperform STA/LTA picking for small events. In the late 2010's models were developed using much larger datasets which, combined with increases in computational power, enabled the use of more complex neural network architectures and automated feature extraction for phase picking.
Zhu et al. [114]  developed a U-Net type network for picking estimations, trained on over 600000 earthquakes in Northern California. The network takes as input the three components of seismic waveforms, and outputs either a P arrival, S arrival, or noise class. Wang et al. [115]  trained two separate convolutional networks to pick P and S arrival times, on a database of over 700000 picks in Japan. Zhu et al. [113]  also utlized on a CNN to study the aftershock sequences of the 2008 M W 7. 9 Wenchuan Earthquake. For all studies, results in testing were found to outperfom existing methods. An online version of their algorithm, identifying picks on continuous waveforms, was also proposed. A sketch of the associated classification pipeline is shown in Figure  1. 12. Besides picking arrival times, a few studies attempt to estimate P-wave first motion polarity.
Automatic detection of geodetic deformation Interferometric Synthetic Aperture Radar (InSAR) is a powerful geodetic technique developed in the early 90's  [136, 137]  for measuring ground surface displacements. InSAR has been successfully applied to monitor large displacements due to earthquakes  [136, 138, 139] , as well as smaller displacements related to interseismic deformation  [140, 141, 142, 143] , slow moving landslides  [144] , and slow slip events  [145, 146]. However, in general these measurements have been successful only through painstaking manual exploration of the deformation data by InSAR experts. In the past low satellite revisit rates typically resulted in large time delays between measurements. The launch of the Sentinel-1 satellite constellation in 2014 has been revolutionary in that it provides systematic radar mapping of all actively deforming regions in the world with a 6-day return period.
Meier et al. [159]  developed a Bayesian algorithm that starts from single station analysis for the very early onset of the detection on nearby stations, and evolves into a more accurate multi-station analysis. Trained on a dataset of 60000+ waveforms, their algorithm can provide a timely assessment of earthquake magnitude for areas close to the epicenter, where damage is greater. Their ML approach of training and testing an empirical model also demonstrates that information about earthquake magnitude saturates at magnitudes that depend on seismic record length, but are in general about magnitude 7-7. 5.
As a counterpart, DAS are also likely to generate staggering amounts of data. The sheer volume of currently available InSAR data already provides a challenge for existing processing methods and algorithms which sufffer from scalability issues and can require manual intervention. Next generation InSAR satellites will soon launch and dramatically increase the amount of acquisitions, making the automation of InSAR data analysis an even more pressing challenge. As the stage is set for geophysical data to become ever harder to analyze using traditional methods, ML and data science techniques may well become fundamental tools of the trade in the geosciences. Notably, as field and laboratory studies of earthquakes and earthquake nucleation processes incorporate ML in the search for ever finer observations and slip characterization, new information may be revealed concerning fault behavior.
Experimental data stored at the Rock Mechanics Lab at Penn State. The database's size has been doubling every year since 2015. Courtesy of Chris Marone, Penn State University. IRIS DMC Archive Size 638. 8 Tebibytes (TiB) as of 1 April 2020 Portable (PH5) Other Engineering US Regional Tebibytes International FDSN EarthScope Portable (PASSCAL, SEIS-UK, OBSIP, SISMOB) 1/1/92 1/1/93 1/1/94 1/1/95 1/1/96 1/1/97 1/1/98 1/1/99 1/1/00 1/1/01 1/1/02 1/1/03 1/1/04 1/1/05 1/1/06 1/1/07 1/1/08 1/1/09 1/1/10 1/1/11 1/1/12 1/1/13 1/1/14 1/1/15 1/1/16 1/1/17 1/1/18 1/1/19 1/1/20 GSN FIGURE 1. 1 IRIS Data Management Center archive size since its inception. The archive's size has been doubling every 3 years since the 2000's. Courtesy of IRIS. FIGURE 1. 2.
Introduction Modern research relies on software, but it has only gained recognition recently. While strategies for articles and even data preservation are already the norm, software is still a unique artifact for which it is rare to find dedicated deposits and preservation mechanisms in institutional repositories  (Milliken, 2019). We need to preserve source code alongside scientific articles and datasets to scaffold future work on top of these open science pillars. As declared on the Inria /UNESCO Paris call: 'Recognise software source code as a fundamental research document on a par with scholarly articles and research data;' (UNESCO-Inria, 2019) Today, software is still too often considered as just data, even though data is gathered through observations or experiments, whereas software is a product of human ingenuity, written by authors and contributors, and embodying the logic of the data transformation.
Hyper Articles en Ligne a. k. a HAL The first actor in this collaboration is HAL, the French national open access repository, created in 2000 by the French National Centre for Scientific Research (CNRS 6 ) and maintained by the the Center for Direct Scientific Communication (CCSD), destined to provide tools for archiving and dissemination of scientific outputs openly. HAL is a repository where researchers can deposit their academic outputs compliant with their copyrights 7. Since its creation, HAL has supported different types of deposits: publications, documents (e. g pre-prints and reports), academic work (e. g theses) and research data (e. g images, videos). HAL's goal is to make research as accessible and open as possible.
IES Inria Another collaborator in this effort is Inria, the French National Institute for computer science and applied mathematics. Inria Created in 1967 , it currently hosts in its teams over 3000 researchers 8  and supports the creation of a broad spectrum of open source software, including award winning projects such as Coq, OCaml, and Sckit-Learn. The research center has a dedicated scientific and technical information service, denoted IES-Inria, which played a major role when specifying the new type of research output: software source code, shown in  (Barborini et al. , 2018).
A workflow for curating the deposit of software artifacts From the earliest open repositories to now, moderation has been a key part of the submission workflow when self-archiving research outputs. One prominent example is ArXiv 9  , founded in 1991 and operated by the Cornell University. Today, there exist platforms that offer source code deposit, such as Zenodo and Figshare, but do not have any pre-submission checks for the self-archived content. HAL chose to follow ArXiv's example 10  and implement a sophisticated moderation workflow in order to ensure that quality metadata is attached to every deposit into the platform. In order to extend the existing HAL moderation workflow to support deposits of research software, a similar workflow had to be implemented to handle the following aspects: • artifacts attribution, • classification, • compliance with metadata requirements, • and appropriate content.
The submission form Contributors must fill out a descriptive metadata form on submission, to ensure the most accurate information about the source code is captured. The metadata is used for moderating the submission and is preserved with the software in both HAL and the SWH archive. The design of the form was adapted from the pre-existing deposit form for scientific articles, see figure  3  where you can choose the software type and add a software license. The HAL metadata schema included terms that are applied to all deposits (e. g author, title and keywords, etc. ) However, it wasn't sufficient to describe software artifacts. Software requires more specific elements in addition to these to adequately describe its complexities. We researched the software vocabulary landscape for a vocabulary adapted to scientific software, and we found that the CodeMeta vocabulary was a perfect fit.
Curating software -including humans in the loop The professionals curating deposits into HAL are librarians and archivists. They are employed by specific institutions, if the institution has authority over its institutional repository (e. g Inria and University of Lorraine) or directly by the CCSD which operates HAL and all attached services. The curation of deposited digital artifacts is one of the roles they assume as information experts. Most of these librarians and archivists have a background in academic institutions, and curating these deposits is one of their key responsibilities. The process of moderating source code deposits requires human intervention, which leads to direct interactions between the submitting researcher and these curators. These consultations center around the metadata attached to the deposit rather than the source code itself, although a mild inspection of the code is done to ensure the metadata describing it is correct.
Our experience over the first two years of operation shows that, with the support of the guidelines, the software moderation process does not add greatly to the workload of digital archivists, and can be performed by digital archivists. The IES-Inria and CCSD teams, which play the role of digital archivists for HAL platform, are used to working with articles, reports and other textual deposit types. The software deposit was very different from that which they were used to review. When doi:10. 2218/ijdc. v0i0. 0 establishing the requirements for a software deposit, we realized that there is no need, at this point, to act as an AEC and verify the functionality, the quality and reproducibility of the artifact itself. Therefore, the main actions the digital archivist performs while reviewing software deposit are: • detecting extraneous or abusive content (illegal or harassing), • verifying consistency between the metadata and the software source code itself, • completing or correcting the deposit metadata if needed.
Deposits without source files During the test phase, researchers could also deposit metadata records about source code without the source code itself, similar to "bibliographic records. " Occasionally, users have chosen to deposit only descriptive information about a software artifact, because they needed the reference to the software record in their activity reports. The clear drawback is that it is impossible for the digital archivist to check the information deposited. One approach is to prevent software deposit without the software source code itself, which would be a compressed static archive without its development history. While this approach is reasonable for researchers that do not use collaborative development platforms, it turns out to be an annoyance for those that have made their software source code available online, or even archived it already in SWH.
We are working on a dedicated standard for this particular use case; • Software collections: sometimes researchers try to deposit a single archive containing many different software tools or software libraries; • Research experiments that do not really qualify as a software tool on their own; for this particular use case, the researchers usually only need long term archival and intrinsic identifiers: we plan to refer them to the dedicated guidelines for source code archival and reference available on the Software Heritage website (Di Cosmo, 2019); • Software source code deposited that include large datasets, instead of a reference to a separate data deposit.
Publishing versus sharing Research software has been around for decades, and some research institutions have a long experience in managing it as a valuable output of research  (Alliez et al. , 2019) , but only very recently attention has started to grow in the broader scholarly ecosystem. This new interest has spawned a rich discussion about what actually could be a software publication. In this context we would like to stress the importance of remembering that in the scholarly world there is a precise semantics attached to the term publication: an academic publication is a research result that has been qualified through some form of peer review; a result that has been simply shared, for example by making it available somewhere on the Internet, is usually not regarded as a publication 17. When we come to software, that is in its vast majority developed outside of academia, and in particular to open source software, it is common practice to share it broadly on code hosting platforms like GitHub, GitLab, and many other ones, but this act of sharing does not carry the same meaning as the act of academic publishing, and code hosting platforms do not play at all the same role as publishers in the academic world.
Hence we should refrain from using the term "publication" when we talk about software that is simply shared on the Internet, even when its source code is deposited on institutional archives. The research community is still exploring how to exactly handle software when it comes to credit and academic recognition, with various ongoing experimentations like the AEC, IPOL, the Journal of Open Source Software  (JOSS doi:10. 2218 (JOSS doi:10. /ijdc. v0i0. 0 team, 2019;; Smith et al. , 2018) , the Dagsthul DARTS series 18  , ACM Badges, etc: it is up to researchers to reach an agreement on this very sensitive issue. For this reason, in the metadata for software deposited via HAL, we do not indicate HAL as a publisher.