dataesr/scientific-paragraphs-categorization huggingface dataset:
Labeled data set For supervised machine learning algorithms, a set of labeled data has to be provided. Here, we focus on seismic data from three sensors of the ILL array recorded in 2017 and the threesensor LERA array recorded in 2018 and 2019. We compiled an event database for both study sites by visual inspection of the seismic waveforms and spectrograms of seismic stations at Illgraben and Pizzo Cengalo and close-by stations of the Swiss Seismological Service (SED) (Illgraben: stations CH. LKBD and CH. VANNI; Pizzo Cengalo: stations XP. PICE1, CH. VDL, and CH. FIESA). Additionally, we use a list of observed slope failures made available by the canton of Grisons for Pizzo Cengalo and earthquake catalogs from SED and the European-Mediterranean Seismological Centre (EMSC). For the classification, we decided to use three different classes: noise (NO), slope failures (SFs), and earthquakes (EQs).
The EQ class contains a set of local, regional, and teleseismic events. Extensive testing showed that lumping all earthquakes in one class does not negatively affect classifier performance (Fig. A2 ). An example signal of each class is presented in Fig. 2. For the continuous noise, we choose random times over the year. For noise signals, earthquake signals, and slope failure signals, we manually picked the events' start time and end time when the signal exceeds the noise level. The number of events in each class for both study sites is presented in Fig. 3a  and Table  B2. Note that there are only five events that are related to slope failures at Pizzo Cengalo in 2018. Due to instrument malfunction, these events were not captured by the automatic camera. The sparsity of recorded events leaves us with a poor data set for this class.
This issue is addressed further in Sect. 3. 2. Figure  3b  shows a boxplot of the signal-to-noise ratios of SF and EQ events at both sites. The signal-to-noise ratio for earthquakes ranges between 1 and 10 3 , with a mean of 5 (Fig. 3b ). The signal-tonoise ratio for slope failures varies between 1 and 30, with a mean of 4 for the Illgraben data set and 22 for the Pizzo Cengalo data set.
Data stream handling To avoid the extra step of detecting events with a trigger such as the STA / LTA algorithm, we classify a running window on the continuous data stream with an overlap of 2/3 of the window length. The overlap was chosen to avoid missing events on the window margins but was not tested for optimal performance. We transform the event catalog with start times and end times of all events into a catalog containing the times of all running windows that include an event for both the Illgraben and the Pizzo Cengalo data set. For the Pizzo Cengalo data set, we make use of the network configuration at the study site, in order to increase the number of training samples. At the frequency band of interest (1-10 Hz), associated wavelengths are larger than the inter-station distance, resulting in waveforms with only small differences.
For earthquakes and slope failures, instead of using the same onset for the sliding windows at all stations, we choose a random onset with a maximum of 2/3 of the sliding window before the event start time. This way, we catch different windows of the signal and increase the training data set by a factor of 3 without using the same window several times. This procedure is unnecessary for the Illgraben data set, as the sourcereceiver distances vary for each station. For discrete noise signals often recorded at only one station, we choose sliding windows at the recording station, again with a random onset up to 2/3 of the sliding window before the event start time. For the continuous noise, we choose a random station at each time step. We divide the catalog with labeled events into a training and test data set, with 70 % of all events as training data and 30 % as test data.
This partition was chosen to be able to meaningfully assess the algorithm performance for the small number of slope failures in the Pizzo Cengalo data set. This way, there are windows of three slope failure events in the training and validation data set and windows of two slope failure events in the test data set. We choose a window length of 20 s as an initial guess. Numbers for resulting training and test data set sizes for Pizzo Cengalo and Illgraben are presented in Table  B2. Following  Provost et al. (2017) , we then compute features of these sliding windows. As we do not use the entire waveform of the event, but only the parts that appear in the sliding window, we exclude features related to the entire waveform of the signal (e. g. , duration and rise time). Additionally, the network configuration at Pizzo Cengalo does not allow network features (signals are too similar between stations), nor does it allow for polarity features (only vertical component available).
For the sake of comparison, we also disregard network and polarity features for the Illgraben data set. We are left with a total number of 55 features, including waveform characteristics in the time and frequency domain (see Table  B ). These features have been proven significant for accurate seismic signal classification (e. g. ,  Hibert et al. , 2017; Provost et al. , 2017). Additionally, we tested a Python tool for automatic feature generation for time series (TSFRESH;  Christ et al. , 2016) , which did not improve classification results compared to the features proposed by  Provost et al. (2017). Before feature calculation, we apply a four-corner Butterworth band-pass filter (1-10 Hz). For feature generation after  Provost et al. (2017) , we choose frequency bands of 1-3, 3-6, 5-7, 6-9, and 8-10 Hz.
Imbalanced data set handling The limited amount of data and, more specifically, the small number of SF events that happened in 2018 at Pizzo Cengalo lead to an imbalanced data set. As shown in Fig. 3a , the number of events is unevenly distributed among classes. This poses a problem for machine learning algorithms, as they generally optimize the score, i. e. , the number of correctly labeled classes. In a highly imbalanced data set, the classification algorithm may be less sensitive to the minority class, as it does not drastically impair the score if it is labeled incorrectly. For our data set, with the events that we are most interested in being the minority class, it is particularly important to address this problem. Therefore, we introduce here different data augmentation and classifier tuning methods to improve the classifiers' performance on the imbalanced Pizzo Cengalo data set.
There are several possibilities to handle imbalanced data sets, based either on manipulating the training data set or on changes within the algorithm  (Chawla, 2010). The most straightforward approaches are random undersampling (US) and naive oversampling (OS) training data. For random undersampling, only a random subset of training data of the majority class is chosen. This way, the data set becomes more balanced by reducing the samples in the majority classes. However, this might mean that important characteristics of the majority class are not captured. In contrast to undersampling, naive oversampling randomly multiplies samples in the minority class but thus increases the risk of overfitting -i. e. , the lack of generalization -within the minority class. A more sophisticated way of increasing training samples in the minority class is synthetic minority oversampling (SMOTE)  (Chawla et al.
, 2002). SMOTE is based on the idea of creating new training samples in the minority class by interpolating between a sample and a random set of its k-nearest neighbors in feature space. Therefore, a new sample is generated with features similar to already-existing samples. This increases the sample size of the minority class but minimizes the problem of overfitting. On the algorithm level, random forest opens two possibilities for imbalanced data: setting a class weight on the minority class or undersampling the training data for every single tree, a so-called balanced random forest (BRF)  (Lemaître et al. , 2017). Here, we use a BRF classifier that undersamples the majority class and weighs the classes inversely to the number of samples in each class.
Data set comparison We trained a random forest classifier on two different types of data sets: the Illgraben data set, with a balanced abundance of all types of classes, and the Pizzo Cengalo data set, with a highly imbalanced number of events. Both the Illgraben and the Pizzo Cengalo data sets were aggregated by manual inspection of the seismic data and, for the Pizzo Cengalo data set, direct observations at the site. Even though both data sets have been carefully examined, a misclassification of events cannot be excluded. For earthquakes, a misclassification is highly improbable, as all events were cross-checked with publicly available earthquake catalogs. However, crosschecking databases for rockfall events are rare, especially for events that have not affected infrastructure. Therefore, an in- herent bias by manual labeling and falsely labeled events is possible but can hardly be avoided for these data sets.
Performance enhancement One way to enhance the classifier's performance on the Illgraben data set could be to add network and polarity features. Provost et al. (2017)  have shown that network features count among the most important features for classifying detected events. However, the setup of the LERA network at Pizzo Cengalo has an unfavorable aperture for detecting and classifying detected seismic signals, as the stations are set up in a line with inter-station distances of 9 and 31. 5 m. This prohibits the usage of network characteristics, like arrival time differences and amplitude ratios. To address the problem of the imbalanced Pizzo Cengalo data set and resulting misleading scores, we test several techniques to handle such data sets and use receiver operating characteristic curves for performance assessment.
This does not mean, however, that imbalance countermeasures did not work. Compared to the initial test (Fig. 5b ), any of the measures mentioned above improves the classification result. We choose to use a 40 s window as an operative window size to test on the 2019 data, even though 10 and 60 s show equally high AUC values and low standard deviations. On a continuous data stream a 10 s window with 6s of overlap does not leave enough time to compute features and classify the event in real time. The 60 s window, on the other hand, results in a classification delay of 1 min, and we assume that chances are higher of missing smaller events which are masked by a large amount of noise in the 60 s window. Generally, an imbalanced data set can be tackled by increasing the amount of training data in the minority class.
A classifier trained for an area that is more active or has been monitored during a longer period is expected to give better results with higher accuracy. Additionally, the small number of events in the slope failure class can lead to overfitting, i. e. , an insufficient generalization of the model. Hence, small deviations in signal characteristics can lead to misclassification and undetected slope failures. However, as seen on tests on the Illgraben data set, solely a larger training data set will not give a perfect classifier either. We attribute the low SNR to the classifier's relatively poor performance of 30 % misclassified events in the slope failure class at Illgraben.
One-step vs. two-step method at Pizzo Cengalo Several studies have shown that classification algorithms accurately classify events detected with the STA / LTA approach (e. g. ,  Hibert et al. , 2017; Provost et al. , 2017). The benchmark analysis performed on the Pizzo Cengalo data set indicates that our continuous approach (CSI of 0. 375) performs slightly better than a two-step approach (CSI of 0. 2). However, the small number of events prohibits a statement on robustness. Interestingly, there is a large overlap in the earthquakes being misclassified as slope failures between the two approaches. Feature importance analysis for the continuous one-step approach presented here shows that the classifier predominantly uses spectral features to distinguish between different classes. This is consistent with the fact that the windowing eliminates information from the entire waveform.
Furthermore, the automatic-camera pictures show no apparent detachment zone (Fig. 8 ). This either means that the break-off happened outside of the camera field or validates our assumption of a small-volume event. Two of the events classified as slope failure are snow avalanches in April 2019 (Fig. 8 ). At this time, Pizzo Cengalo's active slope was partially covered by snow. As both avalanches happened during snowfall periods with obscured view from the automated camera to Pizzo Cengalo, the events were only validated by pictures several days after the actual event. The random forest classifier was only trained on data from the summer period, and therefore snow avalanches were not part of the training data set. However, the seismic signature of snow avalanches is highly similar to those of other mass movements (e.
g. ,  Suriñach et al. , 2001; van Herwijnen and Schweizer, 2011; Heck et al. , 2018). This also validates our assumption that signal differences between different mass movements are less prominent than differences associated with noise and earthquakes. Hence, for training purposes, snow avalanches could be included in the slope failure class. This would allow the detection of slope failure events in winter and in summer months and increase the number of events in the slope failure class, resulting in a less imbalanced training data set. We chose a minimum of three consecutive windows classified as slope failure for a slope failure event detection. If we were to increase the number to four, the number of slope failure event detections would be lowered to four. Out of these four, only one is an actual slope failure, and three are earthquakes.
We explore this approach with our classification problem of earthquakes falsely classified as slope failures. We train a model for a seismic station of the Swiss Seismological Service (CH. FIESA) located 130 km away from Pizzo Cengalo. At this distance local and regional earthquakes that were misclassified as slope failures at Pizzo Cengalo are still recorded, but not slope failures. We use the EMSC earthquake catalog to compile a training data set containing 45 earthquakes and 513 noise samples in 2018. We then train the model on 20 s windows of seismic data. For the decision-making process, we test the trained model on seismic data recorded when a slope failure was classified at Pizzo Cengalo in 2019. All four earthquakes falsely classified as slope failures at Pizzo Cengalo were classified as earthquakes at FIESA.
All time windows around the slope failures at Pizzo Cengalo were classified as noise at FIESA. Therefore, a simple inclusion of a station at a regional distance to Pizzo Cengalo reduced the false alarms to zero. Our approach is computationally inexpensive, as the classification model only has to be trained once. From there on, computational power is only needed to compute features of the 40 s windowed seismic data. For this near-real-time sim-ulation, we used a standard machine (2017, Inter Core i7), which was able to compute all features and classify the windowed data within 3 s. Combined with the consecutive window threshold, this allows for a short warning time within tens of seconds after the event start.
Acknowledgements. We thank  Geopraevent Ltd. , who installed and maintain the seismic stations, provided the data and an online data portal, and implemented the algorithm to run in real time. We are thankful for constructive feedback from  Velio Coviello  and an anonymous reviewer on an earlier version of the manuscript. We also want to acknowledge the Obspy developer team (Beyreuther et al. , 2010; Krischer et al. , 2015) for providing an easy-to-use framework for seismic data handling. We thank  Yves Bühler  and two anonymous referees for insightful reviews who helped to greatly improve the manuscript. Financial support. The project was funded by  WSL's strategic initiative Climate Change Impacts on Alpine Mass Movements (CCAMM). Fabian Walter 's salary was funded by the  Swiss National Science Foundation  ( GlaHMSeis  Project  PP00P2_157551  and  PP00P2_183719 ).
• An example workflow of object detection with deep learning that shows how a simple GitFlow-based software development process can be extended for MLOps with selected MLOps tools. Demystifying MLOps and selection procedure of tools is of utter importance as the complexity of ML systems grows with time. Clarification of stages, roles, and tools is required so that everyone involved in the workflow can understand their part in the development of whole system. This paper is structured as follows. In Section 2, a quick overview of work mentioning and using MLOps techniques is given and is followed by the depiction of workflows in MLOps in Section 3. With a description of the various roles in Section 4 and a comparison of MLOps supporting tools and applicable monitoring metrics in Section 5, a use case for automating the workflow for object detection with deep neural nets is discussed in Section 6.
Muralidhar et al. summarized commonly practiced antipatterns, solutions, and future directions of MLOps in  [11]  for financial analytics. In applying the stated recommendations, error sources in ML projects can be reduced. Next to conforming with the stated collateral best practices in MLOps workflows, the work on hand also tries to extend and generalize the author's definition of involved actors, specific to the financial use case. An approach for easy-to-use monitoring of ML pipeline tasks, while also considering hardware resources, was presented by Silva et al. in  [12]. Concerning the required time for completing tasks (e. g. , operations in batch or stream processing data sets) and focus on resource utilization, thoughts on benchmarking ML-based solutions in production were given. The authors overall compared nine datasets consistent with binary and multiclass regression problems, where three datasets were based on images.
Their approach was evaluated by benchmarking all datasets in batch mode and applying five datasets for online tasks with every 50 and 350 threads. Monitoring aspects of resources in the different stages of an MLOps workflow is included in the work on hand, and there is no focus on benchmarking the chosen toolchain environment. Concerning data warehousing applications, Sureddy and Yallamula proposed a monitoring framework in  [13]. In defining various requirements for monitoring such complex and distributed systems, the framework helps in building effective monitoring tools. As MLOps systems require a monitoring solution, too, aspects of monitoring the server as well as the application are treated in the work on hand. Various aspects and best practices of monitoring the performance and planning infrastructure for specific projects were outlined by Shivakumar in  [14].
Regarding the data completeness, consistency, and statistics, the authors proposed a set of constraints and the respective computable quality metrics. Although the principles of a declarative quality check for datasets are applicable during an MLOps workflow, the enumeration of this approach is a surrogate for the definition of quality check systems and services. Taking the big data value chain into consideration, there are similar requirements to the domain of ML quality. Various aspects of demands on quality in big data were surveyed by Taleb et al. in  [17]  and answered by a proposed quality management framework. While considering the stated demands on data quality during the various sections of the work on hand, no specific framework or quality management model for big data value chains, as introduced by the authors, is proposed.
Barrak et al. empirically analyzed 391 open-source projects which used Data Version Control (DVC) techniques with respect to coupling of software and DVC artifacts and their complexity evolution in  [18]. Their empirical study concludes that using DVC versioning tools becomes a growing practice, even though there is a maintenance overhead. As DVC is a part of the work on hand, it neither exclusively focuses on versioning details nor takes repository and DVC-specific statistics into consideration. Practical aspects of performing ML model evaluation were given by Ramasubramanian et al. in  [19]  concerning a real life dataset. In describing selected metrics, the authors introduced a typical model evaluation process. While the utilization of well-known datasets is out of scope for the work on hand, the principles of ML model evaluation are picked up during the various subsequent sections.
Based on the Findable, Accessible, Interoperable, and Reusable (FAIR) principles, different data lifecycle stages and quality dimensions help in systematically processing and organizing data sets, their quality information, respectively. While determining and monitoring the quality of datasets and processed data structures are vital to different operations in MLOps, the work on hand does not address the sharing of preprocessed records. A systematic approach for utilizing MLOps principles and techniques was proposed by Raj in  [24]. With a focus on monitoring ML model quality, end-to-end traceability, and continuous integration and delivery, a holistic overview of tasks in MLOps projects is given, and real-world projects are introduced. As the authors focused on practical tips for managing and implementing MLOps projects using the technologies Azure in combination with MLflow, the work on hand considers a broader selection of supportive frameworks and tools.
Considering automation in MLOps, various roles and actors' main tasks are supported by interconnected tooling for the dedicated phases. Wang et al. surveyed the degree of automation required by various actors (e. g. , 239 employees of an international organization) in defining a human-centric AutoML framework in  [25]. With visualizing the survey answers, an overview of the different actor's thoughts on automating the various phases of end-to-end ML life cycles was given and underlined the author's assumption of only partly automating processes in such complex and error-prone projects. Additionally, the landscape of MLOps tools has evolved massively in the last few years. There has been an emergence of high-quality software solutions in terms of both open-source and commercial options. The commercial platforms and tools available in the MLOps landscape make ML systems development more manageable.
ML and ML Pipeline An ML system is also a software system, but the development of an individual ML model is essentially experimental. A typical ML workflow starts by gathering more insights about the data and the problem at hand. The data need to be analyzed (Exploratory data analysis (EDA)), cleaned, and preprocessed (feature engineering and selection) such that essential features are extracted from the raw data (with the support of Data Stewards). This is followed by dataset formation for training, testing, and validation. Training, validating, and testing ML algorithms is not a straightforward method. The iterative process involves fitting the hyperparameters of an ML algorithm on the training data set while using the validation set to check the performance on the data that the model has not seen before at every iteration.