dataesr/scientific-paragraphs-categorization huggingface dataset:
The project was started in 2015 as a repository to share pieces of code within the LaMcube laboratory, with the specific aim to perform video-extensometry driven tests. As the project kept growing, it became the reference Python module for driving experimental setups in the laboratory. We do think that it is now mature enough to be distributed to a wider audience. CRAPPY's purpose is now to provide an open-source software canvas for developing experimental tests in a versatile and accessible way. Its strength lies in the ease with which new devices or tools can be added by users, making it adaptable to most experimental fields. We chose not to implement a graphical interface to avoid any loss of generality and to keep CRAPPY as versatile as possible. The programming knowledge required for the user to write a script was however kept as low as possible.
Python was chosen to this end, as it is both popular in the scientific community and widely taught in universities. Before CRAPPY a few other projects with a similar purpose have been released, the most popular being the LABVIEW suite  [1]. Its target audience is however quite different as it is both closedsource and not free, and requires learning its own graphical programming language. Other similar projects also rely on Graphical User Interfaces and show various differences with CRAPPY. MyOpenLab  [2]  is intended to drive hardware in real-time rather than running the test following a given pattern, ACQ4  [3]  is explicitly oriented towards neurophysiology, and PyMoDAQ  [4]  has a strong focus on nanotechnologies. Qudi  [5]  and CRAPPY share the goal of implementing a unified interface for a wide variety of hardware, but users need advanced Python skills to add new features to Qudi.
Software architecture CRAPPY is a Python module that can be installed on Linux, Windows, or Mac OS. It only requires Python 3. 6 (or higher) and Numpy  [7]. Some functionalities however require installing additional Python modules. For instance, Matplotlib  [8]  is required for real-time graph plotting and OpenCV  [9]  for image acquisition and processing. An error message informs the user if a module is missing for running a script. Installing CRAPPY can be done either by downloading the project from GitHub and running python setup. py install or by running pip install crappy as the project is now distributed on PyPI. CRAPPY is the result of a compromise between versatility and accessibility. It can interact with a wide variety of hardware such as cameras, Digital Acquisition (DAQ) boards, motors, but also with virtual elements like PID controllers, graphs, files, and virtual actuators allowing to test scripts without requiring any actual hardware.
CRAPPY is entirely operational on Raspberry Pi OS running on a Raspberry Pi 3 or 4, making it suitable for low-cost setups. The GPIO port also gives access to the wide ecosystem of sensors and actuators built around the Raspberry Pi, which is otherwise hardly accessible from regular PCs. Furthermore, the computing power of the newest Raspberry Pi even allows performing tasks as complex as real-time DIC, as shown in  [10]. Using OpenCV's DISflow  [11]  on a Raspberry Pi 4 it was possible to run DIC at 6 Hz using the piCamera v2 without overclocking, offering an excellent low-cost setup with real-time strain monitoring using nothing but a low-cost camera. To write a script with CRAPPY, the user mostly works with Blocks. They are similar to blocks in graphical languages, except CRAPPY presents no GUI to create the blocks and connect them.
Each Block is meant to perform a single task in a separate process. Depending on its function, it may have a variable number of inputs and outputs. It should be noted that a Block is not limited to a single process: it can use more than one core for CPU-intensive tasks such as DIC and video-extensometry. Blocks communicate together by sending Python dicts over ''Links'', a class relying on Pipes to transmit data between processes. The code snippet Listing 1 illustrates the instantiation of two Blocks, a Generator and a Grapher, and the connection between them using crappy. link(). In this script, the Generator outputs a sine wave signal with given characteristics and the Grapher displays the signal. This script runs until the user presses the CTRL+C key, which also closes the Grapher window.
Note that Matplotlib is required for displaying the graph. The program can be modified to automatically stop after 5 s by setting the 'condition' to ''delay=5'' instead of None. To add a Recorder block that saves the output of the generator, the two following lines can be added just before crappy. start() rec = crappy. blocks. Recorder ( 'my data. csv ' ) crappy. l i n k ( gen , rec ) A block with outputs can be linked to any number of blocks, the same data will be sent to all downstream blocks. Also, data going through a Link can be edited on the fly using Modifiers. They can perform filtering, apply formulas, edge detections, and more. The example file modifiers. py shows how to use and create Modifiers. The documentation lists all the existing modifiers and their arguments. Using Blocks is convenient to split complex experiments into simple tasks.
It also provides a great scaling capability by using multi-core processors to their full extent. One limit of this architecture is the bandwidth of data exchanged between processes, which depends on the memory's effective speed. It was experimentally measured to be around 1GB/s on recent mid-range laptops. This is more than enough for sending analog measurements at a few kHz but may become limiting when sending uncompressed high-resolution images. As a workaround, it was chosen to make all the image-processing blocks also handle the acquisition and the saving, so that the image does not have to be transferred between processes. This is a clear departure from CRAPPY's aim of modularity, but an option allows separating the different tasks for a more modular yet less efficient layout. Apart from Blocks and Links, CRAPPY also features several groups of objects at the root of the module: Camera, InOut, Actuator, and Tool.
Fig. 1  summarizes the architecture of the module and the location of the different objects. The rightmost column enumerates a few objects or functions for each category. The classes in bold are parents of all the other classes within a given category. Camera is meant for image acquisition devices, InOut for hardware with physical inputs or outputs (usually DAQ boards), and Actuator for motors. Tool does not relate to hardware but rather to data processing classes. All these classes are used by the Blocks and are usually never called directly. For example, the block VideoExtenso, which performs video-extensometry, takes an argument specifying which Camera class should be used for image acquisition. It will at some point instantiate this Camera class, which will itself instantiate a CameraConfig Tool to adjust the image settings before starting the test.
Software functionalities The interest of CRAPPY resides in its modular architecture and the possibility to simply add new hardware and features, although the readily implemented elements already provide a wide range of functionalities. The main blocks will be quickly presented here. For an exhaustive list, refer to the online documentation at: https://crappy. readthedocs. io. Examples demonstrating the use of the most common blocks can be found on GitHub: https://github. com/LaboratoireMecaniqueLille/crappy. Different layers of complexity can be included in a script, depending on the task to perform. The easiest way to run a CRAPPY script is to instantiate existing blocks, link them together and call crappy. start(), as showed in Listing 1. The implemented blocks should be enough for running most of the simple tests.
Tensile test Getting back to the tensile test mentioned in the introduction, this section will present an implementation of such a test using CRAPPY. Most modern tensile test machines can read the command from an external analog voltage. It means that the machine can be driven using only a DAC with an appropriate output range. In this example, a Labjack T7 board is used for generating the signal and reading the sensors. It is wired to an Instron 5882 electromechanical tensile machine, but the presented program is not specific to this machine. This paragraph refers to the example Examples/tensile_1. py that can be found on GitHub or in Appendix A. The first step is to create a Generator block. As the analog voltage controls the position of the machine and a constant speed is needed, the Generator should output a ramp.
A shutdown procedure then ensures that the program ends properly and no data is lost. Now that a basic test was detailed, we can add new elements to illustrate the possibilities offered by CRAPPY. A new setup now using an in-house tensile test machine ''Biotens'' is presented here. We now have look at tensile_2. py, again located on GitHub or in Appendix B. In order to better capture the strain in the Region Of Interest (ROI), this setup uses video-extensometry. Four   pattern, as shown in Fig. 3. The entire setup is represented in Fig. 2  and has been previously described here  [15]. To add video-extensometry, a Video_extenso block is needed. The camera used and other parameters must be specified. Since a real strain measurement is now performed, we would like to apply cycles of loading to a chosen strain and unloading to a 0N state.
Furnace Another example of an application of CRAPPY to experimental setups is a solidification furnace featuring five independent resistors, as shown in Fig. 5. The temperature of each section is measured using a thermocouple and adjusted automatically to follow a pre-defined cooling pattern. This setup allows setting a different temperature in each section to create controlled temperature gradients, thus giving a fine control of the microstructure during the solidification of an alloy. Each resistor is controlled by its own PID. The command is sent as a Pulse Width Modulation (PWM) signal using a Labjack board to control the transistors that ultimately power the resistors. The script runs on a Raspberry Pi 3, reducing the costs of the overall setup. For reference, the code is available under impact/furnace.
The usual testing patterns include braking at a constant speed, simulate braking a train of a given mass, braking at constant power dissipation, or just waiting for a condition to verify (e. g. for the pad to cool down below a given temperature). The force applied on the pad is controlled by a PID loop. An external device (Labjack T7) was preferred to CRAPPY's PID, as it was not meeting the high-frequency requirement for this setup. CRAPPY only writes registers on the Labjack to activate or deactivate the PID, and set the target value. It is also possible to drive the torque instead of the force, to perform tests at constant braking power. For more information on this setup, the documentation and the code can be found at https://github. com/LaboratoireMecaniqueLille/ Tribo/. It took two months for an engineer to write the entire code including the GUI, the documentation, and the bindings for the ''Spectrum'' high-frequency acquisition board library (that are now part of CRAPPY).
Impact As demonstrated in the previous section, CRAPPY is a powerful tool for building complex setups. It is not meant for the user to create graphical interfaces or drive complex systems in a few clicks. Instead, the user is expected to have in mind the interactions between the components of his setup and to write code accordingly. Because it manages most of the software complexity during a test, CRAPPY makes it much easier to design innovative setups in a quick and accessible way. As any device may be added to the framework as long as it can interface with Python, almost any kind of experiment can be controlled. For experimental mechanics, CRAPPY also includes cutting-edge tools for image processing such as real-time DIC and video-extensometry. Propositions of code for applications in other domains would of course be happily considered.
Conclusions This paper introduced CRAPPY, an open-source Python module developed at the LaMcube to make the programming of experimental tests in Python easier. Examples of code were explained and detailed to present the basics of the module and give an overview of its capabilities. This project has been used internally for years and we believe it has now become stable enough to be beneficial to other institutes and to the industry. The code is available on GitHub under the GPLv2 license. Contributions are welcome, please use GitHub to ask questions, report issues, or contribute.
Acknowledgments We would like to thank  Robin Siemiatkowski  who worked on C++ bindings for Cameras and on porting the code to  Windows. Also , we would like to acknowledge the users of CRAPPY who brought important feedback, suggestions, and improvements since its creation including  Laure Astruc ,  Annie Morch , and  David Roucou. This work has been financially supported by the  AID (Agence Innovation Défense)  and the  Région Hauts-de-France. The authors gratefully acknowledge the support of these institutions.
Therefore, unlike CH-PeEn, MuPEn depends on the forecast horizon, which is a natural consequence of multivariate forecasts. The main contribution of this paper is the introduction of a multivariate probabilistic solar forecast benchmark that allows forecasters to compare their forecasts with, in what is expected to become a relevant category of forecasting  (Li and Zhang, 2020). Using temporal GHI data recorded by the surface radiation (SURFRAD) budget network and spatio-temporal GHI data recorded by the Oahu Solar Measurement Grid  (Sengupta and Andreas, 2010) , I show that MuPEn is time invariant, insensitive to the training data length and insensitive to the number of ensemble members or trajectories used to construct the multivariate forecasts. At this point, it is helpful to note that ensemble members and trajectories are used interchangeably throughout this paper.
The remainder of this paper is structured as follows. Section 2 introduces multivariate probabilistic forecasting and important aspects thereof, whereas Section 3 outlines the construction of MuPEn. Section 4 describes the data sets used in the paper, as well as the univariate and multivariate forecast verification tools. Continuing, Section 5 presents the temporal and spatio-temporal case studies, including a univariate analysis in the former case study. Finally, Section 6 concludes this paper.
Empirical evidence will be provided in Section 5. 1. 1. The R code that is written for this study is available on https://github. com/ DWvanderMeer/multivariateBenchmark in order to promote open research  (Yang et al. , 2018). The data used in this study can freely be downloaded using SolarData  (Yang, 2018). Finally,  Yang (2019b)  noted that the choice of clear-sky model affects CRPS; in this study, the effect of the clear-sky model on the multivariate probabilistic forecasts is examined.
Data Two data sets are used to evaluate the time and space-time trajectories of the proposed benchmarks. As mentioned above, both data sets can be downloaded using SolarData  (Yang, 2018). To evaluate the MuPEn time trajectories, GHI measurements from the SURFRAD network are used  (Augustine et al. , 2000; Augustine et al. , 2005). The network consists of 7 stations spread out over the United States and each station measures GHI at 1 min resolution since 2009-01-01. Identical to  Yang (2019b) , the years 2016-2017 are used as testing data to evaluate MuPEn at each of the 7 stations (abbreviated BON, DRA, FPK, GWN, PSU, SXG, and TBL, see https://www. esrl. noaa. gov/gmd/grad/surfra d for an overview). To evaluate the time invariant property of MuPEn, 2 year periods between 2005 and 2015 are used as training data.
The second data set comprises 1 s GHI measurements from the Oahu Solar Measurement Grid, which is a dense network of 17 pyranometers covering roughly 1 km × 1 km  (Sengupta and Andreas, 2010). In this study, station AP3 is removed because it contains substantially more zero values than the other 16 stations. The data set covers the period from 2010-03 to 2011-10 and herein, the 5 month period between June up to and including October 2011 is used as testing data. The preceding 15 months serve as training data in which the training data length increases with 3 months to evaluate whether MuPEn is sensitive to the length of training data. In addition, the second data set is used to evaluate the sensitivity of MuPEn to the number of ensemble members m. In order to empirically validate that MuPEn is equivalent to CH-PeEn in case h = 0 and m = n, both data sets are averaged to 15 min resolution.
Thorarinsdottir et al. (2016)  showed that employing only one prerank function could lead to incorrect conclusions regarding miscalibration because a single prerank function cannot detect all types of miscalibration. The script containing the above-mentioned prerank functions can be downloaded as supplementary material to the publication by  Thorarinsdottir et al. (2016) , which also includes simulation studies to replicate their results. The reader is therefore referred to  Thorarinsdottir et al. (2016) ,  Wilks (2017)  and  Thorarinsdottir and Schuhen (2018)  for extensive simulation studies and resulting histograms displaying the various types of miscalibration that can occur in multivariate ensemble forecasts.
1. 2. The CRPS results in Table  1  can be directly compared with the CRPS results of CH-PeEn in Table  2  by  Yang (2019b). The mean absolute percentage deviation in CRPS between MuPEn and CH-PeEn is approximately 0. 28% for the Ineichen-Perez clear-sky model and approximately 0. 23% for the McClear clear-sky model, which provides empirical evidence that MuPEn and CH-PeEn are equivalent in case h = 0, z = 1 and m = n. Since MuPEn and CH-PeEn are equivalent and CH-PeEn minimizes the forecast resolution by its design  (Yang, 2019b) , it can be concluded that MuPEn similarly minimizes forecast resolution. Finally, Table  1  shows that MuPEn is almost independent of the choice of training data and is therefore nearly time invariant. Besides the numerical score, it is pertinent to evaluate the marginal calibration.
Multivariate distribution As mentioned above, 40 trajectories are used in this case study to construct the MuPEn. Table  3  shows the ES of the multivariate probabilistic forecasts for each of the 7 stations. Similar to the univariate case, Table  3  shows that MuPEn is nearly time invariant, which is to say that MuPEn is nearly independent of the choice of training data. To further verify that the dependence structure between the forecast horizons remains similar when using various training data sets, I present the VS 0. 5 in Table  4. Again, the results confirm that MuPEn is nearly independent of the choice of training data. Interestingly, the differences between the clear-sky models in terms of CRPS are larger in relative terms than the differences between the clear-sky models in terms of VS.
Space-time trajectories In the previous section, it was shown that the MuPEn is nearly time invariant in that various periods of training data result in similar CRPS, ES and VS. In the present case study, I assess the impact of the training data length on the performance of the MuPEn, as well as the number of trajectories used to approximate the multivariate predictive distribution. Note that the REST2 clear-sky model is henceforth omitted for brevity as it improved marginally over the McClear clear-sky model in terms of ES and performed slightly worse in terms of VS. The decision to disregard REST2 is supported by  Yang (2020) , who noted that the  simplest clear-sky model should be chosen in case their performances are similar 2. Moreover, the univariate distribution evaluation is omitted for brevity as well but it is noted that this is an important aspect of multivariate forecast evaluation  (Scheuerer and Hamill, 2015).
Tables  6  and 7  present the results in terms of ES and VS 0. 5 , respectively, where the rows indicate the increasing number of trajectories and the columns the increasing training data length. In this case study, the test set comprises the months June up to and including October 2011. It should be noted that this setup results in a rather low number of forecastverification pairs (approximately 4000) relative to the high number of dimensions (384), which in turn results in some uncertainty related to the scores presented in this section  (Pinson, 2013). However, the purpose of this exercise is to show the dependence of the MuPEn on the training data length and on the number of ensemble members m rather than setting absolute benchmark values. If the MuPEn is used as a benchmark against more advanced models, it is recommended to increase the number of forecast-verification pairs by, e.
Furthermore, the figures show that there is limited improvement in terms of calibration when increasing the data length once the MuPEn is constructed using more than three months of data, which further attests to its insensitivity to data length. Generally, the calibration of the space-time trajectories compares favorably to the calibration of the time trajectories in Section 5. 1. 2, which could be due to the fact that the training and testing periods are closer in time or due to the local climate.
A benchmark for multivariate probabilistic solar irradiance forecasts should furthermore feature minimal forecast resolution, be time invariant and be insensitive to training data length and parameters. By design, MuPEn inherently features minimal forecast resolution when the number of sampled trajectories is sufficiently large. In addition, the results of the case studies showed that MuPEn is time invariant and insensitive to both the training data length and the single parameter it requires, i. e. , the number of trajectories. As a result, MuPEn can be considered as a benchmark for multivariate probabilistic solar forecasts because it will generate consistent trajectories for a given forecast horizon, location and data resolution in terms of energy score (ES), variogram score (VS) and calibration.
Acknowledgments This work forms part of the Swedish strategic research program StandUp for Energy. Part of the work in this paper was carried out in the frame of the  EU  project  Smart4RES  (project No. 864337 ). I thank  Dr. Dazhi Yang  for assisting with the REST2 code and MERRA-2 data and  Dr. Pierre Pinson  for discussions on multivariate forecast evaluation. I also thank the anonymous reviewers for their comments that improved the quality of the manuscript.
From Pattern Matching to Memory Moves We now describe the Rew core language. Rew allows to describe Algebraic Data Types and simple rewriting rules on them. For conciseness, we omit the type system and dynamic semantic for this simple system (see  [11, 15]  for exhaustive descriptions of rich pattern languages) and focus on the compilation. We start with our running example before proceeding with the definition of the language. Example 1 (Simple example of Rew program). As a running example, we consider the code below which defines the algebraic data type for binary trees containing integers and a transformation pull-up which pulls the right subtree up, also represented graphically. A Rew function is similar to explicitly typed functions using pattern-matching, but defines a rewrite. Here, expressions only allow constructors and variables.