dataesr/scientific-paragraphs-categorization huggingface dataset:
g. , forecast point at sea) that may induce large temperature biases. Within this simplified benchmark exercise, the only predictor that could be used to perform the postprocessing is the temperature at 2 m itself. Additionally the use of the (static)   metadata was allowed, and some methods used latitude and longitude, elevation, land use, model orography, lead time, and also the day of the year. The 11-member reforecasts produced during the 2017-2018 period were considered training data, while the 51-member forecasts for the same period were used as test data for verification. This setup introduced some challenges for the implementation of some of the postprocessing methods described below. To avoid a potential overlap between the reforecasts and the forecasts, the forecasts from 2017 that were included in the reforecasts of 2018 have been removed from the training dataset.
In particular, the postprocessed forecasts at the station locations are compared with the station observations within the test dataset  (2017) (2018). Forecast quality is multifaceted  (Murphy, 1993) , and no single score can capture all aspects. Here we use four metrics year Dimension to identify the year in the past; year = 1 means a forecast valid 20 years ago at the reforecast day and month, and year = 20 means a forecast valid one year before the reforecast date. Only valid for reforecasts. step Step of the forecast (the lead time). surface Layer of the variable considered (here there is just one, at the surface). isobaricInhPa Pressure level in hectopascal (or millibar). valid_time Actual time and date of the corresponding forecast data. Remark: bold metadata denote dimensions indexing the datasets.
A reliable ensemble would lead to a flat rank histogram. The shape of the rank histogram can help to detect deficiencies in ensemble calibration, e. g. , a U-shaped rank histogram indicates underdispersion or conditional biases. The verification using the different measures allows us to detect if the compared postprocessing methods have systematic errors or biases and if the postprocessed ensembles are well calibrated, overconfident or underconfident. The reference forecast dataset will be the raw IFS ensemble at the nearest forecast grid point from every station. The difference between IFS orography and station elevation is taken into account by applying a constant lapse-rate correction of 6. 5 • C km -1. In this study, some results are also presented conditioned on the station elevation to detect remnant orographic influences.
Postprocessing methods Along with the dataset and verification framework described above, the present work further includes a collection of forecasts of exemplary postprocessing methods along with corresponding code for their implementation. Note that with providing forecasts of a selected set of methods, we do not intend to provide a comprehensive or systematic comparison to establish the best approach but rather aim to present an overview of both commonly used and more advanced methods ranging from approaches from statistics to machine learning. Those can be used in subsequent research for developing extensions to existing approaches and for comparing novel methods to established baselines. Short descriptions of the methods available in the present benchmark are provided below, and verification results are presented in Sect.
Accounting for systematic and representativeness errors (ASRE) ASRE postprocessing tackles systematic and representativeness errors in two independent steps. A local bias correction approach is applied to correct for systematic errors. For each station and each lead time, the averaged difference between reforecasts and observations in the training dataset is computed and removed from the forecast in the validation dataset. The difference averaging is performed over all training dates centered around the forecast valid date within a window of ±30 d. Representativeness errors are accounted for separately using a universal method inspired by the perfect prog approach  (Klein et al. , 1959; Klein and Lewis, 1970). A normal distribution is used to represent the diversity of temperature values that can be observed at a point within an area given the average temperature of that area.
Member-by-member postprocessing (MBM) The member-by-member approach calibrates the ensemble forecasts by correcting the systematic biases in the ensemble mean with a linear regression-based MOS technique and rescaling the ensemble members around the corrected ensemble mean  (Van Schaeybroeck and Vannitsem, 2015). This procedure estimates the coefficients α MBM , β MBM and γ MBM in the formula providing the corrected ensemble:  1 ) by optimizing the CRPS separately for each station and for each lead time. x m (t) = x m (t)-µ ens (t) here denotes the deviation of the member m from the ensemble mean. The results were obtained with the Pythie package  (Demaeyer, 2022a) , training on the 11 members of the training dataset to obtain the coefficients α MBM , β MBM and γ MBM and then using them to correct the 51 member forecasts of the test dataset.
We fit a single DRN model per lead time jointly for all stations and encode the station identifier and land-use via embedding layers to make the model locally adaptive. Since the use of additional input information has been a key aspect in the substantial improvements of DRN and subsequent extensions in other NN-based methods over EMOS, similar benefits are less likely here due to the limitation to ensemble predictions of the target variable only in the experimental setup; see  Rasp and Lerch (2018)  for more detailed comparisons.
Please note that the strong underdispersion of ECMWF DMO as indicated by the spread-error ratio in Figs. 3  and 6  is slightly reduced when systematic biases are removed (not shown). For the postprocessed forecasts, the effect of remaining systematic biases on the spread-error ratio is negligible. More detailed analy-  ses of the results would of course be possible but are beyond the scope of this publication on the benchmark dataset, and they will be the subject of a dedicated work.
Code and data availability The most straightforward way to access the dataset is through the climetlab (https://github. com/ecmwf/climetlab, last access: 22 December 2022) EUMETNET postprocessing benchmark plugin at https://github. com/EUPP-benchmark/ climetlab-eumetnet-postprocessing-benchmark (last access: 31 December 2022). This plugin provides easy access to the dataset stored on the ECMWF European Weather Cloud. An example showing how to use the plugin is documented in the Supplement, along with other unofficial ways to access the data. In addition, the dataset has been uploaded in Zarr format on Zenodo for long-term storage. See Demaeyer (2022b) (https://doi. org/10. 5281/zenodo. 7429236) for the gridded data and  Bhend et al. (2023)  (https://doi. org/10. 5281/zenodo. 7708362) for the station data.
However, the Switzerland station data which are part of the dataset are not presently freely available. These sta-tion data may be obtained from IDAWEB (https://gate. meteoswiss. ch/idaweb/, last access: 15 May 2023) at Me-teoSwiss, and we are not entitled to provide it online. Registration with IDAWEB can be initiated here: https://gate. meteoswiss. ch/idaweb/prepareRegistration. do (last access: 15 May 2023). For more information, please also read https: //gate. meteoswiss. ch/idaweb/more. do?language=en (last access: 15 May 2023). The documentation of the dataset is available at https://eupp-benchmark. github. io/EUPPBench-doc/files/ EUPPBench_datasets. html (EUMETNET, 2023) and is also provided in the Supplement. The code and scripts used to perform the benchmark are available on GitHub and have been centralized in a single repository: https://github.
com/EUPP-benchmark/ ESSD-benchmark (last access: 24 April 2023). This repository contains links to the scripts sub-repositories along with a detailed description of each method. In addition, these codes have been also uploaded to Zenodo: verification code (https://doi. org/10. 5281/zenodo. Mlakar et al. , 2023b). Finally, to allow for further studies and a better reproducibility, the output data (the corrected forecasts) provided by each method have also been uploaded to Zenodo (See https://doi. org/10. 5281/zenodo. 7798350,  Chen et al. , 2023a).
Conclusions and prospects A benchmark dataset is proposed in the context of the EU-METNET postprocessing (EUPP) program for comparing statistical postprocessing techniques that are nowadays an integral part of many operational weather-forecasting suites. This dataset includes ensemble forecasts and reforecasts of the ECMWF for the period 2017-2018, as well as the corresponding gridded and station observations, over a region covering a small portion of western Europe. This region covers a variety of topographies including coastal, flat and mountainous areas. To illustrate the usefulness of this dataset, a standardized exercise is established in order to allow for an objective and rigorous intercomparison of postprocessing methods. This exercise included the contribution of many wellestablished state-of-the-art postprocessing techniques.
Despite the limited scope of the presented exercise, this collaborative effort will serve as a reference framework and will be strongly extended. The whole process includes (i) the download of the data or their access on the European Weather Cloud (where the dataset is stored; see Sect. 6), (ii) the application of the different techniques by the contributors and (iii) the verification of the results by the verification team. This proof-of-concept proved to be very successful. While the authors constructed and performed this benchmark, some lessons were learned along the way: -As much as possible, avoid maintaining an archive or a database of scores for the experiments. Instead compute verification results for the experiments on the fly and only store the summary results. This has the advantage that you can easily add (or remove) scores, summaries of scores, without going through the complex process necessary to update an archive (with new submissions and additional scores).
-Either be very strict about the format of submitted predictions, or use software that is aware of the NetCDF data model and that can handle slight inconsistencies (e. g. , reordering of dimensions or dimension values) -Quality control is imperative. While the verification results generally quickly indicate whether there are any major issues with the submitted predictions, issues may already arise earlier than that (making verification impossible). Catching these errors and establishing a feedback loop with the submitters is important. One way to solve this with NetCDF format is to check the NetCDF header of the submission for format compliance. These points are important for the next EUPP projects, which will aim to harness the full potential of this dataset, by postprocessing other, less predictable variables (e.
For instance, all the methods produce a large fraction of significantly better forecasts (i. e. , with a lower CRPS) than the ECMWF DMO, while ANET, EMOS, DVQR and AR-EMOS outperform the other methods. Supplement. The supplement related to this article is available online at: https://doi. org/10. 5194/essd-15-2635-2023-supplement. Author contributions. JD led the overall coordination of the benchmark, as well as the collection and dissemination of the dataset. He further contributed to the verification setup, implemented the MBM method (Sect. 4. 3) and coordinated the writing of the paper. JC and NH implemented the DRN method (Sect. 4. 7). SL coordinated the writing of Sect. 4 and contributed to the implementation of  DRN (Sect. 4. 7). AM implemented the AR-EMOS method (Sect. 4. 5). CP coordinated the verification work and the writing of Sect.
3. 2. GE and BH utilized the IMPROVER codebase to implement the reliability calibration method, along with bias correction approaches (Sect. 4. 2). MD participated in the collection of the data and provided EMOS corrected forecasts  (Sect. 4. 4). BVS contributed to the verification results (Sect. 3. 2). DJ implemented the DVQR method (Sect. 4. 6). AA contributed to the verification work. JB participated in the collection of the data, contributed to the verification work by producing Figs. 3-6 and drafted the Results (Sect. 5). ZBB implemented the ASRE postprocessing method (Sect. 4. 1). SV leads the PP module of EUMETNET within which the benchmark has been conceptualized. PM developed and implemented the ANET method (Sect. 4. 8). JM and JFP contributed to the development and implementation of the ANET method (Sect.
Competing interests. The contact author has declared that none of the authors has any competing interests. Disclaimer. Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Special issue statement. This article is part of the special issue "Benchmark datasets and machine learning algorithms for Earth system science data (ESSD/GMD inter-journal SI)". It is not associated with a conference. Review statement. This paper was edited by Giuseppe M. R. Manzella and reviewed by three anonymous referees.
The land usage is extracted from the CORINE2018 (https://land. copernicus. eu/pan-european/corine-land-cover, last access: 30 July 2022) dataset(Copernicus Land Monitoring Service, 2018). More details are provided in the legend entry of the metadata within each file. The model terrain height is extracted from the EU-DEM v1. 1 (https://land. copernicus. eu/imagery-in-situ/eu-dem, last access: 1 August 2023) data elevation model dataset(Copernicus  Land Monitoring Service, 2022). Finally, the model orography can be obtained by dividing the surface geopotential by g = 9. 80665 m s -2. * Last access: 22 December 2022.
Acknowledgements. Jonathan Demaeyer thanks Florian Pinault and Baudoin Raoult from ECMWF for their help on the setup of the climetlab plugin, and Francesco Ragone, Lesley De Cruz and David Docquier from Royal Meteorological Institute of Belgium (RMIB) for their support to gather the gridded data. Jonathan Demaeyer also thanks Veerle De Bock and Joffrey Schmitz for their help with the RMIB data. The authors thank Tom Hamill for his guidance on the selection of variables during the dataset design phase. They also thank Thomas Muschinski and Reto Stauffer for raising an important issue about the station data which has since been corrected. This benchmark activity organized within the postprocessing module of the 2019-2023 EUMETNET program phase is largely supported financially by its members, constituted by a large number of European national meteorological and hydrological services.
The EUMETNET postprocessing module also thanks the ECMWF for its support on the European Weather Cloud. Jieyu Chen, Nina Horat and Sebastian Lerch gratefully acknowledge support by the Vector Stiftung through the Young Investigator Group "Artificial Intelligence for Probabilistic Weather Forecasting". Annette Möller acknowledges support by the Deutsche Forschungsgemeinschaft (DFG, grant no. MO 3394/1-1); by the Hungarian National Research, Development and Innovation Office (under grant no. NN125679); and by the Helmholtz Association's pilot project "Uncertainty Quantification". Finally, the authors would like to thank the three anonymous reviewers who helped us to increase the value of this paper. Financial support. This research has been supported by the 2019-2023 EUMETNET program, the Vector Stiftung (Artificial Intelligence for Probabilistic Weather Forecasting), the Deutsche Forschungsgemeinschaft (grant no.
station_id Unique identifier for the station. depthBelowLandLayer station_name Name of the station. model_latitude Latitude of the model grid point. model_longitude Longitude of the model grid point. step Step of the forecast (the lead time). surface Layer of the variable considered (here there is just one, at the surface). isobaricInhPa Pressure level in hectopascal (or millibar). station_land_usage Land usage at the station location, extracted from the CORINE 2018 (https://land. copernicus. eu/pan-european/corine-land-cover * ) dataset. model_altitude True elevation (in meters) of the model grid point, extracted from the EU-DEMv1. 1 (https: //land. copernicus. eu/imagery-in-situ/eu-dem * ) data elevation model dataset. model_orography Surface height (in meters) in the model at the model grid point.
model_land_usage Land usage at the model grid point, extracted from the CORINE 2018 dataset (https://land. copernicus. eu/pan-european/corine-land-cover * ). valid_time Actual time and date of the corresponding forecast data. Remark 1: bold metadata denote dimensions indexing the datasets. Remark 2: the metadata with "model" in their name indicate properties of the closest model grid point to the station location, and at which the forecasts corresponding to the station observations was extracted from the gridded dataset. * Last access: 1 August 2022.
Synthetic Data Generation Following  [16] , we go through three steps to generate a synthethic dataset of preferences: 1) sampling a random θ-additive function, 2) associating it to a ranking function r, 3) using the function to collect a set R of preferences. Sampling a random θ-additive function. This phase is conditioned by two parameters α and p. We start by sampling a model θ. To do so, we initialise it to θ = {{a 1 },. , {a n }} and then we add a proportion α of all subsets S ⊆ F to it, sampled as follows: 1. We initialise S with a random item ai. 2. We augment the set S by appending an element that is randomly sampled within F \ S. 3. We exit the process with probability p, otherwise go back to 2. After that, each coefficient w S for S ∈ θ is sampled w. r. t. a normal distribution N (0, 100).
These vectors were used to sample 1000 models θ with the posterior distribution P(θ|R) and with a burn-in period of 200. The value of was set to 1e-3 using trial-and-error iterations, by looking for a value ensuring that when |w S | ≤ , the parameter w S can be safely assumed as not significant. The tests were conducted on 400 datasets R. The size of R varies from 10 to 600 pairwise comparisons. Remark that |F | = 8 implies a total of 32640 possible pairwise comparisons between subsets of F; 600 comparisons is less then 2% of this total number. After each training, the models were evaluated by considering the predictions on the comparisons of 100 pairs of alternatives. Results and discussions. The results are summarized in Table  2. To facilitate our analysis, we divided our dataset of 600 preference sets into three groups based on their size.
The first group contains datasets involving between 10 and 200 preference, the second group contains datasets involving between 200 and 400 preferences, and the third group contains datasets with more than 400 preferences. In this experiment, we focus on two key metrics to evaluate the performances of our models: Prediction Rate and Prediction Correctness. We see in Table  2  that the predictions made using θ map are more frequent but less accurate than those made using θ d. Thus θ map offers an interesting tradeoff between the prediction rate and the prediction correctness. This is due to the fact that the polyhedron of compatible utilities yielded by the use of θ map is contained in the one yielded by the use of θ d (with a few exceptions that happen when θ map is not compatible with R and the set R is post-treated to remove some preferences).
Therefore, the predictions that are made using the robust ordinal regression with θ d are most of the time also made using θ map. However, as the dataset size increases, the loss in accuracy attributable to θ map progressively decreases while the gain in prediction rate increases resulting in an even more favourable tradeoff between prediction rate and accuracy for θ map compared to θ d for datasets above 200 preferences. It is worth noting that although θ true is the model used to generate the data, it achieves the worst compromise and especially for small datasets (less than 200 preferences) where the prediction rate is only about 6%. This has to do with the fact that the polyhedron of compatible utilities is much larger for θ true than for θ map and θ d , which badly impacts the prediction rate.
The obtained results depend on the size |R| of the dataset: • Figures 1 and 2 illustrate the evolution of the metrics when the size |R| varies between 10 and 250 (the curves show the mean and 95% confidence interval). From Figure  1  we can see that our approach performs relatively poorly on very small datasets (remember that 250 comparisons is less than 0. 7% of all possible comparisons) in terms of the tradeoff between the quantity and the quality of predictions. However, as the dataset size increases, the quality of the tradeoff improves significantly. Figures 2-a and 2-b allow to better understand the reasons for this evolution. They indeed show that the correctness of the predictions made by ROR consistently outperforms that of the other approaches. Interestingly, as the data set size increases, our approach (ROR) makes more predictions, which leads to an improvement in the quality of the tradeoff, reflected by the F-score.
• Generally speaking, as the size of R increases, there is a corresponding increase in all the metrics for the ROR model. Nevertheless, our tests demonstrate that once we reach a certain percentage threshold for the size of R (that seems to be around 0. 6% to 0. 7%, meaning 200-250 comparisons for |F | = 8), the results begin to exhibit minimal variations. For this reason, we opted to present the results in an aggregated manner, for |R| ranging from 250 to 600 (equivalent to 0. 7% to 1. 8% of the total number of comparisons). As Table  3  shows, our approach significantly outperforms the other models in terms of the F-score. Although our ROR approach, being robust, generates fewer predictions than SVC or Gaussian Process models, it achieves a prediction rate of 75% with less than 2% of comparisons in the dataset.
3D and 2D visual exploration of the architecture Two types of architectural visualization were designed: i) a 3D canopy exploration and ii) a 2D individual schematic representation based on the MTG within the OpenAlea. Strawberry Python package (see in Result section hereafter). The schematic 3D visualization of the canopy (Fig. 2B ) highlights the 3D spatial organization of the plant, including phyllotaxy. Vegetative development was represented using a color scale based on module order: zeroth-order (order 0, green color), first-order (order 1, red color), second-order (order 2, blue color), third-order (order 3, yellow color), fourth-order (order 4, purple color). The timing and intensity of flowering were indicated by modulating the size and shape of the light blue boxes representing the inflorescences (Fig.
Open-source software and web application for 3D visualization and architectural analysis Three software artifacts (Fig. 2 ) have been developed for the analysis and visualization of strawberry architectural data and development: i) the OpenAlea. Strawberry Python package, ii) a set of interactive Jupyter Notebook tutorials, and iii) an end-user web application. First, OpenAlea. Strawberry is an open-source Python package (https://github. com/openalea/strawberry), available in the OpenAlea platform  (Pradal et al. , 2008 (Pradal et al. , , 2015)). OpenAlea. Strawberry relies mainly on   (Pradal, et al. , 2009) , and OpenAlea. MTG  (Pradal and Godin, 2020). It provides methods for the architecture reconstruction, analysis at the node and module scale, and 2D and 3D visualization. As shown in Fig.