dataesr/scientific-paragraphs-categorization huggingface dataset:
E = (L 2 * -L 1 * ) 2 + (a 2 * -a 1 * ) 2 + (b 2 * -b 1 * ) 2 (1) We relied on the implementation of E provided by the colormath Python library  [29]. We used the GeoPy library  [30]  and the Nominatim geocoder to access OpenStreetMap data  [31]  and map regions of origin to coordinates. Some region names in our dataset do not exist in OpenStreetMap. It is for example the case for the subdivision of France into four main quadrants. In such cases, we considered the list of named areas composing the region and associated it with the mean latitude and the mean longitude of the areas in the list as a proxy for its location.
Building distance matrices for the ontologies In order to build the distance matrix for the colour and region ontologies, each pair of concepts in the ontology file is processed and the distance is calculated according to the previously defined methods. The OWL file containing the ontologies we engineered is read using the Python Owlready2 library  [33]. We thus retrieve the list of concepts for each ontology along with the pairwise distances. These are formatted as distance matrices stored in a global Microsoft Excel file. The ranges of distance values for each variable are very different. Each distance matrix is normalised on a [ 0 -100] scale to prevent some variables from out-weighting the others in future calculations.
Indeed we have missing data in our dataset. If either A i or B i or both are missing then d A,B (i) is missing too. The weight for each variable is defined in a configuration file. An example for a subset of variables and equal weights is presented in Fig. 5. The computation is made for all pairs of individuals to build the final distance matrix. This matrix contains both the D(A, B) distance and the number of variables used in the calculation (i. e. cardinal of M) for each pair of individuals. We decided to also calculate a distance matrix based on Gower's distance  [6]  for comparison purposes. We could not find an implementation of this distance in Python libraries. Moreover, we needed it to process missing values the same way as our semantic distance. We therefore developed our own based on the Dice and Manhattan distances as found in the scikit-learn library.
Projection in coordinate space and clustering The next stage is to group similar individuals based on the distance matrix. Since different clustering algorithms can produce different results depending on the structure of the population to classify, we chose to test several algorithms. However, not all clustering algorithms can use a distance matrix as input. We therefore had to perform a dimension reduction  [34]  to project the distance matrix in a coordinate space and use the projection as input for all clustering algorithms, as indicated in step (6) of the Fig. 1  pipeline. Numerous methods exist  [35]  and we considered the following: • Classical Multi-Dimensional Scaling, also known as Principal Coordinates Analysis, as implemented in the scikit-bio Python library  [36] , which solves a symmetric eigenvalue problem (PCoA), • Metric Multi-Dimensional Scaling as implemented in the scikit-learn Python library  [25] , which uses an iterative minimisation procedure (mMDS), • Laplacian eigenmaps, as implemented in the scikit-learn Python library  [25].
Classical analysis: MCA We started by exploring a classical analysis approach before applying the pipeline presented here. Given the types of the variables in the dataset, a Multiple Correspondence Analysis (MCA)  [45]  was seemingly appropriate after removing the only quantitative variable "number of flowers by volume". Indeed this variable has around 99% missing values. We applied MCA with the prince Python package  [46] , after removing all missing values (1029 individuals removed out of 1434). We present in Table  3  the percentage of inertia explained by the first six components. With the first two components, the total inertia was 7. 5% and reached 19. 2% with six components. Therefore the projection in the MCA space was very poor. Another point that one needs to be careful about in MCA is the presence of rare categories (categories of small size).
Distance matrices Our dataset was then subjected to the pipeline from Fig. 1. We first of all produced distance matrices between individuals using both distances: Gower's and our semantic one. These matrices are displayed as heatmaps in Figs. 6  and 7. The two matrices are quite different. Gower's matrix has a larger proportion of values closer to the maximum compared to the semantic matrix. This can be explained by the way the two distances are constructed. The distance between individuals is based on a majority of qualitative variables and a single quantitative variable. The quantitative variable is associated with a very limited amount of data. In Gower's case, we mainly represent the proportion of variables whose values are different between individuals. Indeed qualitative variables are somewhat "interchangeable" given the distance between two individuals is binary.
Axis 4 mainly contrasts cluster 5 with clusters 0 and 2. One can compare our representations (top right corner of the scatterplot matrix) with the whole dataset scatter plot (bottom left corner of the matrix). Our representations Fig. 15  Heatmap of the concordance table between KMeans clusters for 6 clusters built with Gower's distance (rows) and the semantic distance (columns) Fig. 16  and archetype visualisations, single archetype, for the semantic distance, KMeans algorithm, 6 clusters. The NA_Variables legend illustrates some of the dot sizes representing the uncertainty: the larger the dot, the more variables are missing (top right corner of the scatterplot) provide a direct overview of the dispersion of the population and its structuring as groups compared to a classical representation (bottom left corner of the scatterplot).
The plane corresponding to components 0 and 1 are on the left and components 1 and 2 on the right variable is therefore not very discriminating, probably because the distance between the beginning and the end of the century is not greater than the distance between two consecutive decennial periods. In all cases, the structuring with the semantic distance increases the distance between groups of modalities and reduces the intra-group spread. We also used the catdes function from the FactoMineR R package  [47]  to evaluate the variable modalities that are significantly under or over-represented in each group according a Chi 2 test. With the semantic distance, cluster 0 is composed of polyploid varieties which were not or very little selected: "before 1700" and/or wild varieties. Clusters 1 and 2 regroup the first recurring flowering hybrids, but cluster 2 specifically regroups tetraploid rosebushes with pink or red flowers.
The influence of the size of the clusters on the structuring when using Gower's or semantic distances could be further investigated in future work. We do not have a ground truth to compare our results with since no clustering based on this dataset has been previously performed. Therefore we cannot conclude whether one distance leads to a better clustering than the other. We can only evaluate their consistency from a biological view point. Both cases lead to relevant clusters. However each distance allows to explore the dataset from a different point of view.
Discussion The introduction of our new semantic distance and of the archetype concept allowed us to build a comprehensive representation of an incomplete dataset characterised by a large proportion of qualitative data. Even if the current prototype is closely linked to the example dataset, it can be used as a proof of concept for a more general methodology. This can be useful from several perspectives. Incomplete datasets including mixed (quantitative and qualitative) data are becoming more and more common in life sciences. The approach we developed allowed us to fully explore the available dataset, even though we simply ignored missing data. Indeed, as long as a pairwise distance can be calculated for all pairs of individuals, they can all be taken into account in the subsequent dimension reduction, clustering, archetype definition, and visualisation.
Regarding distances, we introduced a semantic distance as an alternative to distances tailored to mixed data such as Gower's. This semantic distance (Eq. (  3 )) accounts for the underlying meaning of qualitative variables. It can be attached to distances calculated in ontologies, real-life measures such as geographical distances or associated with specific calculations such as the distances between time periods we defined in Eq. (2). It can also be based on expert knowledge regarding both the structuring of the modalities of the variable (i. e. , the concepts graph of an ontology) and the distance values between two concepts. This semantic distance brings more precision regarding how two individuals relate to each other compared to Gower's, which is more binary. The semantic distance therefore leads to a wider range of possible distance values in the dataset and as a consequence a more realistic spread of the data points in the new coordinate space.
We did just that in OWL format: we defined a has_distance relationship. This relationship links two concepts and stores the distance value within a distance property attached to it. Giving a numeric value to the distance between two concepts is difficult for domain experts, but such an approach also present advantages. These distances are data and they can be easily changed, which once again brings flexibility to the way we process the datasets. Moreover, our semantic distance is defined as a weighted sum. Therefore it gives more or less importance to some variables compared to others, thus granting the ability to fine tune the way each facet of the dataset is managed. Finally, we tackled the problem of cluttered scatter plots by reducing the number of displayed individuals. From the application point of view, we illustrated our approach with passport and phenotypic data of rose varieties.
But it could be used for any dataset describing a large set of organisms, for instance in ecology, and including other types of data, for instance genomic. More widely, it could be used for any incomplete dataset mixing qualitative and quantitative variables. A problem that presents similar premises (reduce the number of individuals representing a population) is the constitution of core-collections by BRCs. Indeed BRCs store large collections of biological material and associated information, and they often need to constitute sub-samples of a more manageable size e. g. for experimental purposes. These core-collections include, with a minimum of repeatability, the maximum diversity of the species in question  [48]  and are designed by exploiting the maximum amount of data available: the origin of the samples, genetic and phenotypic characteristics, etc.
The existing strategies for the selection of inputs are diverse: random sampling, partitioning (also called "stratification"), maximisation, and some other so-called "hybrid" strategies  [49]. The methodology presented here could add a new tool to the arsenal of BRCs. However, even if it is functional, the methodology presents some limitations. First of all the method relies on ontologies. Reference characterisation of individuals in the plant sciences domain is becoming more common. Examples include the MIAPPE (Minimum Information About a Plant Phenotyping Experiment)  [50]  minimum requirements or the ontologies of the Planteome (https://planteome. org) databank  [51] , in particular the Plant Trait Ontology. Sharing more reference ontologies would spread the knowledge engineering effort further.
Methods are described in the literature, for instance for fuzzy MDS  [52]  or fuzzy clustering  [53]. We, however, have to study the topic more thoroughly and find implementations of the described approaches or develop our own. Thirdly the approaches we used to build the archetypes representing the clusters may not be the most relevant. We might want to better associate the construction of these individuals with the values of the variables in the original dataset. A better archetype might indeed be an "artificial" one whose variable values are the most represented in its cluster. Defining an archetype this way however introduces new problems. It would have to be projected in the new coordinate space created by the dimension reduction so that it could be represented in the visualisations. It isn't a trivial task given we can only calculate distances between individuals.
Conclusion In this paper, we presented a new method to analyse heterogeneous datasets. This approach relies on a new semantic distance which is designed for both quantitative and qualitative variables. This distance can be considered as an alternative to distances designed for mixed data such as Gower's. We associated this new distance definition with an archetype concept to overcome the cluttered displays issue. Indeed we defined archetypes as individuals representing groups of similar individuals from the dataset. Limiting the visualisations to these archetypes led to a sparser representation which still provided valuable insight into the data. For easier handling by biologists, we have incorporated their use into a complete tool, from raw data file to visualisation, implemented in Python 3.
7. Following the distance calculation, the next steps performed by the tool consisted in (i) grouping similar individuals, (ii) representing each group by emblematic individuals we call archetypes and (iii) building sparse visualisations based on these archetypes. The semantic distance allows for a more realistic representation of the relationships between individuals and a wider spread of the data points. It can be linked to real-life knowledge regarding the modalities of the underlying variable or to distance measures captured in ontologies. In this respect, we defined how to describe the distance value between two concepts in OWL format. The methodology described here was applied to a dataset describing rosebush passport and phenotypic traits but it could have wider uses. Indeed we could apply the same approach to any mixed dataset.
Moreover, the selection of a representative subset of a population is a widespread problem. It is a concern for Biological Resources Centres willing to build core collections for the species they are conserving. Our technique could provide a complementary methodology to existing ones. The method is fully functional. However, some aspects imply future work. The method relies on ontologies which may have to be constructed. However, in the near future, the expansion of FAIR data principles should lead to sharing more reference ontologies and reduce the knowledge engineering work. Taking into account missing data and some kind of confidence in the pairwise distance between individuals based on the number of variables used to calculate this distance has to be studied further. Future work also implies implementing an interactive visualisation to improve the data mining by biologists.
Acknowledgements We thank all rose gardens owners for providing access to their precious resources: Loubert rose garden (Rosiers-sur-Loire, France),  Val-de-Marne rose garden (Haÿ-les-roses, France) ,  La Cour de Commer rose garden (Commer, France) ,  Jumaju rose garden (Montchamp, France) ,  Jardin botanique de la Tête d'Or (Lyon, France) ,  Désert rose garden (Bouzon Gellenave, France) ,  Grande roseraie de Lyon (Lyon, France) ,  La Beaujoire rose garden (Nantes, France) ,  SCRADH (Hyères, France) ,  Arboretum des Barres (Nogent-sur-Vernisson, France) , the members of the  FloRHiGe  project and the  Pome Fruits and Roses BRC (Angers -Beaucouzé, France). We also thank  Jordan Marie-Magdelaine  for the preparation of the initial dataset. We finally thank  Antonin Bosc ,  Erwan Charles ,  Titouan Grenier ,  Robin Luquet  and  Evan Motte  for their help with the cluster analysis.
Availability of data and materials The software developed to implement the pipeline presented in this paper is available as follows: • Project name: DIVIS • Project home page: https://forgemia. inra. fr/irhs-bioinfo/Divis • Archived version: v1. 2 • Operating system(s): Platform independent • Programming language: Python 3. 7 • Other requirements: Described as requirement. txt file for pip in the code repository • License: CeCILL. See LICENCE file in the code repository • Any restrictions to use by non-academics: None The OWL ontology (in French) is bundled with the code. The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
Snow and Cloud Discrimination in Optical Remote Sensing Concerning the question of snow cover changes at a regional scale adapted to mountainous watersheds, long-term databases offer regular snow map collection (albedo, Snow Cover Area (SCA)); i. e. , the National Snow and Ice Data Center (NSIDC)  [20]  or the Copernicus Snow and Ice Monitoring Service  [21]. For this purpose, different methods have been developed to compute changes in terms of the snow cover duration (SCD in days) and snow cover area (in km 2 ). A complete review is given by  [22]. The most efficient method to separate clouds and snow is given by the NDSI (Normalized-Difference Snow Index)  [23]  based on the normalized ratio between Green and SWIR (Short-Wave Infrared) spectral bands: NDSI = R green -R swir R green + R swir (1) where R green is the surface reflectance in the green spectral band and R swir is the surface reflectance in the SWIR (1.
6 µm) spectral band. Indeed, snow has a very low reflectance in the SWIR band  [24]. Mainly based on the NDSI method, MODIS datasets at regional scale have been widely used  [25] [26] [27] [28] [29] , as well as Sentinel-2 and -3 products  [30, 31]  to compute SCD and SCA. Unlike Sentinel-2 images, the lack of 1. 6 µm SWIR spectral band on the VENµS sensor prevents direct snow and ice mapping through the NDSI calculation  [28] ; thus, alternative methods are needed. Only a few studies have already addressed this issue in other sensors. A method has been developed on Worldview-2 sensor by Bühler et al. [32]  using the visible and the highest NIR (Near Infra-Red) band in a normalized-difference band ratios index. It should be noted that no atmospheric or radiometric correction has been applied to these satellite images.
In Marchane et al. [33] , they apply a supervised classification using the parallelepiped approach with the green, red and NIR bands of the FORMOSAT-2 sensor to retrieve the snow cover. Recently, to overcome this constraint with VENµS, Baba & Gascoin  [34]  suggested an approach using VENµS database (named the NDSIµ index), based on the Red-edge and NIR spectral bands with classification method for SCA mapping in mountainous environments. However, the 15 FORMOSAT-2 images used in Marchane et al. had almost no clouds, which avoids the difficulty of differentiating snow and cloud. Baba & Gascoin  [34]  used the VENµS cloud mask to exclude clouds in the process. Therefore, the discrimination between snow and clouds remains unaddressed. Cloud cover is a major constraint in optical remote sensing, impacting the retrieval of surface properties.
The illumination angle γ (magenta) is the relative angle between the normal to the surface (slope) and the solar ray. All these angles are linked by Equations (  3 ) and (4) using Euler's rotation theorem. Our objective here is to test whether a simple correction of direct illumination (cosine correction), if based on a higher-resolution DEM (8-m HMA DEM in this case), would be more accurate than the Gamma reflectance correction applied by CNES. In order to apply the cosine correction to the VENµS SRE product, we developed a Python script whose major steps are described below and in Figure  3. As illustrated in Figure  2 , the use of a DEM makes it possible to determine the respective illumination cos(γ) of each specific slope applying the following formula: cos(γ) = cos(θ s ) cos(θ n ) + sin(θ s ) sin(θ n ) cos(ϕ s -ϕ n ), (3) where θ s and ϕ s are the sun zenithal and azimuthal angles; θ n and ϕ n are the slope value and orientation, respectively.
We aim to leverage and refine the CNES cloud masks to retrieve mor e information about the snow cover on the ground and create cloud and SCA maps databases with a 2-day frequency, at a 5 m spatial resolution. We therefore propose a complementary process based on a hybrid textural/radiometric approach in order to improve and optimize the CLM_XS product. The Python algorithm to retrieve cloud masks for VENµS is part of the VENuS_cosine_correction algorithm. Mainly applied during the 1970s for geographical application, textural features extraction from optical data are based on mathematical morphology applied to grey-tone images. The Haralick textural decomposition is applied on co-occurrence matrices  [62, 63]  and offers attractive results using entropy and energy as parameters. Energy indicates the angular second-moment feature (ASM), which is a measurement of local grey-level uniformity.
Snow-covered lakes are included, and if not snow-covered, are classified as 0 (other land use). To compare those four products, we select a subset (Figure  1 ) in the NE part of the VENµS tile, representative of fully snow-covered areas without cloud coverage (Figure  A2 ). Figure  6  presents the frequency of reflectance occurrences for SRE, FRE and the two TopCos products, HMA and SRTM. Note that for Figure  6 , normalization of all data was performed using Equation (  20 ): z i = x i -min(x) max(x) -min(x) , ( 20 ) where z i is the ith normalized data and x = (x 1 ,. , x n ) is the reflectance for each pixel. The plots in Figure  6  are representative of the general pattern observed in the complete database. On the one hand, for SRE and FRE products, over the entire range of values, the frequency of occurrence of each reflectance value is homogeneous.
• Temporal evolution Here we compare satellite data registered in a spectral band of near-infrared (B11, 865 µm), with a narrow bandwidth of 40 µm, with broadband albedo sensors on the ground (0. 305 < λ < 2. 8 µm) for incident and reflected short-wave radiation. Although not a likefor-like comparison, we pursue this comparison to explore the possibility of using the remotely sensed data to infer broadband albedo relevant for the surface energy balance. Figure  8  illustrates the results for the three ground measurement sites. It is important to note that during the monsoon (JJAS), Pyramid site is not covered with snow due to its lower elevation (5 035 m a. s. l. ), leading to low reflectance values. Apart from the monsoon, the four reflectance products follow the ground albedo pattern with each instance of snowfall characterized by a sharp albedo increase, followed by a decay.
Comparison of Snow Cover with Meteorological Data For unavailable VENµS data due to high cloud cover, we selected the MAJA chain cloud percentage and filled the cloud cover percentage database retrieved from available images. We compared ground data versus estimations of the extent of snow and cloud covers in Pheriche basin from VENµS images, in order to verify their consistency (Figure  13 ). We consider here the complete VENµS time period database (November 2018 to June 2020). Figure  13  (top) shows the precipitation at Pyramid site. Outside the rainy season, high cloud cover events are observed intermittently. During the monsoon, periods of high cloud cover persist for several days, whereas outside the rainy season they are more short-lived (Figure  13 , bottom). Looking at Pyramid site's precipitation, two phenomena are apparent: (i) the high frequency of pre-monsoon precipitation coincides with the snow height in this period (and so with the cloud cover percentage increase), and (ii) the even greater precipitation frequency during the monsoon.