dataesr/scientific-paragraphs-categorization huggingface dataset:
A Database for Inductive Analysis of Indian Space 6 To deal with the complexity of the Indian space, a conceptual model has been developed  (Perez, 2015)  to guide the feature selection. 55 spatial indicators were selected to cover six main domains of analysis: economic activity, urban structure, socio-demographic development, consumption levels, infrastructure endowment and basic geographical positioning within the Indian space. All indicators are calculated at the scale of every district of the Indian Union (640 spatial units in 2011) and on a ten-year timeframe whenever possible  (2001) (2002) (2003) (2004) (2005) (2006) (2007) (2008) (2009) (2010) (2011). Once calculated, the indicators make up a geographic database covering all the Republic of India 1 and made of 35200 values. Districts are practical observing windows for India's diversity: some are almost completely rural (with practically no urban areas within them), others host several small and mid-sized cities and most of them contain one or two main urban areas organizing a regional urban system.
The inclusion of a few basic assumptions on the role of geographic space in the organization of India's diversity is a way to introduce spatial relations in an otherwise a-spatial data-driven approaches. Inductive reasoning is thus somehow coupled with basic theoretically-driven hypotheses, which will in the end be confirmed or infirmed by inductive data-clustering. 5. 8% of the 35200 values of the database were missing for different reasons. These missing values have been inferred through a Bayesian statistical procedure 2 (4. 8% of database). The remaining missing values (1%) are more a question of non-applicability of indicators and could not be removed (impossibility of calculating ratios related to Scheduled Castes population in districts with no Scheduled Castes). We also stress that the clustering of Indian districts aimed at is not a dual clustering problem  (Lin et al.
10 Multidimensional datasets pose the problem of potentially redundant variables. Principal component analysis (PCA) is thus commonly used before performing hierarchical clustering. PCA is aimed at reducing the effect of highly correlated variables and can be A major problem remains in the inability of PCA to deal with missing data and as discussed previously, India's dataset contains 1% of those. Most of the time, missing values are imputed (which leads to whole rows ignored in the analysis) or replaced by variable's means or median in such standard procedures. Clustering using Bayesian Networks  Pearl (1985)  coined the term "Bayesian Networks" to describe a method performing probabilistic Bayesian inference (deriving logical conclusions from known statements and assigning a probability degree to them) between the nodes (representing variables) of a directed acyclic graph (DAG).
It therefore forces the algorithm to explore all the possibilities within its neighborhood beyond its local optimum. Bouckaert (1995)  adapted this strategy to learn the structure of a Bayesian Network. 16 Unsupervised structure learning can become a preliminary phase of Bayesian clustering of records in a database. A frequent problem of naive Bayesian classifiers is indeed variable redundancy. When the dimensionality of the problem is particularly high, feature selection alone cannot ensure the absence of groups of redundant variables which would heavily influence the clustering results. The Bayesian network produced through structure learning can then be used to cluster variables (for example through a hierarchical clustering algorithm, HCA) according to the mutual information distance  (MacKay 2003)  which is encoded in the probabilistic relationships of the model  (Heller and Ghahramani, 2005, as  applied for example in  Fusco, 2016).
A i,* current input vector, weight vector of the current input vector for each map node d Euclidean distance from current vector to map node weights. 20 Map nodes are characterized by a weight vector for the different variables of the analysis. This weight vector evolves during the self-organization process, as input vectors (statistical units) are assigned to the node. Nevertheless, map node weights must be initialized. They can, for example, be set to small standardized values using random initialization. Database records are presented to the SOM in random order. The map node whose weight vector is the closest to a given input vector becomes the best matching unit (BMU) for this record. When the BMU is found, the associated map node gets its weights updated and the input vector under analysis will then be associated with this node.
Assigning an input vector to a map node amounts to assigning a record to a cluster. At the difference of K-means, the topological properties of SOMs result in clusters which are organized in terms of reciprocal proximity among them. The specificity of the Self-Organizing Map is that when the BMU is found, a radius parameter will allow the update of the neighboring nodes within this radius. This is particularly useful in order to compare geographic-space proximity and variable-space proximity, as it is often the case in spatial analysis. However, just like K-means, the number of clusters must be specified a priori. 21 SOM clustering is also confronted to the problem of redundant variables in highdimensional datasets. Wehrens and Buydens (2007)  introduced an algorithm that allows using separate layers for different kind of input data: the Super-Organized Map (superSOM).
Each layer used in the superSOM algorithm can be seen as a subset of the dataset to be trained. The aim of these subsets is to gather a predefined number of input vectors together in order to reduce the redundant information. In SOM, the input vectors directly feed the map while in superSOM, the elements of the input vectors are first divided between a predetermined numbers of layers before feeding the map. The resulting architectures of SOM and superSOM for clustering purposes are represented in Figure  2. Just like in Bayesian Networks, SuperSOM seems at first glance perfectly suited for processing latent factors, summarizing groups of strongly related variables. Moreover, SuperSOM can process missing and non-applicable values by removing the records before training the Map. They will be mapped later since they are retained in the data  (Wehrens and Buydens, 2007).
The Bayesian Protocol Data Preprocessing 24 Due to intensive computation, continuous variables are difficult to manage in Bayesian Networks (probability distributions in BN are multinomial). Variables are thus usually discretized into numerical intervals. The discretization process necessarily produces an information loss, which is the price to pay for efficient Bayesian inference. Since this project's database is mostly made of continuous variables, the 55 indicators have been discretized into 4 classes using automated k-means algorithms. 5 classes per variable would have led to an overly complex conditional probability table ("Clustering using Bayesian Networks" section) while 3 classes amount to too much information loss. Yet, according to the variable distribution functions, 4 classes were in a few cases not the optimal partition.
Unsupervised Learning of a Bayesian Network 25 The second step is the unsupervised search of probabilistic links among the 55 discretized indicators. A Tabu Order heuristic algorithm has been executed in order to obtain the initial Bayesian network 5. Tabu Order is a specific version of the famous Tabu Search heuristic algorithm, influenced only by the order thorough which variables feed the network. The most robust arcs of this network have been cross-validated by an arc confidence analysis. The dataset has been randomly divided into 10 subsets (k-fold Jackknife resampling). Given the dataset D and k the number of subsets of D, the Tabu Order algorithm has been executed 10 times for each of the D -D k sets. The edges found in both 100% of the ten attempts and in the initial network have been extracted and permanently fixed over the network.
Variable Clustering The obtained network is then analyzed by a hierarchical clustering algorithm (HCA). The aim is to detect groups of closely linked variables using the obtained network. However, HCA is a method highly sensitive to random parameters such as the order from which the inputs will feed the algorithm ("Clustering using Bayesian Networks" section). In order to assess the variable clustering, once again the dataset is randomly divided into 10 subsets (k-fold Jackknife resampling) and a HCA is executed each time after removing one of the 10 subsets. Applying ten HCA through a k-fold cross-validation process allows studying the most robust associative patterns among the input variables for each HCA performed. The overall score of the cross-validation for this initial HCA reached 81.
Extraction of Latent Factors Each cluster of variables is subsequently summarized by a latent factor, corresponding to a more general concept. A naive classifier is built between the individual variables and the non-observable, newly created target node. 17 latent factors plus a single variable, Urban Density (which represents a factor by itself) have been identified (Appendices, Table  A. 3). 12 of the 18 clusters of variables are particularly robust since clustered together in each of the ten k-folds, as well as on the whole dataset. Latent factors are effective summaries of the information content of the original variables, with an average contingency table fit score of around 80%, as compared to a retained variance of 53% for the standard PCA discussed in the previous section 6.
R script Presentation and Data Preprocessing In order to automate SOM/superSOM clustering, an R script has been developed using existing packages (kohonen, class, MASS, dendextend) 8. SOMs use mean and variance of continuous variable values, which presupposes normal, almost-normal or at least symmetric distributions of values. Thus, a symmetrization and normalization of values is also necessary, in order to fully exploit the continuous data of heterogeneous variables. The script loads the necessary packages, imports and preprocesses data, implements automated functions, performs SOM and SuperSOM clustering on both variables and inputs and finally performs a one factor ANOVA to remove non-significant variables. The development of automated functions concerns: the generation of a set of prime number seeds for initialization, the calculation of the Fowlkes-Mallows index associated with each prime seed and the selection of the best initialization.
Variable Clustering Variable grouping in layers within SuperSOM is usually chosen qualitatively prior the treatment 9. In this research, the dataset is first transposed in order to use the variables as inputs within the SOM function. Since SOM cannot handle missing values, the 1% of input records still presenting missing values are temporarily removed. The output results are a clustering of the original variables thus providing the equivalent of the Bayesian latent factors. The number of clusters has to be decided beforehand in the SOM function and corresponds to the size of the grid. This will be a crucial point in record clustering (see below) but is of secondary importance in variable clustering, whose goal is just to reorganize variables in order to reduce redundancies in the original database.
We thus set the grid size to be as close as possible to the number of clusters obtained by the Bayesian analysis. The SOM grid was set to a 4x4 hexagonal grid in order to cluster the 55 variables into 16 latent factors (Bayesian: 17 latent factors + 1 variable). The original nontransposed dataset is then automatically grouped into a subset of layers according to the clustered variables and the missing values are re-entered since SuperSOM functions are able to handle missing values.
The SuperSOM algorithm can be performed directly on input layers of data. Since this was not possible for hierarchical clustering, this method has been performed on the results of a PCA. Both methods identify clusters which are as homogenous as possible on all the factors of the analysis. Yet, topological properties of neighboring clusters and iterative learning in SOM/SuperSOM enable the constitution of clusters that are statistically far from each other. HCPC, by successive aggregation of most similar clusters leads to clusters closer to the average of the whole dataset. This is well reflected within Figure  7 , especially for the traditional and intensive farming model families. Once projected (Figure  6 ), these clusters are more scattered over the Indian geographic space than their SOM/SuperSOM counterpart.
Yes Yes 52 Since SOM/superSOM and HCPC algorithms are always looking for homogeneous features within the whole dataset, the choice of indicators to be used as inputs (feature selection) must be wisely and thoughtfully considered. This is especially true in SOM/SuperSOM since each variable or each dimension (layer) possesses the same weight, while working on a PCA matrix allows avoiding this issue when using hierarchical clustering. Therefore, removing non-significant variables through an ANOVA substantially improves the results of a SOM/SuperSOM application. This step is also dispensable in a Bayesian application. Indeed, for each cluster, only relevant information is taken into account by the Bayesian algorithms. A non-significant factor for the overall model can by very significant for a very specific profile only (e.
4. The R-script and the data set are available at https://zenodo. org/record/2560527 5. The Bayesian algorithms here discussed are implemented using the software BayesiaLab (Bayesia 2010). 6. We will nevertheless highlight the different nature of the two quality parameters and the fact that variables in the Bayesian protocol have been discretized, so that latent factors now account for around 80% of a somewhat impoverished information content.
7. Without this penalizing term the most accurate number of clusters is equal to the number of inputs. That is why the contingency table fit score is constantly increasing according to the number of states in Table  1. Several additional constraints can be given to the random walk algorithm exploring the solution space (e. g. the maximum number of clusters, the minimum content of records per cluster, the maximum number of steps in the random walk, pseudorandom seed, etc. ). 8. The R-script and the data set are available at https://zenodo. org/record/2563213. The protocol coupling SOM and SuperSOM has been fully presented in  Fusco and Perez, 2015. 9. An interesting choice here would be to set the SuperSOM layers according to the latent factors obtained during the Bayesian procedure. However, this solution has not been selected to preserve the independence of the two protocols.
This paper compares Hierarchical Clustering, Bayesian Networks and Self-Organizing Map Neural Networks (SOM and superSOM) approaches used for clustering purposes in geographic space. The same dataset, covering the Republic of India and made of 55 indicators for 640 spatial units (administrative districts), is used in the three analyses. Indicators descry the several aspects of urban, economic and socio-demographic development in India. Bayesian Networks use a likelihood function while SOM/SuperSOM and Hierarchical Clustering minimize variance of Euclidean distance in variable space, the former by preserving the topological properties within the output space and the latter by successively combining similar items. Relatively similar multistep protocols have been implemented for the three techniques, to take into account variable redundancy.
Bayesian Network Clustering and Self-Organizing Maps under the Test of Indian. considered as a dimension reduction step. PCA allows reducing the dimensional space of the data through an orthogonal transformation retaining a maximized variance per dimension (principal component) in decreasing order. The output can be considered as a new dataset made of linear combinations of the original variables. A Hierarchical Clustering can be applied to this new dataset, resulting in a Hierarchical Clustering on Principal Components (HCPC). Cybergeo : European Journal of Geography , Systèmes, Modélisation, Géostatistiques.
5706% 3 Share of Urban Area surface Extended Urban Areas Footprint; Urban Areas 2011 Author's work/e-Geopolis within EUA; Size Main EUA Size Main Extended Urban Area Layer 6 9. 0018% 3 Km² 2011 Phone; Phone Evolution; Bank Account Author's Geopolis work/e- Urban Compactness Layer 3 8. 9159% 3 Ratio of surfaces UA/EUA 2011 Literacy; Secondary and Tertiary workers; Female Author's work/e-Geopolis w/i Secondary and Tertiary workers Administrative Density Inhabitants / Km² 2011 Census of India 4: Cluster correspondences Results between Methods NOTES 1. The dataset, described in Appendices Table A. 1, is available at https://zenodo. org/ record/2563213.
ACKNOWLEDGEMENTS The authors thank  Marian Schmidt  (study development, data management),  Johannes Stadler  (statistical analysis),  Dr Maria Niki Aigyptiadou  and  Bianca Arendt  (medical writing) and all  Kantar Health GmbH, Munich, Germany , as well as  Lorraine Zakin ,  Vifor Pharma Group, Glattbrugg, Switzerland  (input to the data analysis and manuscript), for their services.
Today, crop residues are no longer considered as waste: their contributions to the agronomic and environmental balance of cropping systems are recognized. In this context, knowing their quantity on the soil surface after harvest, becomes an important stake of the management of organic matter in the crop rotation. Generally, the quantitative estimation is carried out by taking samples, in order to determine the dry mass (DM) in the laboratory. The aim of this work is to propose a rapid method of quantification of soil residues, the assumption of which is based on a method for estimating standing grass biomass. This work has been carried out on farms practising conservation agriculture in a crop farming region. The residues were ground during the harvest or a subsequent specific operation. Biomass data were collected using 0.
Relation (épaisseur de résidusquantité de matière sèche totale de résidus) La base de données issue du travail  de Thiébeau et Recous (2016)  a été le support de cette investigation. L'unité de prélèvement pour l'estimation de la matière sèche totale (MST) des résidus est constituée d'un cadre de dimensions intérieures de 0,5 Â 0,5 m. Les prélèvements comprennent plusieurs cadres unitaires, constituant chacun des placettes de prélèvement dont le nombre dépend de la largeur de coupe de la récolteuse. Il en est prélevé par mètre linéaire de largeur de coupe. Dans l'exemple montré à la figure 1A, cinq placettes de prélèvement ont été réalisées car la largeur de récolte était de 4,80 m. En pratique, la mesure de l'épaisseur du transect de largeur de récolte complet est réalisée en premier lieu ; puis, derrière cette ligne de mesure, la récolte des placettes de résidus.
Traitements statistiques des données Toutes les moyennes présentées sont accompagnées de leur erreur d'estimation standardisée (SE) afin de pouvoir être comparées entre elles. Les ajustements linéaires et non linéaires ont été réalisés à l'aide du programme statistique de Sigma-Plot 12 (Systat Software, 2010) après validation de la normalité des variables par le test Shapiro-Wilk. Les critères d'évaluation des ajustements sont l'erreur quadratique au modèle (RMSE) et le coefficient de détermination (R 2 ). Ils sont définis comme suit : RMSE ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 n ⋅ X n i¼1 ðP i À O i Þ 2 r : O et P sont respectivement les valeurs observées et prédites, n est le nombre de paires observées-simulées.
Those are the carbon emissions that would result from the operation of existing fossil-fueled power plants and other carbon-intensive equipment during their typical lifetime. The same concept has been applied to planned power plants that are announced, authorized, being procured, or under construction  (Edenhofer et al 2018 , Pfeiffer et al 2018 , Shearer et al 2017). Here, we assess committed emissions from operational and planned power plants in LAC. We use the Power Plan Tracker (PPT) database from  Enerdata (2019b) , which provides information on power plants classified by fuel type, age, capacity, historical output, and operational status. We compare committed emissions with carbon budgets for the LAC power sector from IPCC (2018). In this paper, we call carbon budget the cumulative emissions from the power generation sector extracted from global pathways that keep global warming in the 1.
, 2018a). we find to be 1. 1 to 13. 5 GtCO 2. However, it is above 80% of 1. 5°C-compliant carbon budgets reported in the IPCC database, and above 50% of 2°C carbon budgets. These findings suggest that to meet the average allowable carbon budget for 2°C (6. 2 GtCO 2 ) or 1. 5°C (5. 8 GtCO 2 ), utilities in the region would need to close prematurely 10% to 16% of the existing fossil-fueled capacity, respectively, or reduce the utilization rate of existing plants to the same effect. If all planned power plants are built, we find that committed emissions would rise to 13. 6 GtCO2, which is more than 90% of LAC power carbon budgets reported in any 1. 5°C or 2°C scenario Closing plants early to meet climate targets would result in losses of revenues for the owners of fossil fuel power plants, that is in stranded assets, and potentially in sudden losses of jobs for the workers and communities who depend on those assets, which could make the political economy of climate policies more difficult to manage  (Bertram et al 2015 , Gambhir et al 2018 , Hallegatte et al 2013 , ILO 2018 , Jenkins 2014 , Nemet et al 2017 , Rozenberg et al 2018 , Vogt-Schilb and Hallegatte 2017).
Carbon emissions per generator from Enerdata and IEA We compute committed emissions in two basic steps. In the first step, we assess current emissions by generator. We decompose CO 2 emissions  (tCO2/year), as the product of capacity  (GW), utilization rate   � where E is electricity output (GWh/year), and carbon intensity of electricity generated   � (tCO2/GWh). Each quantity is computed per country ,fuel , and status s: 𝐹𝐹 𝑖𝑖,𝑓𝑓,𝑠𝑠 = 𝐶𝐶 𝑖𝑖,𝑓𝑓,𝑠𝑠 × � 𝐸𝐸 𝑖𝑖,𝑓𝑓 𝐶𝐶 𝑖𝑖,𝑓𝑓 � 𝑠𝑠 × 𝐹𝐹 𝑖𝑖,𝑓𝑓 𝐸𝐸 𝑖𝑖,𝑓𝑓 (Eq 1) We take existing and planned capacities  ,, from the Power Plant Tracker (ENERDATA 2019b). The PPT reports unit status, date of commissioning, fuel type, net capacity, electricity output and localization in January 2019. The database reports 14,816 generators in Latin America and the Caribbean, 34% of which (5,048) are fossil-fuel-based (oil; coal, peat and oil shale; and natural gas).
In very rare cases, production from PPT is slightly larger than production reported in  Enerdata (2019a). For those cases, we scale down linearly the electricity output in the PPT database to match the statistics. We take CO 2 emissions by country and fuel F from Enerdata (2019a). Since the last year full reported for CO 2 emissions is 2016, we compute the carbon intensity of electricity per country and fuel based on electricity output for 2016 reported in  Enerdata (ENERDATA 2019b , 2019a). We latter test the sensitivity of our results to the data sources chosen.