dataesr/scientific-paragraphs-categorization huggingface dataset:
The stakeholder breakdown can be found in Table  1. The interviews took place from 16th March to 7th April 2017. Of the 10 interviews, eight were conducted over the phone, one was face to face and the last was completed online. All interviewees were made aware of the purpose of the study, its methodology and intended use of the research. All interviewees were asked to complete a confidentiality and consent form prior to the interview, and data were analysed and reported accordingly. All interviewees received a copy of the manuscript once completed to comment on and approve. A suggested time of no more than 30 minutes was given to each stakeholder, although this varied depending on the stakeholders' availability and desire in wanting to talk about the curlew further. The study was carried out using a semi-structured interview process with an interview T A B L E 1 Distribution and codes of interviewees (used to anonymize responses), according to background.
Corporate landowners SH1 guide consisting of 20 broad and specific questions (see Appendix SA1 in Supporting Information for the interview guide). All interviews were transcribed verbatim in Microsoft Word 2010. For the analysis of the interviews, Adobe Acrobat 9. 0 Pro was used to scan both the individual interview transcripts and the dataset as a whole to identify significant themes and issues within it. Establishing common themes and opinions within the interviews created percentages of catalysts, barriers or issues of most significance to the interviewees.
Understanding and reducing the in-orbit instrumental backgrounds are essential to achieving high sensitivity in hard X-ray astronomical observations. The observational data of the Hard X-ray Imager (HXI) on board the Hitomi satellite provides useful information on the background components, owing to its multi-layer configuration with different atomic numbers: the HXI consists of a stack of four layers of Si (Z = 14) detectors and one layer of CdTe (Z = 48, 52) detector surrounded by well-type BGO (Bi 4 Ge 3 O 12 ) active shields. Based on the observational data, the backgrounds of top Si layer, the three underlying Si layers, and the CdTe layer are inferred to be dominated by different components, namely, low-energy electrons, albedo neutrons, and proton-induced radioactivation, respectively.
Monte Carlo Simulations of the HXI Background In order to verify the hypothesis of the origins of the HXI backgrounds in the previous section, we performed a full Monte Carlo simulation by taking account of the in-orbit radiation environment. We used a Monte Carlo simulation framework COMPTONSOFT,  11  identical to that used in a previous work on the radioactivation background of the CdTe-DSD. 7 3] [14]  In addition, detailed detector responses due to the charge carrier transport inside semiconductor detectors are also implemented. In this work, we used the same parameter values for the detector response as those tuned to reproduce the experimental data. 15 ost of the passive materials as well as the imager module and active shields are implemented in the simulation to accurately calculate interactions in the radiation environment.
Conclusions The observational data of the HXI indicates the origins of their instrumental backgrounds: the backgrounds of top-layer Si, middle-layer Si and bottom-layer CdTe are inferred to originate from low-energy electrons, albedo neutrons and proton-induced radioactivation, respectively. We successfully reproduced the spectral shape and flux of the observed HXI backgrounds using the full Monte Carlo simulations. Based on the results from this simulation, we conclude that the electron background is expected to be reduced after applying appropriate shielding, which can further benefit future missions.
Acknowledgments We acknowledge all the Hitomi team members, including many graduate students, for their great contributions to the  HXI  and the  Hitomi  project. We acknowledge the support of  JSPS/MEXT KAKENHI  grant numbers  24105007 ,  15H03639 ,  25287059 ,  24244014 ,  16H02170 , and the  JSPS Core-to-Core Program. All the members from the  U. S. acknowledge support received from the  NASA Science Mission Directorate. Stanford and SLAC members acknowledge support via DoE contract to  SLAC National Accelerator Laboratory   DE-AC3-76SF00515  and  NASA  grant  NNX15AM19G. French members acknowledge support from the  Centre National d'Etudes Spatiales (CNES).
Study population The Survey of Health, Ageing and Retirement in Europe (SHARE) is a multidisciplinary and cross-national panel database with data collected on health, socio-economic status, and social and family networks. SHARE comprises approximately 140,000 participants aged 50 and older from 27 European countries and Israel. Follow-up of respondents was carried out in waves (Wave 1 to 7). SHARE was described elsewhere in more detail. 12 We use information from Wave 2 to 7,  [13] [14] [15] [16] [17] [18]  as from Wave 2 forward the information regarding dementia diagnosis was collected from respondents aged 60 and older. Wave 3 was not included, as it focused on the childhood of respondents. 14   3. Future directions: We would like to encourage researchers to adapt the use of splines in dementia risk prediction research.
Strengths and limitations This study has several limitations. First, due to the design of the used datasets, interval censoring is present. This means that the exact date of diagnosis is not known and occurred at some point during the interval between the waves. This might have resulted in biased results, likely an overestimation of the predictor coefficients. 39 Second, censoring due to death, which is a competing event, was not taken into account. There are existing methods to incorporate competing risks in survival analyses  40  as well as generalizations of these models. 41 However, the information on death in SHARE are recorded by proxy questionnaire and the use of these information might have hampered the results even further. 21 Third, the quality of the data about dementia diagnosis in SHARE is not optimal, as it is only recorded by self-report and no further testing of the diagnosis is made.
ACKNOWLEDGMENTS SHARE: This paper uses data from SHARE Waves 2, 4, 5, 6, and 7 (10. 6103/SHARE. w2. 700 https://doi. org/10. 6103/SHARE. w3. 700https://doi. org/10. 6103/SHARE. w4. 700https://doi. org/10. 6103/SHARE. w5. 700https://doi. org/10. 6103/SHARE. w6. 700https: //doi. org/10. 6103/SHARE. w7. 700), see Börsch-Supan et al. 12 for methodological details. The SHARE data collection has been funded by the  European Commission  through  FP5  ( QLK6-CT-2001-00360 ),  FP6  ( SHARE-I3 :  RII-CT-2006-062193 , COMPARE:  CIT5-CT-2005-028857 , SHARE-LIFE:  CIT4-CT-2006-028812 ),  FP7  ( SHARE-PREP: GA N • 211909 ,  SHARE-LEAP: GA N • 227822 ,  SHARE M4: GA N • 261982 ) and  Horizon 2020  ( SHARE-DEV3: GA N • 676536 , SERISS:  GA N • 654221 ) and by  DG Employment, Social Affairs & Inclusion. Additional funding from the  German Ministry of Education and Research , the  Max Planck Society for the Advancement of Science , the  U.
Introduction From string theory to science fiction, the idea that we might be "glued" onto a low-dimensional surface embedded in a space of large dimensionality has tickled the speculations of scientists and writers alike. However, when it comes to multidimensional data such a situation is quite common rather than wild speculation: data often concentrate on hypersurfaces of low intrinsic dimension (ID). Estimating the ID of a dataset is a routine task in machine learning: it yields important information on the global structure of a dataset, and is a necessary preliminary step in several analysis pipelines. In most approaches for dimensionality reduction and manifold learning, the ID is assumed to be constant in the dataset. This assumption is implicit in projection-based estimators, such as Principal Component Analysis (PCA) and its variants  1  , Locally Linear Embedding 2 , and Isomap 3 ; and it also underlies geometric ID estimators  [4] [5] [6]  , which infer the ID from the distribution of between-point distances.
The hypothesis of a constant ID complies with simplicity and intuition but is not necessarily valid. In fact, many authors have considered the possibility of ID variations within a dataset  [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22]  , often proposing to classify the data according to this feature. However, the dominant opinion in the community is still that a variable ID is a peculiarity, or a technical detail, rather than a common feature to take into account before performing a data analysis. This perception is in part due to the lack of sufficiently general methods to track ID variations. Many of the methods developed in this field make assumptions that limit their applicability to specific classes of data. ] [13] [14]  use local ID estimators that implicitly or explicitly assume a uniform density.
Our approach works even if the data are embedded on highly curved and twisted manifolds, topologically complex and not isomorphic to hyperplanes, and if they are harvested from a non-uniform probability density. Moreover, it is specifically designed to use only the distance between data points, and not their coordinates. These features, as we will show, make our approach computationally efficient and more robust than other methods. Applying our approach to data of various origins, we show that ID variations between different regions are common. These variations often reveal fundamental properties of the data: for example, unfolded states in a molecular dynamics trajectory of a protein lie on a manifold of a lower dimension than the one hosting the folded states. Identifying regions of different dimensionality in a dataset can thus be a way to perform an unsupervised segmentation of the data.
Let x = {x i } be i. i. d samples from a density ρ(x) with support on the union of K manifolds with varying dimensions. This multi-manifold framework is common with many previous works investigating heterogeneous dimension in a dataset  8-11, 14-16, 18-20, 24, 25. Formally, let ρ(x) = ∑ K k=1 p k ρ k (x) where each ρ k (x) has support on a manifold of dimension d k and p. = (p 1 , p 2 ,. , p K ) are the a priori probabilities that a point belongs to the manifolds 1,. , K. We shall first assume that K is known, and later show how it can be estimated from the data. The distribution of the µ i is simply a mixture of Pareto distributions: P(µ i |d, p). = K ∑ k=1 p k d k µ -(d k +1) i. (2) Here, like in ref. 23  , we assume that the variables µ i are independent. This condition could be rigorously imposed by restricting the product in eq.
To analyze such dataset we again choose the hyperparameters q = 3 and ξ = 0. 8. We do not fix the number of manifolds K to its ground truth value K = 5, but we try to let the method estimate K without relying on a priori information. We perform the analysis with different values of K = 1,. , 6 and compute an estimate of the average of the logarithm of the posterior value L for each K. Results are shown in Fig. 1e. We see that L increases up to K = 5, and then decreases, from which we infer that the optimal number of manifolds is K = 5. In Fig. 1f  we illustrate the final assignment of points to the respective manifolds together with the estimated dimensions, upon setting the number of manifolds to K = 5. The separation of the manifolds is very good. Only a few points of the manifold with dimension 1 are incorrectly assigned to the one with dimension 2 and vice versa.
The values of normalized mutual information between the ground truth and our classification is 0. 89. Finally, we show that our method can handle data embedded in curved manifolds, or with complex topologies. Such data should not pose additional challenges to the method: Hidalgo is sensitive only to the local structure of data. In terms of the distribution of the first and second nearest-neighbor distances, a 2-d torus looks exactly like a 2-d plane, and thus we should be able to correctly classify these objects, that are so topologically different, as 2-d objects regardless of their shape. We generated a dataset with 5 Gaussians in dimension 1, 2, 4, 5, 9 (Fig. 2a ). Each Gaussian is then embedded on a curved nonlinear manifold: a 1-dimensional circle, a 2-dimensional torus, a 4-dimensional Swiss roll, a 5-dimensional sphere, and a 9-dimensional sphere.
ID variability in a protein folding trajectory As a first real application of Hidalgo, we address ID estimation for a dynamical system. Asymptotically, dynamical systems are usually restrained to a low-dimensional manifold in phase space, called an attractor. Much effort has been devoted to characterizing the ID of such attractor  4. However, in the presence of multiple metastable states an appropriate description of the visited phase space may require the use of multiple IDs. Here, we consider the dynamics of the villin headpiece (PDB entry: 2F4K). Due to its small size and fast folding kinetics, this small protein is a prototypical system for molecular dynamics simulations. Our analysis is based on the longest available simulated trajectory of the system from Ref. 33. During the simulated 125 µs, the protein performs approximately 10 transitions between the folded and the unfolded state.
We expect to find different dimensions in the folded and unfolded state, since these two states are metastable, and they would be considered as different attractors in the language of dynamical systems. Moreover, they are characterized by different chemical and physical features: the folded state is compact and dry in its core, while the unfolded state is swollen, with most of the residues interacting with a large number of water molecules. We extract the value of the 32 key observables (the backbone dihedral angles) for all the N = 31, 000 states in the trajectory and apply Hidalgo to this data of extrinsic dimension D = 32. We obtain a vector of estimated intrinsic dimensions d and an assignment of each point i to one of the K manifolds. We find four manifolds, three low-dimensional ones (d 1 = 11.
ID variability in financial data Our final example is in the realm of economics. We consider firms in the well-known Compustat database (N = 8309). For each firm, we consider D = 31 balance sheet variables from the fiscal year 2016 (for details, see Table  S2  in the SI). Applying Hidalgo we find four manifolds of dimensions d 1 = 5. 4, d 2 = 6. 3, d 3 = 7. 0 and d 4 = 9. 1. To understand this result, we try to relate our classification with common indexes showing the type and financial stability of a firm. We start by relating our classification to the Fama-French classification  37  , which assigns each firm to one of twelve categories depending on the firm's trade. In Fig. 5 (top) we separately consider firms belonging to the different Fama-French classes, and compute the fraction of firms assigned to each the four manifolds identified by Hidalgo.
Comparison with other methods We have compared our method with two state-of-the-art methods in the literature, considering the synthetic datasets as well as the protein folding dataset (for which the folded/unfolded classification yields an approximate "ground truth"). The first  Table  1. Performances of Hidalgo, restricting analysis to points with non-overlapping first and second neighbor For each data set we show the number of points in the original data set (N 0 ), the number of points upon restriction N, the estimated value of the d k , and NMI betzeen the assignment and the ground truth. Qll results were obtained with ξ = 0. 8, q = 3, repeating the sampling M = 10 times and considering the sampling with the highest average log-posterior. method we consider is the "sparse manifold clustering and embedding" (SMCE, Ref.
01 ≤ λ ≤ 20, and we report the results corresponding to the highest NMI with the ground truth. Furthermore, we repeated the estimation M = 10 times and kept the results with the highest NMI with the ground truth. The second method is the local ID (LID) estimation by Carter et al. 10  , which combines ID estimation restricted to a (large) neighborhood and local smoothing to produce an integer ID estimate for each point. We used the implementation provided by Kerstin Johnsson  38  (https://github. com/kjohnsson/intrinsicDimension) using a neighborhood size k = 50 which is a standard value. Note that the LID estimation is deterministic, so we do not need to repeat the estimation several times. For LID, we can compute the NMI between the (integer) label given by the local ID estimate and the ground truth; moreover, we can compute the average ID, d for the points with a given ground truth assignment.
The results of the comparison are presented in Table  1. For the examples with two manifolds, SMCE  19  is able to correctly retrieve the manifolds with high accuracy (NMI≥ 0. 94). The estimate of the lower dimension d 2 = 4 is always correct, while the highest dimension is underestimated for d 1 ≥ 7. Overall, SMCE performs very well (even better than Hidalgo) on the datasets with two manifolds, as the two manifolds are well separated, matching well the assumptions of the model. On the examples with five manifolds, the performance of SMCE is worse, as the method cannot correctly retrieve intersecting manifolds. In the example with five linear manifolds, SMCE cannot retrieve the manifold of dimension d 1 = 1, which is intersecting the manifolds of dimension d 3 = 4 and d 4 = 5. Moreover, the ID estimates are quite poor.
The degree of underestimation increases with increasing d 1. This is expected, since the ID estimates are produced by assuming a uniform density in a large neighborhood of each point, an assumption that fails in these data, especially for points on the manifold with higher dimension. Overall, LID performs quite well (comparably with Hidalgo) in separating regions with different IDs in the datasets with two manifolds, but gives worse ID estimates than Hidalgo. In the examples with five manifolds, the performance of LID is slightly worse. In the example with five linear manifolds LID merges the manifolds of dimension d 3 = 4 and d = 5, and misassigns some of their points to the manifold of dimension 1. This is because relatively large neighborhoods are used for the estimation of the local ID: this leads to a difficulty in discriminating close regions with similar ID ( d 3 = 4 and d 4 = 5) and to a "contamination" of results by points of the manifold with d 1 = 1 intersecting the two.
In the example with five curved manifolds, LID can correctly identify the manifolds with d 2 = 2, d 4 = 5, and d 5 = 9, even if it underestimates the ID in the latter two cases (high dimension, large density variations). Many points in the manifold of dimension d 3 = 4 are mis-assigned to the manifold of dimension d 1 = 1, due to the fact that the two manifolds are highly intersecting. Finally, for the protein folding data, where the density is highly variable, LID shows a tendency for unfolded configurations to have a higher local ID than unfolded ones. However, it cannot discriminate between folded and unfolded configurations as Hidalgo. The ID of the folded configurations is highly underestimated compared to the Hidalgo estimate, probably because of density variations. In Figure  6  we compare the results of the three methods (Hidalgo, SMCE, and LID) for the dataset with five curved manifolds.
Discussion The increasing availability of a large amount of data has considerably expanded the opportunities and challenges for unsupervised data analysis. Often data come in the form of a completely uncharted "point cloud" for which no model is at hand. The primary goal of the analyst is to uncover some structure within the data. For this purpose, a typical approach is dimensionality reduction, whereby the data are simplified by projecting them onto a low-dimensional space. The appropriate intrinsic dimension (ID) of the space onto which one should project the data is not constant everywhere. In this work, we developed an algorithm (Hidalgo) to find manifolds of different IDs in the data. Applying Hidalgo, we observed large variations of the ID in datasets of diverse origin (a molecular dynamics simulation, a set of time series from brain imaging, a dataset of firm balance sheets).
This finding suggests that a highly non-uniform ID is not an oddity, but a rather common feature. Generally speaking, ID variations can be expected whenever the system under study can be in various "macrostates" characterized by a different number of degrees of freedom due, for instance, to differences in the constraints. As an example, the folded state the protein is able to locally explore the phase space in many independent directions, while in the unfolded state it only performs wider movements in fewer directions. In the case of companies, a financially stable company may have more degrees of freedom to adjust its balance sheet. In the cases we have analyzed, regions characterized by different dimensions were found to host data points differing in important properties. Thus, variations of the ID within a dataset can be used to classify the data into different categories.
several values of q ∈ {2, 3,. , 10} and ξ ∈ [0. 5, 1) for all the artificial datasets used for validation: i) two manifolds of dimension 4 and 5, with a Gaussian density of points; ii) five manifolds of dimension 1, 2, 4, 5, 9, with a Gaussian density of points; iii) five manifolds of dimension 1, 2, 4, 9, with a Gaussian density of points, embedded on a 1-dimensional circle, a 2-dimensional torus, a 4-dimensional Swiss roll, a 5-dimensional sphere and a 9-dimensional sphere. The crucial figure of merit to assess the performance of the method is the normalized mutual information (NMI) between the estimated and the true assignment of z, which measures the quality of the assignment of points to manifolds. Indeed, once the manifolds are correctly separated, the problem is essentially reduced to a dimension estimation within the single manifolds (which is successfully solved by TWO-NN).
S13) converged to 1/2. While an excessive variance around 1/2 is a sign of residual non-stationarity, failure of E t to approach the value 1/2 for increasing t is a likely sign of the non-independence of the samples used  43. Therefore, if E it did not approach to 1/2, we increased the subsampling at θ = 20, θ = 50, θ = 100, θ = 200, θ = 500 until E it 1/2 for large t. We divided the D 1t and D 2t in non-overlapping windows of 20 successive points and checked whether E it was within the bounds (S14) with α = 0. 05 for 95% of the times (at 19 times out of 20) within each window. We then defined as convergence time T conv the time corresponding to the first window where both E 1t , E 2t were satisfying the criterion. For each dataset size N, we repeated the assessment of T conv for M = 20 independent Gibbs samplings.
Acknowledgements MA was supported by  Horizon 2020 FLAG-ERA  ( European Union ), grant  ANR-17-HBPR-0001-02  during completion of this work. We thank  Giovanni Barone Adesi  and  Julia Reynolds  ( Institute of Finance, USI, Lugano, Switzerland ) for helping us with the Compustat dataset and offering precious suggestions for its analysis. We thank  Giulia Sormani  ( SISSA, Trieste, Italy ) for suggesting, and providing us with the protein dynamics data. We thank  Alex Rodriguez  ( SISSA, Trieste, Italy ) for helping with the analysis of protein data. M. A. thanks the  SISSA community at large  for the substantial moral and intellectual support received, which has been critical for completion of this work. The authors declare that they have no competing financial interests. Correspondence and requests for materials should be addressed to A.