dataesr/scientific-paragraphs-categorization huggingface dataset:
The agencies responsible for the maintenance of archival standards have developed several important initiatives in order to gain interoperability. EAD3, developed by an international subcommittee sipported by the Society of American Archivists 2  , with the cooperation of many archivists worldwide, and maitained by the Library of Congress 3  , is a big step towards interoperability even though many archives consider the change towards EAD3 as a midterm perspective. ). This initiative will also contribute to providing a solid framework for exchanging archival data more easily. The maintenance of EAD, undertaken by the Society of American Archivists and the Library of Congress, is still a big issue. The maintenance of a standard requires in all cases discussions to achieve consensus, and any major revision should undergo a precise and complete process, although some minor corrections and adjustments are sometimes welcome in the meantime.
Between EAD 2002 and EAD3, more than ten years passed and some features introduced in 2015 had been requested by the community many years before. For example, some users asked for a typing attribute for the <ead:addressline> element, a child of <ead:address> [EAD working group AFNOR]. This small modification was introduced in EAD3, as part of the general revision process, that lasted five years. One might have hoped for a smoother evolution of the standard, based on continuous maintenance, which is the case for the TEI consortium that updates its standard regularly on GitHub. In this respect, the fact that the development of EAD3 actually took place on GitHub opens the way to this more continuous maintenance.
Archive portals and EAD: use cases The experience gained from concrete use cases has shown how strong the need is to build interoperability solutions between heterogeneous archival descriptions in EAD. (Metadata Encoding and Transmission Standard). Specific schemas were created, in particular apeEAD, a subset of EAD2002, which "was drafted on the basis of a comparison of EAD profiles and practices of the National Archives participating in the project" 4. European funded research infrastructures and projects have also tackled this issue of interoperability of archival descriptions, with an additional focus on specific research communities, taking into account the specific needs of each. In the context of two H2020 Research infrastructure projects which manipulate archival data, Cendari (Collaborative European Digital Archival Research Infrastructure) 5  on medieval and modern European history and EHRI, various solutions were proposed.
In the Cendari virtual research environment, where researchers have the possibility to select descriptions originating from various sources and create their own collections, EAD was customized with the addition of elements dedicated to the researchers' uses  [Romary et al. 2017 ]. We should note here that the necessity to be able to customize the EAD model in order to add e. g. a more precise model for describing bibliographic sources  [Medves, Romary, 2013]  has been in tension in projects such as Cendari with a possible tendency to simply get rid of EAD as a reference format for integrating researchers' information and design an ad hoc format  [Gartner 2015 ]. For the EHRI project, on which we will focus, the problem is slightly different. Researchers are not (yet) allowed to create, or pin and select, their own descriptions, but the heterogeneity of the archival descriptions, which have to be collected in a single pool and processed uniformly, make it necessary to create a straightforward workflow for the ingestion of archival data in the portal database.
EHRI coordinates the activity of 24 institutions (research bodies, archives, libraries, etc. ), but its archival portal hosts descriptions from around 1 900 institutions. An extra challenge for EHRI is that Holocaust archives are hidden, often dispersed over several institutions or several fonds. Moreover, EHRI intends to focus on Eastern Europe, where few archives are digitally advanced, due to lack of funding or technical infrastructures. These descriptions can be entered manually by EHRI staff, but the preferred method is semi-automatic ingestion of XML files in the database. The portal database is a graph database whose data model is based on the ICA standards -ISAD(G) and ISAAR-CPF, combined with extra administration fields. Due to the variety of institutions providing data, EHRI has to deal with a great heterogeneity of data formats and of EAD flavours, even if EHRI has used EAD2002 since the beginning of the project in 2011.
In 2015, a discussion arose about the opportunity to move towards EAD3, which had just been released, but the lack of visibility on its uptake in the archival community during the four years of the project made this choice too risky. The use of EAD in EHRI takes place at the two sides of the workflow: the ingestion and the export of archival descriptions. EAD2002 is the pivot format for the semi-automatic ingestion of data in the EHRI database. It is also used as an exchange format, with users being able to download any content of the portal in XML-EAD (or EAC and EAG for authorities and institutions). Therefore, there is a strong need for both valid and customized EAD (and EAC-CPF) schemes, for two kinds of tasks: The first one is the possible mapping of the data to XML-EAD if the descriptions are not provided in this format.
The second one is a validity check to be sure that the EAD conforms to EHRI requirements. format. If they are in EAD2002, we automatically check if the EAD flavour in input is compatible with the customized EHRI EAD format. If some adjustments are necessary, they can either be made by the provider himself of by the same EHRI mapping mechanism mentioned before. Then, the formatted XML document can be processed again in order to populate the EHRI database.
More importantly, this solution doesn't change the core EAD schema, but adds more specific rules in a comprehensive and human-readable format, by combining the EAD schema (expressed in Relax NG) with ISO Schematron rules. Schematron is an ISO/IEC Standard (ISO/IEC 19757-3:2016) that parses XML documents and makes "assertions about the presence or absence of patterns" 6. It can be used in conjunction with a large number of grammar languages such as DTD, relax NG, … To do so, we made a natural choice, from a researcher's point of view, integrating EAD into an environment which is familiar to us, the Text Encoding Initiative (TEI), broadly recognized as the de facto standard for the representation of text in digital form. This choice is also coherent from an historical perspective, considering that the development of EAD was greatly influenced by the TEI.
Originally, ODD relied on RelaxNG snippets encapsulated in TEI specifications, but has recenlty evolved towards a uniform language called PureODD, with a better power of expression and conciseness. However, the transition between these two methods is smooth and allow users to use both in one single specification, making it possible to move to PureODD gradually. ODD was initially created to give TEI users a straightforward way to customize the TEI schema according to their own practices and document this customization. But it is possible to describe a schema and the associated documentation of any XML format, for example the W3C standard International Tag Set (ITS) 8  [Lieske et al. 2006 ]. Moreover, although ODD is a description language, it can be processed to generate an actual schema (a DTD, a Relax NG XML or compact schema and an XML schema), and documentation in various formats (XHTML, PDF, EPUB, docx, odt).
We used ODD to completely encode the EAD standard, as well as the guidelines provided by the Library of Congress 9. 2 The EAD specification in ODD 10 For clarification purposes, the code samples presented below come with prefixes : "rng" for Relax NG elements, "tei" for TEI(ODD), "ead" for EAD and "sch" for Schematron. The EAD ODD is a XML-TEI document made up of three main parts. The first one is, like any other TEI document, the <tei:teiHeader>, that comprises the metadata of the specification document. Here we state, among other pieces of information, the sources used to create the specification document in a <tei:sourceDesc> element. Our two sources are the EAD Tag Library 11 and the Relax NG XML schema 12 , both published on the Library of Congress website. The second part of the document is a presentation of our method (the foreword) with an introduction to the EAD standard and a description of the structure of the document.
Typology of the constraints The constraints that we need to add to EAD in order to ensure a smooth ingestion of descriptions in the database are of two types. First, some EAD elements are required for the correct functioning of the database, for instance unique identifiers for all the descriptions (i. e. the value of <ead:eadid>). Second, some elements are made mandatory for more qualitative reasons: for instance, to ease the discoverability of its resources, EHRI requires that a minimal description in English is provided with each description unit. Another example is the fact that EHRI encourages the use of ISO standards for the representation of languages, scripts, dates, etc, as well as the interlinkage of entities, via the use of authority lists. Many specific validation rules were already used in EHRI, so we integrated them directly in the EHRI-EAD ODD  16.
The annexed table lists additional constraints found in the EHRI guidelines. Other constraints were found by EHRI database managers to ease the process of importing EAD documents into the database or were gathered by analysing samples from collection holding institutions (CHI). This approach will make it possible to ensure a very good quality of the EAD files, based on the very specific remarks made on relevant sample files. All the constraints were sorted in categories, which we call roles. The different roles are: • MUST: mandatory for the import process or according to the EAD (when we want to particularly highlight a requirement) • SHOULD: mandatory for the description process, i. e. in terms of archival description. The SHOULD rules are not technically mandatory, but if they are not respected in the input description, it would be considered to be incomplete, with potential comprehension issues • COULD: Non mandatory rules.
Some rules reflect the requirements of EHRI database content model. For instance, it asks that the <ead:date> elements contain a @normal attribute whose content respects the ISO8601 standard on the representation of dates and time. This constraint is expressed in the ODD file with embedded schematron rules in the following way: <elementSpec ident="date" module="EAD" mode="change"> <constraintSpec ident="dateNormal" scheme="isoschematron" type="EHRI" mode="add"> <desc>All the <gi>date</gi> elements MUST have a <att>normal</att> attribute whose pattern respects the ISO8601 standard and take the following form: YYYY-MM-DD</desc> <constraint> <sch:rule context="ead:date"> <sch:assert role="MUST" test="matches(@normal,'^(( [0-9]|[1-9][0-9]|[1-9][0-9]{2}|[1-9][0-9]{3}))-(0[1- 9]|1[012])-(0[1-9]|[12][0-9]|3[01])$')">@normal attribute MUST respect ISO8601 pattern = YYYY-MM-DD</sch:assert> </sch:rule> </constraint> </constraintSpec> </elementSpec> The second rule is also a requirement, but for different reasons.
Use of the schema in EHRI's mapping and validation workflow The schema created from the ODD file is used for the mapping and the validation processes of archival descriptions in the EHRI database. As the EHRI EAD schema is a Relax NG schema with embedded schematron rules, these combined languages are used for different parts of the process, in a two-step validation. The first step transforms the input files in EAD2002 if needed. In this case, the EAD schema used is the EHRI EAD schema, without the schematron rules. The second step is the validation with respect to the schematron rules. The schematron rules embedded in the EHRI-EAD schema are meant to be presented as a diagnosis to the content providers. This diagnosis will point to elements of the EAD that, even if they are in valid EAD, are not in line with the EHRI requirements.
We present the public release of Mr-Moose, a fitting procedure that is able to perform multi-wavelength and multi-object spectral energy distribution (SED) fitting in a Bayesian framework. This procedure is able to handle a large variety of cases, from an isolated source to blended multi-component sources from an heterogeneous dataset (i. e. a range of observation sensitivities and spectral/spatial resolutions). Furthermore, Mr-Moose handles upper-limits during the fitting process in a continuous way allowing models to be gradually less probable as upper limits are approached. The aim is to propose a simple-to-use, yet highly-versatile fitting tool fro handling increasing source complexity when combining multi-wavelength datasets with fully customisable filter/model databases. The complete control of the user is one advantage, which avoids the traditional problems related to the "black box" effect, where parameter or model tunings are impossible and can lead to overfitting and/or over-interpretation of the results.
Also, while a basic knowledge of Python and statistics is required, the code aims to be sufficiently user-friendly for non-experts. We demonstrate the procedure on three cases: two artificially-generated datasets and a previous result from the literature. In particular, the most complex case (inspired by a real source, combining Herschel, ALMA and VLA data) in the context of extragalactic SED fitting, makes Mr-Moose a particularly-attractive SED fitting tool when dealing with partially blended sources, without the need for data deconvolution.
drouart@curtin. edu. au † Contact e-mail: tfalkend@eso. org the regime of "overfitting", where degeneracies between parameters dominate, and therefore, no meaningful constraints can be extracted from the dataset (usually keeping in mind the Ockham razor principle). This optimal regime can be quantified thanks to the Bayes theorem, where likelihood of different models can be compared to find the relative likelihood probability of one model compared to another. The particular case of spectral energy distribution fitting is an old problem and can be traced as far as the 1960s (e. g. Tinsley 1968 ). This technique has seen a considerable development starting at the end of the twentieth century and is now widely used in most fields of Astrophysics: from stars in our Galaxy to very distant galaxies during the epoch of reionisation.
Mr-Moose is written in python 2. 7, developed in PyCharm Community Edition and complies to the PEP8 writing standard. It relies as much as possible on already existing packages (see full list in Appendix) to further increase robustness. The code was designed to be user friendly but not user opaque. A large (and necessary) control of the data, model and fit setting inputs is allowed to test the reliability of the output and therefore avoid the "blackbox" effect. However, the code aims to be sufficiently simple to be used with a minimal statistical and computational knowledge. We first present the core of the fitting procedure, the maximum likelihood estimation and review the inputs and outputs files. Table  1  summarises the different file extensions used in Mr-Moose. We also report a simplified flowchart of the code structure in Figure  1  and refer the user to a more detailed flowchart in the GitHub repository 2.
Therefore, P(M |D) is the final parameter distribution (that we are looking for) and P(D|M) is likelihood to be determined and to maximise (or minimise given our estimator, a slightly modified version of the goodness-of-fit). This maximum likelihood (abbreviated P onwards) corresponds to the product of all probabilities of the model given the data and can be written as: P ∝ i P i j P j , (2) with i the detections and j the upper limits. We stress that the prior on the model parameters (set to uniform, corresponding to an uninformative prior in Mr-Moose version 1). Each individual probability distribution, respectively for detection and upper limit (following a normal distribution), are defined as follows: P i ∝ exp       - 1 2 S dat a i -S model i σ i 2      ∆S i , (3) P j ∝ ∫ S d at a l i m, j -∞ exp       - 1 2 S j -S model j σ j 2       dS j , (4) where S dat a i is the measured flux and σ i , its standard deviation, S model i is the predicted flux, all in filter i for a detection and j for a non-detection.
2013) , allowing a choice of different samplers to probe the parameter space. The emcee package follows the classic Markov chain Monte Carlo (MCMC) approach, consisting of randomly moving walkers exploring the parameter space. The main goal is to sample of the posterior probability density function of the parameters with a sufficiently high number of independent realisations. This is the best compromise between a brute force approach where each solution of the parameter space is calculated for a given grid and a standard least-square minimisation where only the best solution is provided (which can be a local minima). This approach also presents the advantage to estimate uncertainties on the model parameters, even with non-normal distributions thanks to the exploration the parameter space, but without the computing cost (increasing exponentially with the number of parameters).
In particular, emcee uses affine-invariant transformations  (Foreman-Mackey et al. 2013; Goodman & Weare 2010)  to sample the parameter space, using a stretch move based on the other walker positions. This presents the advantages of being less sensitive to strongly correlated parameters allowing a minimum loss of performance and therefore a quicker convergence. Also, this algorithm enables two valuable simplifications compared to the standard M-H: (i) the performance of the code is basically relying on two parameters: a, a gain parameter and n, the number of walkers and (ii) the possible parallelisation, splitting the chains into complementary sub-ensembles to predict iteratively the next walker moves, enabling even larger performance gains for specific problems. We refer the reader to  Foreman-Mackey et al.
Data file The data-file requires the most attention in its design as it will set up the number of arrangements (the various data/model combination). The Mr-Moose procedure doesn't check for inconsistency in the data, and is the responsibility of the user to provide reliable or relevant photometric points (well calibrated and corrected for any photometric effects). To help with this step, corresponding images will be generated for each reported filter in the data file (see § 2. 3). Multi-wavelength SED fitting is particularly challenging in this regard, therefore specific care should be taken in order to have consistent data throughout the fitting range (blending, filtering of scale/flux by interferometers, instruments corrections, aperture effects, absolute calibration uncertainties) by adding extra models/arrangements if necessary.
We refer the user to the examples available in the repository. The library of models will increase in future updates depending on the need, we refer to the user manual for an up-to-date library of available models. We remind that the package astropy (Astropy Collaboration et al. 2013) provides a large library of the most commonly used function in astronomy. Also, we suggest to design the model with parameters spanning a large range in logscale (such as normalisation factors, as done in the examples in this paper), making sure the corresponding function is making the adequate transformation to calculate flux. The initial position of the walkers are set following the emcee documentation: a Gaussian ball of walkers, centred on the median of the parameter interval. This has important consequences and should be taken into account when defining this interval (see § 4).
Setting file The setting file (see full structure in Table  4 ) contains the parameters for Mr-Moose to perform the fit such as the number of walkers (nwalkers), the number of steps (nsteps), the limit (nsteps_cut) at which the chains should be used to plot the results (the limit from which convergence of the fit is reached). We provide examples with typical numbers. For more information, we invite the user to read the recommendations from the emcee documentation. In a nutshell: the more walkers and steps, the better. However a balance has to be found to keep the code executable on a machine. From experience, one effective strategy consists to run some first blind tests to have a feeling of the likely requirements for the number of walkers and the number of steps. For relatively simple case, we advice to start with 200 walkers and run for 500 steps, with a cut at 400 (nwalkers=200, nsteps=500 and nsteps_cut=400).
Filter database The filter database is essential in the fitting routine as allowing to predict the flux of a given combination of components at a given wavelength. For each step, the average flux of the model is calculated through the transmission curve of the filter given in the data file. This transmission curve represents the response of a telescope for a constant incident flux (in the frequency space) for a given frequency range. The convolution of the incident flux with the telescope response is important as any large variation in the SED within a spectral window can bias the estimated flux, for instance in the case of an emission line or a strong gradient within the averaged frequency range. This effect is particularly strong in optical and NIR where the telescope response varies significantly from filter to filter.
During the fitting, we therefore calculate for each filter, the corresponding telescope response convoluted with the modelled source with the following equation: S i = ∫ ν ma x ν mi n S model ν t ν dν ∫ ν ma x ν mi n t ν dν (11) where S i the averaged flux for a given filter i, ν the frequency, S model ν the flux of the model and t ν the transmission curve. Each of the filter used in the data file must be present in the database, or the code will fail to execute. Addition of new filters is easy, a simple text file with two columns, frequency and corresponding transmission can be added in the filters folder. For optical and NIR, given the large amount of filters used in astronomy, we point the reader to Asiago Database on Photometric Systems 4  to add the relevant filters for a specific analysis  (Fiorucci & Munari 2003).
Outputs Mr-Moose produces series of outputs, containing different information (see Table  1 ). The outputs can be divided in two parts: the supporting files (". fits", ". pkl", ". sav", ". sed") and the result files (". pdf"). We focus first on the support files and their contents, allowing to dig further in the interpretation if necessary and allow for an easy reproduction of a given fitting. We designed Mr-Moose to allow users familiar with the emcee package to use their custom tools from the ". pkl". However, Mr-Moose comes also with plotting functions to display results in order to interpret the result from the fit. There are several graphs ". pdf" file generated by the procedure: chain convergence, triangle and SED plots. These plots works "in unison" to understand the results of the fitting.
pkl" file, contains the object from the emcee procedure saved making use of the cPickle function from the pathos package allowing a serialisation, necessary in case of parallelised use. No modifications are implemented from the original emcee sampler, and as such, emcee-familiar users can make use of this output into custom graphical tools. This file contains the complete chain for each walker and other diagnostics provided by the emcee package. The ". sav" file contains the information from the fitting, including the setting file and the model structures, modified during the fitting and storing the final results in a YAML format. The file contains the final percentiles for each parameter, the best fit value, the name and path of the output files generated by the procedure. A ". sed" file is created reporting each component of the models from the global best fit.