dataesr/scientific-paragraphs-categorization huggingface dataset:
This triangle diagram is particularly helpful to (i) reveal any degeneracies among parameters (ii) reveal if parameters are constrained, and how well, given the dataset. The SED plots show the fit of each arrangement on the data, the final results of the models and data combination. The final number of generated plot changes, depending on the complexity of the source considered. First, the SED is coming in two versions, a best fit and a "spaghetti" version. The former shows the best overall fit to the data, each colour corresponding to one model (the maximum likelihood reached in the chains). The latter shows all the solutions provided by each walkers at each step after the convergence cut (same than the triangle plot). This is particularly useful to see how good the models are constrained with the data in combination with the triangle and convergence plots.
EXAMPLES In order to demonstrate Mr-Moose capabilities, we focus here on three specific extragalactic examples which where used to build this procedure (the code is provided with several examples with increasing complexity, we focus here on the artificial ones produced by the file example_1. py and example_6. py) as well as a previous result from literature. For the sake of simplicity, we will refer to them as example #1, #2 and #3, respectively. The first is a simple case of fitting, with only one model onto a dataset, here synchrotron emission from a radio source in the 74 MHz-8. 4 GHz range. The second case is much more complex: we here aim to disentangle the contribution of various components, spatially non-collocated and originating from different physical processes. We take the example of a powerful radio galaxy and a star forming galaxy companion with separate emission from the lobes (of synchrotron origin), the dust heated by star formation and the dust heated by AGN, in the 8 µm-1.
Example #3: real dataset We finally run Mr-Moose on a published dataset to ensure the code is able to recover previous results. We take the example from Gilli et al. (  2014 ) as photometry and parameter results are reported in the publication and allow for a direct comparison. Briefly, the data, a combination of ALMA and Herschel observations with upper limits and detections, are used to fit a modified black body component (reported in § 2. 2. 2), in order to access the dust properties of a z=4. 75 quasar. We run the code in two configurations a and b: one with only the detected filters and a second adding the upper limits to the fit. For both configurations we set β=2 and ν 0 =1. 5 10 12 Hz as in the original publication. We list the files in the Appendix and run the procedure twice. We focus only on recovering the temperature parameter as the other parameters are derived either from the best fit temperature or directly from observed fluxes.
KNOWN LIMITATIONS AND FUTURE DEVELOPMENTS At the publication of this paper, the code will be available online on the GitHub platform at the following url: https://github. com/gdrouart/MrMoose, under a GPLv3 license (allowing re-use and modifications, see License. txt in Mr-Moose folder). While Mr-Moose is under continuous development and will see periodic upgrades, the code also knows some limitations. We list some of the most important ones and refer the reader to the README for a more exhaustive list of all known issues/limitations. • The execution time is highly dependent on the complexity of the configuration: the most parameters/components/arrangements, the longer the time for the fitting to converge. Typical execution times are from couple of minutes to several hours (see Table  9 ) for a reasonably recent machine.
More particularly, it assumes that the user defined models are as efficient as possible because these models are called at each iteration for each walker and is therefore the bottleneck of the code. Large number of walkers and steps can lead to significantly larger files, slowing the process. Even if Mr-Moose is designed to run in one step, we recommend to split the run in several steps, first generate the images only to check your datafile, hence a second run to perform the fit and a third to generate the ". pdf" files (by changing the keywords values in the setting file). • Choosing initial values close to the "true" values is important, otherwise the code will take a long time to converge (if converging at all). This choice of initial guess is particularly important when including upper limits.
By design, when the parameter values produce a model above an upper limit, the likelihood is very quickly set to infinity due to the error function (see § 2). Therefore a slight underestimation of parameters is preferable, particularly on any normalisation parameters (to keep the second term of the χ 2 in Eq. 6 close to null at the beginning). We remind that the initialisation of the walkers is made following the emcee recommendation: a Gaussian "ball" of walkers centred on the median value of the interval parameters. • The code provides two levels of parallelisation (multi-core processing) for the calculation of the likelihood: the parallelisation on one source and the parallelisation on a sample of sources. The former tends to deteriorate the performance, most likely due to the large overheads when the likelihood is relatively simple (see emcee documentation).
CONCLUSION We presented a new fitting tool allowing the user to fit SEDs for a variety of source configuration from relatively simple to complex source configuration in a bayesian framework using the emcee package. The code, highly customisable, allows a wide range of configurations between data and models, with the aim to stay user-friendly. The code consistently handles upper limits along with detections, each data point being calculated through the filter database, also managed and defined by the user. The large flexibility on creating models/data/filters makes this code virtually adaptable to any case, independently of wavelengths, instruments/telescopes combinations or science cases, providing a robust tool to interpret increasingly complex multi-wavelength datasets. In particular, the code appears ideally suited (but not limited) to fit a combination of Spitzer, Herschel, ALMA and JVLA dataset, and/or in the presence of constraining upper limits in the data even in the case of a single unresolved source.
Figure5. Scheme of the combination of observations at different wavelengths from the data file to illustrate the considered example. The circle sizes are relative to the resolution of the observation but not to scale (see Fig. 6). Note that Herschel, LABOCA, one ALMA, one VLA and ATCA observations are not resolving the different components. From bigger to small circles: Herschel, ATCA, VLA, LABOCA, ALMA, double circles are VLA and ALMA(closer to center), and in the center Spitzer.
astropy v2. 0rc1, http://www. astropy. org,AstropyCollaborationet al. (2013)• corner v2. 0. 1, https://pypi. python. org/pypi/corner • emcee v2. 2. 1, http://dan. iel. fm/emcee/current/#, Foreman-Mackey et al. (2013) • guppy v0. 1. 10, https://pypi. python. org/pypi/guppy/ • matplotlib v1. 5. 1, http://matplotlib. org/ • numpy v1. 9. 1, http://www. numpy. org/ • pathos v0. 2. 0, https://pypi. python. org/pypi/pathos/0. 2. 0 • pycallgraph v1. 0. 1, http://pycallgraph. slowchop. com/en/master/#, • PyYAML v3. 12, http://pyyaml. org/ • tqdm v4. 8. 4, https://github. com/tqdm/tqdm • scipy v0. 14. 0, https://www. scipy. org/.
ACKNOWLEDGEMENTS GD would like to thanks  Steven Murray  for insightful discussions on Bayesian inference and the  Curtin PyClub  for useful programming tips. We also thank  Matt Lehnert ,  Carlos De Breuck ,  Joel Vernet  and  Nick Seymour  for advices and help in the conception of this code. GD acknowledges the  ESO Scientific Visitor Programme  in supporting the development of this code.
astropy v2. 0rc1, http://www. astropy. org,AstropyCollaborationet al. (2013)• corner v2. 0. 1, https://pypi. python. org/pypi/corner • emcee v2. 2. 1, http://dan. iel. fm/emcee/current/#, Foreman-Mackey et al. (2013) • guppy v0. 1. 10, https://pypi. python. org/pypi/guppy/ • matplotlib v1. 5. 1, http://matplotlib. org/ • numpy v1. 9. 1, http://www. numpy. org/ • pathos v0. 2. 0, https://pypi. python. org/pypi/pathos/0. 2. 0 • pycallgraph v1. 0. 1, http://pycallgraph. slowchop. com/en/master/#, • PyYAML v3. 12, http://pyyaml. org/ • tqdm v4. 8. 4, https://github. com/tqdm/tqdm • scipy v0. 14. 0, https://www. scipy. org/ INPUT FILES FOR BOTH EXAMPLES.
Given the importance of the atmospheric aerosol, the number of instruments and measurement networks which focus on its characterization are growing. Many challenges are derived from standardization of protocols, monitoring of the instrument status to evaluate the network data quality and manipulation and distribution of large volume of data (raw and processed). CAELIS is a software system which aims at simplifying the management of a network, providing tools by monitoring the instruments, processing the data in real time and offering the scientific community a new tool to work with the data. Since 2008 CAELIS has been successfully applied to the photometer calibration facility managed by the University of Valladolid, Spain, in the framework of Aerosol Robotic Network (AERONET). Thanks to the use of advanced tools, this facility has been able to analyze a growing number of stations and data in real time, which greatly benefits the network management and data quality control.
Ground-based and orbital instruments have been applied to monitor aerosol properties. Combining instruments is also possible to maximally exploit synergies. For example, satellites have demonstrated the potential of high spatial coverage and resolution, and standardized ground-based networks have the benefit of high accuracy. A common exercise is to validate satellite data with ground-based networks. One of these ground-based networks is the Aerosol Robotic Network (AERONET;  Holben et al. , 1998). Led by NASA (National Aeronautics and Space Administration; http://aerosnet. gsfc. nasa. gov) and PHOTONS (PHOtométrie pour le Traitement Opérationnel de Normalisation Satellitaire; http://loaphotons. univ-lille1. fr/), AERONET is built as a federation of sub-networks with highly standardized procedures: instrument, calibration, processing and data distribution.
It was created in the 1990s with the objective of global monitoring of aerosol optical properties from the ground, as well as validating satellite retrievals of aerosols. The standard Published by Copernicus Publications on behalf of the European Geosciences Union. D. Fuertes et al. : CAELIS software instrument used by the network is the photometer Cimel-318. It is an automatic filter radiometer with two-axis robot and nine spectral channels covering the spectral range of 340 to 1640 nm. It collects direct solar and lunar measurements, and sky radiances in the almucantar, principal plane and hybrid geometrical configurations. Once the data are validated through instrument status and cloud screening, aerosol optical depth (AOD) can be obtained as direct product for the nine wavelengths. Using inversion algorithms  (Dubovik and King, 2000; Dubovik et al.
The GOA calibration facility is in charge of the calibration and site monitoring of about 50 AERONET sites in Europe, North Africa and Central America. AERONET standards stands for annual instrument calibration, maintenance and weekly checks on the observation data. The calibration process takes about 2-3 months and includes post-field calibration for sun, moon and sky channels, maintenance of the instrumentation and pre-field calibration for the next measurement period. In order to avoid gaps in the data sets during calibration periods, frequently one instrument is swapped out with a freshly calibrated one. The network management determines where each instrument is located, what its exact configuration and calibration coefficients are and how many days remain until the next calibration is needed.
During the regular deployment period the instrument has to be regularly checked to guarantee the data quality. A routine maintenance protocol is performed by the site manager, but the network is ultimately responsible for data quality. The routine maintenance helps in reducing instrument failure and data errors, but even with the best daily protocol, instrumentation problems may occur. Data monitoring at the calibration center helps in early identification of instrument issues. However such work cannot be accomplished manually in near-real time (NRT) for a large number of sites. In this context, it was necessary for the calibration facility at GOA to implement an automatic mechanism (in addition to the standard mechanism of AERONET) to help manage the network and facilitate weekly data checks needed to guarantee the quality of the data.
The motivation of the CAELIS system is to fulfill these two requirements. The system has to be designed to save all data, metadata and ancillary data (assimilated from other sources) in order to, on the one hand, support the management, maintenance and calibration of the network, and on the other hand, process the raw data in NRT with different algorithms and provide network managers, site managers and ultimately the scientific community with a very powerful and modern tool to analyze data produced at the observation sites. This work shows the fundamentals of CAELIS system which have been developed since 2008, both with respect to the scientific background and the information technology employed. There was no predecessor software at Valladolid and these tasks were done manually before CAELIS was developed.
The other two AERONET calibration centers at NASA and University of Lille have their own tools. Some ideas implemented in CAELIS are inspired by these tools. 2 General architecture CAELIS has been designed to run on a server which, connected to the internet, allows for external communication via a web interface. The software contains a "daemon" (a background process that offers a service) which is responsible for selecting and launching tasks. These tasks, later explained, are responsible for downloading new data whenever available and processing them. Each task reads the required input information from the database and writes the output there. Some tasks use direct internet access to retrieve data, e. g. , downloading ancillary data from an FTP server. All information downloaded and treated by the CAELIS tasks is stored in the database.
This allows for the following actions to retrieve all information required from the database (quick extraction). External users (organized by role with various privilege levels) can connect through the web interface to watch what tasks are being executed and explore the results of finished tasks. All actions required by the system administrators can easily be done through the web interface. Network management is also performed through the web interface which allows for, for example, setting up the installation of an instrument at a measurement station. The same information will be used by the system when data from the instruments reach the server and CAELIS will compare the received information (instrument number, parameters, location, dates, etc. ) with reference registers stored in the database (installation periods, configuration parameters, etc.
) to know if the instrument is working properly and using the correct configuration. External systems, such as measurement stations, can also be connected to the server and submit data. Thanks to the web interface, it can be done using port 80 (standard HTTP), which avoids many problems derived from security rules of the measurement stations and hosting institutions (some of which are in military areas). The current system manages 120 users and 80 stations. Each station can send thousands of aerosol observations every year and the system is constantly growing. A benchmark has been applied to confirm that the current architecture can support a network 100 times bigger, so the database can grow safely in the future. As shown in Fig. 1 , CAELIS is composed of a database, a processing module and a web interface.
These modules can be deployed independently even in different computers. The users and the stations interact with the system through the web interface. In the database, the raw data and metadata are stored, as well as the retrieved products, ancillary data, user information, etc. The NRT processing module is composed of the system daemon and a set of processing routines that extract information from the database, calculate products and store them in the database. The web interface is the platform designed to manage the system, to manage the network and to provide visual access to the data and metadata, with tables, plots, searching capability, etc. Each of these elements will be explained in detail in the next sections.
Database model Databases are one of the main concepts developed in the 1980s in the computer sciences. Many different approaches in terms of technology and data models have been developed with varying success. There are many types of databases, classified depending on their characteristics. A database management system (DBMS) is software with an interface to a database system that provides the user with advanced characteristics such as the management of concurrency or a query language. The decision about what kind of database and which specific DBMS software to select is one of the main design decisions because all further development will be impacted by it. Relational databases are a traditional and well-known model, and have been successfully applied to many different fields. In relational databases the information is organized in tables or relations which represent entity types  (Chen, 1976).
A good database modeler is able to identify those entity types that are relevant with the information that describes them. The tables or relations are composed of columns with the attributes that describe them, and rows which represent different individual entities that are identified by an unique key (one or more attributes that cannot be repeated in different rows). The tables are linked, creating a relational model. The keystone of a database is good design which needs to take into account the information targeted for modeling as well as the way in which the data is going to be accessed (to optimize performance). Complex models with many groups of entities need to be planned in advance by creating an entityrelationship diagram. This diagram then helps final implementation of the database, which can be a direct translation of the diagram just taking some implementation decisions about a balance between data redundancy and performance.
The main elements of the entity-relationship diagram of CAELIS database are shown in Fig. 2. The central entity is the photometer, which produces raw data. The photometer, with given hardware configuration and calibration coefficients, is installed at one site of the network. The ancillary data for the site (e. g. , meteorological data, ozone column, surface reflectance) need to be stored. Finally the measurement stations are supported by institutions, which can also own other instruments. Each of these elements is in many cases representing a group of entities. For instance, calibration coefficients include extraterrestrial signal for the different solar spectral channels, radiance calibration coefficients for sky channels, coefficients for temperature correction of the signals, instrument field of view, etc.
), solar and lunar extraterrestrial irradiance spectra and spectral absorption coefficients for several species (ozone, NO 2 , water vapor, etc. ). Many different DBMS can be used to implement such a model: OracleDB, SQLite, PostgerSQL, etc. CAELIS is based on a MySQL database. MySQL software is widely used by many different communities. Therefore the software is very robust, complete, stable and well documented, and it can be run in many different architectures. The entity-relationship diagram for CAELIS, illustrated using the model defined by  Chen (1976) , is shown in Fig. 2. This diagram shows the fundamental part of the database, called layer 0. On top of that, direct products -obtained with the combination of raw data, calibration coefficients and ancillary data -are stored. This represents "layer 1" products, physical quantities with their corresponding units and estimated uncertainties (derived from the calibration uncertainties).
Processing chain and near-real-time module CAELIS system provides many different data products. To provide each product, some input data has to be processed in a specific way. This is what we call a "task". The job is divided into a set of simple tasks. The system works as a state machine: one task cannot start until the previous one is finished, no matter if the second task is dependent on or independent of the previous one. When many tasks work sequentially to achieve a common objective, we create a chain of tasks. The daemon running in the server is responsible for coordinating the different tasks, as it will be explained in the next section. The main processing chain in CAELIS is the set of the tasks that are performed once new photometer data are uploaded into the system, as shown in Fig.
4. The pre-filter checks that the file uploaded to the server is a valid data file pertaining to the AERONET instruments managed by CAELIS. If this is true, the "filter" checks that the basic information (instrument number, coordinates and dates, configuration parameters) is in accordance with the stored information about instrument installations. If any of these parameters do not correspond to its expected value, the data file is put to quarantine and the network managers receive an email notification ("send notification"). If all parameters are correct, the measurements are inserted in the database and the data file is forwarded to any desired destination: our data file repository, an AERONET server at NASA, etc. With the raw measurements inserted in the database, the processing of layer 1 products is triggered: the aerosol optical depth, water vapor and radiances in several geometries (almucantar, principal plane, etc.
). With all raw data and layer 1 products, a set of flags concerning data quality control are produced by the alarms task. These flags are produced in near-real time, as soon as new data are submitted to CAELIS from a particular site. Since the automated quality control (QC) analysis needs all available information, the alarms task is always the last one in the processing chain. The QC flags in near-real time are a very important element in the network management. More details are given in Sect. 7. 2. Any new implementation, for instance a new layer 1 product, needs to be inserted in the processing chain taking into account its dependency on any other elements in the chain. The last step in a certain task is to trigger the next one in the chain. Whenever new data are found (photometer, ancillary, other) or new information is inserted by the managers (new calibration, installation, etc.
), a processing chain is triggered. The management of all chains in CAELIS is carried out by the daemon, which is explained in detail in the next section. This kind of task organization is highly modular, so new elements in CAELIS, either data or different instruments can be added by creating new chains that can be connected or not to the existing ones. The near-real-time processing module (see Fig. 1 ) is composed of a set of programs and libraries that are related to all the above mentioned tasks. These are programmed mainly in C for fast computation with large data sets and interoperability with other technologies, allowing for the use of other languages in the future. A Git repository is used to facilitate version control and deployment of the software.