dataesr/scientific-paragraphs-categorization huggingface dataset:
The Portal generates galaxy samples in the form of pruned lightweight catalogs containing only the columns required by specific science analysis, which may also be integrated into workflows  (Fausti Neto et al. , 2018). In this paper, we present, in particular, the capabilities of the Portal to produce photo-zs. It provides an integrated environment where all the steps necessary to compute photo-zs can be carried out in a controlled and consistent way. The automatic provenance, configuration management, and the computing facilities that sustain the Portal allow for a selection of many photo-z algorithms or settings, which would be highly time-consuming without infrastructure such as this. The need for the Portal capabilities will increase as the DES databases grow, and more generally, as we enter an era of big data astronomy.
In  Cavuoti et al. (2015) , the authors of the PhotoRApToR algorithm discussed the advantage of linking automatically different steps of photo-z calculation. The Portal surpasses Pho-toRApToR in the sense that it is method agnostic: any photo-z algorithm can be incorporated into the portal framework, which becomes especially interesting when the investigation aims to compare results using different methods. We present a sequence of tasks that include the preparation of a spectroscopic sample by combining data from different redshift surveys, the creation of training sets, the training and validation procedures for several algorithms, and the computation of photo-zs for large datasets. To show these examples, we used the DES first year data release, referred as Y1A1  (Drlica-Wagner et al. , 2018; Abbott et al.
Stage A group of pipelines in the same phase of data management. The E2E comprises the three stages: Data Installation > Data Preparation > Catalog Creation, and it is directly connected to the Science Workflows' stage. This last stage comprises the suite of scientific pipelines to support the research done by members of DES-Brazil Consortium (details in the supplemental video V0 4 ). The outline of this paper is as follows. In Section 2 we present the general technical aspects of the Portal. In Section 3 we go deeper in details of the processes related to the production of photo-zs. Still in Section 3 we present a use case of how the Portal can aid to determine reliable photo-zs through examples of runs using data from DES. The data is described in Appendix A. Finally, our conclusions and a summary of the paper are presented in Section 4.
DESDM at NCSA receives the data and reduces and co-adds the raw images  (Morganson and Dark Energy Survey Data Management Team, 2016). The photometry tables of objects detected from the coadded images are called coadd tables. After that, these tables are downloaded and ingested into the Portal@LIneA's database, where the science-ready catalogs are created. Finally, the catalogs and science products are transferred to Portal@NCSA, which provides visualization tools through the data release interface. In this work, we only present the technical aspects of the Portal@LIneA, which are related to the production of photo-zs. Among these features, we highlight those that apply to many other tasks: • Storing and registering of survey data to serve as input for analysis pipelines. • Maintaining analysis codes (and their development history) ready to run on registered products.
Portal Infrastructure The Portal framework relies on two databases, a mass storage file system, a web interface, a workflow system and a cluster of computers, as illustrated in Figure  2. Both databases uses PostgresSQL 5  object-relational database management system. The catalog database stores the catalogs retrieved from NCSA database and the catalogs produced by the Portal pipelines. The total storage capacity of the machine available for the catalog database is 23 TB, from which, ∼17 TB is already occupied. This device has a hot backup duplication in another connected machine. Both will be replaced in the future by new devices with larger capacity. The administrative database keeps track of metadata such as available releases, ingested products, product information, such as file and table names, storage location, classification, provenance, etc.
All the operations and steps are logged in the administrative database, so it allows to detect errors and investigate them posteriorly, and also to produce reports on the resources usage that can be filtered by a user or by an application. The mass storage device has the capacity for 59 TB, from which 39 TB is already used. It keeps data in three separate spaces: • Archive area, where the original catalogs FITS files are preserved (so the catalogs are duplicated in the database and mass storage's archive). • Scratch area, where are placed the directories and files that are created during the process executions. They include tables, images, flat files of any kind, run logs, error logs, etc. They remain in this area until the user saves the process. Periodically, a tool called garbage collector removes old files from this area.
• Process area, is a safe area where the directories and files from saved processes are kept permanently. The location of all the process directories, both in Scratch and Process areas are registered in the administrative database. The web application front-end uses the Model-View-Controller (MVC,  Burbeck, 1987)  software architecture and is developed in both Hypertext Markup Language 6  (HTML) and JavaScript 7  languages. It connects to the databases via the back-end infrastructure Python components.
The pipelines highlighted in gray are those related to the photo-z calculation. They will be discussed in details in Section 3. For now, it is not possible to run part of a pipeline stand-alone. For instance, one can perform repeated tests, varying configurations, until it converges into a result with the desired quality. In our case of study, we have already run training, validation, and computing for nine different photo-z codes, varying their configurations, more than 300 times in total, considering the several datasets of the Y1A1 release. That would be complicated to manage using directories and command-line runs.
Nowadays it is kept frozen and is being gradually replaced by EUPS. The pipelines of the first stage, Data Installation (the first block of pipelines in Figure  4 ), are only executed by the portal developers (LIneA's IT team), except for the Upload Tool, that can be executed by any science user. The initial step of Data Installation is the retrieval and ingestion from NCSA, of coadd tables. A daemon process called Data Retriever periodically inspects the NCSA database to discover new releases of the DES Survey data. For each new release recognized, the Data Retriever registers its existence in the Portal Administrative database. At this point, the Portal system "knows" the presence of a new release and a Portal operator can start the specific installation procedure that takes care of downloading the data in an optimized network route.
The download is done using a server in a "demilitarized zone network" and then storing it into a mass storage server. Later it is ingested into the Portal Catalog database. Finally, the Administrative database registers the 8 https://git-scm. com/ 9 https://git. linea. gov. br 10 https://github. com/RobertLuptonTheGood/eups new tables and makes them available to be accessed by visualization tools and to serve as input to be processed by pipelines in the Portal. A system of "classes" of products connects pipelines via inputs and outputs. In this context, a class is a unique keyword that identifies a specific product data structure. Hence, we can define the type of products that each pipeline receives as input and returns as output. As an example, the catalogs mentioned above, which are ready to be used in the portal, pass through the pipeline Install Catalog, where they are classified as products with class ="Object Catalog".
Parallelization The photo-zs estimation is one of the most computationally intensive tasks of the E2E process, due to the size of datasets involved. The first solution adopted in the Portal is the so-called Embarrassing Parallelization  (Herlihy and Shavit, 2011) , which is the division of the data into small partitions and their processing is done simultaneously by several computers. The large volume of data requires a high-performance system to transfer such data inside and outside each computer node avoiding the creation of an I/O bottleneck. This is a common problem in data-intensive computing that does not have a unique solution for all possible use cases. We solved this obstacle by implementing different I/O service strategies that can be used according to each specific problem, as detailed below.
For the very basic DES datasets (table of objects identified in coadded images) frequently used in the majority of the algorithms, we use the high responsive Hadoop Distributed File System (HDFS,  Shvachko et al. , 2010)  to distribute data uniformly across the cluster nodes. Despite the fact that the processing is done by several computers, the data itself also needs to be distributed. Otherwise, the simultaneous reading from the same database by several parallel tasks (jobs) would establish a significant bottleneck. The Portal's computers cluster contains 38 nodes with 24 central processing unit (CPU) cores each. The management of job submission is performed by an orchestration system, together with the HTCondor management system  11. The orchestration system interprets the parallelization strategy defined by each pipeline and its configuration, then it calculates the number of jobs necessary, and gives instructions to HTCondor.
The tasks are organized in the cluster nodes based on the data, such that it always prioritizes the runs to process the data that are already stored in each node, avoiding unnecessary data transfer overheads. If necessary, it allows reading additional data from other nodes. There is a mirroring in the data storage to help on this optimization. Each data chunk is stored triplicated in three different nodes. Hence, if one node needs data from a neighbor node that is, by chance, very busy with some intense process, the former still has two other options of nodes from where to get the data. For less frequently accessed data used in our algorithms, the Portal use PostgreSQL 12 , a fat node database with 24 cores and 256 gigabytes of RAM that supports multiple queries concurrently, allowing the fast retrieval of the portion of the data that each node needs.
postgresql. org/docs/9. 6/static/ parallel-query. html it follows one of the three options of data partitioning strategies: • Tiles: This is the original data division from DES. Tiles are tangent projections of an array equivalent to 5000 × 5000 CCD pixels (0. 7306 • on each side, Morganson and Dark Energy Survey Data Management Team, 2016). Each tile corresponds to one file stored in HDFS. The file size is strongly dependent on the density of objects detected, which is related to the depth of observations. For the Y1A1 depth, one tile contains, typically, ∼ 40k objects detected, and the file occupies ∼130 MB (for data description, see Appendix A). The number of tiles processed by each job submitted to the cluster is a free parameter in the configuration of the pipelines. The optimal choice depends on the dataset size (number of tiles), compared to the number of CPU cores available, as well as the memory consumption of the algorithm run.
• HEALPix 13  indexation: This is a more flexible way of splitting the entire sky projected area into pieces (pixels) of variable size. The division based on HEALPix pixels is available both for the coadd tables, stored in HDFS, and some of the other datasets stored in the database. Concerning the first, the data is distributed in files containing pixels of NSide = 32 (∼1. 6 GB each for Y1A1 wide datasets), using nested pixel ordering. NSide is a quantity that represents the resolution of the HEALPix map, so that the total number of pixels covering the whole sphere is N = 12 × (NSide) 2. Therefore, the larger the NSide, the smaller the pixel size. Similarly to the tilebased partitioning, the choice of NSide has implications on the processing performance. The available options for NSide values are 2 n , n = 2 to 10.
Stress tests have shown that large pixels (n < 4) should be avoided due to random-access memory limitations, depending on the dataset. On the other hand, scaling-out to use small pixels (n > 8) are also not recommended, because they convert in too many jobs, increasing the transferring overhead and stressing the cluster management system. One advantage of the partitioning based on HEALPix is that it applies to any data set which has celestial coordinates, therefore connected to spherical geometry. It is particularly useful to organize simulated data, that is not related to any observational strategy. • Custom: This option defines the data partitioning based on one of the data attributes (i. e. , any table's column). It queries the data and distributes, among the several cluster nodes, using intervals of one attribute.
Therefore, new techniques are being developed to estimate photo-z in surveys without the need of spectroscopic data (see, e. g. ,  Hoyle et al. , 2017). In the Portal, we have implemented tools to estimate the redshift of sources using standard techniques based on spectroscopic redshifts to train and validate the algorithms. In the following sections, we describe the methodology of each pipeline related to the photo-z estimation in the Portal. They operate in sequence, as illustrated in Figure  8. For each pipeline described, we add an example of a result obtained using data from DES Y1 release, as an illustration, and proof of concept. A description of the data used is available in Appendix A.
Spectroscopic Sample In the Portal the first step to obtain photo-zs is the creation of a spectroscopic sample that will be matched with DES photometric catalog to define a training set used for training and validation. The goal is to create a sample with as many sources as possible avoiding effects of cosmic variance or under-representation of particular spectral types. The database associated with the Portal serves as a centralized spectroscopic database for DES  (Hoyle et al. , 2017) , being continually updated, in particular, by ongoing follow-up observations from DES collaborators such as the OzDES project  (Yuan et al. , 2015; Childress et al. , 2017) , as well as with substantial new spectroscopic galaxy samples made public. The current spectroscopic redshift samples available are indicated in  Table A.
4. Once a new spectroscopic survey of interest is identified, the data is downloaded to our archive and ingested in the database to be accessible by the Spectroscopic Sample pipeline. At this stage of the process, it is necessary to provide some predefined information to be associated with a given spectroscopic sample in the registration database (Figure  9 ). When building a spectroscopic sample from heterogeneous sources, we need to take into account the following: • Each source might have different column names for the same quantities, e. g. , redshift represented by "z", "specz", "zspec", etc. We associate mandatory columns (RA, Dec, redshift, redshift quality, source, redshift error) when uploading each sample to ensure that the columns are properly delivered as input to the pipelines (see Figure  10 ).
• Each catalog may have different quality estimates of redshift. To normalize quality flags to a single schema, we take the approach of OzDES Survey  (Yuan et al. , 2015) , with qualities (Q spec ) ranging from 0 to 4. The numbers 0 and 1 are two types of unknown redshift, 2 is only a guess, 3 is above 90% confidence, and 4 attributed to a trusted redshift. When a survey is uploaded, we need to tell the centralized spectroscopic database that a new catalog has arrived and how to translate the quality information to the numerical system explained above. • Elimination of duplicates. Internally, the pipeline handles possible multiple measurements for same objects. There is also another pipeline (Photo-z PDF, Section 3. 5) not present in this illustration, because it is placed after the construction of scientific catalogs.
The next step is to choose the configuration parameters, as can be seen in Figure  12. There one can make decisions about the criterion adopted to handle duplicates, quality threshold, spectral classification, etc. The result of this pipeline is a table registered in the database containing: spec-zs, errors, quality flags, sources, and coordinates. At this stage, there is no association with DES objects. This table becomes available to be an input for the next pipeline or to be delivered to the collaboration. Relevant information about the spectroscopic sample created in a particular run is in the process' product log. In Figure  13  we show the result of creating a spectroscopic sample containing data from all surveys available in the Portal, selecting only the best measurements (Q spec =4), and resolving duplicates by the default criteria mentioned above.
This plot shows the spatial distribution of spectroscopic redshifts included in the    10 : Upload tool -column association screen. In this screen, the user is asked to drag the original column names (on the right side) to those names expected by the pipelines (on the left). With this information, the Upload tool creates a table in the database with the contents of the uploaded catalog and the columns names appropriate to be read by the pipeline Spectroscopic Sample. Both the original and the "translated" names are saved as metadata, to keep the history. sample, and we can verify that part of the spectroscopic sample spills over the DES footprint. The product created contains redshifts of 1,408,138 unique galaxies, from 34 surveys (see Appendix A. 2). Furthermore, the supplemental video V1 14 shows an example of a guided run of the Spectroscopic Sample pipeline and a quick exploration of its results.
Training Set Maker After creation of the spectroscopic sample, the next step is to match the photometric data to the spectroscopic catalog. This is done by the Training Set Maker pipeline, designed to build training (and validation) samples by matching a photometric sample (among the datasets available in the Portal) with a spectroscopic sample, which comes from the Spectroscopic Sample pipeline, as shown in Figure  14. In this stage the user can also include outputs of the Star/Galaxy Separation pipeline, identifying and removing point sources, avoiding the mismatch between spec-zs from galaxies merged to photometry from stars and removing QSOs. Also, optionally, it is possible to apply corrections in the observed magnitudes, like zero-point based on stellar locus regression calibrations  (High et al.
, 2009)  or galactic extinction. In the configuration menu (Figure15), the user selects the parameters to make quality choices and a search radius used for matching. Similarly to the previous pipeline, the matching is also done based on the angular separation between the objects at the database level using Q3C, but here with the spectroscopic and photometric catalogs. We also selected the radius to 1. 0 arcsec as a default configuration. If two or more objects from the photometric sample are within the search radius, the nearest object to the spectroscopic one is selected. The result of this pipeline is a table registered in the database under the class "training set", containing the columns from the spectroscopic sample plus some columns from photometric data (DES IDs, magnitudes, and errors).
Photo-z Training and Validation After the matched sample is created and registered in the portal's database, the next step before calculating photo-zs for the whole photometric dataset is to train the empirical algorithms and, optionally, calibrate the template-fitting ones. For 14 https://youtu. be/1mu-PqOvK88?list= PLGFEWqwqBauBIYa8H6KnZ4d-5ytM59vG2 15 https://youtu. be/2nA1PFGCnEM?list= PLGFEWqwqBauBIYa8H6KnZ4d-5ytM59vG2 several science applications, it is necessary to know the quality of the photo-z estimations requiring a validation step also done with a sample with known spectroscopic redshifts. Although it is not the pipeline with the largest number of components, Photo-z Training is the most complex among pipelines related to photo-z, because it performs several different tasks, as detailed below.
But in this case, this one is used only to compare photometric properties, to check whether the training and validations subsets are representative of the photometric set or not. In the first implementation, we created a unique pipeline to do both steps, the training and validation (The pipeline Photoz Training). Thus, a first component is necessary to separate the data in training and validation subsamples (to avoid the biases introduced by validating with the same galaxies used for the training). Afterward, we created another pipeline to do the validation step separately (details below), but we kept the first option available, optionally. The Subsets Separation component splits the matched catalog randomly, where the fraction of the data delivered to the training subset (and consequently the remaining portion for validation) is a free parameter in the component's configuration.
The user builds the weighted sample by repeating galaxies multiple times in the proportion of their weights, with their magnitudes spread according to their errors (assumed Gaussian) to avoid generating identical cloned galaxies. Applying this algorithm, we obtain a weighted sample that presents distributions of colors and magnitudes very similar to     On this screen, the user is asked to chose one matched sample (recognized by its class "Training Set"), and a photometric sample of reference ("Objects Catalog"). Similarly to any other pipeline, each row with a check box refers to a product previously generated by another pipeline, and registered in the database. The process number is a link that redirects to the process' product log, which helps the users on their choice. those of the photometric sample, as shown in Figure  19.
Photo-z Training In recent years the number of photo-z algorithms has increased enormously. The Portal is an interesting environment to compare different methods, since they can be applied to datasets under similar conditions. So far, the following codes are implemented in the Portal: Annz  (Collister and Lahav, 2004) , Annz2  (Sadeh et al. , 2016) , ArborZ  (Gerdes et al. , 2010) ,  BPZ (Benítez, 2000) , DNF  (De Vicente et al. , 2016) , LePhare  (Arnouts et al. , 2002; Ilbert et al. , 2006) , Pofz  (Cunha et al. , 2009) , Sky-Net  (Graff et al. , 2014 ), and TPZ (Carrasco Kind and Brunner, 2013 , 2014). We refer to  Hildebrandt et al. (2010)  and Carrasco  Kind and Brunner (2014)  for a review of the particularities and comparison of their performances. Empirical methods are the basis for the majority of the algorithms, except for BPZ and LePhare, two template fitting codes, for which a training sample can be used to improve photo-z quality through systematic shifts in the theoretical magnitudes from the spectral energy distribution (SED) templates.
Besides the training files, the component Photo-z Training also registers the code configurations used, so it is also applied by the pipeline Photo-z Compute, where the photo-zs are estimated for the DES datasets. The main advantage of performing this step independent from the actual photo-z estimation is that training files from one training procedure can be used in the photo-z calculation several times, for different photometric datasets. On the other hand, one can make training and validation several times, until gets a satisfactory result and then apply it to the photo-z central estimate.