dataesr/scientific-paragraphs-categorization huggingface dataset:
2019). DES collected ∼300 science exposures during a full night of observing, with variations depending on the season and survey strategy. These raw data were transferred to NOIRLab on a nightly basis for archiving and public distribution  (Fitzpatrick 2010; Honscheid et al. 2012). The wide-area survey was observed with 90 s exposures in griz. The Y-band observations used a 45 s exposure time for Y1-Y3 and a 90 s exposure time in Y4-Y6. 71 The median overhead between DES wide-area survey exposures is 29 s, which includes readout, filter change, and slew. Each position in the DES DR2 footprint was typically observed in 7-10 overlapping DECam exposures in the griz bands (Figure  2 ). A more complete description of the DES survey strategy can be found in  Neilsen et al. (2019). DES DR2 includes a total of 96,263 DECam exposures (each comprising 62 science CCDs), including 83,706 exposures acquired as part of the wide-area survey and 12,557 exposures acquired as part of the supernova survey and other programs.
Data Processing The DES image reduction, detrending, and processing were performed by the DESDM system at NCSA  (Morganson et al. 2018 ). Here we briefly summarize the processing pipeline and describe changes that were implemented in DES DR2 relative to previous data releases (i. e. , DES Collaboration 2018b). The DES Final Cut single-epoch processing pipeline removes instrumental signatures from individual exposures to produce reduced, science-ready, single-epoch CCD images. The Final Cut pipeline performs flat-fielding, overscan removal, crosstalk correction, nonlinearity correction, bias subtraction, gain correction, correction for the brighter-fatter effect, bad-pixel masking, astrometric matching, flagging of saturated pixels and bleed trails, principal-components background subtraction, secondary flat-field correction, and the masking of cosmic rays, artificial satellites, and other imaging artifacts.
The resulting processed images from this pipeline are released through the NOIRLab Astro Data Archive. Those images are provided as FITS-formatted files and contain extensions for the science data (SCI), a mask of bad pixels (MSK), and an inverse-variance weight (WGT). As in DR1, the weight images do not include flagged defects. A summary of the flags is provided in Table  9  of  Morganson et al. (2018). The Final Cut pipeline also performs PSF model fitting with PSFEx (Bertin 2011) and source detection and measurement with SourceExtractor  (Bertin & Arnouts 1996). Photometric zero points (ZP) for the reduced single-epoch images are derived with the Forward Global Calibration Method (FGCM;  Burke et al. 2018). The values of the single-epoch image pixels, I( pix), are provided in units of total corrected photoelectrons and can be converted to flux density in units of nJy as ( ) [ ] ( ) (( ) ) = ´´- F I 3.
63 pix 10 nJy. 1 ZP 30 2. 5 The typical depth of the Final Cut images estimated from the PSF magnitude at a signal-to-noise ratio (S/N) = 10 is g = 23. 57, r = 23. 34, i = 22. 78, z = 22. 10, and Y = 20. 69 (Morganson et al. 2018). At the other extreme, a DECam CCD with a typical full-well depth starts to saturate for point sources observed at the median seeing and exposure time (90 s for griz and 45 s for Y) of the DES wide-field survey at about g = 15. 2, r = 15. 7, i = 15. 8, z = 15. 5, and Y = 13. 6 mag. A summary of data quality metrics for the single-epoch data is reported in Table  1. The multiepoch pipeline produces coadded images and catalogs of astronomical objects organized within a tiling scheme that subdivides the full celestial sphere into manageable-sized regions for parallel processing  (Morganson et al.
2018). Each coadd "tile" is a square region of 0°. 7306 × 0°. 7306 with adjacent tiles overlapping by ∼ 30″. Coadded images are created in each tile with dimensions of 10, 000 pix × 10, 000 pix and a pixel scale of 0 263 (approximately the native pixel scale of DECam) using a sinc interpolation (LANCZOS3). Tile images are rescaled such that the magnitude ZP is fixed to 30 for all filters and the values of the pixels are given in units of picomaggies. 72 This makes the conversion between flux and magnitude the same for all bands. The set of input single-epoch images are based on data quality assessments from the Final Cut pipeline and FGCM. In addition, single-epoch images with severe scattered light, ghosting, or bright transient defects (e. g. , comets, meteors, airplanes, and earthquakes) are rejected.
We note that the absolute photometric systematic uncertainties quoted here simply reflect the difference in the C26202-and the DA-based methods of absolute color calibration and do not result from nonuniformities in the DES DR2 data. The nonuniformities are considered in the absolute photometric statistical errors. Multiepoch, multiband, and multiobject fitting that overcomes the PSF discontinuities has been applied for DES cosmology analyses and will be released in data products associated with those publications.
Changes in Single-epoch Processing Below we highlight several changes in the DES DR2 Final Cut processing of individual DECam exposures relative to the processing described in  Morganson et al. (2018). 1. New image calibration products (e. g. , master biases, super flats, pupil corrections) were created for the additional epochs of DES data taking. The process for creating these products is described in Section 3 and Appendix A of  Morganson et al. (2018). 2. Gaia DR2 (Gaia Collaboration 2016) was used for the external astrometric reference catalog. This yielded improved absolute calibration as documented in Section 4. 1. 3. ZP for all DES single-epoch images were rederived using FGCM  (Burke et al. 2018). Because the FGCM relies on overlapping exposures to derive relative ZP, DES DR2 improves the accuracy and uniformity of ZP derived for exposures included in DES DR1 (see Section 4.
2). 4. Starting in DES Year 5, a "light bulb" defect (a pixel with large spurious charge production) manifested itself at x, y = 795, 2620 on CCD = 46 (detector N15). The amplitude of this feature was found to depend on the state of the camera and the sky level of the image. An algorithm was used to detect the extent of this feature and set BPM_BADPIX over the affected region (MSK = BADPIX_BPM = 1). 5. Increased charge transfer inefficiency was detected sporadically in Amp B of CCD 41 (detector N10) during data taking in DES Year 6. An algorithm was developed to identify and mask affected pixels on this amplifier. The BPM_BADAMP bit was set for these pixels in the mask plane (MSK = BADPIX_BADAMP = 8). 6. The algorithm for detecting and masking artificial satellites was adjusted to avoid masking large, nearby galaxies.
186, R r = 2. 140, R i = 1. 569, R z = 1. 196, and R Y = 1. 048. The photometric measurements included in the DES DR2 database tables are not dereddened by default. However, the DR2_MAIN table includes additional columns for the dereddened versions of MAG_AUTOand WAVG_ MAG_PSF indicated by a _DERED suffix (Appendix E).
Catalogs and Database Tables The coadd source detection process cataloged 691,483,608 distinct objects. Object information includes object centroids, shape parameters, HEALPix indices, and processing flags. Several different photometric measurements and associated uncertainties are provided, including AUTO, PETRO, WAVG_PSF, and assorted aperture magnitudes (Table  D2 ). These measurements are distributed in three database tables: DR2_MAIN, DR2_MAGNITUDE, and DR2_FLUX served from an Oracle database at NCSA. The DR2_MAIN table contains all object information that is not a photometric measurement or uncertainty, augmented by MAG_AUTO and WAVG_MAG_PSF (with associated uncertainties) and interstellar-extinction-corrected versions of these magnitudes. The other two tables contain additional magnitude and flux measurements (with associated uncertainties).
Coverage Maps DR2 includes high-resolution HEALPix maps of the survey footprint in units of partial coverage (see Section 4. 4). The fractional coverage within each spatial pixel is calculated separately for each of the grizY bands. The maps are distributed at resolution nside = 4096, corresponding to a pixel side length of ~¢ 0. 86. The coverage maps distributed in FITS file format contain two columns, PIXEL and FRACDET, and are saved as partial HEALPix maps for storage efficiency. They can be found on the DES DR2 release page, as well. Coverage maps can also be found in the database table  DR2_COVERAGE.
Data Access This section describes the tools and interfaces that provide access to DES DR2. Data access is supported by NCSA 84 , LIneA 85 , and Astro Data Lab. 86 5. 7. 1. NCSA DESaccess NCSA continues to provide access to DES DR2 (and previous releases) with DESaccess (https://des. ncsa. illinois. edu/desaccess/), a collection of web applications connected to the Oracle database hosting the DES DR2 catalogs. The DESaccess API server and job management system were recently revamped to enhance the service and ensure its longterm sustainability. The user-facing web interface (UI) was also updated for compatibility with the new API and to add new features, including integrated online documentation of the available tools and a detailed specification of the DESaccess API to support programmatic use of the service.
Here we present a brief description of the main services provided by DESaccess: 1. The DES Tables app provides a list of all tables the user can access. The user can also view descriptions of columns in the chosen schema. 2. The DB Access app provides an SQL web client that allows the user to send queries directly to the Oracle 12 database hosting the DES DR2 tables. Queries with a small result set (currently limited to 1000 lines) may be sent synchronously as a "quick query," where the results are displayed on the web page within a few seconds. Typically queries are submitted for asynchronous execution, with an option to notify the user by email when the results are available for download. Results can be retrieved in the following formats: csv, FITS  (Wells et al. 1981) , or HDF5. 87 Compression is optional for the csv and HDF5 formats.
LIneA Science Server The images and catalogs generated by DESDM for DES DR2 can also be accessed through a web interface developed by the Laboratorio Interinstitucional de e-Astronomia (LIneA) for DES DR1 and available at NCSA since early 2018. 95 A new landing page integrates the new APIs implemented at NCSA, including a new cutout service. The platform has been designed to offer different ways to visualize images, catalogs (from DES and other surveys), and target lists, which can be either user uploaded or created from queries to the database carried out using the platform's user query functionality. The platform includes the following services: 1. The Sky/Image Viewer combination integrates thirdparty tools to allow the user to visualize the entire sky map produced by DES (Aladin Lite 96 developed by CDS 97 ) in the form of a HiPS (Hierarchical Progressive Surveys) image, as well as each individual coadd tile using VisiOmatic  (Bertin et al.
The upload can currently be done manually by pasting lines of (R. A. , decl. ) coordinates to the interface or selecting a table in the user's database. In fact, the user can paste a list of (R. A. , decl. ) positions or a copy of the contents of a CSV file with a header (the first line) containing multiple columns, but with columns labeled R. A. and decl. (case insensitive). Future versions will allow files to be uploaded in CSV and VOTable format. Once a list is pasted or uploaded into the tool, the image surrounding an object/ position can be visualized by selecting an entry in the list and the corresponding position. The user can then select the columns to be shown, sort according to a given attribute, and comment, rank, and reject an entry. One can also apply a filter, and the filtered list can be downloaded (as csv or FITS file) or saved for future use.
This tool is useful, for instance, to analyze a list of candidates with specific properties (e. g. , galaxies of a given morphological type, stars of a given spectral type, etc. ). It is also possible to explore a given object's properties from this tool, as described above. 3. The Tile Viewer is a simplified way to locate and visualize a specific DES tile or sky location. 4. The User Query service provides access to the database table available in the DES DR1/DR2 data from which SQL queries can be written, validated, and executed. The resulting table is displayed under "My Tables" where it can be renamed, a few lines of its content listed, and deleted. Objects selected can be immediately viewed in the Target Viewer after the resulting table columns are properly associated with those recognized by the tool.
Astro Data Lab The Astro Data Lab  (Fitzpatrick et al. 2016; Nikutta et al. 2020)  science platform (https://datalab. noirlab. edu) is one of the access portals for DES DR2. It is being developed at the Community Science and Data Center (CSDC), which is hosted by NOIRLab. Astro Data Lab enables efficient exploration and analysis of large data sets including surveys that use DECam and the Mosaic cameras  (Dey et al. 2016) ; high-value external photometric surveys; and, since recently, massively multiplexed spectroscopic surveys. Astro Data Lab hosts the DES DR2 catalog tables using a PostgreSQL v13 database backend. The data are identical to those hosted by the DES Collaboration through the NCSA portal but with a few additions. First, the Astro Data Lab database contains tables with crossmatches to other large catalogs such as Gaia and AllWISE.
Second, the main table contains several extra columns, such as ecliptic and Galactic coordinates, a Hierarchical Triangular Mesh (HTM) 102  index, supplementary HEALPix indices in RING (nside =256) and NESTED (nside =4096) schemes, S/Ns, and precomputed colors. Finally, the data are clustered and indexed using the Q3C scheme  (Koposov & Bartunov 2006) , which enables fast spatial queries. Astro Data Lab also serves the coadded images and single-epoch tiles, including a Simple Image Access (SIA) metadata database, and an image cutout service. Access to DES DR2 data, to colocated auxiliary high-value data sets, and to various data services is possible in a number of ways: 1. Database access to catalogs. Facilitated through a More information on using the data services and our data holdings is available on the Astro Data Lab web page: https:// datalab.
We solved this by making use of Peter Stetson's database of Johnson-Cousins standard stars  (Stetson 2009; Stetson et al. 2019). 114 We note that Stetson's photometry is calibrated to the photometric system defined by  Landolt (1992)  and is zeropointed to the Vega system rather than the AB system. Other than these complications, the process was very similar to fitting the other transformation equations in this section. We matched stars from DES DR2 to stars in the Stetson database, using the WAVG_PSF_MAGʼs from DES and the mean magnitudes from Stetson. To reduce the effects of photon noise, we only considered stars with rms magnitude errors of 0. 02 in either DES or the Stetson sample, and we only included Stetson stars that had at least 5 observations each in B, V, R, I. These cuts left us with ≈10,000 matches.
Based in part on observations at  Cerro Tololo Inter-American Observatory at NSF's NOIRLab (NOIRLab Prop. ID  2012B-0001 ; PI:  J. Frieman ), which is managed by the  Association of Universities for Research in Astronomy (AURA)  under a cooperative agreement with the  National Science Foundation. The DES data management system is supported by the  National Science Foundation  under Grant Numbers  AST-1138766  and  AST-1536171. The DES participants from Spanish institutions are partially supported by  MICINN  under grants  ESP2017-89838 ,  PGC2018-094773 ,  PGC2018-102021 ,  SEV-2016-0588 ,  SEV-2016-0597 , and  MDM-2015-0509 , some of which include ERDF funds from the  European Union. IFAE is partially funded by the  CERCA program  of the  Generalitat de Catalunya. Research leading to these results has received funding from the  European Research Council  under the European Union's  Seventh Framework Program  (FP7/ 2007-2013) including  ERC  grant agreements  240672 ,  291329 , and  306478.
This paper makes use of software developed for the  Large Synoptic Survey Telescope. We thank the  LSST  Project for making their code available as free software at http://dm. lsst. org. The  PanSTARRS1 Surveys  ( PS1 ) have been made possible through contributions of the  Institute for Astronomy , the  University of Hawaii , the Pan- STARRS  Project Office, the  Max-Planck Society  and its participating institutes, the  Max Planck Institute for Astronomy, Heidelberg , and the  Max Planck Institute for Extraterrestrial Physics, Garching , The  Johns Hopkins University, Durham University , the  University of Edinburgh, Queenʼs University Belfast , the  Harvard-Smithsonian Center for Astrophysics , the  Las Cumbres Observatory Global Telescope Network Incorporated , the  National Central University of Taiwan , the  Space Telescope Science Institute , the  National Aeronautics and Space Administration  under grant No.
NNX08AR22G  issued through the  Planetary Science Division of the NASA Science Mission Directorate , the  National Science Foundation  under Grant No. AST-1238877 , the  University of Maryland , and  Eötvös Loránd University (ELTE) , and the  Los Alamos National Laboratory. This work has made use of data from the  European Space Agency (ESA) mission Gaia  (https://www. cosmos. esa. int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www. cosmos. esa. int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement. The authors would also like to thank  R. C. Bohlin  and  S. E. Deustua  for useful comments on the absolute calibrations sections. Facilities:  Blanco (DECam), Astro Data Lab.
106 It allows the easiest mode of access to all data holdings and analysis tools, without the need to install any software at all or to download any data locally; only a web browser and an Astro Data Lab account are needed. All common astronomy and data science Python libraries are pre-installed, including the client interfaces specific to Data Lab (for queries, storage, access to a spectral service, and authentication). 6. Remote storage. Astro Data Lab provides users with space for files and database tables, enabling persistent notebooks and user data. We reserve 1 TB per user for file storage (VOSpace) and 250 Gbyte for personal DB tables (MyDB). 7. All-sky viewer based in Aladin Lite. The viewer supports multiple-layer transparency and hosts various wide-area surveys. 8. Curated set of example notebooks.
The extraction of WFS data used the wfs Stata command  (48). The metaregression model was written in Stan  (49). All other analyses were performed in R (50) using the package rdhs for the extraction of DHS data (51), the package scam for the estimation of shape-constrained additive models  (52) , and the package survey for bootstrap resampling  (53). Data Availability. DHS data are freely available at https://www. dhsprogram. com for authorized users. All computer codes needed to replicate the analysis are available in GitHub at https://github. com/nptodd/BAR.
The response surface typically represents unobserved spatially smooth predictors. Using realistic simulated data, we show that the method can alleviate bias in sampling effort and species niche estimates while allowing computational efficiency for large occurrence datasets. We further examined the method's robustness to the approximation of constant sampling within cells by varying the amplitude of spatial variation and the curvature of the sampling effort within cells. We also analysed a real dataset including opportunistic occurrence data of plant species stemming 1 https://www. gbif. org/ large opportunistic presence-only datasets, with broad spatial variation in sampling effort but also many species, such as datasets from citizen science programmes.
Our implementation is based on the glmnet library, in R language. It is similar to the implementation in  Renner et al. (2015) , except that it is extended to a multi-species case and with a cell-wise constant sampling effort. glmnet handles sparse matrices and is very efficient in terms of memory and computational load, given the structure of the model design matrix. The R code for reproducing the results and fitting the model is provided in a publicly available Github repository  (Botella, 2020).
| Simulated species density We simulated n o = ( n 1 , …, n 50 ) occurrences of 50 virtual species over D. The n i s were chosen from real occurrence data (i. e. the 50 most represented plant species in the Pl@ntNet queries dataset;  Botella et al. , 2019). This resulted in the following statistics: min(n o ) = 1502, max(n o ) = 5002 and ∑ i n i ∕50 ≈ 2206. All the virtual species densities ( i for species i in our model) were defined as Gaussian functions of the same single environmental variable (two cases considered: elevation/alti or annual precipitation/chbio_12, see Appendix E). The expectation of the Gaussian density was drawn uniformly inside the quantiles 0. 1 and 0. 9 of the environmental variable range of values while the standard deviation was drawn according to a gamma distribution of shape parameter 3 and scale parameter 50.
| Sampling effort The spatial density of sampling effort, s H was parameterized by a bandwidth parameter H > 0, which controlled the level of spatial smoothness. It was a continuous approximation of the density of real occurrences, obtained by filtering the Pl@ntNet dataset  (Botella et al. , 2019)  over D. More precisely, s H was an (3) log(p(Z 1 , …, Z N | )) = N ∑ i = 1 n i ∑ k = 1 log s z i k i z i k -∫ D s(z) i (z)dz.
| Application to a real dataset We also fitted the model to real occurrences recorded in the Pl@ ntNet query dataset  (Botella et al. 2019). The occurrences in this dataset were collected by citizens using the Pl@ntNet mobile application  (Joly et al. , 2016). They were automatically identified by the Pl@ntNet AI engine. Details on the identification system and the database infrastructure of Pl@ntNet are provided in Affouard et al. (2017). The dataset is publicly available on the open-access repository Zenodo (http://doi. org/10. 5281/zenodo. 2634137). The code for extracting occurrence and environmental data and fitting the model is provided in the code repository accompanying this article  (Botella, 2020).
| Environmental data A set of nine environmental variables was used to model the environmental density of species. These were selected carefully to model the macroecological niche of plant species, following the recommendations in  Mod et al. (2016). The set included mean and annual temperature variation, annual precipitation, potential evapotranspiration, available soil water capacity and a soil pH proxy. The variables are presented in Table  1  of Appendix E. We got the environmental data from multiple sources  (Karger et al. , 2016; Panagos, 2006; Panagos et al. , 2012; Van Liedekerke et al. , 2006; Zomer et al. , 2007 Zomer et al. , , 2008)). We extracted the values at occurrence points from the geographical rasters described and downloadable at Botella (2019). 2.
| RE SULTS The simulation study allowed us to evaluate the reliability of our joint model estimation method (see Section 2. 4). The two performance metrics obtained for the 10 simulation scenarios (2 environmental variables and 5 sampling effort profiles) are summarized in Figure  2. We were also able to illustrate the effectiveness of the method on the Pl@ntNet queries dataset as described in Section 2. 5. The estimated sampling effort is displayed in Figure  3. The estimation results for an exotic invasive plant, Phytolacca americana L, are provided in Figure  4.