dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgments: We thank  A. Gorelik ,  Ø. Strømland , and  R. Smith  for comments on the manuscript and  M. Schuller ,  E. Prokhorova , and  T. Agnew  for experimental assistance. We thank  C. Colas  and the MS subplatforms of the MO2VING platform ( CNRS Orléans ) and of the FR2708 federation for the HRMS analyses, as well as the NMR subplatform of MO2VING for help with NMR analyses. Funding: The work in I. A. 's laboratory is supported by the  Wellcome Trust  ( 101794  and  210634 ),  Biotechnology and Biological Sciences Research Council  ( BB/ R007195/1 ),  Ovarian Cancer Research Alliance  ( 813369 ), and  Cancer Research United Kingdom  ( C35050/A22284 ). The work in D. A. 's laboratory is supported by the  Edward Penley Abraham Research Fund. M. J. S. is supported by the installation grant of the  INC CNRS.
(2020)  and scipy  Virtanen et al. (2020) , visualization frameworks like matplotlib  (Hunter 2007) , machine-learning and data analysis packages like tensorflow  (Abadi et al. 2015) , pymc3  (Salvatier et al. 2016) , and emcee (Foreman-Mackey et al. 2013), domain-specific libraries like yt  (Turk et al. 2011 ), plasmpy (PlasmaPy Community et al. 2021) , sunpy (SunPy Community et al. 2020), Biopython  (Cock et al. 2009) , and sympy  (Meurer et al. 2017) , to name a few in each category. The astropy  (Astropy Collaboration et al. 2013 , 2018)  core package began in this vein, as an effort to consolidate the development of commonly used functionality needed to perform astronomical research into a communitydeveloped Python package. The astropy core package was one of the first large, opensource Python packages developed for astronomy and provides, among other things, software functionality for reading and writing astronomy-specific data formats (e.
g. , FITS), transforming and representing astronomical coordinates, and representing and propagating physical units in code. An early description of the core functionality in astropy can be found in the first Astropy paper Astropy  Collaboration et al. (2013)  or in detail in the core package documentation. The code base in the astropy core package is now largely stable, in that the software interface does 122 TARDIS Collaboration. 123 Astropy Coordination Committee. 124 NSF Graduate Research Fellow. * The author list has two parts: the authors that made significant contributions to the writing and/or coordination of the paper (in order of contribution), followed by maintainers of and contributors to the Astropy Project (in alphabetical order). The position in the author list does not correspond to contributions to the Astropy Project as a whole.
Because of this natural expansion, the name Astropy has grown in scope beyond a single Python library to become "the Astropy Project. " The Astropy Project is a community effort that represents the union of the astropy core package, the ecosystem of astronomy-specific software tools that are interoperable with astropy (Astropy Coordinated and Affiliated Packages), and the community of users, developers, and maintainers that participate in Astropy efforts. However, there is no single institution responsible for managing the Astropy Project, for funding or maintaining its development, or sustaining it into the future: The Project is maintained and coordinated by both volunteers and contributors who are primarily paid for other reasons (i. e. , Astropy work is just a means to an end). While new Astropy-affiliated packages are being developed that expand upon the core functionality in the astropy package, representing a natural expansion of the Astropy Project ecosystem, the needs and challenges faced by the Project are evolving.
Highlighted Feature Development The astropy core package is mature and stable, in that many features have been part of the core package for years with purposefully few changes to the software interface. This maturity and stability allow for the broader astronomy Python community to rely upon the astropy core package and build specialized packages within the Astropy ecosystem. Even many non-astronomical Python libraries have come to rely on astropy. The relationship between Astropy and the broader Python community is reciprocal, with the evolving needs of the community and the maturation of the broader scientific Python ecosystem driving much of the development of the astropy core package. Within astropy, development may be roughly split into several categories: new features, intra-and inter-package interoperability, improvements to numerical precision, accuracy, and reproducibility, performance and documentation enhancements, and bug fixes.
We discuss some of these in the subsections below. Additionally, some features have been moved from the core package, either because they are of more general use outside of astropy, or because they are too specialized and belong in an Astropy-affiliated package. An example of the former is the copy of the IAU Standards Of Fundamental Astronomy (SOFA) software Collection 125   Hohenkerk (2011)  that Astropy carried, as well as the associated Python wrappers. Since this is basic infrastructure, with a release cycle set by SOFA, it made more sense to create separate packages that serve as astropy dependencies:  ERFA Tollerud et al. (2021)  and  PyERFA (van Kerkwijk et al. 2021b ). An example of something that was too specialized was a Virtual Observatory subpackage; the Simple Application Message Protocol (astropy.
astropy. units provides the ability to associate numbers with units, and propagate these correctly, using the Quantity class. However, the ability to associate and propagate uncertainties is relatively limited: in astropy. nddata, there are options to associate errors with data arrays, but covariances are not tracked. Error propagation is difficult, and often is best approximated using Monte Carlo methods. The new astropy. uncertainties1 subpackage is our first step toward enabling seamless error propagation. The package allows one to generate, for each variable, randomly drawn samples in a Distribution, and then propagate these by passing them through the normal analysis, producing a Distribution of final results that can be inspected. This Distribution can be any type of array, including a Quantity.
The masked quantities work largely without any changes in higher-level objects such as SkyCoord, but more work is needed to better integrate it with the rest of astropy, e. g. , using masked arrays in Time instead of NaN. Time Series-From sampling a continuous variable at fixed times to counting events binned into time windows, many different areas of astrophysics require the manipulation of time series. The new astropy. timeseries subpackage extends the QTable class to support tables of data as a function of time, where the data can either represent samples or averages over particular time bins. The new classes offer a number of special methods to manipulate time series (e. g. , folding and resampling) and to read different data formats (e. g. , Kepler light curves). Also part of astropy.
This mapping is the essence of the World Coordinate System (WCS) concept. From its inception, the astropy. wcs subpackage allowed access to WCS information provided in, e. g. , FITS files, but it became clear that other WCS standards and representations had to be supported for new missions and observatories (e. g. , JWST and the Rubin Observatory). To harmonize these needs, a new high-level interface was created based on a formal design first proposed in an Astropy Proposal for Enhancement (APE 14). The hope is that by having a formal design, other packages implementing WCS objects can straightforwardly modify their classes to conform to the new interface or build thin wrappers that conform. The implementation in astropy. wcs interacts well with other astropy objects such as SkyCoord and Time.
Interoperability Because Astropy is modular and situated at the nexus between scientific computing and astronomy, interoperability is an important focus for Astropy, both within the various astropy subpackages and with other Python libraries. We aim for a seamless user experience, where different code blocks would just work when put together. Here, we describe a few particular efforts toward this goal. numpy on Units-Quantity is a backbone of Astropy, leveraging the power of numpy and adding units. Previously, many numpy functions would strip a Quantity of its units (or fail outright), limiting Quantity 's potential. Now, advances in numpy function overloading (e. g. , NEP 18) mean Quantity works with almost all numpy (v1. 17+) functions. In the remaining gaps, the numpy and Quantity interoperability efforts are ongoing.
For some functions, such as in numpy' s module for manipulating structured arrays, compatibility only requires extending the existing numpy-astropy bridge frameworks. Community interest, in the form of a Feature Request (or better yet Pull Request) would be sufficient to see this compatibility completed. For a few remaining functions, discussed further in the Astropydocumentation, the numpy framework does not yet allow for full interoperability with Astropy. A goal of Astropy and the scientific Python community is to enhance and implement the frameworks, allowing Quantity to propagate units seamlessly across the whole ecosystem of numpy-like projects. astropy on Units-Units were also integrated further within astropy. For an already-defined unit-less Model, e. g. , those imported from another library, astropy.
modeling can now coerce units. A new module-astropy. cosmology. unitshas been added to the cosmology subpackage for defining and collecting cosmological units and equivalencies. astropy. cosmology. units has a unit and equivalencies for littleh, and a new unit, redshift, for factors of cosmological redshift. redshift has a number of equivalencies for converting between cosmological distance measures, e. g. , cosmic microwave background temperature or comoving distance. Table  Column  Types-Within astropy and Python, there are numerous different ways to represent and store array-valued data. Some differences are historical: astropy. table and io. fits. BinTableHDU are both for holding tabular data but the latter was developed for and remains largely specific to FITS. Some differences are computational: numpy, cupy, and dask arrays have almost identical APIs, but are optimized for different use cases.
Lastly, some differences are inherent: Quantity, Time, and SkyCoord represent fundamentally different types of objects. astropy. table is astropy's most general container for array-valued data. Rather than requiring component data to be converted to a table. Column, a well-defined protocol has been developed for any array-valued data to be used in a astropy. table, without having to convert the data. The protocol for mixing column types, which astropy terms "mixin" columns, provides the same familiar table API, yet loses none of the object's data or attributes. Note that mixin here is not implemented in the commonly used Python scheme of multiple inheritance. Rather, it is a composition protocol that makes it possible to store Astropy native objects-including Time, Quantity, and SkyCoord -within a Table and write these to various file formats, such as FITS.
For objects not already covered by the astropy mixin protocol, functions can be registered with Table to enable interoperability. As an example, astropy already implements these mixin functions to integrate dask arrays with Table, allowing cloud-stored or cluster-scale data to be used as a column in a Table. Astropy FITS in Time-The FITS standard was extended to rigorously describe time coordinates in the WCS framework  (Rots et al. 2015). Compared to other types of coordinates in WCS, time requires more metadata: format, scale, position, reference, etc. This metadata had to be manually specified and the nuances understood by the user. astropy. fits could read this data as a standard Column, but would not interpret the time-related metadata and attributes. Through the support of the Google Summer of Code 2017 program, 126  the astropy.
fits package now interprets time data correctly, using Time as a column. For backward compatibility, this feature may be turned off. Persistent Storage-Astropy's efforts to increase support for column types in a table are mirrored by efforts to expand storage format options. New formats have been added, and existing formats updated to support more column types. Astropy now supports reading and writing tables in the American Astronomical Society Journals' machine-readable table (MRT) format. This ASCII format has long been missing and was added to Table  I /O in a Google Summer of Code 2021 project. In addition, Table may also read from and write to ASDF, 127  Parquet, and QDP formats. The ECSV standard has been updated to v1. 0, adding support for three additional data subtypes: multidimensional column data (both masked and unmasked) with fixed dimensions in all table cells, multidimensional column data with variable-dimension arrays similar to FITS variable-length arrays, and object-type columns with simple Python objects.
As mentioned previously, dask arrays may be used as a column in Table. Now, dask may also be a data array in FITS HDU and only computed while written, avoiding excessive memory use. Furthermore, Table can be appended to an existing FITS file, where dask columns can interoperate seamlessly between them. Unified I/O Architecture-astropy. io. #1 is a powerful way to define input and output (I/O) functions for astropy objects, such as reading from or writing to a file, following a given standard. astropy uses this internally for the I/O methods in Table  and  Cosmology. Users can also register custom I/O to extend the options on these classes. As of astropy v5. 0, the I/O registry subpackage has been generalized to enable a number of new use cases; for instance, the class-based architecture allows streamlining of the creation of custom registries while minimizing code duplication.
Performance In some cases, feature enhancements and corner-case handling were added at the cost of performance. Hence, GitHub pull requests that improve performance are welcomed, and were even the main goal for the v3. 1 release. For most subpackages, performance was improved in the Python code, but for some, the most time-critical pieces were rewritten in C and wrapped with Cython  (Behnel et al. 2011 ; e. g. , for convolution of images, sigma clipping, and converting time strings to binary). Furthermore, a particular effort was made to ensure astropy is threadsafe, so that it can be used on supercomputing clusters. Performance in astropy is addressed based on feedback and requests, and through contributions from community members: if you find that parts of astropy are not performant, we encourage you to open a GitHub issue to start a discussion with the astropy core maintainers.
Project Governance As part of the process of developing astropy into a longterm sustainable product, and to improve transparency and accountability, the Project agreed to write down and formalize our governance structure (partly supported by explicit funding for this purpose; see Section 3. 4). At the 2019 Astropy Coordination Meeting, input was gathered from participants on what governance structures existed in the associated Open Source Software communities, and what would fit well with the needs of Astropy. This led to a "retreat" planned for March 2020, but due to the COVID-19 pandemic, this became a series of virtual meetings of the "Astropy Governance Working Group. " This group drafted the APE 0 document (Aldcroft et al. 2021), which was eventually ratified and implemented by the "Astropy Governance Implementation Working Group" in Fall 2021.
Inclusion, Diversity, and Equity Programs With support from the Moore Foundation, the Astropy Project was allocated funding to support mentoring programs. In 2020 a call was made to submit proposals for Inclusion, Diversity, and Empowerment (IDE) initiatives for project-wide consideration on GitHub. This process was deemed the most open because it allowed for community-wide feedback to focus and improve proposal initiatives. Two highlighted programs that were selected and implemented via this process are described below. Women of Color Code (WoCCode) is a peer-mentoring network for coders from traditionally marginalized groups, most notably women of color. Participants were invited to the WoCCode Slack space and encouraged to attend monthly webinars to share skills in the context of open-source software libraries.
Astropy Representation at National Diversity Conferences: Several members of the Astropy community attended virtual conferences of the National Society of Black Physicists (NSBP) and SACNAS (focusing on Hispanic and Native American scientists across all STEM fields). Astropy representatives noted that one underlying topic came up multiple times: a major barrier for persons who come from underrepresented communities is the lack of pedagogical resources and expertise. For instance, software engineering skills and best practices are not part of the standard physics and astrophysics curriculum. Additionally, the larger astronomical community is still in the process of transitioning toward Python -based tools, hindering students not currently under advisement by someone with extensive Python expertise from getting involved with the open-source Python community.
Projects such as Learn Astropy (see Section 4. 5) could have a profound impact by empowering underrepresented groups because it provides a free, searchable, and accessible introduction to Python tools for astrophysical research. Representatives also noted that offering training and teaching materials to PIs of Research Experience for Undergraduate (REU) programs throughout the United States could be helpful for encouraging advisors to teach students how to use Astropy, create their own libraries, and use version control on open-source platforms like GitHub. Such materials could also be offered as a workshop at national diversity conferences themselves, as SACNAS solicits special session proposals each year.
Community-oriented Infrastructure The Astropy Project supports the broader ecosystem by providing preconfigured infrastructure packages that the community can use to support and maintain their own software package infrastructure. These include tools and extensions that enable generating documentation and setting up automated testing, as well as providing package scaffolding for new projects. Sphinx is a common and useful tool for generating documentation for Python packages. The Astropy Project maintains a default Sphinx configuration along with Astropyspecific extensions that can be easily added to community projects via the sphinx-astropy meta-package. This tool provides a preconfigured Sphinx setup compatible with Astropy projects, which includes several extensions useful for generating API documentation (sphinx-automodapi), allowing for numpy docstring parsing (numpydoc), embedded image handling (sphinx-gallery; pillow), advanced documentation testing support (pytest-doctestplus), and providing a custom documentation theme ideal for analysis packages (astropysphinx-theme).
Community package testing infrastructure is supported through the pytest-astropy meta-package, providing a unified testing framework with useful extensions compatible with both Astropy-and non-Astropy-affiliated community packages. This meta-package pulls in several pytest plugins to help with custom test headers (pytest-astropy-header), accessing remotely hosted data files in tests (pytest-remotedata), interoperability with documentation (pytest-doctestplus), dangling file handle checking (pytest-openfiles), data array comparison support in tests (pytest-arraydiff), subpackage command-line testing support (pytest-filter-subpackage), improved mock object testing (pytest-mock), test coverage reports and measurements (pytest-cov), and configuring packages for property-based testing (hypothesis).
Astropy Affiliated Packages As defined in Astropy  Collaboration et al. (2018) , Astropy affiliated packages are astronomy-related Python packages that provide functionality that builds upon and extends the astropy core package that have requested to be included as part of the Astropy Project community. These packages support the goals and vision of Astropy of improving code reuse, interoperability, and embracing good coding practices such as testing and thorough documentation, but are not solely developed and maintained by members of the Astropy Project team. However, a subset of the affiliated packages-the "Astropy Coordinated Packages"-are maintained by the Project. All Astropy-coordinated and other affiliated packages are listed with detailed information on the Astropy website; a table summarizing key aspects of the affiliated packages is included in the Appendix below (Table  1 ).
astroquery  (Ginsburg et al. 2019 ) has added access to roughly a dozen new missions and data services, including the JWST archive, and the Project has switched to a continuous release model: every time a change is committed to the main development branch it is published on the Python Package Index (PyPI) and available for installation. (However, formal releases are still done a few times per year. ) photutils  (Bradley et al. 2022a ) released its first stable version, indicating that the API will change less frequently, and there have been several significant performance improvements. ccdproc  (Craig et al. 2015)  released a new major version, bringing better performance to some image combination operations. The regions package  (Bradley et al. 2022b) , for manipulating ds9-, CASA-, or FITS-style region definitions  (Joye & Mandel 2003) , added new ways to manipulate regions, introduced new region types and obtained expanded visualization and interactive region editing capabilities.
reprojectʼs (Robitaille 2018) major new feature is a function to align and co-add images to create a mosaic; better support for parallelization was also added. specutils, the package that defines containers for 1D and 2D spectra  (Earl et al. 2021) , had its first stable release and the addition of classes to read JWST data. stingray  (Huppenkothen et al. 2019)  has had a major performance overhaul. The package gammapy  (Deil et al. 2017)  has had major performance improvements and has unified its API in preparation for its first stable release.
Connections with Data Archives astroquery  (Ginsburg et al. 2019)  is the Astropycoordinated package for interacting with online archives of astronomical and related data. It contains over 50 modules for querying astronomical databases, large and small. In particular, since the publication of Astropy  Collaboration et al. (2018) , significant contributions to astroquery have come from several of the major archives, including the European Space Agency (ESA), the Mikulski Archive for Space Telescopes (MAST) at STScI, the Infrared Science Archive at the NASA Infrared Processing and Analysis Center (IRSA at NASA IPAC), the Canadian Astronomical Data Center (CADC), and the Atacama Large Millimeter/Submillimeter Array archive (ALMA). These contributions represent a formal acknowledgment of the utility of a centralized tool suite for archive interaction from Python.
Software developers at STScI, the operations center of JWST, have been developing Python -based tools for JWST since 2010 (starting with the JWST Calibration Reference Data System) and have provided major contributions to Astropy from its inception. The JWST instrument calibration pipelines, exposure-time calculators, and data analysis tools are all written in Python and depend on the astropy core package and some coordinated and affiliated packages. For the astropy core package, the JWST mission has provided extensive contributions to the astropy. modeling, astropy. units, astropy. coordinates, astropy. wcs, astropy. io. fits, astropy. io. votable, astropy. stats, astropy. visualization, and astropy. convolution subpackages as well as to the general package infrastructure and maintenance.