dataesr/scientific-paragraphs-categorization huggingface dataset:
Based on the filtering results, an alert sentinel will post candidates and annotations. Once the enhanced alerts are ingested into Kowalski's database, users can query them by running MongoDB Query Language (MQL)-based queries using Kowalski's API. The package penquins 10 provides a python client to interact with the Kowalski API. Kowalski is containerized using Docker allowing for simple deployment on the cloud or on-premises. The instance of Kowalski that serves as the backend for fritz is deployed on-premises at Caltech. As of March 2023, it stores ∼38 TB of data including ∼ 555 million ZTF alerts and ∼ 20 other catalogs (e. g. , 2MASS,  Cutri et al. 2003; ALLWISE, Cutri & et al. 2012). It processes millions of requests daily, received either from fritz or ∼10 individual users for whom direct access is allowed.
AMPEL In addition to Kowalski, some science working groups in ZTF use AMPEL  (Nordin et al. 2019)  to process the alert stream and display the results on fritz. AMPEL divides its processing into four tiers. Each tier is handled by a set of units (pluggable, configurable modules) that each perform a specific task. At Tier 0 (ingest), alert packets are filtered, and selected alerts are decomposed into constituent data points (e. g. , indi-vidual photometric points) and stored. At Tier 1 (combine), data points are combined into states (e. g. , light curves). At Tier 2 (augment), data points and states are enhanced with auxiliary or derived information, e. g. , catalog matches for photometric points or model fits for light curves. At Tier 3 (react), documents are aggregated by stock (transient object being tracked) and can be reported to an external service, e.
g. , to request followup for the N most interesting objects, or simply provide a target list for a traditional scanning workflow. Users can customize each of these tiers to create a channel, a real-time analysis plan specific to their intended science program. This could consist of e. g. , a custom alert filter, datapoint association scheme, set of light curve analyses, and reporting scheme. Tiers 0-2 are driven by the alert stream, while Tier 3 is run at fixed intervals. AMPEL integrates with SkyPortal via SkyPortalPublisher, a Tier-3 unit distributed as part of the Ampel-ZTF package 11. Each stock that SkyPortalPublisher processes is saved as a candidate for a set of filters. It assumes a one-to-one mapping between AMPEL channels and (dummy) filters stored on fritz by default, but a custom set of filters may be configured instead.
Additionally, a stock is saved as a source in a set of target groups if so configured. Light curve analysis results from Tier 2 are serialized to JSON and posted as comments on the source. This behavior predates the analysis platform described in Sec. 3. 5. 3, and we expect to migrate the analysis reporting of AMPEL to this service in the near future. If photometry and image cutouts were loaded as part of the Tier 3 process, these are posted as well. This makes it possible to use AMPEL with a standalone SkyPortal instance, but this is disabled in the ZTF instance, as competing backends tend to confuse users and lead to unnecessarily high database loads. Finally, SkyPortalPublisher keeps a record of its communication with SkyPortal in the journal associated with each stock, and only attempts to post if it was updated since the last successful attempt.
Fink The GRANDMA collaboration uses the Fink broker  (Möller et al. 2020)  to select promising alerts from the 11 https://github. com/AmpelAstro/Ampel-ZTF public ZTF alert data stream and display the results on icare. The current Fink platform works in four steps. First, alerts from multiple streams are continuously ingested and stored on disk. Second, alerts satisfying the quality cuts defined by the broker team are processed by a set of science modules. These science modules-currently a dozen, spanning solar system objects to galactic and extragalactic science-are independent processing units developed by the community of users, and deployed in the Fink platform. They can work on a single input alert stream, or combine several streams together. These science modules enrich the initial alert packets using several techniques such as cross-match with external catalogs of astronomical objects, or classification using machine or deep learning based algorithms.
All addedvalues are made public. Third, enriched alert packets are filtered based on their content, and the most promising events are redistributed to the scientific community in real-time. The filtering is again community-driven, and users design and deploy filters to receive tailored information in real-time. Finally, all enriched alert packets are stored in a database for permanent access and for further analyses. Fink exposes a number of services to the users to design modules, filters, and access the data  12. Fink integrates with icare with the SkyPortal Fink Client 13. Alerts are first polled using the Fink Livestream service, and then pushed to icare using core functionalities of SkyPortal. The SkyPortal Fink Client runs as an optional microservice inside icare which simplifies the deployment and the maintenance for users.
At the core of baselayer, we find sqlalchemy 15  , an Object Relational Mapper (ORM) written in Python that allows easy interaction with a relational database, here being PostgreSQL 16   (Stonebraker & Rowe 1986 ). Several base models-which are the abstractions of tables in the database-are defined to introduce the concept of users, permissions, roles, API tokens, and groups. An addon of Python Social Auth provides the necessary models for user authentication using many openid authentication services. In both baselayer and SkyPortal, authentication through Google services is the method of choice. sqlalchemy also allows handling multiple parallel and separate connections to the database named sessions. This is an important requirement for any API-based application, where multiple users need to read/write/modify/delete from the database simultaneously.
Wherever interact-ing with the database is necessary, we open a session, add/modify/delete from that session, and then commit the changes. Changes are made to the database only once the commit method is called, and PostgreSQL is capable of resolving conflicts between multiple commit sessions if necessary. For the users to interact with the API-based application that is baselayer, we define API Handlers, which are simply python classes with the standard GET, POST, DELETE, PUT, and PATCH method an API is expected to have. Then, Tornado maps each handler to the associated API endpoint. It is capable of calling the right API method of the handler and passing in-path parameters for each API call. In SkyPortal, everytime a new feature is needed, database models and handlers to interact with them can be added to the application following the same process.
When it comes to permissions, baselayer provides python decorators that can be specified above each API method to specify what permissions or roles a user needs to use it. Similarly, database-level access restrictions can be defined in every sqlalchemy model, to ensure that users only have access to the data points that have been shared with them or with the groups they belong to. Using a microservice architecture managed with supervisord, we can run multiple instances of the application in parallel to ensure availability and reduce downtime. That way, the app can easily scale horizontally based on the number of users or when new features are added. Moreover, we can add microservices to run background operations continuously. In baselayer, this is used to run the application of course, but also the database migration manager, the webpack builder, websocket server, cron jobs, external logging, and nginx.
HEALPix-Alchemy To ensure calculations are sufficiently computationally expedient in SkyPortal, we represent the complex geometry of regions of interest (such as multimessenger skymaps) and the instruments' fields of view using the Hierarchical Equal-Area isoLatitude Pixelization (HEALPix;  Górski et al. 2005 ) framework; to this end, we have developed a PostgreSQL extension enabling HEALPix-based cross-matches to make rapid processing on these large alert databases possible (Singer.
Probability Sky Map Multi-resolution HEALPix Tiles Sky Map Tiling In version 14, PostgreSQL introduced a new multirange type, consisting of an array of ranges, with a fast aggregation function, which takes ranges as its input and returns their union as a multirange; when combined with the multi-order-coverage (MOC) representation of sky maps  (Fernique et al. 2014) , this tandem yields highly efficient database-side processing of skymap overlaps. Most multi-messenger events do not provide us with a precise localization to observe, but instead a map of probability density (see e. g. , Fig. 2 ) using MOC to divide the sky into tiles of diverse resolutions. Each tile has an associated probability density that the event occurred in it. MOC uses the HEALPix tessellation algorithm to divide the spherical observable sky into 12 diamond shapes of equal area.
Each of them can be recursively divided into 4 to obtain the level of resolution needed to represent the variation of probability from one point of the sky to another; recursively dividing these tiles into 4 sub-diamonds increases the order and therefore the resolution. At order 0, tiles have indexes from 1 to 12, and at order 1, from 1 to 64. For instance, tile 1 at order 0 is now made of tiles 1 to 4 at order 1, and tiles 1 to 16 at order 2, and so on recursively until the highest order is order 29, where a tile side size is only 0. 38 milliarcseconds; this minimum resolution is set by requiring that the corresponding pixel index can be stored as a 64 bit signed integer without overflow. Thanks to the recursive indexing schemes of the HEALPix algorithm, when saving MOC maps in the database, we can represent each tile as the range of tiles that it is composed of at order 29.
This is convenient as it allows us to represent tiles as a probability density associated with a range set. To do so, Healpix-Alchemy combines the power of Healpix and SQLAlchemy, a python package used to communicate with PostgreSQL databases using python. Healpix-Alchemy uses the range sets feature of PostgreSQL databases to represent tiles and single points with 64 bits integers and provides the tools necessary to cross-match tiles and points. For software stacks that rely on relational databases, this range sets representation of HEALPix tiles is the best (and only viable) implementation to perform rapid spatial queries. As MOC has become the standard representation for skymaps in multi-messenger astronomy  (Fernique et al. 2022) , Healpix-Alchemy enables efficient queries of instrument footprints for rapid cross-matches with historical γ-ray burst, neutrino, and gravitational-wave skymaps.
Statistics on each source are collected and updated upon the insertion of new data points. This includes, e. g. , the brightest magnitude, the time when the source was last detected, and so on. These statistics are updated on insertion to avoid the costly re-calculation over hundreds of data points per source (in fritz, there is currently an order of 4 × 10 8 individual photometry points across all sources). Such statistics are used to quickly filter sources based on their light curves, without needing to calculate each statistic for each source in each query. Another way to store large photometric datasets has recently been added to SkyPortal: the photometric series. This data product will be used by surveys with very high cadences and continuous coverage of the same field, e. g. , the Transiting Exoplanet Survey Satellite (TESS;  Ricker et al.
2014 ) takes images at a maximum cadence of 20 s over 27-day series. In this case, saving each of the ∼ 10 5 data points as individual rows in the database becomes unmanageable, both in terms of storage and in terms of communicating the data in and out of SkyPortal. Instead, we store the underlying data as HDF5 files and keep a queryable record in the main database of the file (along with some statistics). Spectra are uploaded either manually or via API. The spectra are associated with each source but the platform also enables the users to provide comments, allowing users to discuss either the source or the individual spectra. There are also standard line lists available to perform line identification, enabling classification.
GCN Events Much of the existing infrastructure for communicating multi-messenger observations originated from the GRB detection community in the 1990s  (Barthelmy et al. 1995) , with the decades-old "Gamma-Ray Coordinates Network" (GCN) continuing to play a central role in modern time-domain astronomy. This system includes the machine-readable "GCN Notices," which uses three legacy communication formats (text, binary, and VO-Events;  Allan et al. 2017) , each with its own communication protocols. Recently, there have been concerted efforts to reimagine this system as a modern "General Coordinates Network," transitioning to a unified communication protocol using open-source Apache Kafka, with the ultimate aim of relaying all data via the industrystandard JSON format 18. SkyPortal is fully integrated with this new GCN Kafka format and can ingest notices shared via this protocol.
Which of the available streams should be ingested is specified in the configuration of the application. These GCN notices typically contain either full MOC sky maps  (Fernique et al. 2022) , coordinates in right ascension and declination along with an associated error radius, or the 18 https://gcn. nasa. gov/ vertices of a region shaped like a polygon. In all cases, a new entry is added to the database containing the localization itself (transformed to a MOC for efficient and standardized queries) and other available event properties such as localization area, median distance, or astrophysical probability. These events then can be subsequently queried, cross-matched to sources or galaxies, and used to coordinate subsequent follow-up. SkyPortal supports the analysis of these events by allowing for different queries within the localization map: telescope observation footprints, galaxies, and transient sources.
Observation: Planning and Execution A central component of multi-messenger astronomy is the efficient coordination of follow-up for a given event, especially for gravitational-wave and GRB triggers which are initially poorly localized. Astronomers may have access to both wide field-of-view instruments and narrow field-of-view instruments of different depth and wavelength coverage. Developing strategies for optimal follow-up with a particular telescope has been an area of active research in recent years (e. g. ,  Coughlin et al. 2018 Coughlin et al. , 2019c;; Almualla et al. 2020). SkyPortal has extensive functionality to automatically perform these optimizations in a user-friendly GUI. Upon ingestion of localizations, SkyPortal can generate optimized observing schedules using the opensource "Gravitational-wave Electromagnetic Optimization" (gwemopt) software package  (Coughlin et al.
Galaxy Catalogs Due to the relatively local nature of multi-messenger astronomy, both follow-up focusing on galaxy cata-logs  (Arcavi et al. 2017; Valenti et al. 2017 ) and comparison of transient locations to catalogs is important. Within the database, galaxies can have several attributes: name, alternative name, catalog name, RA, Declination, redshift (z), and distance in Mpc (if known), as well as other properties and their associated margin of error if provided. Concerning the usage of catalogs, in fritz, we use the Census of the Local Universe (CLU)  (Cook et al. 2017 ) catalog, which is complete to 85% in star formation and 70% in stellar mass at 200 Mpc; this catalog currently remains proprietary. In icare, GLADE+  (Dalya et al. 2021 ), an open-source catalog containing more than 22 million galaxies, complete up to 44 Mpc and nearly 90% complete at even 500Mpc, is used.
InferenceData format; Fig. 8 ), and/or diagnostic plots. Site administrators first create analysis services, by providing a URL (and authentication credentials) to an existing service, the schema of required and optional parameters, and data types to be sent from SkyPortal to the analysis service. When an analysis is initiated (either via the front end or via API), all data viewable by the user in the analysis data types are packaged and sent to the analy- sis service. Via a unique unauthenticated webhook, the analysis service then asynchronously returns analysis results. (Analysis results generated without SkyPortalpackaged source data and without a webhook can also be saved by an authenticated token. ) The results are stored in a persistent datastore by SkyPortal and can be retrieved by analysis id or by querying for analysis results by source name.
AI Summarization & Embeddings One challenge for end-users in the face of hundreds (or thousands) of active events is maintaining a current understanding of the state of knowledge and activity for each event. To facilitate "quicklook" insights into sources, following the emergence and general availability of highly capable large language models (LLMs), the SkyPortal team released a pre-built analysis service to provide human-readable summaries of individual sources, using redshift, classifications, and comments. When such an analysis is initiated on a source, the summarization service formats available source data along with a predefined prompt requesting a summary; the OpenAI API (openai. ChatCompletion) returns a summary which is then displayed on the source (fig. 9 ). This summary can be easily edited, with the summary history being saved.
A sitewide API key to OpenAI can be used or individual users can configure their own key and specific completion parameters. SkyPortal defaults to use the gpt-3. 5-turbo model but users can opt to employ the more capable (but currently slower) gpt-4 model. Once a summary is created, an OpenAI embeddings service (using the text-embedding-ada-002 model) is called with the summary that returns a vector embedding of the source in n = 1536 dimensions. These em-beddings are saved in a pinecone database. Using these embeddings, similar sources can be retrieved by rankordering cosine distance metrics or natural-language queries (first embedded, then compared against the summary corpus) can be answered. Future embeddings will incorporate photometry, spectra, and annotations to further facilitate AI-guided exploration of SkyPortal datasets.
Recognizing that many workflows also happen programmatically outside of SkyPortal the major functionality also is accessible via a well-documented API infrastructure 22. It will come as no surprise that working across teams in different institutions-sometimes with aligned and sometimes with competing incentivesrequires modern frameworks like SkyPortal to enable the establishment and curation of strict data access controls. Every photometry, spectroscopy, comment, annotation, classification entry in SkyPortal is permissioned by group. SkyPortal is, first and foremost, scientific software developed to enable astronomical data science. However, a close secondary goal is to train students in proper software development practices, which has been identified as a gap between coursework and eventual readiness for academic and industry practice  (Licorish et al.
2022 ). These required skills include software engineering skills (e. g. , identifying requirements, prototyping, designing, and using version control) as well as transferable and soft skills (e. g. , teamwork, and communication). SkyPortal has had significant contributions from students at a variety of levels, with the faculty / permanent researchers associated with the project focused on bringing in students into the fold. For example, two sets of ten students from the Leonard de Vinci Engineering School have participated in the development of the features, focusing on the icare version of the application. This particular effort has yielded multiple internships at member institutions for these students. However, bringing on students continues to raise a number of challenges for a small project like SkyPortal.
2014) , this tandem yields highly efficient database-side processing of skymap overlaps. Most multi-messenger events do not provide us with a precise localization to observe, but instead a map of probability density (see e. g. , Fig. 2) using MOC to divide the sky into tiles of diverse resolutions. Each tile has an associated probability density that the event occurred in it. MOC uses the HEALPix tessellation algorithm to divide the spherical observable sky into 12 diamond shapes of equal area. Each of them can be recursively divided into 4 to obtain the level of resolution needed to represent the variation of probability from one point of the sky to another; recursively dividing these tiles into 4 sub-diamonds increases the order and therefore the resolution. At order 0, tiles have indexes from 1 to 12, and at order 1, from 1 to 64.
For instance, tile 1 at order 0 is now made of tiles 1 to 4 at order 1, and tiles 1 to 16 at order 2, and so on recursively until the highest order is order 29, where a tile side size is only 0. 38 milliarcseconds; this minimum resolution is set by requiring that the corresponding pixel index can be stored as a 64 bit signed integer without overflow. Thanks to the recursive indexing schemes of the HEALPix algorithm, when saving MOC maps in the database, we can represent each tile as the range of tiles that it is composed.
MWC, JL, and VS are supported by the National Science Foundation with grant number OAC-2117997; MWC is also supported by PHY-2010970. MWC and RWK were supported by the Preparing for Astrophysics with LSST Program, funded by the Heising Simons Foundation through grant 2021-2975, and administered by Las Cumbres Observatory. The Gordon and Betty Moore Foundation, through both the Data-Driven Investigator Program and a dedicated grant to develop SkyPortal, provided critical funding for this project without which this project could not have succeeded. J. S. B. , G. N. , A. C-Q. , D. A. D. were partially supported by the Gordon and Betty Moore Foundation. BP thanks the Northeastern Lawrence Co-op Fellowship for their continuous support. JP thanks LSST-France and CNRS/IN2P3 for supporting Fink. The Kilonova-Catcher program is supported by the IdEx Université de Paris Cité, ANR-18-IDEX-0001 and the MITI CNRS Sciences participative program.
obtained with the Samuel Oschin Telescope 48-inch and the 60-inch Telescope at the Palomar Observatory as part of the Zwicky Transient Facility project. ZTF is supported by the National Science Foundation under Grant No. AST-2034437 and a collaboration including Caltech, IPAC, the Weizmann Institute of Science, the Oskar Klein Center at Stockholm University, the University of Maryland, Deutsches Elektronen-Synchrotron and Humboldt University, the TANGO Consortium of Taiwan, the University of Wisconsin at Milwaukee, Trinity College Dublin, Lawrence Livermore National Laboratories, IN2P3, University of Warwick, Ruhr University Bochum and Northwestern University. Operations are conducted by COO, IPAC, and UW.
To that end, we developed a calibration protocol and numeric tools to map the 3D holographic stimulation path with the multiplane two-photon imaging path. This calibration empirically accounts for aberration and deformities that are introduced both by the SLM and associated lenses, but also by the optomechanical defocusing method (in this example an electro tunable lens; ETL). This strategy provides both an improved calibration over less thorough procedures, and even more critically accurate measurements of the size of the holographic disk throughout the useable volume. While this procedure is quite slow, it is fully automated and can be run overnight (see Note 2. 3. 1). Scripts are available at https://github. com/ adesniklab/3D-SHOT/AutoCalib In addition to the 3D-SHOT and 2P imaging system described above, the calibration requires a substage camera (Fig.