dataesr/scientific-paragraphs-categorization huggingface dataset:
Beyond More analysis tools leveraging the libraries to create derived datasets and produce quantitative insights into OME-Zarr data are already planned. For example, the BigStitcher  (Hörl et al. 2019 ) toolchain which currently relies on a custom XML/HDF5 format will gain support for OME-Zarr, making it easier to parallelize distributed access for HPC and avoid additional file conversion. This is particularly important for very large datasets and means that NGFF will be natively exported by BDV in Fiji. Work on this is underway in the bigdataviewer-n5 25  and bigstitcher-spark 26  repositories. At the time of writing, BigStitcher has support for large file export and limited direct support of OME-Zarr datasets via the bigdataviewer-omezarr module 27. This module provides a transient bridge between the current XML-based dataset definition and OME-Zarr images until native support is implemented.
It allows multiple images stored in one NGFF dataset to be defined as separate view setups in BDV/ BigStitcher (Fig. 10 ). Other tools such as RS-FISH  (Bahry et al. 2022)  for spot detection and STIM for visualizing and reconstruction of spatial transcriptomics datasets  (Preibisch et al. 2022 ) are currently being extended to support OME-Zarr and other formats. Additionally, registration tools that generate the developing spatial transformation specification are planned to enable quantitative comparison of compatible datasets. To track these developments, the community will maintain https:// ngff. openm icros copy. org/ tools for finding the status of other new and exciting developments.
Python ome-zarr-py 28  , available on PyPI, was the first implementation of OME-Zarr and is at the time of writing considered the reference implementation of the OME-NGFF data model. Reading, writing, and validation of all specifications are supported, without attempting to provide complete highlevel functionality for analysis. Instead, several libraries have been built on top of ome-zarr-py. AICSImageIO is a popular Python library for general 5D bio data loading. In addition to loading OME-Zarr data, AICSImageIO provides OmeZarrWriter using ome-zarr-py under the hood. In this way, format conversion is possible by loading data with AICSImageIO and immediately passing the Dask array to OmeZarrWriter, though improvements in the metadata support are needed. Fractal 29  is a framework to process high-content imaging data at scale and prepare it for interactive visualization.
Fractal is focused on processing images in the OME-Zarr format and saving results in forms of images, label images and feature tables back into OME-Zarr, while keeping orchestration of these processing steps cluster friendly via slurm 30. It allows users to build workflows to process images in OME-Zarr files at the TB scale to segment images and make measurements. As a result, large-scale OME-Zarr image datasets can be processed by Fractal and then browsed interactively across pyramid levels in viewers like napari 31  (Fig. 2 ). Fractal is in its early stages and currently contains workflows to be controlled from the command line interface. A web client to build workflows and manage the processing is currently being built. BFIO 32 is a Python library that supports reading of all 160 + Bio-Formats supported file formats, with opinionated but highly optimized reading and writing of OME-TIFF and OME-Zarr for use in large scale applications.
11 ). It uses the ome-zarr-py readers and writers for raster data (images and labels) and implements features of the upcoming version of the NGFF specification, including tables and transformations. Furthermore, it experimentally supports the representation of additional modalities commonly found in spatial omics data, such as points (e. g. transcripts locations) and shapes (e. g. spatial transcriptomics circular array capture locations and generic polygonal ROIs). The library implements a set of operations such as aggregation of molecular measurements (associating transcripts locations to cells) and efficient spatial queries that are interoperable across representations. Finally the Spatial-Data framework provides readers for datasets produced by the most popular spatial omics technologies, a static plotting library based on matplotlib (Hunter 2007) as well as a napari plugin for interactive visualization and annotation of the data.
Example An example analysis of a dataset from the Image Data Resource (IDR)  (Williams et al. 2017 ) helps to illustrate the possibilities available to the end-user from an analysis perspective. A light sheet fluorescence microscopy image published by  McDole et al. (idr0044)  39 is composed of 2 channels, approximately 1000 z-sections and 500 timepoints. Each 2D-plane is of dimension 2 k × 2 k. Numerous z-sections were acquired but the relevant planes are the middle z-sections. Such data is particularly useful for teaching and training purposes, so it is usually only necessary to access a limited subset of an image. Traditionally, two options are available in order to analyze an image. One can download the full image. This approach is far from ideal since only a portion of the data is required for analysis.
Generators For the foreseeable future, datasets will exist in one of the many forms they exist in today. As outlined in  (Moore et al. 2021) , translating those on the fly brings delays in visualization and analysis that can be solved by performing a single conversion to OME-Zarr. This process captures all metadata that Bio-Formats is aware of in an open format, and decouples the user from the version of the vendor software used to capture the data. Several generation tools (Table  4 ) are available based on the particular environment you are working in.
Bioformats2raw Bioformats2raw 43 is the original command-line utility to convert various image file formats into OME-Zarr format. Bioformats2raw offers rich and flexible parameter options, giving the user extensive control over the conversion process as well as freedom to specify various features of the output Zarr datasets. A few of the interesting input parameters are the chunk dimensions, the number of resolution levels, the compression algorithm, and the number of workers. Bio-formats2raw can read all proprietary file formats supported by Bio-Formats as well as a select few file format readers supported only in bioformats2raw, including the 3D-Histech. mrxs format. The input, therefore, can be single or multiple series as well as high-content screening (HCS) data. The conversion will be performed according to the respective OME-NGFF specification.
Kerchunk Within the Python ecosystem, it is also possible to "simulate" an OME-Zarr dataset with kerchunk 46. This library pre-processes existing data formats like TIFF and HDF5 to generate a JSON file containing all metadata and chunk locations. Support for other file formats can be added if each chunk can be represented by a combination of path to a file, location in that file, and length of the chunk. Using this mechanism, it is possible to leave data in a monolithic format but still achieve some of the benefits of OME-Zarr. Support in other programming languages is possible based on community interest.
Examples of shared OME-Zarr data Not just individual users or research projects are faced with the issues of format compatibility. Large-scale bioimaging resources are also moving to OME-Zarr to ease access across a range of storage options. Below we discuss several of the ways in which these and other institutions are sharing their data with the OME-Zarr format as examples of what is possible as well as where you might find existing data today, all summarized in Table  5. However, as with the tools above, these and other resources are being actively updated. Users interested in re-using datasets can refer to https:// ngff. openm icros copy. org/ data for an up-to-date version of this table maintained by the community or submit new resources as they become available. Though central registry does not exist for other file formats, the ease of access to OME-Zarr on the web, e.
Cell painting gallery The Cell Painting Gallery 49 contains Cell Painting  (Bray et al. 2016 ) images and image-based profiles from many publicly available datasets, hosted by AWS Open Data Registry. Currently, the LINCS Dataset  (Way et al. 2022 ) is available in the Cell Painting Gallery in OME-Zarr format. In LINCS, 110,012,425 A549 human lung carcinoma cells across 136 plates were treated with 1,571 compounds across 6 dose points. Morphology was captured by a standard Cell Painting workflow of five fluorescent channels covering eight organelles. Image data was converted to OME-Zarr using bioformats2raw with the Distributed-OMEZarrCreator wrapper  (Weisbart and Cimini 2022). 1,790 morphological measurements were taken using CellProfiler  (Kamentsky et al. 2011)  which are also available in the Cell Painting Table  5  List OME-Zarr resources data belonging to the authors, broken down by storage types in the order they are described below along with rough estimates of their size at the time of publication These include catalogs and dashboards which will help the reader discover datasets as well as some resources which are migrating to OME-Zarr.
DANDI allows any researcher to view or compute on the data. Lightly pre-processed data and metadata are being made available via DANDI for the BRAIN Initiative and other projects. Two such datasets were produced by the BRAIN Initiative Cell Census Network (biccn. org) comprising 370 TB of data. About 60% of a single human brain hemisphere is available in a dataset with vascular, neuronal, and cellular stains (290 TB; an example 3D slab shown in Fig. 13 ) with an additional 65 TB from the brains of two other participants. These data can be visualized by anyone on the planet due to horizontal scalability provided by AWS S3, the use of a streaming data storage format (OME-Zarr), and interactive visualization tools such as Neuroglancer, ITK-VTK-Viewer, and others described earlier. For the OME-Zarr data, each such object provides an option to directly open the object in the ITK-VTK-Viewer or for checking using the OME-Zarr Validator.
BIA and IDR An example of an institute providing a large-scale cloud repository is the European Bioinformatics Institute (EBI) who has done so by investing in object, or "cloud", storage. The Image Data Resource (IDR)  (Williams et al. 2017 ) and the BioImage Archive (BIA)  (Hartley et al. 2021 ) are free public digital repositories for biological images hosted at EBI that are associated with peer-reviewed publications or are of value beyond a single experiment. The BIA strives to provide hosting for all such publications while the IDR focuses on a selection of reference datasets which are further curated. They accept submissions of many different image formats to enable easy and fast sharing. However, most of these formats have issues relating to metadata and to online visualization as discussed above.
50 https:// bids-speci ficat ion. readt hedocs. io/ en/ stable/ 04-modal ityspeci fic-files/ 10-micro scopy. html# file-forma ts. 51 https:// github. com/ dandi sets. 52 https:// github. com/ dandi zarrs. 53 https:// regis try. opend ata. aws/ allen-nd-open-data/. 54 https:// glenc oesof tware. com/ ngff.
Moreover, it can accommodate extensive image level metadata, and has substantial community support. To enable easier sharing and investigation of long-term solutions, images are converted from selected datasets, including most machine learning datasets, into OME-Zarr format. The converted representation is stored transiently on fast, S3 compatible storage. This enables serving the images for on-the-fly visualization in embedded viewers, visualization by desktop clients that can access OME-Zarr remotely such as napari or MoBIE, and provides fast access for on-the-fly analysis via Jupyter notebooks. The converted image representations are transient due to the need to balance resources between cheaper scalable storage for long term archival and fast storage for access to in-demand datasets. The OME-Zarr converted image representations are currently presented on separate websites, with plans to integrate these within the primary dataset pages.
Images can be displayed using Vizarr, and an option to view them with the ITK viewer is also provided. As with all submissions, each image representation has a unique URI therefore the users can access and view them with their favorite online viewers, without the need to download. Already it is possible to submit datasets that have been converted into OME-Zarr and as this solution matures, submitters will be encouraged to pre-convert complicated datasets to lower the overhead on the curation teams.
SSBD SSBD is a platform for Systems Science of Biological Dynamics to share and reuse bioimaging data hosted at RIKEN, Japan  (Tohsato et al. 2016). It consists of two systems; SSBD:database for sharing highly reusable data with rich metadata and SSBD:repository for rapid sharing of all bioimaging data in journal papers. To support the development and validation of software tools for NGFF, we converted selected datasets in SSBD:database into OME-Zarr format using the bioformats2raw converter. The datasets include those obtained with light sheet fluorescence microscopy, STED super-resolution microscopy, FIB-SEM, and other state-of-the-art microscopies. They are currently stored and available on S3-compatible storage in the RIKEN HOKUSAI SS system.
Zebrahub Zebrahub 55  is a comprehensive atlas of zebrafish development  (Lange et al. 2023 ) paired with a fully interactive website to explore both single-embryo single-cell RNA sequencing datasets and light-sheet microscopy time-lapses. Light-sheet microscopy is a powerful imaging technique that, when combined with transgenic zebrafish expressing fluorescent proteins, can reconstruct the entire embryonic development at a cellular level. This results in terabyte-scale time-lapse datasets that contain 3D images acquired at regular intervals, often every 30 s. Acquiring multiple channels is necessary to answer specific biological questions, which further increases the data's size and dimensionality. Storing, processing, visualizing, and disseminating such data becomes challenging. The OME-Zarr format addresses several of these issues by providing a well-established and maturing standard storage format supported by various image processing and visualization tools.
Hosting your own Alternatively there are existing software solutions that expose an object-storage interface to files on disk like Minio 56. The European Molecular Biology Laboratory in Heidelberg is currently hosting all of the reference data for the development of the MoBIE tool by running Minio on their own hardware. Examples of datasets formatted in OME-Zarr include https:// github. com/ mobie/ covid-if-proje ct, https:// github. com/ mobie/ htm-test and https:// github. com/ mobie/ spati al-trans cript omics-examp le-proje ct. In the README of those repositories, a list of all S3 OME-Zarr URLs can be found which can be opened with viewers other than MoBIE. Sample data for the SpatialData library can also be found on EMBL's S3 resource. A list is available under https:// github. com/ scver se/ spati aldata-noteb ooks/ tree/ main/ datas ets.
Alternatively, dedicated, bioimaging server solutions are available for hosting and serving OME-Zarr data. As mentioned above, webKnossos makes all datasets accessible in the OME-Zarr format. To import data into webKnossos a conversion is required, whether automatically on browser 1 3 upload or manual through filesystem import to OME-Zarr or WKW, the "webKnossos Wrapper" format. Once imported into webKnossos, datasets can be made publicly available via the dataset settings. This is an easy way to publish datasets directly from an institutional HPC storage. A collection of published OME-Zarr data is available at https:// zarr. webkn ossos. org/. Another possible mechanism for hosting OME-Zarr data is OMERO  (Allan et al. 2012). The OMEZarrReader plugin to Bio-Formats is included in OMERO since version 5.
6. 5, enabling import of OME-Zarr data into any OMERO including the IDR. These are stored in OMERO's internal repository with other file types and appear as images, or in the case of HCS data plates, in the standard hierarchy. Users can then use the OMERO API or clients (OMERO. figure, QuPath, or even R) to access the data. To enable OME-Zarr access to end users, services are available which expose an OME-Zarr compatible endpoint to clients.
Brain Image Library Institutes with large scale parallel (HPC) file systems like the Pittsburgh Supercomputing Center, home to the Brain Image Library 57  (BIL), are building tools to share OME-Zarr formatted data without object storage and without modifying the existing data repository 58. Still in early development, these services will make 1000 s of whole brain microscopy datasets, including murine, NHP and human, some expected to be over a petabyte in size, from the BICCN (BRAIN Initiative Cell Census Network) (BICCN Data Ecosystem Collaboration et al. 2022) and BICAN (BRAIN Initiative Cell Atlas Network) available in OME-Zarr where appropriate starting this year. To achieve this, microservices will automatically provide an OME-Zarr representation of the day on the fly to prevent data duplication.
HuBMAP The HuBMAP ("Human Biomolecular Atlas Program") consortium (HuBMAP Consortium 2019) adopted openstandard imaging formats and developed novel web-based technologies in order to simplify the server-side infrastructure traditionally required to support visualization of such a bioimaging resource. Consideration of infrastructure costs and a commitment to FAIR data access principles led the consortium to adopt OME-TIFF as an internal standard. In conjunction, members of the consortium developed Viv  (Manz et al. 2022 )-a client-side bioimaging visualization library-to remove a dependency on server-side rendering and enable flexible browsing of multi-terabyte datasets directly within the HuBMAP data portal website. Currently, HuBMAP is working to adopt OME-Zarr in order to support a wider diversity of modalities (e.
OpenOrganelle The OpenOrganelle  (Heinrich et al. 2021 ) from the Janelia Research Campus hosts all of the datasets listed under https:// openo rgane lle. janel ia. org/ datas ets on AWS's Open Data Registry. The data is visualizable in the browser using neuroglancer or via Fiji N5-plugins. The data is currently stored in a precursor to OME-Zarr known as the N5 format with "NGFF-compatible" metadata with conversion to OME-Zarr on the future roadmap. Data is stored at BIL in an alternative format and is dynamically converted to OME-Zarr chunk-by-chunk and delivered to clients upon request 1 3.
Discussion Over the last twenty years, the number of bioimaging file formats has been a constant source of confusion and frustration. While that has often been a struggle that each user manages in isolation, increasingly data sizes from more sophisticated hardware and more advanced modalities are leaving users with significant infrastructure burdens for efficiently converting and sharing their imaging data. Data acquisition systems often design formats specifically for writing data quickly to timely capture the scale and the breadth of modern experiments. The tension between the requirements of quickly writing and quickly reading bioimaging datasets force both data providers and consumers to be aware of the costs of converting, relinking, downsampling, or otherwise modifying datasets for reuse.
com/ Polus AI/ nyxus. 36 https:// github. com/ ome/ ZarrR eader. 37 https:// github. com/ saalf eldlab/ n5-ij. 38 https:// github. com/ saalf eldlab/ n5-viewer. 39 https:// idr. openm icros copy. org/ webcl ient/? show= proje ct-502. 40  https:// www. dask. org/ 41 https:// colab. resea rch. google. com/. 42 https:// mybin der. org/.
An up-to-date version of the table is maintained at https:// ngff. openm icros copy. org/ data and contributions are welcome Gallery. More Cell Painting datasets in the Cell Painting Gallery are planned for both conversion to OME-Zarr and browsability through IDR. Storage Catalog Dashboards/datasets Zarr files Size DANDI DANDI, "Distributed Archives for Neurophysiology Data Integration" is supported by the US BRAIN Initiative as an open-access data repository for publishing and sharing neu- rophysiology data including electrophysiology, optophysi- ology, and behavioral time-series, as well as images from immunostaining experiments. As datasets get larger, down- loading whole datasets for analysis or visualization becomes increasingly impractical. Thus, DANDI allows data to be operated on in the cloud using standardized methods and computational servers near the data.
It enforces the organ- ized structure of the Brain Imaging Data Structure (BIDS) and its microscopy extensions (Bourget et al. 2022), with the latest specification 50 allowing such data to be stored in the OME-Zarr format. DANDI uses standardized metadata (JSON, JSON-LD), data organization (BIDS, or BIDS-like subset), and data storage formats (e. g. , OME-Zarr) that allow the metadata to be queried before data access, and data to be accessed partially and at different resolutions. The archive stores open and embargoed data on Amazon S3 in the US and is supported by the Amazon Open Data program. The DANDI API has full programmatic support for uploading and accessing Zarr objects, and researchers can use the DANDI command line interface tool to upload BIDS organized Zarr data to DANDI. All datasets are also available as DataLad (Halchenko et al.
This paper presents a Python package called pycapacity, which provides a set of tools for evaluating task-space physical ability metrics for humans and robots, based on polytopes and ellipsoids. The aim of pycapacity is to provide a set of efficient tools for their evaluation in an easy to use framework that can be easily integrated with standard robotics and biomechanics libraries. The package implements several state of the art algorithms for polytope evaluation that bring many of the polytope metrics to the few milliseconds evaluation time, making it possible to use them in online and interactive applications. The package can be easily interfaced with standard libraries for robotic manipulator rigid body simulation such as robotictoolbox  (Corke & Haviland, 2021)  or pinocchio  (Carpentier et al.
, 2019) , as well as human musculoskeletal model biomechanics software opensim  (Delp et al. , 2007)  and biorbd  (Michaud & Begon, 2021). The package can also be used with the Robot Operating System (ROS)  (Quigley et al. , 2009). The package additionally implements a set of visualization tools for polytopes and ellipsoids based on the Python package matplotlib intended for fast prototyping and quick and interactive visualization.
The evaluation of polytopes is often a computationally expensive task, as their resolution requires using different vertex and facet enumeration algorithms  (Fukuda, 2004). Therefore, their computation time is often the limiting factor for the use of polytopes in real world applications, especially when it comes to their online use. Furthermore, even though there are several open-source projects that implement polytope evaluation algorithms, such as pypoman  (Caron, 2023) , Multi-Parametric Toolbox 3  (Herceg et al. , 2013)  or cddlib  (Fukuda, 1997; Zurich, 2023) , they are often very generic and not easy to use with standard physical ability polytopes. On the other hand, more specific polytope resolution software solutions, such as Constrained Manipulability package (Philip Long, 2023; P.