dataesr/scientific-paragraphs-categorization huggingface dataset:
Building Deconvolution Inputs Aside from the pseudo-bulk data and the corresponding proportions, deconvolution methods require different references to process the inputs. Within the pipeline, these references are encoded as C0, C1, C2, phenData and refVar. C0 is a generic single-cell matrix, with rows being genes and columns being cells. C1 is the averaged (arithmetic mean) gene expression matrix by cell types, with rows being genes and columns being cell types. C2 is a list of marker genes obtained from differential gene expression tests through the R package Seurat, specifically using a combination of NormalizeData and FindAllMarkers functions. phenData is the metadata object that is associated with C0. Finally, refVar is the row standard deviations matrix by cell types, similar in structure to C1.
Evaluation of Performance -Metrics The pipeline will evaluate the predictions through various metrics, where the overall score of a method for a given metric is the mean over all samples (columns) within the pseudo-bulk proportions matrix. The metrics available in the pipeline are: All the metrics have been estimated in R and the functions can be found here https://github. com/Functional-Genomics/CATD_snakemake/blob/main/Modules/Res_ explore/Res_explore. R For each metric, the scores of the methods are saved as named list objects throughout the pipeline. Graphs are also constructed from these lists for initial viewing of the results.
We set 4 different parameters for the simulation: the number of cells to be sampled from the single-cell (pool size (p)), the number of samples in each pseudo-bulk matrix (s)-the default is 1,000 samples, the variance of the proportions (pVar) for the cell-types and finally a logical vector to decide if we use all the cell-type or not (sampleCT, True or False). b. Scatter plot of mean and variance of expression data from 5 different simulation scenarios and matched real bulk data. All the simulated pseudo-bulk data derive from the same mouse 10X single-cell dataset and the real bulk RNA-seq derives from the same study as well (Hrvatin2018). c. boxplot plot showing the cosine similarity of gene expression for the samples of each simulation as well as the cosine similarity of expression across the real bulk samples.
A growing community is constructing a next-generation file format (NGFF) for bioimaging to overcome problems of scalability and heterogeneity. Organized by the Open Microscopy Environment (OME), individuals and institutes across diverse modalities facing these problems have designed a format specification process (OME-NGFF) to address these needs. This paper brings together a wide range of those community members to describe the cloud-optimized format itself-OME-Zarr-along with tools and data resources available today to increase FAIR access and remove barriers in the scientific process. The current momentum offers an opportunity to unify a key component of the bioimaging domain-the file format that underlies so many personal, institutional, and global data management and analysis tasks.
Introduction The exchange of scientific data is one of the key hallmarks of scientific practice in the twenty-first century. In 2016 Wilkinson and colleagues provided guidelines for making scientific data findable, accessible, interoperable, and reusable (FAIR) that provide a foundation for future scientific discoveries through data integration, reanalysis and the development of new analytic tools  (Wilkinson et al. 2016 ). In the case of biological and biomedical imaging (collectively, "bioimaging"), the size, complexity and heterogeneity of datasets present several impediments towards that goal, the most immediate of which are the specification and construction of data formats that can meet the requirements of FAIR data  (Könnecke et al. 2015). Any format must support both the pixel measurements that are the core of bioimaging data as well as relevant imaging metadata.
Specifications that enable storage of experimental, acquisition, and analytic metadata are necessary. The implementation of metadata specifications must be both flexible and scalable to handle the large and heterogeneous volumes of analytic metadata generated, for example the definition of the segmentations and annotations on individual cells and tissues that are quite common in biological imaging workflows. Critically, the set of formats available to end users must support local data storage  (laptops, desktop computers, etc. )  as Extended author information available on the last page of the article 1 3 well as cloud-based storage that is becoming more heavily used as dataset volumes grow. Previously, the Open Microscopy Environment (OME) developed OME-TIFF as an open-source file format in bioimaging.
Accompanied by reference software implementations, OME-TIFF is primarily for use in fluorescence imaging workflows and has recently been updated to enable whole slide imaging technologies  (Besson et al. 2019). This format combines the fundamentally 2D TIFF format with metadata cast in XML in the TIFF header. Its structure makes it appropriate for many applications, where the plane-based access pattern is appropriate. For bioimaging applications that require large non-planar access to volume data, e. g. , arbitrary slicing from userdefined angles, a more sophisticated "chunking" of the data is required that defines how data is stored in accessible and reasonable subsections. This means that large, multi-Gigabyte up to Petabyte bioimaging datasets are not accessed all at once but can be accessed in reasonably sized planes or sub-volumes.
In the case of TIFF, the chunk is a tile of the 2D plane allowing data access across time-lapse series or 3D volume. N-dimensional formats like HDF5 ("Hierarchical Data Format") provide much more flexibility and allow chunking across different dimensions chosen by the user. While TIFF and HDF5 are well established, the chunking strategies depend on fast random access to the entire file that is common in laptops, desktop machines and large cluster file systems, but is not provided by large scalable cloudbased object storage. Over the last few years, a new data format, Zarr 1  , has been developed for the storage of large N-dimensional typed arrays in the cloud. The Zarr format is now heavily adopted across many scientific communities from genomics to astrophysics  (Miles et al. 2023). Zarr stores associated metadata in JSON and binary data in individually referenceable "chunk"-files, providing a flexible, scalable method for storing multidimensional data.
In 2021, OME published the first specification and example uses of a "next-generation file format" (NGFF) in bioimaging using the Zarr format  (Moore et al. 2021). The first versions of this format, OME-Zarr, focused on developing functionality that tests and demonstrates the utility of the format in bioimaging domains that routinely generate large, metadata-rich datasets-high content screening, digital pathology, electron microscopy, and light sheet imaging. The discussions necessary to arrive at these specifications have also presented an opportunity to build a coherent development community under the OME-NGFF umbrella, combining a growing range of use cases and requirements with an open, transparent, but technically valid development process. The result has been a thriving community based on open development and open-source principles  (Rueden et al.
2019). This open, collaborative approach has been essential to tackle the addition of complex additional metadata to OME-Zarr. This was important as neither TIFF nor HDF5 has specifications for many of the derived data types that are generated in an analysis workflow, e. g. , regions of interests (ROIs), labels and other derived data which are crucial in modern analysis workflows. In most cases accessory files are generated to handle these limitations but as data volumes grow, these create additional problems for management and linkage of data. Using the established development process, this functionality was first formally adopted into the OME-NGFF specification, then added to the OME-Zarr implementation, but can equally be applied to other formats like HDF5 in the future. In this paper, we review the current status of the OME-Zarr format and focus on resources that are now available to users for creating, accessing, and visualizing data stored in OME-Zarr (Fig.
Growth of a compatible solution The development of a common format is not a light undertaking. Historical approaches to address challenges of scale most often offer a problem-specific and highly-optimized solution, and do not generalize to the wider bioimaging community, reducing interoperability and re-use of datasets and software. Bespoke formats are often incompatible and require significant time and compute resources spent in data wrangling, and generally reduce the amount of FAIR data that is available to scientists. Without a formal body to declare such specifications or dedicated funding to produce a single solution for users, work is left to the community to discuss and implement with the available resources. The larger the community consensus, the more tools can be adapted with the agreed upon solution.
Big data OME-Zarr has been designed for performant reading and writing of large image data. This begins by storing the arrays of data in individual N-dimensional chunks. Since pixels that are shown together in viewers are stored together, they can be loaded more quickly. In a lightsheet dataset, for example, a 3-dimension region of 128 × 128 × 64 pixels might be colocated in a single atomic object. The current specification 3  supports up to 5 dimensional images  (time point, channel, z, y, x). In the forthcoming 0. 5 specification, this constraint will be relaxed to allow N-dimensional arrays. 1 3 To reduce file sizes and transfer times, Zarr supports compression of each of the chunks. The compression algorithm (e. g. , GZIP or Blosc  (Alted 2010 )) and its parameters can be configured transparently at the storage layer.
Collections of images With these combined capabilities, complex, data-rich collections can be constructed to support diverse applications. The OME-NGFF specification for high-content screening (HCS) 4  , for example (Fig. 2 ), defines multiple levels of hierarchy for storing a plate, each of its wells and each of the fields of that well as a separate image pyramid. Similarly, segmentations, known in other domains as "annotations", "regions-of-interest" or "segmentation maps", can be stored as labeled images beside the raw image that was segmented. Multi-modal datasets that represent multiple acquisitions on the same sample can be stored along with location information so that the images can be overlaid on one another for visualization without changing the original underlying data.
With many storage backends, this strategy enables the parallel writing of large image datasets, essential for cluster and cloud-based processing. On the other hand, the potentially large number of files produced by Zarr can create problems on some file systems, generally increasing the time to list, copy, and delete data. Having support for both gives users a choice while the use of a common model in both formats increases overall interoperability. This and future strategies for meeting user requirements will need periodic review. An upcoming version of Zarr, v3, will support a sharded layout which places a configurable number of chunks into a single shard file. This reduces the total number of files at the cost of some writing parallelism. A similar feature is available in HDF5 using "external" files and "virtual datasets" to group many separate files together.
ITKWidgets ITKWidgets (Fig. 5 ) provides interactive widgets to visualize images, point sets, and 3D geometry on the web  (McCormick et al. 2022). The development of ITKWidgets was motivated by the need for interactive insights into N-dimensional scientific datasets, such as three-dimensional, multi-channel bioimages. ITKWidgets is a component of the Fig. 3  The original viewers of OME-Zarr published in  Moore et al. 2021 , from left to right BigDataViewer, napari, and Vizarr, here seen loading a view of the same EM volume of a 6 day old Platynereis larva from  (Vergara et al. 2020 ) available at https:// s3. embl. de/ i2k-2020/ platy-raw. ome. zarr. These three applications provided broad coverage over the most common bioimaging platforms like Fiji and napari but critically also a web viewer that could stream data on the fly Table  1  List of visualization tools in the order they are described below along with their primary platform of use and the software frameworks used to build them  Designed for web-first visualization and large-scale data, ITKWidgets is built on universally deployable technologies and the OME-NGFF and ITK data models.
ITKWidgets communicates with Google CoLab, Jupyter Notebooks, Jupyter-Lab, and JupyterLite with ImJoy, a modern remote procedure communication library and plugin interface for biomedical computing in the deep learning era  (Ouyang et al. 2019). ITKWidgets is available as a Python package or the clientside viewer application can be loaded by visiting its webpage. In Python, NumPy, PyImageJ, PyTorch, the Visualization Toolkit (VTK), ITK, Xarray, Zarr, and Dask data structures are transformed on-demand to multiscale OME-Zarr. In the browser, ITKWasm will generate a multiscale OME-Zarr ondemand from common bioimaging file formats  (McCormick 2022). In Python, a simple view command accepts datasets for interactive exploration. In the client-side application, a local dataset can be selected or a shareable URL with the dataset and rendering parameters can be generated.
With a focus on supporting registration (alignment), ITK-Widgets is recommended for the comparison of datasets. Spatial metadata on multi-dimensional raster images along with associated point-based volumetric data, geometries, and annotations are supported to understand their relationship in space. Additionally, this provides a foundation for the creation of spatial transformations that define or improve on the alignment of datasets. ITKWidgets is particularly focused on providing elegant renderings to elucidate insights from volumetric information. Advanced rendering capabilities, such as light scattering, are supported. Intuitive and efficient interactive widgets are available to select rendering parameters, such as color maps and opacity transfer functions. The rendering system leverages OME-Zarr chunked, multiscale architecture to automatically load the optimal amount of data for a selected volumetric region by accounting for the current system's hardware capabilities.
MoBIE/BigDataViewer MoBIE 7  (Fig. 3 ) is a Fiji plugin  (Schindelin et al. 2012)  for the exploration of big, possibly remote, image datasets  (Pape et al. 2022). The development of MoBIE was initiated in 2018 at EMBL Heidelberg in order to solve the challenge of browsing and publicly sharing a large CLEM dataset consisting of one high-resolution TB sized 3D volume EM dataset, cell and tissue segmentations of the EM data, tables with segmentation annotations, and around 200 registered lower resolution LM images  (Vergara et al. 2020)  and is still in daily use across the institute. The main usage of MoBIE is to continuously browse complex image datasets from the moment they are produced up until publication. A typical workflow is to use other applications for image and data analysis and add the output of those applications such as segmentations and tables into the corresponding MoBIE project for visual inspection, exploration and quality control.
An exception is the possibility to perform semi-manual image registration directly within MoBIE by means of an integration with the BigWarp Fiji plugin  (Bogovic et al. 2016). MoBIE is a desktop application written in Java that heavily relies on BigDataViewer 8   (Pietzsch et al. 2015)  for image rendering and the N5 library 9  for (remote) image I/O, described below. It supports viewing locally (e. g. file-system) and remotely (e. g. "Simple Storage Service", or S3) hosted OME-Zarr image data as well as HDF5 and the eponymous N5 multi-scale image data format. In addition to simply viewing OME-Zarr images in Fiji, the main usage and feature of MoBIE is the ability to structure potentially thousands of images into a "MoBIE project" and define and configure useful views into that dataset. An important application of those features are "MoBIE views" that can be configured to conveniently browse the raw data associated with figures in publications.
While the core napari package is focused on interactively viewing and annotating data, it can be extended to other use cases via custom scripts or through the plugin interface. OME-Zarr data can be viewed in napari via the napariome-zarr plugin 11. Users can load OME-Zarr datasets through the command line interface or via the Python API. Datasets can be loaded from both local and remote data sources. Local OME-Zarr files can also be loaded via drag & drop. Developers can use the ome-zarr-py library to load datasets and add them to the viewer via the Python API. The Fractal framework uses Dask lazy loading with the napari-ome-zarr plugin and the experimental napari asynchronous loading feature (under development, NAP-4 12 ) to interactively view 2D multichannel datasets from 100 s of GBs to 1 TB in size (see Fractal section below).
Neuroglancer Neuroglancer 14 (Fig. 6 ) is an open-source web-based visualization tool for multi-dimensional volumetric data. Originally designed for visualizing petabyte-scale volume electron microscopy datasets of brain ultrastructure, it is now widely used to visualize scientific imaging data in many different application areas, including connectomics, lightsheet functional calcium neuroimaging, fMRI, and high-throughput screening. Key functionality includes: • Scalability to petabyte and larger datasets through the use of multi-resolution data streaming for OME-Zarr and other chunked formats • Cross-section views at arbitrary oblique angles • Rendering of segmentations and meshes • Arbitrarily many datasets may be displayed tiled side-byside, or overlaid as separate "layers" • Mapping from stored data to displayed RGBA values may be customized through user-defined "shader" functions that can be edited directly within the tool, and these shaders can make use of user-defined UI controls such as sliders and checkboxes • Experimental volume rendering support • Supports Zarr data served from arbitrary HTTP servers, as well as Google Cloud Storage (GCS) and Amazon S3 Neuroglancer is built using WebGL and relies on advanced GPU rendering and compression techniques to achieve high performance despite the limitations of the web platform.
As a web-based tool, Neuroglancer is particularly convenient for collaborating on datasets; users can share particular views of a dataset simply by sharing a URL. As a purely client-side web application, Neuroglancer can be used directly from the official hosted instance, or it can be deployed to any static file web server. There is also a Python package (`neuroglancer` on PyPI) that allows for full interaction with the viewer state from Python, defining of custom key and mouse bindings that invoke Python callbacks, and also allows Neuroglancer to display inmemory NumPy arrays, as well as arrays from other packages such as TensorStore, zarr-python, Dask and h5py that provide a similar NumPy-like interface. The Python package can be used both in standalone Python programs and shells and also from Jupyter notebooks, and provides a convenient way to quickly build ad-hoc data analysis and proofreading tools.
11 https:// github. com/ ome/ napari-ome-zarr. 12 https:// napari. org/ stable/ naps/4-async-slici ng. html. 13 https:// github. com/ scver se/ napari-spati aldata. 14 https:// github. com/ google/ neuro glanc er. 15 https:// github. com/ ome/ ome-ngff-valid ator. 1 3 share a URL that shows their data in the ome-ngff-validator  16. This improves the ability of the community to discuss and work with public OME-Zarr files.
In addition to the bioimaging rendering challenges that Viv addresses in production, it has also served as a testbed and mechanism to prototype new data standards. Vizarr 18  (Fig. 3 ) is a bioimaging visualization tool built using Viv that served as one of the first implementations of a reader and renderer for the HCS metadata standard in OME-Zarr 19. Using Viv as the core rendering library, Vizarr is able to simultaneously render hundreds of images from an HCS dataset. The design of Viv as a UI-agnostic library that separates rendering from data loading means that it will remain possible to quickly develop or adapt existing applications to the evolving and increasingly flexible OME-NGFF specification. • visualize very large 3D datasets, • manually segment or skeleton data efficiently, • proof-read automatic segmentation, • share data with collaborators, • manage data in one place with access restrictions, and • stream data through an OME-Zarr-based API or use a Python library to enable interoperation with other tools.
webKnossos allows users to store all data in one place and access it from wherever you are, no matter the size of the data, like a Google Drive for large microscopy images. The server component stores all the image and annotation data as well as user settings. In addition to server-stored image data, remote datasets stored in OME-Zarr can also be accessed from HTTP, S3 and GCS sources. webKnossos uses GPU rendering for high-performance visualization of the data. Users can access webKnossos through a browser without the need for additional installations. The easiest way to use webKnossos is to open an account on webknossos. org. It is free for limited storage. Alternatively, both a Docker-based setup and commercial private hosting services are available. webKnossos is a mature software and in routine use for more than 10 years.
It gained support for OME-Zarr in 2022 and implements many features of the format. OME-Zarr datasets can be imported via URL, optionally with credentials. OME-Zarr data is read on the backend with on-the-fly re-chunking and additional webKnossos-managed access controls. Additionally, all data in webKnossos is accessible as OME-Zarr data to be used in other tools via token-protected dynamic URLs. Therefore, the software can be used as a central hub for teams to manage OME-Zarr datasets. webKnossos is actively developed by a dedicated development team with monthly releases. Today, most users are from the Volume EM community, especially EM connectomics and cell biology  (Rzepka et al. 2023 ). However, light microscopy users are also well-represented. In upcoming versions, webKnossos will add support for image transformations, multi-modal datasets, and time-series data as well as the ability to run AI-based segmentations and to show segment statistics for quantitative analysis.
Website-3d-cell-viewer (and volume-viewer) from AICS The Allen Institute for Cell Science is an open science organization, interested in producing results, data, and code that can be shared with the world. Volumetric microscopy datasets are presented for interactive exploration through the Cell Feature Explorer 21  , a web-based application. As with so many microscopy efforts, ever larger and larger datasets are the norm. As an early part of transitioning storage and processing to the cloud, the interactive 3D viewer component of Cell Feature Explorer has been extended to read OME-Zarr data. The core of the viewer component is called volumeviewer which implements all of the data loading and 3D rendering, using WebGL and Typescript (Fig. 9 ). The front end, called website-3d-cell-viewer, is a reusable React component that has also been published as a standalone viewer 22  and embedded in a Jupyter notebook plugin 23.