Support scientific data formats
Opened by lhoestq (open)

List of formats and libraries we can use to load the data in `datasets`:

- [ ] DICOMs: pydicom
- [x] NIfTIs: nibabel
- [ ] WFDB: wfdb

cc @zaRizk7 for viz

Feel free to comment / suggest other formats and libs you'd like to see or to share your interest in one of the mentioned format

Comments:
Please add the support for `Zarr`! That's what we use in the Bioimaging community. It is crucial, because raw upload of a *single* bio image can take _terrabytes in memory_!

The python library would be `bioio` or `zarr`:
- [ ] Zarr: `bioio` or `zarr`

See a [Zarr example](https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD845/796b9fb8-f4ec-4c4b-bfc3-5cb00ccf19fe/796b9fb8-f4ec-4c4b-bfc3-5cb00ccf19fe.zarr)

cc @joshmoore

@stefanches7 `zarr` is already usable with the hf hub as an array store. See this example from the [docs](https://huggingface.co/docs/huggingface_hub/en/guides/hf_file_system):

```python
import numpy as np
import zarr

embeddings = np.random.randn(50000, 1000).astype("float32")

# Write an array to a repo
with zarr.open_group("hf://my-username/my-model-repo/array-store", mode="w") as root:
   foo = root.create_group("embeddings")
   foobar = foo.zeros('experiment_0', shape=(50000, 1000), chunks=(10000, 1000), dtype='f4')
   foobar[:] = embeddings

# Read an array from a repo
with zarr.open_group("hf://my-username/my-model-repo/array-store", mode="r") as root:
   first_row = root["embeddings/experiment_0"][0]
```

Is there additional functionality that would not be covered by this?

@cakiki I think some tiling capabilities, as well as metadata / labels handling. Consult ome-zarr doc here: https://ome-zarr.readthedocs.io/en/stable/python.html
Visualization would be the cherry on the top. 

cc @joshmoore @lubianat @St3V0Bay: curious what you think

zarr-specific dataset viewer would be very cool

A support for BIDS it would be perfect, I think it's possible to do all the biosinal can be done with mne. There's a cool community for decoding brain signals, and now with EMG. The new META bracelet EMG is saving things in BIDS.

I can help to interface, coding and try to make this happen. I am available at hugging face discord with the username aristimunha, if some 1-to-1 discuss it would be necessary :)

@lhoestq , @cakiki , do you think we can make this happen?

If you give me the OK, I'll create the PR to make everything for a Biosignal Reader logic, I already studied the nilabel PR :)

That would be an amazing addition ! Feel free to ping me in your PR for review or if you have questions / if I can help

@bruAristimunha @lhoestq I've recalled a gold of a resource for BIDS: https://openneuro.org/

Do you think there is a data-easy way to make those visible here on HuggingFace? Afaik they use `datalad` to fetch the data. Maybe the best way is to leave OpenNeuro as-is, not connecting it to HuggingFace at all - just an idea I had spontaneously.

I know an "easy" way to create interoperability with all biosignal datasets from OpenNeuro =) 

For biosignal data, we can use [EEGDash](https://eegdash.org/) to create a Pytorch dataset, which automates fetch, lazy read, and converts to a pytorch dataset. 

I have a question about the best serialization for a Hugging Face dataset, but I can discuss it with some of you on Discord; my username is aristimunha.

I can explain it publicly too, but I think a short 5-minute conversation would be better than many, many texts to explain the details.

It's ok to have discussions in one place here (or in a separate issue if it's needed) - I also generally check github more often than discord ^^'