500 internal server error when trying to open a dataset composed of Zarr stores
Author: jacobbieker
State: closed
Labels: name: bug

## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine. 

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format? 

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0



Comments: Hi @jacobbieker, thanks for reporting!

I have transferred this issue to our Hub team and they are investigating it. I keep you informed. 

Hi @jacobbieker, we are investigating this issue on our side and we'll see if we can fix it, but please note that your repo is considered problematic for git. Here are the results of running https://github.com/github/git-sizer on it:

```
Processing blobs: 147448                        
Processing trees: 27                        
Processing commits: 4                        
Matching commits to trees: 4                        
Processing annotated tags: 0                        
Processing references: 3                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |   167 k   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of files        [2] |   189 k   | ***                            |

[1]  aa057d2667c34c70c6146efc631f5c9917ff326e (refs/heads/main:2016.zarr/unknown)
[2]  6897b7bf6440fdd16b2c39d08085a669e7eaa59d (refs/heads/main^{tree})
```

You can check https://github.com/github/git-sizer for more information on how to avoid such pathological structures.

Hi, thanks for getting back to me so quick! And yeah, I figured that was probably the problem. I was going to try to delete the repo, but couldn't through the website, so if that's the easiest way to solve it, I can regenerate the dataset in a different format with less tiny files, and you guys can delete the repo as it is. Zarr just saves everything as lots of small files to make chunks easy to load, which is why I was preferring that format, but maybne that just doesn't work well for HF datasets.

Hi @jacobbieker,

For future use cases, our Hub team is still pondering whether to limit the maximum number of files per repo to avoid technical issues...

On the meantime, they have made a fix and your dataset is working: https://huggingface.co/datasets/openclimatefix/mrms