Repository: dask/dask

Issue:
unable to read multiple hdfs after dask update
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
after upgrading dask version above >=2021.06.1 reading with dd.read_hdf("*.hdf5") code is failing with:
```
      5 hdf5files="/lustre/arm2arm/ipython/GaiaEDR3/tmp/*.hdf5"
----> 6 ddf=dd.read_hdf(hdf5files,key="G18")

~/.conda/envs/py38dask2/lib/python3.8/site-packages/dask/dataframe/io/hdf.py in read_hdf(pattern, key, start, stop, columns, chunksize, sorted_index, lock, mode)
    423 
    424     # Build parts
--> 425     parts, divisions = _build_parts(
    426         paths, key, start, stop, chunksize, sorted_index, mode
    427     )

~/.conda/envs/py38dask2/lib/python3.8/site-packages/dask/dataframe/io/hdf.py in _build_parts(paths, key, start, stop, chunksize, sorted_index, mode)
    449     for path in paths:
    450 
--> 451         keys, stops, divisions = _get_keys_stops_divisions(
    452             path, key, stop, sorted_index, chunksize, mode
    453         )

~/.conda/envs/py38dask2/lib/python3.8/site-packages/dask/dataframe/io/hdf.py in _get_keys_stops_divisions(path, key, stop, sorted_index, chunksize, mode)
    520                 stops.append(storer.nrows)
    521             elif stop > storer.nrows:
--> 522                 raise ValueError(
    523                     "Stop keyword exceeds dataset number "
    524                     "of rows ({})".format(storer.nrows)

ValueError: Stop keyword exceeds dataset number of rows (3697)

```
**What you expected to happen**:
reading all data as expected

**Minimal Complete Verifiable Example**:
pip install dask==2021.06.1 distributed==2021.06.1
```python
# Put your MCVE code here
import numpy as np
import dask.dataframe as dd
# only 2 files are there
hdf5files="/lustre/arm2arm/ipython/GaiaEDR3/tmp/*.hdf5"
ddf=dd.read_hdf(hdf5files,key="G18")
```

**Anything else we need to know?**:
```
pip install dask==2021.06.1 distributed==2021.06.1
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting dask==2021.06.1
  Downloading dask-2021.6.1-py3-none-any.whl (973 kB)
     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 973 kB 4.5 MB/s 
Collecting distributed==2021.06.1
  Downloading distributed-2021.6.1-py3-none-any.whl (722 kB)
     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 722 kB 113.7 MB/s 
Requirement already satisfied: fsspec>=0.6.0 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from dask==2021.06.1) (2021.7.0)
Requirement already satisfied: toolz>=0.8.2 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from dask==2021.06.1) (0.11.1)
Requirement already satisfied: cloudpickle>=1.1.1 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from dask==2021.06.1) (1.6.0)
Requirement already satisfied: pyyaml in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from dask==2021.06.1) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from dask==2021.06.1) (1.2.0)
Requirement already satisfied: msgpack>=0.6.0 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (1.0.2)
Requirement already satisfied: tornado>=6.0.3 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (2.4.0)
Requirement already satisfied: setuptools in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (52.0.0.post20210125)
Requirement already satisfied: click>=6.6 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (8.0.1)
Requirement already satisfied: tblib>=1.6.0 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (2.0.0)
Requirement already satisfied: psutil>=5.0 in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from distributed==2021.06.1) (5.8.0)
Requirement already satisfied: locket in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from partd>=0.3.10->dask==2021.06.1) (0.2.1)
Requirement already satisfied: heapdict in ./.conda/envs/py38dask2/lib/python3.8/site-packages (from zict>=0.1.3->distributed==2021.06.1) (1.0.1)
Installing collected packages: dask, distributed
  Attempting uninstall: dask
    Found existing installation: dask 2021.6.0
    Uninstalling dask-2021.6.0:
      Successfully uninstalled dask-2021.6.0
  Attempting uninstall: distributed
    Found existing installation: distributed 2021.6.0
    Uninstalling distributed-2021.6.0:
      Successfully uninstalled distributed-2021.6.0
Successfully installed dask-2021.6.1 distributed-2021.6.1

```
**Environment**:

- Dask version: dask-2021.6.1 distributed-2021.6.1 and above is failing
- Python version: 3.8.x
- Operating System: UB 18.x
- Install method (conda, pip, source): ``` pip install dask==2021.06.1 distributed==2021.06.1 ```



Discussion:
Hmm it's hard to reproduce without the files that were causing the error. Can you generate files that trigger this same error or share small versions of your real files? For reference, there are lots of hdf files being generated in  https://github.com/dask/dask/blob/main/dask/dataframe/io/tests/test_hdf.py  
Yes sure you can grub those hdf files and example from: 
https://cloud.aip.de/index.php/s/SkxQTHLfXkP3rMA
(I am worry maybe we use here long64 as an index?)
Ok thanks! I think the issue is that the files are different lengths. I am not sure if this is supposed to be supported. Maybe @martindurant would know? In the meantime, I think you could achieve the desired behavior with map_partitions. I'll give it a go.
Yes this is true, my hdfs are random length, even some of them with 0. I was using dask with the data since 2018, only the recent update broke the pipeline.
Oh! I forgot that this is a recent failure that you've noticed. I'll check for changes. In the meantime here is a workaround:

```python
import glob
import dask.dataframe as dd

filenames = glob.glob("../Downloads/*.hdf5")
ddf = dd.concat([dd.read_hdf(fn, key="G18") for fn in filenames])
```
Seems likely that this is related to https://github.com/dask/dask/pull/7625 @rjzamora 
Looking at the diff in that PR, the workaround that I just gave you is essentially what the multiread code used to do, so I don't anticipate you having any performance issues with it.
Interesting,  thanks for the workaround, but would't explode my memory? or how is the concat managing the partitions, assuming you have 100K files and only 16 workers?  

Resolving patch:
diff --git a/dask/dataframe/io/hdf.py b/dask/dataframe/io/hdf.py
--- a/dask/dataframe/io/hdf.py
+++ b/dask/dataframe/io/hdf.py
@@ -452,14 +452,14 @@ def _build_parts(paths, key, start, stop, chunksize, sorted_index, mode):
             path, key, stop, sorted_index, chunksize, mode
         )
 
-        for k, stop, division in zip(keys, stops, divisions):
+        for k, s, d in zip(keys, stops, divisions):
 
-            if division and global_divisions:
-                global_divisions = global_divisions[:-1] + division
-            elif division:
-                global_divisions = division
+            if d and global_divisions:
+                global_divisions = global_divisions[:-1] + d
+            elif d:
+                global_divisions = d
 
-            parts.extend(_one_path_one_key(path, k, start, stop, chunksize))
+            parts.extend(_one_path_one_key(path, k, start, s, chunksize))
 
     return parts, global_divisions or [None] * (len(parts) + 1)
 


Test patch:
diff --git a/dask/dataframe/io/tests/test_hdf.py b/dask/dataframe/io/tests/test_hdf.py
--- a/dask/dataframe/io/tests/test_hdf.py
+++ b/dask/dataframe/io/tests/test_hdf.py
@@ -199,6 +199,16 @@ def test_to_hdf_multiple_files():
         out = dd.read_hdf(fn, "/data")
         assert_eq(df16, out)
 
+    # saving to multiple files where first file is longer
+    # https://github.com/dask/dask/issues/8023
+    with tmpdir() as dn:
+        fn1 = os.path.join(dn, "data_1.h5")
+        fn2 = os.path.join(dn, "data_2.h5")
+        b.to_hdf(fn1, "/data")
+        a.to_hdf(fn2, "/data")
+        out = dd.read_hdf([fn1, fn2], "/data")
+        assert_eq(pd.concat([df16, df]), out)
+
     # saving to multiple files with custom name_function
     with tmpdir() as dn:
         fn = os.path.join(dn, "data_*.h5")