What’s the best way to load large hdf5 data?

Vandermode: <p>I have large hdf5 database, and have successfully resolved the thread-safety problem by enabling the SWARM feature of hdf5. However, using multiple worker to load my dataset still not achieve normal speed. Typically, I observe the GPU utility circularly rise up to 100%, then drop down to 1%.<br>
Here is my dataset code (seems very naive):</p>
<pre><code class="lang-python">class HDF5Dataset(Dataset):
    """
    Args:
        h5data (HDF5 dataset): HDF5 dataset object
    """
    def __init__(self, h5data):
        self.h5data = h5data
        
    def __getitem__(self, index):
        return self.h5data[index, ...]

    def __len__(self):
        return len(self.h5data)
</code></pre>
<p>then I use the above code this way</p>
<pre><code class="lang-auto">f = h5py.File('Mydata.h5', 'r', libver='latest', swmr=True)
h5data = f['Data']
dataset = HDF5Dataset(h5data)
train_loader = DataLoader(dataset, batch_size=4, num_workers=4)
</code></pre>
<p>how can i solve this problem? what’s the best pratice to load large hdf5 datasets in pytorch? or I should follow <a href="https://discuss.pytorch.org/t/whats-the-best-way-to-load-large-data/2977">What’s the best way to load large data?</a> to migrate my data into lmdb?</p>

liqiang311: <p>Hi, How did you solve this problem? I have encountered the same problem.<br>
Thank you.</p>

piojanu: <p>Hi!</p>
<p><a class="mention" href="/u/vandermode">@Vandermode</a> how did you enable SWARM feature of hdf5 and got you code snippet to work? I can’t hack it for a long long time… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Thanks,<br>
Piotr</p>

dasturge: <pre><code class="lang-auto">class HDF5Dataset(Dataset):
    """
    Args:
        h5data (HDF5 dataset): HDF5 dataset object
    """
    def __init__(self, h5data_file):
        self.h5data_file = h5data_file
        
    def __getitem__(self, index):
        return self.h5data[index, ...]

    def __len__(self):
        return len(self.h5data)

    def h5py_worker_init(self):
        self.h5data = h5py.File(self.h5data_file, "r", libver="latest", swmr=True)
        atexit.register(self.cleanup)

    def cleanup(self):
        self.h5data.close()

def worker_init_fn(worker_id):
    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset
    dataset.h5py_worker_init()


dataset = HDF5Dataset("Mydata.h5")
DataLoader(dataset, batch_size=4, num_workers=4, worker_init_fn=worker_init_fn)
</code></pre>

DDCoan: <p>Thanks a lot for your answer. Unfortunately this code does not work for me:</p>
<pre><code class="lang-auto">AttributeError: 'HDF5Dataset' object has no attribute 'h5data'
</code></pre>
<p>It seems like the h5data was not properly initialized. Do you have an idea how to fix this? Many thanks</p>

dasturge: <p>very late reply as I seldom log in here.  h5data is only initialized when a worker is initialized, which means this only works if you are actually using a pytorch DataLoader as shown at the bottom of the example, by passing the “worker_init_fn=worker_init_fn” in.  Some extra work could make this Dataset more flexible.</p>

preethamam: <p>Hello All:</p>
<p>I have the same problem. I have a large H5 file of images of 500 GB.</p>
<p>Were you able to solve the problem of loading large HDF5 file? Especially, the GPU utility rising to 100% and falling back to 1%.</p>