Recommend the way to load larger h5 files

John1231983: <p>Hello all, I have a dataset that requires to use h5 files. The dataset size has 1000 images with the total size of <strong>100GB</strong>. I have two options:</p>
<ol>
<li>Save each image to an hdf5 file, then I have total 1000 hdf5 file. In dataloader, I will call</li>
</ol>
<pre><code class="lang-auto">class h5_loader(data.Dataset):
    def __init__(self, file_path):
	    self.file_list = [f for f in glob.glob(os.path.join(file_path, '*.h5'))]
    def __getitem__(self, index):
	    h5_file = h5py.File(file_list[index])
        data = h5_file.get('data')
</code></pre>
<ol start="2">
<li>Save all images into a single hdf5 file, then dataloader can call</li>
</ol>
<pre><code class="lang-auto">class h5_loader(data.Dataset):
    def __init__(self, file_path):
        h5_file = h5py.File(file_path)
        data = h5_file.get('data')
    def __getitem__(self, index):
		...

</code></pre>
<p>Which option should I use to speed up data loader time? The first way load each hdf4 in <code>__getitem__</code> function, while the second way load a single (combined of 1000 images) in the <code>__init__</code> function.</p>

justusschock: <p>Depends on some facts:</p>
<ol>
<li>Do you have enough RAM to contain your whole data</li>
<li>How many CPUs do you have, how many GPUs?</li>
<li>How big is your batchsize?</li>
<li>With how many workers do you train?</li>
<li>Do you do some online/offline augmentation?</li>
</ol>

John1231983: <p>Thanks. I have 32 GB Ram with core i7 cpu. 11gb gpu and batch size of 16. I do not use augmentation and number of worker is 1</p>

justusschock: <p>If your whole dataset fits into your ram, you could load it from one large file inside your dataset’s <code>__init__</code>. This would load all your data once and take a bit longer for the initialization, but would be the fastest way during training, since no additional data has to be loaded.</p>
<p>If that is not possible, you could increase your number of workers and load the images inside your <code>__getitem__</code>. If I remember correctly, HDF5 works as a generator, which means you could also use one large HDF5 file. The advantage is that you don’t need that much RAM, on the other side you would have to load your data during training, which may be a bottleneck (depending on the size of your images, the performance of your GPU, your CPU and your number of workers).</p>

Idan_Uri: <p>Hey!<br>
I’m a first time responder and pretty new to pytorch and deep learning. I am also trying to train a CNN (UNet to be specific) with image data and think I’m running into a problem related to this topic.</p>
<p>I have a h5py data base file that is too big to load (~27GB). It has 8,000 sampls and each sample shape is (14,257,256). I think It’s worth to mention that I am creating that data base and I can control the number of files and each file size.</p>
<p>Acording to the solution here I can create 8000 different h5py files and use the <strong>getitem</strong> method to get a different sample from a different file every time (In that case does the <strong>len</strong> method still returns 8000?).</p>
<p>I was thinking of a way to combain these 2 ways and creating 8 files 1000 samples each. In that case the loader would look like(?):</p>
<pre><code class="lang-auto">class h5_loader(data.Dataset):
    def __init__(self):
	    self.file_path = '/content/drive/My Drive/Project Dataset/trainFile'
    def __getitem__(self, index):
        i = index/1000
        mod = index%1000
	    h5_file = h5py.File(file_path+str(i)+'.h5','r')
        input = h5_file['Input'][mod]
        target = h5_file['Target'][mod]
        return input, target

</code></pre>
<p>Would that work?<br>
What would you say is the most “clean” and fastest way to load my data into the model?</p>
<p>I’m working on google colab from my 8GB RAM laptop (I use 0 workers).</p>
<p>Thanks!</p>

Idan_Uri: <p>UPDATE</p>
<p>So apparently this is a very BAD idea. I tried to train my model using this option and it was very slow, and I think I figured out why.<br>
The disadvantage of using 8000 files (1 file for each sample) is that the <strong>getitem</strong>  method has to load a file every time the dataloader wants a new sample (but each file is relatively small, because it contain only one sample).</p>
<p>Doing what I did makes the dataloader to load a 1000 samples file just to get out 1 sample out of it, And by the next time the dataloader will want a new sample it will have to load the same file over again.</p>

SU801T: <p>What did you do to resolve this?</p>

Idan_Uri: <p>I haven’t found an opitmal solution, for the meantime I settled for a smaller dataset.</p>
<p>At first I tried to use John second option (the get item load each sample separately), but it was really slow on my computer (about 28 sec to load each sample).</p>
<p>Than I tried to get creative and divide each epoch into 8 parts (of 1000 samples each), and after each part I deleted the dataloader object and created a new dataloader object with a different dataset, like that:</p>
<pre><code class="lang-auto">for j in np.arange(25):
    ########Starting train epoch######### 
  running_loss = 0
  train_loss_vector = []

  for k in range(8):    
      DB_T = My_Dataloader_Train(k)
      train_loader_train = data.DataLoader(dataset = DB_T, batch_size = 16, num_workers=0)
      
      net.train() # setting the model for training mode
      for batch_inx, (specs,masks) in enumerate(train_loader_train):

          specs = specs.to(device).float()
          .
          .
</code></pre>
<pre><code class="lang-auto">class My_Dataloader_Train(Dataset):
  def __init__(self,i):
    super().__init__()
    self.file_path = '/content/drive/My Drive/Project Dataset/trainFile' +str(i)+ '.h5'
    self.input = h5py.File(self.file_path, 'r')['Input'][:]
    self.target = h5py.File(self.file_path, 'r')['Target'][:]    
    self.len = 1000
</code></pre>
<p>It looked like it was working but after (I think) 10 epoches colab crashed with an ‘Unable to open file’ error (which is weird because it already succeeded opening the same file an epoch ago.</p>
<p>Anyway, I’m still wating for a better solution.</p>

SU801T: <p>I was initially getting an OS B-Tree error when using multiple processes. So I followed the advice in this thread here:</p>
<aside class="quote" data-post="8" data-topic="25643">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/8">DataLoader, when num_worker &gt;0, there is bug</a> 
  </div>
  <blockquote>
    Thanks for the debugging and pointing out strange issues. 
I can reproduce the issues following your information. 
It seems that multiprocessing doesn’t work well with HDF5/h5py. 
<a href="https://discuss.pytorch.org/t/incorrect-data-using-h5py-with-dataloader/7079/2">Here</a> is a similar issue with a link to the <a href="https://groups.google.com/forum/#!topic/h5py/bJVtWdFtZQM" rel="noopener nofollow ugc">known problem</a>. 
Using the spawn method doesn’t solve the issue in this case. I’ll dig a bit deeper.
  </blockquote>
</aside>

<p>And created a dataclass like this:</p>
<pre><code class="lang-auto">class Features_Dataset(data.Dataset):
    def __init__(self, archive, phase):
        self.archive = archive
        self.phase = phase

    def __getitem__(self, index):
        with h5py.File(self.archive, 'r', libver='latest', swmr=True) as archive:
            datum = archive[str(self.phase) + '_all_arrays'][index]
            label = archive[str(self.phase) + '_labels'][index]
            path = archive[str(self.phase) +  '_img_paths'][index]
            return datum, label, path

    def __len__(self):
        with h5py.File(self.archive, 'r', libver='latest', swmr=True) as archive:
            datum = archive[str(self.phase) + '_all_arrays']
            return len(datum)


if __name__ == '__main__':
    train_dataset = Features_Dataset(archive= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=8, batch_size=128)
    print(len(trainloader))
    for i, (data, label, path) in enumerate(trainloader):
        print(path)

</code></pre>
<p>Now I don’t get an error anymore, but loading data is super slow. Because of that, the 4 GPUs that I’m trying to utilize are at zero % volatility. I think there should be a fix, or I have written something completely inefficient. I have 150k instances, where the data, labels and paths are in 3 different datasets within the H5 file. I’m not sure if that plays a problem.</p>

preethamam: <p>Hello All:</p>
<p>I have the same problem. I have a large H5 file of images of 500 GB. I don’t have problem with loading. However, for my batch size of 60, I am able to churn 1-1.2 iteration per second. But GPU utilization alternates 1 and 100%.</p>
<p>Were you able to solve the problem of loading large HDF5 file? Especially, the GPU utility rising to 100% and falling back to 1%.</p>