Save torch tensors as hdf5

username: nabsabs
cooked: <p>Hi guys!</p>
<p>I’m not sure if this is a PyTorch question but I want to save the 2nd last fc outputs from a pretrained vgg into an hdf5 array to load later on. The issue is I would need to save all tensor outputs as one chunk to use an hdf5 dataset (below) however I cannot seem to append tensors to h5 dataset without creating chunks. Does anyone know of an efficient way to save torch tensors into one chunk in an hdf5 file?</p>
<p>Any help appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto"># https://www.tinymind.com/learn/terms/hdf5

import h5py
import torch
import torch.utils.data as data

class H5Dataset(data.Dataset):

    def __init__(self, file_path):
        super(H5Dataset, self).__init__()
        h5_file = h5py.File(file_path)
        self.data = h5_file.get('data')
        self.target = h5_file.get('label')

    def __getitem__(self, index):            
        return (torch.from_numpy(self.data[index,:,:,:]).float(),
                torch.from_numpy(self.target[index,:,:,:]).float())

    def __len__(self):
        return self.data.shape[0]
</code></pre>

username: imaluengo
cooked: <p>HDF5 is not a great format for appending information over-time… It will end up generating a very large binary file to handle new data.</p>
<p>I’d recommend doing it for a fixed size. E.g. first create a dataset of a fixed size:</p>
<pre><code class="lang-python">N = 100 # find the length of my dataset
data = h5_file.create_dataset('data', shape=(N, 3, 224, 224), dtype=np.float32, fillvalue=0)
</code></pre>
<p>Then populate it</p>
<pre><code class="lang-python">for i in range(N):
    img = ... # load image
    data[i] = img

h5_file.close()
</code></pre>
<p>However, if you really really want to do resizable datasets (not recommended, size will exponentially grow), HDF5 and h5py <a href="http://docs.h5py.org/en/stable/high/dataset.html#resizable-datasets" rel="nofollow noopener">supports it</a>, e.g., replacing above:</p>
<pre><code class="lang-python">data = h5_file.create_dataset('data', shape=(N, 3, 224, 224), dtype=np.float32, 
                              maxshape=(None, 3, 224, 224))
</code></pre>
<p>And then at any time you can call the <a href="http://docs.h5py.org/en/stable/high/dataset.html#Dataset.resize" rel="nofollow noopener">resize</a> function of a dataset:</p>
<pre><code class="lang-python">data.resize(10000, axis=0) # now you can fit up to 10K samples!
</code></pre>
<p>Just be careful not calling resize too many times <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>PS: Make sure you open the HDF5 file in read-only for better performance, and add <a href="http://docs.h5py.org/en/stable/swmr.html" rel="nofollow noopener">swmr</a> flag to allow concurrent reads, for which the <code>.h5</code> file has had to be created with <code>swmr</code> too:</p>
<pre><code class="lang-python">h5_file = h5.File(file_path, 'r', libver='latest', swmr=True)
</code></pre>

username: nabsabs
cooked: <p><a class="mention" href="/u/imaluengo">@imaluengo</a> this is awesome thanks so much! I didn’t know I could just keep the H5file open and add to it because when I tried appending fc2 output tensors into a list, I ran out of memory. I guess the h5 file is compressing it when it stores it in the dataset? I’ll read into this some more but thank you again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: imaluengo
cooked: <p>Yup appending doesn’t work great in HDF5.</p>
<p>Resize + inserting data with slices should work fine tho. Just be aware that resizing operation does not physically extend the dataset memory in disk, it does create a separate storage blob and link them under the hood.</p>
<p>Thus, the more you resize, the slower it will be to do consecutive reads, e.g.</p>
<pre><code class="lang-auto">data = h5_file['data'][100:200]
</code></pre>
<p>That will return 100 elements  however, if first 50 elements were initially created, and last 50 are created later as part of a resizing operation, that data retrieval query above will have a decrease in performance, as data is not contiguous in disk.</p>
<p>So, just try to minimize resize calls <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: wolterlw
cooked: <p>Does anyone have any experience storing images in hdf5? I mean compressing the image into a byte string and then writing it into the dataset, then read and decode it when reading.<br>
Would be interesting to compare if this approach would be any faster than filesystem storage.</p>