Pytorch + HDF5 appending spectograms

username: Balazs_Gonczy
cooked: <p>Hi everyone,</p>
<p>My goal:<br>
To load spectograms 1 by 1 (It is because my preprocessing - Has to be done this way) into a HDF5 file then load this file into Pytorch with Customdataset (I am also struggling with this) method and Dataloader. Could you please help me understand the following error code?</p>
<p>Yes I know that it’s probably of the diferring dimensions of the maxshape (96, 96, 3)/ tuple object/ plots (96,96,3). I have no idea how to resolve this conflict. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>My code:</p>
<p>“”"<br>
import h5py</p>
<p>hdf5_dir = “G:/Drive/Folder/spectograms/”</p>
<p>h5_file = h5py.File(hdf5_dir + “train_combined.h5”, “w”)</p>
<p>h5_file.close()</p>
<p>h5_file = h5py.File(hdf5_dir + “train_combined.h5”, “a”)</p>
<p>def store_single_image_to_hdf5(image, label, h5_file):</p>
<pre><code>if not h5_file.keys():

    dataset = h5_file.create_dataset(

    "image", data=image, compression="gzip", chunks=True, maxshape=(None, 96, 96, 3)

    )

    meta_set = h5_file.create_dataset(

        "label", data=label, compression="gzip", chunks=True, maxshape=(None, 1)

    )

else:

    h5_file["image"].resize((h5_file["image"].shape[0] + image.shape[0]), axis=0)

    h5_file["image"][-image.shape[0]:] = image

    h5_file["label"].resize((h5_file["label"].shape[0] + label.shape[0]), axis=0)

    h5_file["label"][-label.shape[0]:] = label
</code></pre>
<p>“”"</p>
<p>Error message:<br>
“”"<br>
<strong>~\AppData\Local\Temp/ipykernel_15372/2618725217.py</strong> in plot_specgram**(waveform, sample_rate, title, xlim, ylim, folder, file_name, x_train_or_x_test)** 41 <strong><span class="hashtag">#image_array</span> = torch.from_numpy(image_array)</strong> 42 <strong>—&gt; 43</strong> store_single_image_to_hdf5**(** image_array**,** label**,** h5_file**)** <strong>~\AppData\Local\Temp/ipykernel_15372/112584867.py</strong> in store_single_image_to_hdf5**(image, label, h5_file)** 10 <strong>def</strong> store_single_image_to_hdf5**(** image**,** label**,** h5_file**)** <strong>:</strong> 11 <strong>if</strong> <strong>not</strong> h5_file**.** keys**(** <strong>)</strong> <strong>:</strong> <strong>—&gt; 12 <strong>dataset = h5_file.create_dataset(</strong></strong> 13 <strong>“image”</strong> <strong>,</strong> data**=** image**,** compression**=** <strong>“gzip”</strong> <strong>,</strong> chunks**=** <strong>True</strong> <strong>,</strong> maxshape**=** <strong>(</strong> <strong>None</strong> <strong>,</strong> <strong>96</strong> <strong>,</strong> <strong>96</strong> <strong>,</strong> <strong>3</strong> <strong>)</strong> 14 ) <strong>~\AppData\Local\Programs\Orange\lib\site-packages\h5py_hl\group.py</strong> in create_dataset**(self, name, shape, dtype, data, <strong>kwds)</strong> 147 group <strong>=</strong> self**.** require_group**(** parent_path**)** 148 <strong> → 149</strong> dsid <strong>=</strong> dataset**.** make_new_dset**(** group**,** shape**,** dtype**,** data**,** name**,** ****** kwds**)** 150 dset <strong>=</strong> dataset**.** Dataset**(** dsid**)** 151 <strong>return</strong> dset <strong>~\AppData\Local\Programs\Orange\lib\site-packages\h5py_hl\dataset.py</strong> in make_new_dset**(parent, shape, dtype, data, name, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times, external, track_order, dcpl, allow_unknown_filter)** 107 compression_opts <strong>=</strong> compression 108 compression <strong>=</strong> <strong>‘gzip’</strong> <strong> → 109 <strong>dcpl = filters.fill_dcpl(</strong></strong> 110 dcpl <strong>or</strong> h5p**.** create**(** h5p**.** DATASET_CREATE**)** <strong>,</strong> shape**,** dtype**,** 111 chunks**,** compression**,** compression_opts**,** shuffle**,** fletcher32**,** <strong>~\AppData\Local\Programs\Orange\lib\site-packages\h5py_hl\filters.py</strong> in fill_dcpl**(plist, shape, dtype, chunks, compression, compression_opts, shuffle, fletcher32, maxshape, scaleoffset, external, allow_unknown_filter)** 175 176 rq_tuple**(** chunks**,** <strong>‘chunks’</strong> <strong>)</strong> <strong> → 177</strong> rq_tuple**(** maxshape**,** <strong>‘maxshape’</strong> <strong>)</strong> 178 179 <strong>if</strong> compression <strong>is</strong> <strong>not</strong> <strong>None</strong> <strong>:</strong> <strong>~\AppData\Local\Programs\Orange\lib\site-packages\h5py_hl\filters.py</strong> in rq_tuple**(tpl, name)** 172 <strong>raise</strong> TypeError**(** <strong>‘"%s" argument must be None or a sequence object’</strong> <strong>%</strong> name**)** 173 <strong>if</strong> len**(** tpl**)** <strong>!=</strong> len**(** shape**)** <strong>:</strong> <strong> → 174 <strong><strong>raise</strong></strong></strong> ValueError**(** <strong>‘"%s" must have same rank as dataset shape’</strong> <strong>%</strong> name**)** 175 176 rq_tuple**(** chunks**,** <strong>‘chunks’</strong> <strong>)</strong> <strong>ValueError</strong> : “maxshape” must have same rank as dataset shape<br>
“”"</p>