Create and save .h5 file for image datasets in PyTorch

username: Niraja
cooked: <p>I am working on how to create HDF5 file for age and gender image recognition datasets. I want to produce an HDF5 file for the aforementioned dataset so that I may utilise it .h5 file for image recognition in a webcam application.<br>
I made a.h5 file.<br>
‘’’</p>
<pre><code>import h5py as h5
def train_model(model, dataloaders, optimizer, criterion1, criterion2, num_epochs):  
    since = time.time()
    history = {
    #'train_acc': [],
    'train_loss': [],
    'val_loss': [],
    'age_accuracy': [],
    'gender_accuracy': []
}

    for epoch in range(num_epochs):
        print("Epoch {}/{}".format(epoch, num_epochs-1))
        print("-" * 10)

    for phase in ["train", "test"]:
        running_loss = 0.0
        running_acc1 = 0.0
        running_acc2 = 0.0
        total_1 = 0
        total_2 = 0
        if phase == "train":
            model.train()
        else:
            model.eval()

        for inputs, label1, label2 in tqdm(dataloaders[phase]):
            inputs = inputs.to(device)
            label1 = label1.long()
            label2 = label2.long()
            label1 = label1.to(device)
            label2 = label2.to(device)
            optimizer.zero_grad()

            with torch.autograd.set_grad_enabled(phase=="train"):
                outputs = model(inputs)
                loss1 = criterion1(outputs[0], label1)
                loss2 = criterion2(outputs[1], label2)
                loss = loss1 #+ loss2
                _, preds1 = torch.max(outputs[0], 1)
                _, preds2 = torch.max(outputs[1], 1)

                if phase == "train":
                    loss.backward()
                    optimizer.step()

            running_loss += loss.item() * inputs.size(0) #
          
            running_acc1 += torch.sum(preds1 == label1.data)
            running_acc2 += torch.sum(preds2 == label2.data)
            total_1 += label1.size(0)
            total_2 += label2.size(0)

        epoch_loss = running_loss / len(dataloaders[phase].dataset)
        age_epoch_acc1 = 100 * running_acc1 // total_1
        gender_epoch_acc2 = 100 * running_acc2 // total_2
        print("{} Loss: {:.4f} Age_Acc: {:.4f} gender_Acc: {:.4f} ".format(phase, epoch_loss, age_epoch_acc1, gender_epoch_acc2))  #, gender_epoch_acc2 "gender_Acc: {:.4f}"


        if phase == 'test':
            history['age_accuracy'].append(age_epoch_acc1)
            history['gender_accuracy'].append(gender_epoch_acc2)
            history['val_loss'].append(epoch_loss)
        if phase == 'train':
            history['train_loss'].append(epoch_loss)

    print()

torch.save(model, '/content/drive/MyDrive/ageNgendermodel.h5')

time_elapsed = time.time() - since
print("Training compete in {:.0f}m {:.0f}s".format(time_elapsed // 60, time_elapsed % 60))

return model, history
</code></pre>
<p>‘’’</p>
<p>My “ageNgendermodel.h5” file is created in Google Drive after training the model. However, when I download this file and utilise it in a web cam programme in a Jupyter notebook, I get the message “OSError: Unable to open file (File signature not found).”</p>
<p>I used the following code to determine if an HDF5 file was valid or not:<br>
‘’’</p>
<pre><code>import h5py
if h5py.is_hdf5('/content/drive/MyDrive/ageNgendermodel.h5'):
    print("File is a valid HDF5 file")

else:
    print("File is not a valid HDF5 file")
</code></pre>
<p>‘’’<br>
for above code it shows “File is not a valid HDF5 file”.<br>
I’m not sure where my code is going wrong. I’ve been attempting to address this problem for two days and haven’t been successful.<br>
Please assist me in resolving this issue.</p>

username: ptrblck
cooked: <p><code>torch.save</code> will not respect the file extension and will store the data in its own format.<br>
You might thus need to create a proper h5file manually and store the data there.</p>

username: Niraja
cooked: <p>Okay. How to create h5 file manually. and Is there another solution to create file.<br>
As I have studied for web cam application it requires<br>
faceProto = “\opencv_face_detector.pbtxt”<br>
faceModel = “\opencv_face_detector_uint8.pb”</p>
<p>ageProto = “\age_deploy.prototxt”<br>
ageModel = “\age_net.caffemodel”</p>
<p>genderProto = “\gender_deploy.prototxt”<br>
genderModel = “\gender_net.caffemodel”</p>
<p>So, how to create .prototxt file and .caffemodel file. basically I want create my own these files for my own model.<br>
Please assist me in resolving this issue.</p>

username: ptrblck
cooked: <p><a href="https://docs.h5py.org/en/stable/quick.html#appendix-creating-a-file">This guide</a> might be helpful.</p>
<p><code>.caffemodel</code> files (and their corresponding <code>.prototxt</code>) were used in <code>Caffe</code> which was deprecated a long time ago, so I would not recommend trying to revive it.</p>

username: Niraja
cooked: <p>Okay. Then how generate .h5 file in PyTorch.<br>
I have search for this it shows<br>
"’</p>
<pre><code>torch.save(model, filepath)
</code></pre>
<p>‘’’<br>
to save the .h5 file.<br>
Or shall I use `<br>
‘’’</p>
<pre><code>torch.save(model.state_dict(), PATH)
</code></pre>
<p>‘’’<br>
to save .h5 file.</p>

username: ptrblck
cooked: <p>As described before, PyTorch will not generate h5 files but use it’s own format.<br>
If you want to create an h5 file (for some reason), refer to the linked guide.<br>
I also don’t know <em>why</em> you want to create an h5 file in the first place as it seems you want to use Caffe based on your previous post and I don’t see the connection to PyTorch here.</p>

username: Niraja
cooked: <p>First, I’d want to make a.h5 file for my own dataset, which will be utilised in a Realtime age and gender identification application. I attempted to generate it but was unsuccessful. Then I looked for a lot of Realtime age and gender recognition programmes that used.caffemodel and.prototxt files.<br>
So, in my opinion, use PyTorch to generate the.caffemodel and.prototxt files if the.h5 file generation fails.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Niraja" data-post="7" data-topic="183552">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niraja/48/61924_2.png" class="avatar"> Niraja:</div>
<blockquote>
<p>So, in my opinion, use PyTorch to generate the.caffemodel and.prototxt files</p>
</blockquote>
</aside>
<p>This won’t work as <code>Caffe</code> is deprecated.</p>

username: Niraja
cooked: <p>So, which files should I create so that I may utilise them in my real-time age and gender recognition model or any future applications?</p>

username: Niraja
cooked: <aside class="quote no-group" data-username="Niraja" data-post="3" data-topic="183552">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niraja/48/61924_2.png" class="avatar"> Niraja:</div>
<blockquote>
<p>Please assist me in resolving this issue.</p>
</blockquote>
</aside>
<p>Please assist me in resolving this issue.</p>