Two image input vgg 16 model

Directorsim: <p>Hello everybody,<br>
I have a video dataset. I extract one video image frame and extract on audio spectrum as image of the video. I have two main folders -one includes video image frames and the other contains audio spectrums of each videos-. Each two main folder have 8 subfolders - which are the classes.</p>
<p>My model has two inputs -one image frame and one audio spectrum image-. Each input is transferred a pretrained model vgg16 paralelly for feature extraction. Then, result of these two inputs are concatinated into 8192 linear and then transferred the classification step.</p>
<p>So firstly, I create a dataset that combines these two dataset -one image frames, the other audio spectrum images-. So that, each same videos of image frame and audio spectrum images are matched for dataloader.</p>
<pre><code class="lang-auto">transform = transforms.Compose([
             transforms.Resize((224, 224)),
             transforms.ToTensor()])

class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, firstDataset, secondDataset):
        self.firstDataset = firstDataset  #video image frames of the videos
        self.secondDataset = secondDataset  #audio image spectrum of videos

    def __getitem__(self, i):
        f_x, f_y = self.firstDataset[i] #f_x: video frame image, f_y: class of the f_x
        s_x, s_y = self.secondDataset[i]  #s_x: audio spectrum image , s_y: class of the s_x
        #I checked that f_x and s_x are came from the same video
        return {"videoFrame": f_x, "audioImage":s_x}, f_y

    def __len__(self):
        return len(self.firstDataset)
    
train_loader = DataLoader(ConcatDataset(datasets.ImageFolder(root=OUTPUT_DIR_OF_VIDEO_IMAGES, transform=transform),
                 datasets.ImageFolder(root=OUTPUT_DIR_OF_SOUND_SPECTOGRAMS, transform=transform)),
                 batch_size=batch_size, shuffle=True)

</code></pre>
<p>My model is</p>
<pre><code class="lang-auto">vggmodel = vgg16(weights=torchvision.models.VGG16_Weights.DEFAULT)#(pretrained=True)
for param in vggmodel.features.parameters():
    param.require_grad = False
class MyModel(nn.Module):
    
    def __init__(self):
        super().__init__()
        
        m = vggmodel
        for param in m.parameters():
          param.requires_grad = False

        m.classifier[6] = nn.Identity() # replaced final FC layer with identity

        self.vgg16_modified = m
        
        self.classifier_last = nn.Sequential(
          nn.Linear(8192, 256),
          nn.ReLU(),
          nn.Dropout(0.4),
          nn.Linear(256, 8),
          nn.LogSoftmax(dim=1))
        
    def forward(self, x):        
        y1 = self.vgg16_modified(x["videoFrame"])
        y2 = self.vgg16_modified(x["audioImage"])
        y = torch.concat((y1, y2), 1) #IS IT TRUE ALSO?
        
        return self.classifier_last(y)  

model = MyModel()
print(model)
</code></pre>
<p>This is my model <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
When I trained, I got low accuracy. I just wondered that my logic is correct. If it is maybe my dataset images are not suitable for this case.</p>
<p>First epoch accuray: 20.000000298023224<br>
Second one: 0.0<br>
Third epoch: :20.000000298023224<br>
Forth one: 60.00000238418579<br>
Fifth one: 20.000000298023224</p>
<p>Thanks for everything,<br>
Best regards.</p>

KFrank: <p>Hi Director!</p>
<p>The short story is that concatenating feature vectors from your image<br>
frame and audio spectrum and then training a final classifier on the<br>
combined features is a reasonable approach.  But I have concerns<br>
about using the same (pretrained) <code>vgg16</code> on the audio spectrum as<br>
on the image frame.</p>
<aside class="quote no-group" data-username="Directorsim" data-post="1" data-topic="171226" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/directorsim/48/56276_2.png" class="avatar"> Directorsim:</div>
<blockquote>
<p>My model has two inputs -one image frame and one audio spectrum image-. Each input is transferred a pretrained model vgg16</p>
</blockquote>
</aside>
<p><code>vgg16</code> is pretrained (<code>VGG16_Weights.DEFAULT</code>) on “normal” images<br>
(perhaps a picture of some cows or a bicycle or a bottle of ketchup).<br>
You haven’t told us what your use case is or what your video dataset<br>
looks like, but if your “image frame” is a “normal” image, then using<br>
(or starting with) pretrained <code>vgg16</code> could make sense.</p>
<p>However, a spectrogram (“audio spectrum image”) is very different in<br>
character from a normal image.  I could believe that the pretrained <code>vgg16</code><br>
weights might not be useful in extracting meaningful features from your<br>
spectrogram.  If you <em>cannot</em> afford to do any training / fine-tuning of the<br>
<code>vgg16</code> part of your model, I suspect that your combined model won’t work<br>
very well and that you might be better off using just your image frame as<br>
input, and leaving the spectrogram out entirely.</p>
<p>If you <em>can</em> afford to train / fine-tune the <code>vgg16</code> part of your model, I would<br>
strongly suggest having two copies of <code>vgg16</code> in your model – one for the<br>
image frame and a second, with its own weights, for the spectrogram.  If<br>
you do this, then given my belief that a spectrogram is very different from a<br>
normal image, you might consider initializing the weights of the <code>vgg16</code> that<br>
you use for the spectrogram randomly, rather than using pretrained weights.</p>
<p>Either way, I would recommend first training (the normal-image part of) your<br>
final classifier and then – if you can afford the training cost – training and<br>
fine-tuning the normal-image and spectrogram <code>vgg16</code>s, while continuing to<br>
train the final classifier.</p>
<aside class="quote no-group">
<blockquote>
<p>First epoch accuray: 20.000000298023224<br>
Second one: 0.0<br>
Third epoch: :20.000000298023224<br>
Forth one: 60.00000238418579<br>
Fifth one: 20.000000298023224</p>
</blockquote>
</aside>
<p>These results suggest that you only have <em>five</em> samples in your training<br>
set.  (From your quoted accuracies it looks like you get none right, one<br>
out of five right, or three out of five right.)  If so, you would need <em>much</em><br>
more training data to perform any useful training.</p>
<p>Also, it looks like you have only trained for five epochs – also much less<br>
than you would need for any effective training or fine-tuning.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Directorsim: <p>Thank you so much!<br>
It is very informative</p>