Concatenating more input features onto convolutional layer output

Ethan_R: <p>Hello Everyone,</p>
<p>I am currently implementing a regression model in pytorch utilizing a CNN. The model takes in an image as input, but also has to take in some vectorized information, i.e. 1-D input vector.</p>
<p>My images are fairly large (256x512), but my dataset is also very large (5 million). Long story short, I only have to store 500 images since every chunk of 10k vectorized inputs uses the same image.</p>
<p>I want to utilize this redundancy in my training as well, so instead of running the same image through my CNN 1000 times (my batch size is 1000) I can run it through the convolutional layers 1 time and then only change the vectorized input.</p>
<p>The model I’m using is a VGG with the following code:</p>
<pre><code class="lang-auto">cfg = {
    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}
class VGG(nn.Module):
    def __init__(self, vgg_name, imgH, imgW, numClasses):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        #Figure out the shape for flatten
        x = torch.ones(1,1,imgH, imgW)
        outputShape = self.FigureOutFlattenShape(x)
        self.classifier = nn.Linear(outputShape[1] + 208, numClasses)
        #self.classifier = nn.Linear(512, 10)

    #Added by K to figure out the flattening layer 
    def FigureOutFlattenShape(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        return out.shape

    #x is a tuple, x[0] contains a single image, x[1] contains vectorized input vector
    def forward(self, x):
        out = self.features(x[0])
        out = out.view(out.size(0), -1)
        out = out.repeat(len(x[1]), 1)
        out = torch.cat((out, x[1]), 1)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg):
        layers = []
        in_channels = 1
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=False)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)
</code></pre>
<p>The key part of this post is in forward() where I run “out = out.repeat(len(x[1]), 1)”. My concern is that, for training, this will somehow mess things up since the actual training batch size is 1000 but the convolutional layers only got 1 input. My code, as it is set up now, is running and the model is improving, but I’m worried that performance will either be really slow or plateau very soon.</p>
<p>One symptom I’m already seeing is that my learning rate had to be very small (1e-5) otherwise the training loss would grow exponentially.</p>
<p>I feel like this is somewhat of a niche concern, but if anyone has any experience with similar configurations and can give some advice or pointers that would help greatly.</p>