Modify ResNet or VGG for single channel grayscale

Alex_Ge: <p>Hi,</p>
<p>I’m working on infrared data which I convert to grayscale 64x64 (although I can use other sizes, but usually my GPU runs out of memory). I used VGG11 but I manually recreated the architecture in order to use it, which is a very cumbersome and error prone task.<br>
I was wondering if there is an easier way to modify VGG19 or ResNet architectures in a fast and simpler way to use my 64x64 single channel input, and if yes, would that make sense since those models are fine-tuned for 3 channel RGB?<br>
I’ve read <a href="https://discuss.pytorch.org/t/transfer-learning-how-to-modify-the-first-conv2d-layer-of-alexnet-to-accomodate-for-9-channel-input/4063/4">this discussion</a> and it seems to me that all I have is create a class which inherits from VGG19 or ResNet-152 and modify the first Conv layer?</p>
<p>Many thanks,<br>
Alex</p>

JuanFMontesinos: <p>Modifiying ResNet is very easy and more powerful (than VGG).<br>
This is a copy of official pytorch implementation</p>
<pre><code class="lang-auto">class ResNet(nn.Module):

    def __init__(self, block, layers, num_classes=1000):
        self.inplanes = 64
        super(ResNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
                               bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
        self.avgpool = nn.AvgPool2d(7, stride=1)
        self.fc = nn.Linear(512 * block.expansion, num_classes)
</code></pre>
<p>you just have to check self.conv1 to</p>
<pre><code class="lang-auto">        self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,
                               bias=False)
</code></pre>
<p>And that’s all.</p>
<p>I am afraind that there is no fine tunning… you would be training from the scratch.</p>
<p>ResNet input is 224x224 by default. Code will run with 64 by 64 of course but all the pretraining would be not very useful.</p>
<p>You should also consider what are you using this net to. The output size using a 224x224 input is 8x8 (forgetting about fully connected and these stuff). Using a 64x64 input will generate a much smaller output.</p>
<p>Considering those facts, do the best choise <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptab: <p>If you want to make use of a pretrained network, consider feeding your grayscale image as RGB image to the network, by pasting your grayscale information to all three channels. There might be some clever variants of this technique, Jeremy Howard from fast.ai talked about this a bit in his lectures, unfortunately I don’t remember in which lecture / timestamp exactly.</p>

Alex_Ge: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> Thank you a ton! Didn’t expect to be spoon-fed the answer. Yes, I agree that 64x64 is much smaller, only reason I use it is because the entire dataset fits in my GPU’s RAM (about 10Gb). Using 224x224 makes it considerably slower because every mini-batch must be uploaded to fit.</p>
<p><a class="mention" href="/u/ptab">@ptab</a> Thanks for the suggestion. I am not doubting there might be benefits from doing that, but it seems to me like an overkill, what could it potentially offer? I can use RGB, but I’d rather stick to grayscale for now, however thanks again!</p>

JuanFMontesinos: <p>Yeh I know but… even if you have to do some iterations it’s better than using that size i guess.</p>
<p>Abou that <a class="mention" href="/u/ptab">@ptab</a> said, using grayscale you loose the preteaining, and imagenet weights are very good weights trained on millions of images. It’s an standard when using a pretrained visual architecture for general purposes</p>

Alex_Ge: <p>My task is face valence/expression classification so I doubt imagenet weights will be of much use.<br>
I’ll try and see if I can somehow speed up training in mini-batch allocation since I’ve got a Titan Xp which seems to idle when I don’t fit the data-set in GPU memory. Thanks for the advise and help, I’ll give it a try and report back!<br>
My current overall accuracy appears to be around 69-70% using a greyscale derivate of VGG11 with batch normalisation.</p>

JuanFMontesinos: <p>Not sure right now, you can check dlib which is a library for face recognition.  I don’t know the architecture it uses</p>

Deeply: <p>Pre-rained imageNet models did well even with OCR data, so, I would not be surprised if they do well with face data too.</p>

Alex_Ge: <p>I can’t seem to get it to work.<br>
I’ve tried a slightly different approach:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models

class resnet152_mech(models.resnet152(pretrained=False)):

    def __init__(self, block, layers, num_classes=4):
        self.inplanes = 64
        super(resnet152_mech, self).__init__()
        self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,
                               bias=False)
</code></pre>
<p>However when I create such an object, I get an error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “resnet_test.py”, line 1, in <br>
import resnet_mechion as mechion<br>
File “/home/zuperath/code/mechion_core/python/resnet_mechion.py”, line 7, in <br>
class resnet152_mech(models.resnet152(pretrained=False)):<br>
File “/home/zuperath/anaconda2/lib/python2.7/site-packages/torchvision/models/resnet.py”, line 106, in <strong>init</strong><br>
self.layer1 = self._make_layer(block, 64, layers[0])<br>
File “/home/zuperath/anaconda2/lib/python2.7/site-packages/torchvision/models/resnet.py”, line 123, in _make_layer<br>
if stride != 1 or self.inplanes != planes * block.expansion:<br>
AttributeError: ‘str’ object has no attribute ‘expansion’</p>
</blockquote>
<p>I’m not sure what this is, seems to me some logic test tabout planes and blocks fails?</p>

JuanFMontesinos: <aside class="quote no-group quote-modified" data-username="Alex_Ge" data-post="9" data-topic="22762">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_ge/48/17912_2.png" class="avatar"> Alex_Ge:</div>
<blockquote>
<p>import torch import torch.nn as nn import torchvision.models as models class resnet152_mech(models.resnet152(pretrained=False)): def <strong>init</strong>(): self.inplanes = 64 super(resnet152_mech, self).<strong>init</strong>() self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)</p>
</blockquote>
</aside>
<p>Hi, I think you are commiting a coding error.<br>
You are creating a inherited class from an object. I reviewed the source code and when u call</p>
<pre><code class="lang-auto">models.resnet152(pretrained=False)
</code></pre>
<p>You are defining a model, it means, u are creating an instance of the class ResNet. When you inherit a class from another class you have to call the class, not the instance.<br>
Therefore:<br>
Your fixed code is:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models

class resnet152_mech(models.resnet.ResNet):

    def __init__(self, block, layers, num_classes=4):
        self.inplanes = 64
        super(resnet152_mech, self).__init__()
        self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,
                               bias=False)
</code></pre>
<p>For later on defining your model you have to use a similar instance definition:</p>
<pre><code class="lang-auto"> model = resnet152_mech(models.resnet.Bottleneck, [3, 8, 36, 3], **kwargs)
</code></pre>
<p>Oh, dont forget about modifiying **kwargs xd<br>
Something like that</p>

Alex_Ge: <p>Apologies, I came from a C++ background, of course what you said makes perfect sense.<br>
I’ll try with it again.</p>

fermat97: <p>Hi, I have tried</p>
<pre><code class="lang-auto">class resnet2ch(torchvision.models.resnet.ResNet):
    def __init__(self,block, layers, num_classes=4):
        self.inplanes = 64
        super(resnet2ch, self).__init__()
        self.conv1 = nn.Conv2d(2, 64, kernel_size=7, stride=2, padding=3,bias=False)
</code></pre>
<p>and use</p>
<pre><code class="lang-auto">model= resnet2ch(torchvision.models.resnet.Bottleneck,[2, 2, 2, 2])
</code></pre>
<p>to build resnet18 with two channels. But I got the error:</p>
<pre><code class="lang-auto">super(resnet2ch, self).__init__()
TypeError: __init__() missing 2 required positional arguments: 'block' and 'layers'

</code></pre>
<p>Do you have any idea? thank you.</p>

ptrblck: <p>The parent class expects these arguments, so try to pass them to the <code>super()</code> call:</p>
<pre><code class="lang-python">def __init__(self, block, layers, num_classes=4):
    super(resnet2ch, self).__init__(block, layers)
    ...
</code></pre>

fermat97: <p>Thanks alot.<br>
I have tried something like:</p>
<blockquote>
<p>model = resnet18()<br>
model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)</p>
</blockquote>
<p>from this post <a href="https://discuss.pytorch.org/t/transfer-learning-with-different-inputs/15741/5" class="inline-onebox">Transfer learning with different inputs - #5 by austin</a></p>

akschougule: <pre><code class="lang-auto">model= resnet2ch(torchvision.models.resnet.Bottleneck,[2, 2, 2, 2])
</code></pre>
<p>what does the [2,2,2,2] specify here?</p>

ptrblck: <p>The list specifies the <code>layers</code> argument to create different resnet architectures as seen <a href="https://github.com/pytorch/vision/blob/791c172a337d98012018f98ffde93b1020ba3ed5/torchvision/models/resnet.py#L236">here</a>.</p>

carbocation: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="22762">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>I am afraind that there is no fine tunning… you would be training from the scratch.</p>
</blockquote>
</aside>
<p>I don’t see why this has to be true. For example, <a href="https://github.com/fastai/fastai/blob/87827a10c946ea024fb09ecaf00073395e4d950d/fastai/vision/learner.py#L33" rel="noopener nofollow ugc">fastai automatically sums the 3-channel weights to produce 1-channel weights</a> for the input layer when you provide a 1-channel input instead of the usual 3-channel input.</p>