InceptionV3 with two inputs: changing the first layer

rfmiotto: <p>Hi,</p>
<p>I am modifying pre-trained models so that they accept 2 input images. I tested my code with the <code>vgg11_bn</code>, <code>resnet50</code> and <code>vit_b_16</code> nets and all of them worked pretty well. However, when using a similar approach with the <code>inception_v3</code>, I am getting the following error:</p>
<p>RuntimeError: Given groups=1, weight of size [32, 6, 3, 3], expected input[1, 3, 299, 299] to have 6 channels, but got 3 channels instead</p>
<p>To replicate the problem:</p>
<pre><code class="lang-python">import torch
from torchvision import models

model = models.inception_v3(pretrained=True)

first_layer = model.Conv2d_1a_3x3.conv
weight = first_layer.weight.clone()

# Modify the input layer to take 2 images as input (i.e., 6 channels instead of 3)
first_layer = torch.nn.Conv2d(6, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)

# Copy weights channelwise:
with torch.no_grad():
    first_layer.weight[:, :3] = weight
    first_layer.weight[:, 3:] = weight

model.Conv2d_1a_3x3.conv = first_layer

inputs = torch.randn(2, 6, 299, 299)   # using batch size 2 to avoid problems with batch norm layers
model(inputs)
</code></pre>
<p>Could someone help me please?</p>

ptrblck: <p>I cannot reproduce the mentioned error and get:</p>
<pre><code class="lang-python">ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 768, 1, 1])
</code></pre>
<p>which is expected for a single sample. Using a batch size of 2 works fine for me.</p>

rfmiotto: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Indeed it needs a batch size greater than 1. I edited this in the question, thank you for the heads up!<br>
Anyway, the problem I’m having is not due to the batch size. I also tested this code on another machine and got the same error.</p>
<p>Could you please try to replicate the package versions to see if the problem occurs?</p>
<p>I have a <code>requirements.txt</code> file with the following content:<br>
matplotlib==3.5.1<br>
numpy==1.21.5<br>
pandas==1.4.2<br>
Pillow==9.2.0<br>
torch==1.12.1<br>
torchmetrics==0.10.0<br>
torchvision==0.13.1<br>
tqdm==4.64.0</p>
<pre><code class="lang-auto">python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
</code></pre>
<p>As you noticed I’m using torchvision==0.13.1. The previous code prints a deprecation warning. With this version of torchvision, the proper way to load the model now is:</p>
<pre><code class="lang-python">weights = models.Inception_V3_Weights.IMAGENET1K_V1
model = models.inception_v3(weights=weights)
</code></pre>
<p>However, it doesn’t fix the problem. The error is still raised. <em>RuntimeError: Given groups=1, weight of size [32, 6, 3, 3], expected input[1, 3, 299, 299] to have 6 channels, but got 3 channels instead<br>
</em></p>
<p>Thank you!</p>

ptrblck: <p>You are right and I am able to reproduce the issue when using a pretrained model (my previous code snippet used a randomly initialized model).<br>
The issue is caused by the built-in transformation, which normalizes and slices the first three channels.<br>
Use <code>model = models.inception_v3(pretrained=True, transform_input=False)</code> and it should work.</p>

rfmiotto: <p>Excellent, <a class="mention" href="/u/ptrblck">@ptrblck</a> !<br>
Your solution worked for me. Thank you very much. I didn’t know about these transformations.</p>