Transfer learning with different inputs

Fish: <p>Hi,</p>
<p>I have used the transfer learning example provided on the website and it works pretty well. So I plan to apply pre-trained model, such as ResNet18, to different modalities, and then fuse the two models at FC layer to continue the training? Any thoughts on how this could be implemented here? The network parameters could either be shared or not shared. Thanks.</p>

ptrblck: <p>What do you mean by “different modalities”?<br>
As far as I understand you would like to get two pre-trained models and concat their activations at some point?</p>

Fish: <p>To clarify, for instance, one input is from the raw image, and another input is from the depth image.</p>
<p>Yes, concatenation at an early stage or a late stage should be fine. Thanks.</p>

ptrblck: <p>Ah ok, I see.<br>
So I suppose the images have different numbers of channels, i.e. image has 3 channels while depth has 1 channel?<br>
I created a small code snippet:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        
        image_modules = list(models.resnet18().children())[:-1]
        self.modelA = nn.Sequential(*image_modules)

        depth_modules = list(models.resnet18().children())[:-1]
        self.modelB = nn.Sequential(nn.Conv2d(1, 3, 3, 1, 1),
                                    *depth_modules)
        
        self.fc = nn.Linear(1024, 1)
        
    def forward(self, image, depth):
        a = self.modelA(image)
        b = self.modelB(depth)
        x = torch.cat((a.view(a.size(0), -1), b.view(b.size(0), -1)), dim=1)
        x = self.fc(x)
        x = F.sigmoid(x)
        return x


x_image = Variable(torch.randn(1, 3, 224, 224))
x_depth = Variable(torch.randn(1, 1, 224, 224))

model = MyModel()
output = model(x_image, x_depth)
</code></pre>
<p>Does it suit your needs?</p>

austin: <p>Alternatively, you could add the depth information as a fourth channel and edit the first layer of resnet18 so that it takes 4 input channels instead of three.</p>
<p>to steal from <a class="mention" href="/u/ptrblck">@ptrblck</a>’s nice example:</p>
<pre><code class="lang-python">
x_image = Variable(torch.randn(1, 3, 224, 224))
x_depth = Variable(torch.randn(1, 1, 224, 224))

input = torch.cat(x_image, x_depth, dim=1) # RGBD input

model = resnet18()
model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)

output = model(input)
</code></pre>
<p>I’m not sure which would work better for your purposes but this saves a lot of parameters vs the siamese method.</p>

dimple: <p>what changes are to be done for the data loader here?</p>

ptrblck: <p>Basically none. Your <code>Dataset</code> should provide the samples containing 4 channels.</p>

dimple: <p>but custom class has to be written for the data loader if i am right.</p>

ptrblck: <p>You would write a custom <code>Dataset</code> and just pass it to the <code>DataLoader</code>.<br>
Here is a small example:</p>
<pre><code class="lang-python">
class MyDataset(Dataset):
    def __init__(self, image_paths, targets, transform=None):
        self.image_paths = image_paths
        self.targets = targets
        self.transform = transform
        
    def __getitem__(self, index):
        x = Image.open(self.image_paths[index])
        y = self.targets[index]
        if self.transform:
            x = self.transform(x)
            
        return x, y
    
    def __len__(self):
        return len(self.image_paths)

dataset = MyDataset(image_paths, targets)
loader = DataLoader(
    dataset,
    batch_size=10,
    shuffle=True
)
</code></pre>

Ohm: <p>Hey,</p>
<p>The input channel size in my case is 19, not 3. I am looking to use the pre-trained resnet18. I found a way to handle keras, but not in PyTorch. Do you have any idea?</p>

ptrblck: <p>I don’t know how you handle it in Keras, but you could:</p>
<ul>
<li>replace the first conv layer with a new one accepting 19 channels</li>
<li>manipulate the first conv layer by increasing the <code>in_channel</code> dimension of the weight (e.g. by repeating etc.)</li>
<li>add an entirely new conv layer (let’s call it layer0) which would accept 19 channels and return an activation with 3 channels</li>
<li>manipulate the input data and reduce the channel dimension to 3</li>
<li>split the input in some way and pass only 3 channels to the model</li>
<li>…</li>
</ul>
<p>I don’t know which approach would work the best and which you are already using, but I would probably just replace the first conv layer and make sure the new layer is accepting 19 channels.</p>

Anna_Inberg: <p>Hi!<br>
I’m working on a similar model, just instead of depth images, I need to add numeric input. What does this star mean in line:</p>
<p><code>self.modelA = nn.Sequential(*image_modules)</code></p>

ptrblck: <p>The asterisk (<code>*</code>) unpacks the <code>list</code> and passes the content as positional arguments to <code>nn.Sequential</code>.</p>

Anna_Inberg: <p>Thanks!<br>
I’ve made some changes to the model according to my input.<br>
Image input is torch.Size([1, 4, 224, 224]), numeric input (48 landmarks with 2 coordinates each) torch.Size([1, 48, 2]) and a label with torch.Size([1]).</p>
<pre><code class="lang-auto">class MixedNetwork(nn.Module):
    def __init__(self):
        super(MixedNetwork, self).__init__()
        
        image_modules = list(models.resnet18().children())[:-1]
        self.image_features = nn.Sequential(*image_modules)

        self.landmark_features = nn.Sequential(
            nn.Linear(in_features=2,out_features=16,bias=False), 
            nn.ReLU(inplace=True), 
            nn.Dropout(p=0.25),
            nn.Linear(in_features=16,out_features=2,bias=False), 
            nn.ReLU(inplace=True), 
            nn.Dropout(p=0.25))
        
        self.combined_features = nn.Sequential(
            nn.Linear(32,16),
            nn.ReLU(),
            nn.Linear(16,2)
        )
        
    def forward(self, image, landmarks):
        a = self.image_features(image)
        b = self.landmark_features(landmarks)
        x = torch.cat((a.view(a.size(0), -1), b.view(b.size(0), -1)), dim=1)
        x = self.combined_features(x)
        x = F.sigmoid(x)
        return x
</code></pre>
<p>I keep getting the error:<br>
RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [1, 1, 4, 224, 224]</p>
<p>Here I don’t understand why there’s an extra one in the size - the input tensor is [1, 4, 224, 224]. Secondly, the picture is colored, so 4 channels are okay for resnet18, right?</p>

ptrblck: <p>Based on the error message your input is 5-dimensional, so print the shape inside the <code>forward</code> and check it again.<br>
A 4-dimensional input works fine:</p>
<pre data-code-wrap="python"><code class="lang-python">image_modules = list(models.resnet18().children())[:-1]
image_features = nn.Sequential(*image_modules)

x = torch.randn(1, 3, 224, 224)
out = image_features(x)
print(out.shape)
# torch.Size([1, 512, 1, 1])
</code></pre>
<aside class="quote no-group" data-username="Anna_Inberg" data-post="14" data-topic="15741">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anna_inberg/48/44661_2.png" class="avatar"> Anna_Inberg:</div>
<blockquote>
<p>Secondly, the picture is colored, so 4 channels are okay for resnet18, right?</p>
</blockquote>
</aside>
<p>No, as 3 input channels are expected. You could either remove the alpha channel or replace the first conv layer with a new one accepting 4 channels.</p>

Anna_Inberg: <p>Thanks! First, I got rid of the alpha channel and now the image tensor is of 3 channels.<br>
I also tried the dimensions check:</p>
<pre><code class="lang-auto">for i in range(len(train_dataloader)):
    sample = train_dataset[i]
    print(i, sample['image'].size(), sample['landmarks'].size(), sample['labels'].size())

    if i == 0:
      break

image_modules = list(models.resnet18().children())[:-1]
image_features = nn.Sequential(*image_modules)
image_features.double()

out = image_features(sample["image"])
print(out.shape)
</code></pre>
<p>out.shape was torch.Size([1, 512, 1, 1]), as you wrote.<br>
However, when I tried to apply the whole model with the same dataloader, it didn’t work.</p>
<p>RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [1, 1, 3, 224, 224]</p>
<p>This is the same dataloader, what else could cause this dimensionality issue?</p>

ptrblck: <p>I don’t know as I don’t fully understand the difference between the working use case and the failing one.<br>
Could you post a minimal, executable code snippet showing the failure please?</p>

Anna_Inberg: <p>Sorry for not making myself clear.<br>
What I meant is the following:<br>
I have a training dataset consisting of images, landmarks on the image, and labels.</p>
<pre><code class="lang-auto">train_dataset = FaceLandmarksDataset(train_table, images_path, 
                                    transform=transforms.Compose([scale, ToTensor()]))
sample = train_dataset[3]
print(sample['image'].size(), sample['landmarks'].size(), sample['labels'].size())
</code></pre>
<p>Print results: torch.Size([1, 3, 224, 224]) torch.Size([1, 48, 2]) torch.Size([1, 1])</p>
<p>Then I make a dataloader using this dataset and print out the tensor sizes:</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(train_dataset, batch_size=1, shuffle=False, num_workers=0)

for i_batch, sample_batched in enumerate(train_dataloader):
    print(i_batch, sample_batched['image'].size(), sample_batched['landmarks'].size(), sample_batched['labels'].size())

    if i_batch == 1:
      break
</code></pre>
<p>Print result:  torch.Size([1, 1, 3, 224, 224]) torch.Size([1, 1, 48, 2]) torch.Size([1, 1, 1])<br>
So, while getting a dataloader from a dataset, an extra dimension is added to the tensor. This is what I don’t understand.</p>

ptrblck: <aside class="quote no-group" data-username="Anna_Inberg" data-post="18" data-topic="15741">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anna_inberg/48/44661_2.png" class="avatar"> Anna_Inberg:</div>
<blockquote>
<p>Print results: torch.Size([1, 3, 224, 224]) torch.Size([1, 48, 2]) torch.Size([1, 1])</p>
</blockquote>
</aside>
<p>These shapes are unexpected as the <code>Dataset</code> already returns samples with a batch dimension.<br>
In the standard use case the batch dimension would only be added in the <code>DataLoader</code> e.g. as seen in this example:</p>
<pre data-code-wrap="python"><code class="lang-python">dataset = datasets.CIFAR10(root='./data', download=False, transform=transforms.ToTensor())

x, y = dataset[0]
print(x.shape)
# torch.Size([3, 32, 32])
print(y)
# 6

loader = DataLoader(dataset, batch_size=1)
x, y = next(iter(loader))
print(x.shape)
# torch.Size([1, 3, 32, 32])
print(y)
# tensor([6])
</code></pre>
<p>Check how the <code>__getitem__</code> method is defined in your <code>FaceLandmarksDataset</code> and make sure you are not adding a batch dimension.<br>
Note that calling <code>x = x.squeeze(1)</code> on the data batch inside the <code>DataLoader</code> would also just remove the additional dimension, but it might be a better idea to properly fix the addition of the unneeded dimension.</p>

Anna_Inberg: <p>Thanks a lot! I understand now: fixed the <strong>getitem</strong> and now it’s working properly.</p>