How can I crop half?

wonchulSon: <p>I want to crop each cifar10 images half(0-15) and half(16-31) height.<br>
32x32 -&gt; 2x16x32</p>
<p>I can’t solve this with my code…<br>
Because after crop like that, when I put the data in model, dataloder makes problem…</p>
<p>Thank you for your help.</p>

ptrblck: <p>Could you post the code you’ve used so far?<br>
If should be possible to create these tensors in the <code>__getitem__</code> of your <code>Dataset</code>.</p>

wonchulSon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8092f8987d2084a85fa4e2521c700e77f29bd384.png" data-download-href="https://discuss.pytorch.org/uploads/default/8092f8987d2084a85fa4e2521c700e77f29bd384" title="%EC%A0%9C%EB%AA%A9%20%EC%97%86%EC%9D%8C"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8092f8987d2084a85fa4e2521c700e77f29bd384.png" alt="%EC%A0%9C%EB%AA%A9%20%EC%97%86%EC%9D%8C" data-base62-sha1="ilq018QIfISZCgEmf7Zk2SqMYMk" width="690" height="386" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%EC%A0%9C%EB%AA%A9%20%EC%97%86%EC%9D%8C</span><span class="informations">1010×566 17.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Like this, I want to include crop function into transforms.compose but it didn’t work.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c5a54edba5140a38ca5190b53975f1b7598a04a2.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5a54edba5140a38ca5190b53975f1b7598a04a2" title="%EC%A0%9C%EB%AA%A9%20%EC%97%86%EC%9D%8C2"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c5a54edba5140a38ca5190b53975f1b7598a04a2.png" alt="%EC%A0%9C%EB%AA%A9%20%EC%97%86%EC%9D%8C2" data-base62-sha1="scsepNHrLOKsHlNebJP0HOxJPns" width="690" height="363" data-dominant-color="F4F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%EC%A0%9C%EB%AA%A9%20%EC%97%86%EC%9D%8C2</span><span class="informations">1030×542 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And this is what I tried, it works to crop height 1/2, but it is hard to apply to model train.<br>
Because DataLoader do not support this logic…<br>
If I can include height 1/2 crop function into transform compose, the problem might be solved I think.<br>
Thank you for your kindness.</p>

ptrblck: <p>Could you try to concatenate both images in <code>crop_half</code>?</p>
<pre><code class="lang-python">up = transforms...
down = transforms...
return torch.cat((up, down), dim=2)
</code></pre>
<p>PS: If you post code using three backticks ```, we could easily copy your code and debug it.<br>
Also the search will be able to use your code so that others could easily find your solution to a similar problem. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

wonchulSon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you!!<br>
If you need .ipynb file I can give you by email then send me mail (dnjscjf92@ajou.ac.kr)</p>
<p>Umm…<br>
My purpose is after crop half the cifar10 image like 16x32, make two input up and down data sets, to put in each model input size is 16x32.<br>
After put in these data to VGG16’s first layer, concat two feature map like before crop, and flow it as normal VGG16 model.</p>
<p>Your comment might be save me.<br>
Thank you.</p>
<p>Under is my foolish code!!</p>
<hr>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
from torchsummary import summary
from torchvision import datasets, transforms
import numpy as np
from tqdm import tqdm
import time
import os
import torch.backends.cudnn as cudnn
import matplotlib.pyplot as plt

# Loda data
train_dataset = datasets.CIFAR10(root='../data/cifar10/',
                                 train=True,
                                 download=True)

test_dataset = datasets.CIFAR10(root='../data/cifar10/',
                                train=False)

#crop 
def crop_half(img):
    up = transforms.functional.crop(img,0,0,16,32)
    down = transforms.functional.crop(img,16,0,16,32)    
    return up, down

ToPIL = transforms.ToPILImage()
ToTensor = transforms.ToTensor()

#train
train_up = torch.tensor([])
train_down = torch.tensor([])

for i in tqdm(range(train_dataset.data.shape[0])):
    img = ToPIL(train_dataset.data[i])
    up,down = crop_half(img)
    train_up = torch.cat((train_up, ToTensor(up)), dim=0)
    train_down = torch.cat((train_down, ToTensor(down)), dim=0)
    
train_up = train_up.reshape(train_dataset.data.shape[0],3,16,32)
train_down = train_down.reshape(train_dataset.data.shape[0],3,16,32)

#test
test_up = torch.tensor([])
test_down = torch.tensor([])

for i in tqdm(range(test_dataset.data.shape[0])):
    img = ToPIL(test_dataset.data[i])
    up,down = crop_half(img)
    test_up = torch.cat((test_up, ToTensor(up)), dim=0)
    test_down = torch.cat((test_down, ToTensor(down)), dim=0)
    
test_up = test_up.reshape(test_dataset.data.shape[0],3,16,32)
test_down = test_down.reshape(test_dataset.data.shape[0],3,16,32)

#model
os.environ["CUDA_VISIBLE_DEVICES"] = '0'
start_time = time.time()
batch_size = 128
learning_rate = 0.1

# 'head' means first layer with 16x32 cropped image data 
class Vgg16_head(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16_head, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Linear(2048, num_classes)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

model_head = Vgg16_head()
optimizer = optim.SGD(model_head.parameters(), lr=learning_rate, momentum=0.9, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss().cuda()
model_head = nn.DataParallel(model_head)
cudnn.benchmark = True
model_head.cuda()

model = Vgg16()
optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss().cuda()
model = nn.DataParallel(model)
cudnn.benchmark = True
model.cuda()

targets = torch.tensor(train_dataset.targets).reshape(-1,125)

def train_head(epoch):
#     model_head.train()
    model.train()
    train_loss = 0
    total = 0
    correct = 0
    
    for data_idx, (data, target) in enumerate(zip(train_up, targets)):
        if torch.cuda.is_available():
            data, target = Variable(data.cuda()), Variable(target.cuda())
        else:
            data, target = Variable(data), Variable(target)

        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        train_loss += loss.data
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += predicted.eq(target.data).cpu().sum()
        if batch_idx % 10 == 0:
            print('Epoch: {} | Batch: {} |  Loss: ({:.4f}) | Acc: ({:.2f}%) ({}/{})'
                  .format(epoch, batch_idx, train_loss/(batch_idx+1), 100.*correct/total, correct, total))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    total = 0
    for batch_idx, (data, target) in enumerate(zip(test_up, targets)):
        if torch.cuda.is_available():
            data, target = Variable(data.cuda()), Variable(target.cuda())
        else:
            data, target = Variable(data), Variable(target)

        outputs = model(data)
        loss = criterion(outputs, target)

        test_loss += loss.data
        _, predicted = torch.max(outputs.data, 1)
        total += target.size(0)
        correct += predicted.eq(target.data).cpu().sum()
        if batch_idx % 10 == 0:
            print('TEST : Loss: ({:.4f}) | Acc: ({:.2f}%) ({}/{})'
                  .format(test_loss/(batch_idx+1), 100.*correct/total, correct, total))

# The last cell
# run model train
for epoch in range(0, 165):
    if epoch &lt; 80:
        learning_rate = learning_rate
    elif epoch &lt; 120:
        learning_rate = learning_rate * 0.1
    else:
        learning_rate = learning_rate * 0.01
    for param_group in optimizer.param_groups:
        param_group['learning_rate'] = learning_rate

    train_head(epoch)
    test()
# print("start_time", start_time)
print("--- Total %s seconds ---" % (time.time() - start_time))
</code></pre>
<p>(The last cell occur under error message)</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-115-bdeec26048b8&gt; in &lt;module&gt;
      9         param_group['learning_rate'] = learning_rate
     10 
---&gt; 11     train_head(epoch)
     12     test()
     13 

&lt;ipython-input-113-cb7193d6e59e&gt; in train_head(epoch)
     13 
     14         optimizer.zero_grad()
---&gt; 15         output = model(data)
     16         loss = criterion(output, target)
     17         loss.backward()

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    139         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    140         if len(self.device_ids) == 1:
--&gt; 141             return self.module(*inputs[0], **kwargs[0])
    142         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    143         outputs = self.parallel_apply(replicas, inputs, kwargs)

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

&lt;ipython-input-85-4b17a74fdecd&gt; in forward(self, x)
     32         x = self.features(x)
     33         x = x.view(x.size(0), -1)
---&gt; 34         x = self.classifier(x)
     35         return x

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/.local/lib/python3.5/site-packages/torch/nn/modules/linear.py in forward(self, input)
     65     @weak_script_method
     66     def forward(self, input):
---&gt; 67         return F.linear(input, self.weight, self.bias)
     68 
     69     def extra_repr(self):

~/.local/lib/python3.5/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1350     if input.dim() == 2 and bias is not None:
   1351         # fused op is marginally faster
-&gt; 1352         ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
   1353     else:
   1354         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [125 x 4096], m2: [2048 x 10] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:266
</code></pre>

ptrblck: <aside class="quote no-group" data-username="wonchulSon" data-post="5" data-topic="41767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wonchulson/48/10718_2.png" class="avatar"> wonchulSon:</div>
<blockquote>
<p>My purpose is after crop half the cifar10 image like 16x32, make two input up and down data sets, to put in each model input size is 16x32.</p>
</blockquote>
</aside>
<p>Currently you are cropping each image and append each crop to a new tensor.<br>
To do so I would recommend to avoid concatenating the tensor in each iteration and just append the crops to a list and call <code>torch.stack</code> on it, as it will be much faster:</p>
<pre data-code-wrap="python"><code class="lang-python">#train
train_up = []
train_down = []

for i in range(train_dataset.data.shape[0]):
    img = ToPIL(train_dataset.data[i])
    up,down = crop_half(img)
    train_up.append(ToTensor(up))
    train_down.append(ToTensor(down))

train_up = torch.stack(train_up)
train_down = torch.stack(train_down)
</code></pre>
<p>Once this is done, I would wrap these tensors in a custom <code>Dataset</code> and yield each the up and down crops for each sample:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, data_up, data_down, targets):
        self.data_up = data_up
        self.data_down = data_down
        self.targets = targets
        
    def __getitem__(self, index):
        up = self.data_up[index]
        down = self.data_down[index]
        y = self.targets[index]
        
        return up, down, y
    
    def __len__(self):
        return len(self.data_up)

batch_size = 128
train_dataset = MyDataset(train_up, train_down, train_dataset.targets)
train_loader = DataLoader(
    train_dataset,
    batch_size=batch_size,
    shuffle=True
)
</code></pre>
<aside class="quote no-group" data-username="wonchulSon" data-post="5" data-topic="41767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wonchulson/48/10718_2.png" class="avatar"> wonchulSon:</div>
<blockquote>
<p>After put in these data to VGG16’s first layer, concat two feature map like before crop, and flow it as normal VGG16 model.</p>
</blockquote>
</aside>
<p>Therefore I would just create a <code>head</code> and <code>features</code> layer in your <code>VGG</code> model, pass both crops separately through head, and concatenate the activations afterwards:</p>
<pre data-code-wrap="python"><code class="lang-python">
class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        
        self.head = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.features = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Linear(2048, num_classes)

    def forward(self, x_up, x_down):
        x_up = self.head(x_up)
        x_down = self.head(x_down)
        x = torch.cat((x_up, x_down), dim=2)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>
<p>Note that <code>Variables</code> are deprecated since PyTorch <code>0.4.0</code> and also I would recommend using <code>.to(device)</code> instead of <code>.cuda()</code>, so I’ve rewritten your code to the new style:</p>
<pre data-code-wrap="python"><code class="lang-python">device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = Vgg16()
model.to(device)
optimizer = optim.SGD(model.parameters(), lr=1e-3, momentum=0.9, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss()

def train(epoch):
    model.train()
    train_loss = 0
    total = 0
    correct = 0
    
    for batch_idx, (data_up, data_down, target) in enumerate(train_loader):
        data_up, data_down, target = data_up.to(device), data_down.to(device), target.to(device)

        optimizer.zero_grad()
        output = model(data_up, data_down)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        train_loss += loss.data
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += predicted.eq(target.data).cpu().sum()
        if batch_idx % 10 == 0:
            print('Epoch: {} | Batch: {} |  Loss: ({:.4f}) | Acc: ({:.2f}%) ({}/{})'
                  .format(epoch, batch_idx, train_loss/(batch_idx+1), 100.*correct/total, correct, total))
</code></pre>
<p>Your learning rate was a bit high with <code>0.1</code> so I lowered it also to <code>1e-3</code> so that your model started to learn.</p>

wonchulSon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Wow… I touched…<br>
You are so cool!! Thank you so much for your help!!<br>
I didn’t check this code yet but I will keep going based with your help.<br>
Thank you again^^.</p>

wonchulSon: <p>Hey, <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>Based your code with little custom edit, I solved problem!!<br>
But there is something I cannot understand.</p>
<pre><code class="lang-auto">class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        
        self.head = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.features = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Linear(2048, num_classes)

    def forward(self, x_up, x_down):
        x_up = self.head(x_up)
        x_down = self.head(x_down)
        x = torch.cat((x_up, x_down), dim=2)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>
<p>And how can I show the cropped &amp; concatenated feature map using plt.imshow ?<br>
Thank you so much again.</p>

ptrblck: <p>The current model should work as you’ve described in the sketch, i.e. both cropped inputs will be passed through the <code>head</code> layer, concatenated afterwards and passed through the rest of the model.</p>
<p>If you would like to visualize the cropped and concatenated feature maps, you could add them to the return statement:</p>
<pre><code class="lang-python">def forward(self, x_up, x_down, return_features=False):
    x_up = self.head(x_up)
    x_down = self.head(x_down)
    x_cat = torch.cat((x_up, x_down), dim=2)
        
    x = self.features(x_cat)
    x = x.view(x.size(0), -1)
    x = self.classifier(x)
    if return_features:
        return x, x_up, x_down, x_cat
    else:
        return x
</code></pre>
<p>Now you could just use <code>return_features=True</code>, which will give you the intermediate activations:</p>
<pre><code class="lang-python">output, act_up, act_down, act_concat = model(x_up, x_down, return_features=True)
</code></pre>

wonchulSon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I will check this!<br>
Thank you!! You are gorgeous!!</p>

wonchulSon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Hi^^.<br>
It’e me again.</p>
<p>Now I’m trying to find out which is influencing to the lower performance<br>
between cropped image and model architecture.</p>
<p>So what I want to try is<br>
“train original(uncropped) Vgg16 and with this weight test with cropped images”.</p>
<p>How can I save the trained original vgg weight<br>
and load this to cropped image test in my cropped model?</p>
<p>Thank you.</p>

ptrblck: <p>I assume the model architecture does not change, but only the <code>forward</code> definition.<br>
If that’s the case, you could just save the <code>state_dict</code> using the <a href="https://pytorch.org/docs/stable/notes/serialization.html" rel="nofollow noopener">Serialization semantics</a>.<br>
You should be able to load the <code>state_dict</code> in both models, as long as all submodules are equal.</p>

wonchulSon: <pre><code class="lang-auto">torch.save(model.state_dict(), "./weight/vgg16_baseline.pth")
</code></pre>
<p>After save original vgg weight,</p>
<pre><code class="lang-auto">model = myVgg16()
model.load_state_dict(torch.load("./weight/vgg16_baseline.pth"))
</code></pre>
<p>This code makes under error.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-12-6ceaad033204&gt; in &lt;module&gt;
      1 model = myVgg16()
----&gt; 2 model.load_state_dict(torch.load("./weight/vgg16_baseline.pth"))

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    767         if len(error_msgs) &gt; 0:
    768             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--&gt; 769                                self.__class__.__name__, "\n\t".join(error_msgs)))
    770 
    771     def _named_members(self, get_members_fn, prefix='', recurse=True):

RuntimeError: Error(s) in loading state_dict for myVgg16:
	Missing key(s) in state_dict: "head.0.weight", "head.0.bias", "head.1.running_mean", "head.1.running_var", "head.1.weight", "head.1.bias", "features.0.weight", "features.0.bias", "features.1.running_mean", "features.1.running_var", "features.1.weight", "features.1.bias", "features.4.weight", "features.4.bias", "features.5.running_mean", "features.5.running_var", "features.5.weight", "features.5.bias", "features.7.weight", "features.7.bias", "features.8.running_mean", "features.8.running_var", "features.8.weight", "features.8.bias", "features.11.weight", "features.11.bias", "features.12.running_mean", "features.12.running_var", "features.12.weight", "features.12.bias", "features.14.weight", "features.14.bias", "features.15.running_mean", "features.15.running_var", "features.15.weight", "features.15.bias", "features.17.weight", "features.17.bias", "features.18.running_mean", "features.18.running_var", "features.18.weight", "features.18.bias", "features.21.weight", "features.21.bias", "features.22.running_mean", "features.22.running_var", "features.22.weight", "features.22.bias", "features.24.weight", "features.24.bias", "features.25.running_mean", "features.25.running_var", "features.25.weight", "features.25.bias", "features.27.weight", "features.27.bias", "features.28.running_mean", "features.28.running_var", "features.28.weight", "features.28.bias", "features.31.weight", "features.31.bias", "features.32.running_mean", "features.32.running_var", "features.32.weight", "features.32.bias", "features.34.weight", "features.34.bias", "features.35.running_mean", "features.35.running_var", "features.35.weight", "features.35.bias", "features.37.weight", "features.37.bias", "features.38.running_mean", "features.38.running_var", "features.38.weight", "features.38.bias", "classifier.0.weight", "classifier.0.bias", "classifier.1.weight", "classifier.1.bias", "classifier.2.weight", "classifier.2.bias". 
	Unexpected key(s) in state_dict: "module.features.0.weight", "module.features.0.bias", "module.features.1.weight", "module.features.1.bias", "module.features.1.running_mean", "module.features.1.running_var", "module.features.1.num_batches_tracked", "module.features.3.weight", "module.features.3.bias", "module.features.4.weight", "module.features.4.bias", "module.features.4.running_mean", "module.features.4.running_var", "module.features.4.num_batches_tracked", "module.features.7.weight", "module.features.7.bias", "module.features.8.weight", "module.features.8.bias", "module.features.8.running_mean", "module.features.8.running_var", "module.features.8.num_batches_tracked", "module.features.10.weight", "module.features.10.bias", "module.features.11.weight", "module.features.11.bias", "module.features.11.running_mean", "module.features.11.running_var", "module.features.11.num_batches_tracked", "module.features.14.weight", "module.features.14.bias", "module.features.15.weight", "module.features.15.bias", "module.features.15.running_mean", "module.features.15.running_var", "module.features.15.num_batches_tracked", "module.features.17.weight", "module.features.17.bias", "module.features.18.weight", "module.features.18.bias", "module.features.18.running_mean", "module.features.18.running_var", "module.features.18.num_batches_tracked", "module.features.20.weight", "module.features.20.bias", "module.features.21.weight", "module.features.21.bias", "module.features.21.running_mean", "module.features.21.running_var", "module.features.21.num_batches_tracked", "module.features.24.weight", "module.features.24.bias", "module.features.25.weight", "module.features.25.bias", "module.features.25.running_mean", "module.features.25.running_var", "module.features.25.num_batches_tracked", "module.features.27.weight", "module.features.27.bias", "module.features.28.weight", "module.features.28.bias", "module.features.28.running_mean", "module.features.28.running_var", "module.features.28.num_batches_tracked", "module.features.30.weight", "module.features.30.bias", "module.features.31.weight", "module.features.31.bias", "module.features.31.running_mean", "module.features.31.running_var", "module.features.31.num_batches_tracked", "module.features.34.weight", "module.features.34.bias", "module.features.35.weight", "module.features.35.bias", "module.features.35.running_mean", "module.features.35.running_var", "module.features.35.num_batches_tracked", "module.features.37.weight", "module.features.37.bias", "module.features.38.weight", "module.features.38.bias", "module.features.38.running_mean", "module.features.38.running_var", "module.features.38.num_batches_tracked", "module.features.40.weight", "module.features.40.bias", "module.features.41.weight", "module.features.41.bias", "module.features.41.running_mean", "module.features.41.running_var", "module.features.41.num_batches_tracked", "module.classifier.0.weight", "module.classifier.0.bias", "module.classifier.1.weight", "module.classifier.1.bias", "module.classifier.2.weight", "module.classifier.2.bias". 
</code></pre>
<p>Is this because the different head architecture?<br>
Umm… then how can Ioad the weight?</p>

ptrblck: <p>How did you define the original model? You should also use <code>myVGG16</code> and adapt the forward method to just use the <code>head</code> once.</p>

wonchulSon: <pre><code class="lang-auto">class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Sequential(
            nn.Linear(512, 512),
            nn.Linear(512, 512),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>
<p>Above is original Vgg16<br>
and under is myVgg16 which is you hinted for me.</p>
<pre><code class="lang-auto">class myVgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(myVgg16, self).__init__()
        
        self.head = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
        )
        
        self.features = nn.Sequential(
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
            
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Sequential(
            nn.Linear(512, 512),
            nn.Linear(512, 512),
            nn.Linear(512, num_classes)
        )

    def forward(self, x_up, x_down):
        x_up = self.head(x_up)
        x_down = self.head(x_down)
        x = torch.cat((x_up, x_down), dim=2)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>

ptrblck: <p>You could use the same modules and just derive the “headed” version using a basic vgg16 implementation.<br>
Here is a small example:</p>
<pre><code class="lang-python">class myVgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(myVgg16, self).__init__()
        
        self.head = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
        )
        
        self.features = nn.Sequential(
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
            
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Sequential(
            nn.Linear(512, 512),
            nn.Linear(512, 512),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.head(x)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x




class myVgg16Headed(myVgg16):
    def __init__(self, num_classes=10):
        super(myVgg16Headed, self).__init__(num_classes)

    def forward(self, x_up, x_down):
        x_up = self.head(x_up)
        x_down = self.head(x_down)
        x = torch.cat((x_up, x_down), dim=2)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x


model = myVgg16()
model_headed = myVgg16Headed()

torch.save(model.state_dict(), 'tmp.pt')
model_headed.load_state_dict(torch.load('tmp.pt'))
</code></pre>

wonchulSon: <p>Oh! I think I catched your idea!!<br>
I will try it myself.<br>
Thank you so much.</p>

wonchulSon: <p>Hmm… it doesn’t solve the error.<br>
Please look at my effort.</p>
<p>First, I trained <code>Vgg16</code> (original) model.</p>
<pre><code class="lang-auto">class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        
        self.head = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
        )
        
        self.features = nn.Sequential(
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
            
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        
        self.classifier = nn.Sequential(
            nn.Linear(512, 512),
            nn.Linear(512, 512),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.head(x)
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>
<p>After run and train, saved weight like this.</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), "./weight/vgg16_baseline.pth")
</code></pre>
<p>Second, I made <code>myVgg16</code> (cropped) model.<br>
(Here is the point, I used the same code to make <code>Vgg16</code> and <code>myVgg16</code> model. It’s just have different class name, is this right?)</p>
<p>Third, I made <code>myVgg16Headed</code></p>
<pre><code class="lang-auto">class myVgg16Headed(myVgg16):
    def __init__(self, num_classes=10):
        super(myVgg16Headed, self).__init__(num_classes)

    def forward(self, x_up, x_down):
        x_up = self.head(x_up)
        x_down = self.head(x_down)
        x = torch.cat((x_up, x_down), dim=2)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>
<pre><code class="lang-auto">model_headed = myVgg16Headed()
</code></pre>
<p>(above code make <code>myVgg16</code> too automatically…?)</p>
<pre><code class="lang-auto">model_headed.load_state_dict(torch.load('./weight/vgg16_baseline.pth'))
</code></pre>
<p>But the same error is occurred.</p>

ptrblck: <p>I’m not sure to understand it completely, but I think you would just need two models.<br>
One using a single input and the other one using the cropped inputs, is that correct?<br>
If so, you should just create the two models given in my last example, i.e. <code>myVgg16</code> and <code>myVgg16Headed</code>.</p>

wonchulSon: <p>Oh, with your help I solved problem finally.</p>
<pre><code class="lang-auto">model_headed = myVgg16Headed()
model_headed = nn.DataParallel(model_headed)
model_headed.load_state_dict(torch.load('./weight/vgg16_baseline.wts'))
</code></pre>
<p>In this cell, <code>model_headed = nn.DataParallel(model_headed)</code> solved the error.</p>
<p>And I have one more question about your code.</p>
<pre><code class="lang-auto">class myVgg16Headed(myVgg16):
    def __init__(self, num_classes=10):
        super(myVgg16Headed, self).__init__(num_classes)

    def forward(self, x_up, x_down):
        x_up = self.head(x_up)
        x_down = self.head(x_down)
        x = torch.cat((x_up, x_down), dim=2)
        
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
</code></pre>
<p>Above code means that <code>class myVgg16Headed(myVgg16)</code> override<code>myVgg16</code>’s <code>def forward</code> ??</p>