Loading a checkpoint with different number of layers onto a model

username: Arvind_Subramaniam
cooked: <p>I have applied a pruning method in which the number of channels are reduced in the model. I would like to know if it would be possible to load this checkpoint into a model without a size mismathc error.<br>
Since the model before pruning had 256 channels, 512 channels, etc in each layer (VGG), and the pruned VGG model has 188 and 313 channels, I am getting a size mismatch error when I run:<br>
model.load_state_dict(checkpoint[‘model_state_dict’])<br>
Here’s the error:<br>
size mismatch for feature.50.running_var: copying a param with shape torch.Size([313]) from checkpoint, the shape in current model is torch.Size([512])</p>
<p>How do I save the new pruned checkpoint into another model?</p>

username: ptrblck
cooked: <p>The probably cleanest way would be to load the <code>state_dict</code> into the new model definition.<br>
How did you prune the original model?<br>
This might give us some information about the easiest way to load the parameters.</p>

username: Arvind_Subramaniam
cooked: <p>This might make it easier to read: I have run the same code on google Colab for MLP and a CNN:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6e79e5de4e298f1baea10b5f832bd377af4d4a6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e79e5de4e298f1baea10b5f832bd377af4d4a6c" title="Screenshot%20from%202019-07-21%2000-06-14"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6e79e5de4e298f1baea10b5f832bd377af4d4a6c_2_690x284.png" alt="Screenshot%20from%202019-07-21%2000-06-14" data-base62-sha1="fLjGAez3eWHFA3dok9ElnkUdGJ6" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6e79e5de4e298f1baea10b5f832bd377af4d4a6c_2_690x284.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/6e79e5de4e298f1baea10b5f832bd377af4d4a6c_2_1035x426.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/6e79e5de4e298f1baea10b5f832bd377af4d4a6c.png 2x" data-dominant-color="2A2A29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-07-21%2000-06-14</span><span class="informations">1259×519 72.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>error on CNN:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fd49975630b10bb667bcaf20e206643d3fb2d8e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd49975630b10bb667bcaf20e206643d3fb2d8e2" title="Screenshot%20from%202019-07-21%2000-09-41"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd49975630b10bb667bcaf20e206643d3fb2d8e2_2_690x244.png" alt="Screenshot%20from%202019-07-21%2000-09-41" data-base62-sha1="A8Gt2lYafSt5fpFKxpMCdBgO8E2" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd49975630b10bb667bcaf20e206643d3fb2d8e2_2_690x244.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd49975630b10bb667bcaf20e206643d3fb2d8e2_2_1035x366.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/fd49975630b10bb667bcaf20e206643d3fb2d8e2.png 2x" data-dominant-color="3D3D3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-07-21%2000-09-41</span><span class="informations">1214×430 50.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Executed it piece-wise and got this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3b8a66af89f7ee30fb8a16cde1af1bd23bdf654c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3b8a66af89f7ee30fb8a16cde1af1bd23bdf654c" title="Screenshot%20from%202019-07-21%2000-14-07"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3b8a66af89f7ee30fb8a16cde1af1bd23bdf654c_2_690x163.png" alt="Screenshot%20from%202019-07-21%2000-14-07" data-base62-sha1="8uIH0TcnMQBZidzgEgMmRvIiWMQ" width="690" height="163" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3b8a66af89f7ee30fb8a16cde1af1bd23bdf654c_2_690x163.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3b8a66af89f7ee30fb8a16cde1af1bd23bdf654c_2_1035x244.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3b8a66af89f7ee30fb8a16cde1af1bd23bdf654c.png 2x" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-07-21%2000-14-07</span><span class="informations">1226×291 44.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Arvind_Subramaniam
cooked: <p>Thanks for the reply. I got rid of that error but am facing another one.<br>
File “adj_matrix.py”, line 34, in<br>
dim = child.weight.shape[1]<br>
AttributeError: ‘Sequential’ object has no attribute ‘weight’</p>
<p>adj_matrix.py is the file to get an adjacency matrix from each layer in vgg. The adjacency matrix basically treats each channel in a layer as a node in an MLP and prints the edge weights connecting channels in the form of a matrix. For an MLPon NNIST, if I have 784+(128+64+10), the adj_matrix would be 986x986.<br>
The pruned model with 188 and 313 channels mentioned above has been loaded into the adj_matrix.py. (from main_finetune.py in the repo below)<br>
Code for adj_matrix.py:</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torchvision<br>
from torch.utils.data import Dataset, DataLoader, ConcatDataset, random_split<br>
import numpy as np<br>
from torch.utils.data import SubsetRandomSampler<br>
import shutil<br>
import argparse<br>
import os<br>
import models<br>
from models import *<br>
parser = argparse.ArgumentParser(description=‘Adjacency_Matrix’)<br>
parser.add_argument(’–path’, default=’’, type=str, metavar=‘PATH’,<br>
help=‘path to the model (default: none)’)<br>
parser.add_argument(’–arch’,default=’’,type=str)<br>
parser.add_argument(’–dataset’, type=str, default=‘cifar10’,<br>
help=‘training dataset (default: cifar100)’)<br>
parser.add_argument(’–depth’, default=19, type=int,<br>
help=‘depth of the neural network’)<br>
args = parser.parse_args()<br>
if args.arch == ‘MLP’:<br>
model = MLP()<br>
else:<br>
model = vgg(dataset=args.dataset, depth=args.depth)</p>
<p>#<span class="hashtag">#loading</span> the model<br>
if args.path:<br>
checkpoint = torch.load(args.path)<br>
model.load_state_dict(torch.load(args.path),strict=False)<br>
print(f’Loaded Checkpoint:{args.path}’)</p>
<p>for name,child in model.named_children():<br>
dim = child.weight.shape[1]<br>
break<br>
for name,child in model.named_children():<br>
if isinstance(child,nn.BatchNorm1d) or isinstance(child,nn.BatchNorm2d) or isinstance(child,nn.MaxPool2d):<br>
continue<br>
dim += child.bias.shape[0]<br>
adj_matrix = np.zeros((dim,dim))<br>
print(f’The shape of the matrix is {adj_matrix.shape}’)<br>
k = 0<br>
for child in model.children():<br>
if isinstance(child,nn.BatchNorm1d) or isinstance(child,nn.BatchNorm2d) or isii<br>
nstance(child,nn.MaxPool2d):<br>
continue<br>
for i in range(child.weight.shape[0]):<br>
for j in range(child.weight.shape[1]):<br>
#<span class="hashtag">#Take</span> mirror image later<br>
adj_matrix[k + child.weight.shape[1]+i][k+j] = child.weight[i][j]<br>
k += child.weight.shape[1]</p>
<p>#<span class="hashtag">#Making</span> Symmetrical<br>
for i in range(adj_matrix.shape[0]):<br>
for j in range(<ins>adj_matrix</ins><ins>.</ins><ins>shape</ins><ins>[</ins>1<ins>]</ins>):<br>
adj_matrix[j][i] = adj_matrix[i][j]</p>
<p>The pruning method is basically applying L1 norm on the scaling factor gamma in batchNorm. After that, the small scaling factors are eliminated leadng to elimination of nodes in the hidden layers of the model. In case of VGG, some channels in intermediate layers would vanish.</p>
<p>Here is the repo: <a href="https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/network-slimming" rel="nofollow noopener">https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/network-slimming</a></p>

username: ptrblck
cooked: <p>How are you changing the model architecture based on this adj_matrix?<br>
Are you using this matrix directly or are you at some point resetting the model parameters?</p>
<p>PS: You can add code snippets using three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>