Regarding Weights

username: Ganga
cooked: <p>if not self.Linear:</p>
<pre><code>        self.bn = nn.BatchNorm2d(input_channels, eps=1e-4, momentum=0.1, affine=True)

        self.conv = nn.Conv2d(input_channels, output_channels,

                kernel_size=kernel_size, stride=stride, padding=padding, groups=groups)
</code></pre>
<p>the weights which i printed using self.conv.weight and the weights which i obtained from model.named_parameters() are totally different can anyone please provide the difference between them and also which is the correct weight values</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet, which shows the issue you are seeing?<br>
It’s unclear how and when you are printing the weights based on your code snippet.</p>

username: Ganga
cooked: <p>loading the pretrained model here:</p>
<p>if os.path.isfile(args.resume):</p>
<pre><code>        print("=&gt; loading checkpoint '{}'".format(args.resume))

        checkpoint = torch.load(args.resume)

        args.start_epoch = checkpoint['epoch']

        best_prec1 = checkpoint['best_prec1']

        model.load_state_dict(checkpoint['state_dict'])

        optimizer.load_state_dict(checkpoint['optimizer'])

        print("=&gt; loaded checkpoint '{}' (epoch {})"

              .format(args.resume, checkpoint['epoch']))

        del checkpoint

    else:

        print("=&gt; no checkpoint found at '{}'".format(args.resume))

#print("before fusing model",model)

fuse_module(model)
</code></pre>
<h1>print(model.state_dict())</h1>
<pre><code>cudnn.benchmark = True

for key, value in model.named_parameters():
</code></pre>
<p>this is where i am printing weight</p>
<p>class BinConv2d(nn.Module): # change the name of BinConv2d</p>
<pre><code>def __init__(self, input_channels, output_channels,

        kernel_size=-1, stride=-1, padding=-1, groups=1, dropout=0,

        Linear=False):

    super(BinConv2d, self).__init__()

    self.layer_type = 'BinConv2d'

    self.kernel_size = kernel_size

    self.stride = stride

    self.padding = padding

    self.dropout_ratio = dropout

    if dropout!=0:

        self.dropout = nn.Dropout(dropout)

    self.Linear = Linear

    if not self.Linear:

        self.bn = nn.BatchNorm2d(input_channels, eps=1e-4, momentum=0.1, affine=True)

        self.conv = nn.Conv2d(input_channels, output_channels,
</code></pre>
<p>and this place where i am printimg weight</p>
<pre><code>                kernel_size=kernel_size, stride=stride, padding=padding, groups=groups)

    else:

        self.bn = nn.BatchNorm1d(input_channels, eps=1e-4, momentum=0.1, affine=True)

        self.linear = nn.Linear(input_channels, output_channels)

    self.relu = nn.ReLU(inplace=True)



def forward(self, x):

    x = self.bn(x)

    x = BinActive()(x)

    if self.dropout_ratio!=0:

        x = self.dropout(x)

    if not self.Linear:

        x = self.conv(x)

    else:

        x = self.linear(x)

    x = self.relu(x)

    return x
</code></pre>
<p>class AlexNet(nn.Module):</p>
<pre><code>def __init__(self, num_classes=1000):

    super(AlexNet, self).__init__()

    self.num_classes = num_classes

    self.features = nn.Sequential(

        nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=0),

        nn.BatchNorm2d(96, eps=1e-4, momentum=0.1, affine=True),

        nn.ReLU(inplace=True),

        nn.MaxPool2d(kernel_size=3, stride=2),

        BinConv2d(96, 256, kernel_size=5, stride=1, padding=2, groups=1),

        nn.MaxPool2d(kernel_size=3, stride=2),

        BinConv2d(256, 384, kernel_size=3, stride=1, padding=1),

        BinConv2d(384, 384, kernel_size=3, stride=1, padding=1, groups=1),

        BinConv2d(384, 256, kernel_size=3, stride=1, padding=1, groups=1),

        nn.MaxPool2d(kernel_size=3, stride=2),

    )

    self.classifier = nn.Sequential(

        BinConv2d(256 * 6 * 6, 4096, Linear=True),

        BinConv2d(4096, 4096, dropout=0.5, Linear=True),

        nn.BatchNorm1d(4096, eps=1e-3, momentum=0.1, affine=True),

        nn.Dropout(),

        nn.Linear(4096, num_classes),

    )

def forward(self, x):

    x = self.features(x)

    x = x.view(x.size(0), 256 * 6 * 6)

    x = self.classifier(x)

    return x</code></pre>

username: Ganga
cooked: <p>i am loading the pretrained model and i am printing the weight from model.named_parameter()<br>
and also i am printing the weights from self.conv.weight<br>
but the weight i am getting from these two are different</p>
<p>class BinConv2d(nn.Module): # change the name of BinConv2d</p>
<pre><code>def __init__(self, input_channels, output_channels,

        kernel_size=-1, stride=-1, padding=-1, groups=1, dropout=0,

        Linear=False):

    super(BinConv2d, self).__init__()

    self.layer_type = 'BinConv2d'

    self.kernel_size = kernel_size

    self.stride = stride

    self.padding = padding

    self.dropout_ratio = dropout

    if dropout!=0:

        self.dropout = nn.Dropout(dropout)

    self.Linear = Linear

    if not self.Linear:

        self.bn = nn.BatchNorm2d(input_channels, eps=1e-4, momentum=0.1, affine=True)

        self.conv = nn.Conv2d(input_channels, output_channels,

                kernel_size=kernel_size, stride=stride, padding=padding, groups=groups)

    else:

        self.bn = nn.BatchNorm1d(input_channels, eps=1e-4, momentum=0.1, affine=True)

        self.linear = nn.Linear(input_channels, output_channels)

    self.relu = nn.ReLU(inplace=True)



def forward(self, x):

    x = self.bn(x)

    x = BinActive()(x)

    if self.dropout_ratio!=0:

        x = self.dropout(x)

    if not self.Linear:

        x = self.conv(x)

    else:

        x = self.linear(x)

    x = self.relu(x)

    return x
</code></pre>
<p>class AlexNet(nn.Module):</p>
<pre><code>def __init__(self, num_classes=1000):

    super(AlexNet, self).__init__()

    self.num_classes = num_classes

    self.features = nn.Sequential(

        nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=0),

        nn.BatchNorm2d(96, eps=1e-4, momentum=0.1, affine=True),

        nn.ReLU(inplace=True),

        nn.MaxPool2d(kernel_size=3, stride=2),

        BinConv2d(96, 256, kernel_size=5, stride=1, padding=2, groups=1),

        nn.MaxPool2d(kernel_size=3, stride=2),

        BinConv2d(256, 384, kernel_size=3, stride=1, padding=1),

        BinConv2d(384, 384, kernel_size=3, stride=1, padding=1, groups=1),

        BinConv2d(384, 256, kernel_size=3, stride=1, padding=1, groups=1),

        nn.MaxPool2d(kernel_size=3, stride=2),

    )</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce it, as I’m getting the same values for the direct access and the <code>state_dict</code>:</p>
<pre><code class="lang-python">model = AlexNet()
state_dict = model.state_dict()

print((model.features[4].conv.weight == state_dict['features.4.conv.weight']).all())
&gt; tensor(True)
</code></pre>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Ganga
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="87446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">state_dict = model.state_dict()

print((model.features[4].conv.weight == state_dict['features.4.conv.weight']).all())
</code></pre>
</blockquote>
</aside>
<p>print((model.features[4].conv.weight == state_dict[‘features.4.conv.weight’]).all())<br>
TypeError: ‘DataParallel’ object does not support indexing<br>
can you please provide the solution ,since i am very new to python,pytorch</p>

username: ptrblck
cooked: <p>You might have wrapped <code>model.features</code> into <code>nn.DataParallel</code>.<br>
If that’s the case, you could add the <code>.module</code> attribute to the access:</p>
<pre><code class="lang-python">model.features.module[4]
</code></pre>
<p>If that doesn’t work, could you post the model architecture, so that we could have a look?</p>

username: Ganga
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="87446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>state_dict['features.4.conv.weight']).all())</code></p>
</blockquote>
</aside>
<p>File “/content/XNOR-Net-PyTorch/ImageNet/networks/model_list/alexnet.py”, line 871, in alexnet<br>
print((model.features.module[4] == state_dict[‘features.module.4.conv.weight’]).all())<br>
AttributeError: ‘bool’ object has no attribute ‘all’<br>
“”“(‘after fusing model’, AlexNet(<br>
(features): DataParallel(<br>
(module): Sequential(<br>
(0): Conv2d(3, 96, kernel_size=(11, 11), stride=(4, 4))<br>
(1): DummyModule()<br>
(2): ReLU(inplace=True)<br>
(3): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(4): BinConv2d(<br>
(bn): BatchNorm2d(96, eps=0.0001, momentum=0.1, affine=True, track_running_stats=True)<br>
(conv): Conv2d(96, 256, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))<br>
(relu): ReLU(inplace=True)<br>
)<br>
(5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(6): BinConv2d(<br>
(bn): BatchNorm2d(256, eps=0.0001, momentum=0.1, affine=True, track_running_stats=True)<br>
(conv): Conv2d(256, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu): ReLU(inplace=True)<br>
)<br>
(7): BinConv2d(<br>
(bn): BatchNorm2d(384, eps=0.0001, momentum=0.1, affine=True, track_running_stats=True)<br>
(conv): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu): ReLU(inplace=True)<br>
)<br>
(8): BinConv2d(<br>
(bn): BatchNorm2d(384, eps=0.0001, momentum=0.1, affine=True, track_running_stats=True)<br>
(conv): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu): ReLU(inplace=True)<br>
)<br>
(9): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
)<br>
(classifier): Sequential(<br>
(0): BinConv2d(<br>
(bn): BatchNorm1d(9216, eps=0.0001, momentum=0.1, affine=True, track_running_stats=True)<br>
(linear): Linear(in_features=9216, out_features=4096, bias=True)<br>
(relu): ReLU(inplace=True)<br>
)<br>
(1): BinConv2d(<br>
(dropout): Dropout(p=0.5, inplace=False)<br>
(bn): BatchNorm1d(4096, eps=0.0001, momentum=0.1, affine=True, track_running_stats=True)<br>
(linear): Linear(in_features=4096, out_features=4096, bias=True)<br>
(relu): ReLU(inplace=True)<br>
)<br>
(2): BatchNorm1d(4096, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)<br>
(3): Dropout(p=0.5, inplace=False)<br>
(4): Linear(in_features=4096, out_features=1000, bias=True)<br>
)<br>
))”“”</p>

username: ptrblck
cooked: <p>Could you print the <code>type</code> and the shape of both tensors you are trying to compare, please?</p>

username: Ganga
cooked: <p>print(model.features.module[4].conv.weight.size())<br>
(256, 96, 5, 5)<br>
print(type(model.features.module[4]))<br>
&lt;class ‘model_list.alexnet.BinConv2d’&gt;</p>
<p>print(state_dict[‘features.module.4.conv.weight’].size())<br>
(256,96,5,5)<br>
print(type(state_dict[‘features.module.4.conv.weight’]))<br>
&lt;class ‘torch.Tensor’&gt;<br>
print((model.features.module[4].conv.weight == state_dict[‘features.module.4.conv.weight’]).all())<br>
tensor(True, device=‘cuda:0’)</p>

username: Ganga
cooked: <aside class="quote quote-modified" data-post="1" data-topic="88069">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ganga/48/28549_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/regarding-weight-and-bias-initialization-for-sequential-model-from-the-text-file/88069">Regarding weight and bias initialization for sequential model from the text file</a> 
  </div>
  <blockquote>
    i have reference model which is evaluated using pretrained model,i have weights,bias and input to the function…to check individually i took single conv2d has a sequential model. and i have initialized the input,weights and bias from the text file(values obtained from the refernce model) 
but the problem is output from the Conv2d is not matching with the refernce model output (which is also a Conv2d with all same attributes) 
here is my code… 
“”"import torch 
import torch.nn as nn 
import csv 
i…
  </blockquote>
</aside>

<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  if you don’t mind can you please go through this and provide the solution</p>

username: ptrblck
cooked: <p>This line of code seems to work now:</p>
<pre><code class="lang-python">print((model.features.module[4].conv.weight == state_dict[‘features.module.4.conv.weight’]).all())
&gt; tensor(True, device=‘cuda:0’)
</code></pre>
<p>while it seemed to create the <code>AttributeError</code> before, which is strange.<br>
Anyway, good to see it’s working now.</p>

username: Ganga
cooked: <p>Thanks for your time</p>