Check if models have same weights

lwang77: <p>Hello, I want to be able to check if two models have the same weights in their layers. After poking around, I couldn’t find a function that did this, so I implemented my own.</p>
<pre><code class="lang-auto">def compareModelWeights(model_a, model_b):
    module_a = model_a._modules
    module_b = model_b._modules
    if len(list(module_a.keys())) != len(list(module_b.keys())):
        return False
    a_modules_names = list(module_a.keys())
    b_modules_names = list(module_b.keys())
    for i in range(len(a_modules_names)):
        layer_name_a = a_modules_names[i]
        layer_name_b = b_modules_names[i]
        if layer_name_a != layer_name_b:
            return False
        layer_a = module_a[layer_name_a]
        layer_b = module_b[layer_name_b]
        if (
            (type(layer_a) == nn.Module) or (type(layer_b) == nn.Module) or
            (type(layer_a) == nn.Sequential) or (type(layer_b) == nn.Sequential)
            ):
            if not compareModelWeights(layer_a, layer_b):
                return False
        if hasattr(layer_a, 'weight') and hasattr(layer_b, 'weight'):
            if not torch.equal(layer_a.weight.data, layer_b.weight.data):
                return False
    return True
</code></pre>
<p>It’s recursive because modules can have a tree structure.<br>
If there is already a function that does this, I would love to use it in my code. If not, this might be a useful addition. Please let me know if there are any errors.</p>

smth: <p>you can get <code>model1.parameters()</code> and <code>model2.parameters()</code> and use:</p>
<pre><code class="lang-python">for p1, p2 = zip(model1.parameters(), model2.parameters()):
    if p1.data.ne(p2.data).sum() &gt; 0:
        return False
return True
</code></pre>

roee: <p>Hey, just a small correction:<br>
It should be:</p>
<aside class="quote no-group quote-modified" data-username="smth" data-post="2" data-topic="4351">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>for p1, p2 <strong>in</strong> zip(model1.parameters(), model2.parameters()):<br>
if p1.data.ne(p2.data).sum() &gt; 0:<br>
return False<br>
return True</p>
</blockquote>
</aside>

roee: <p>I don’t think an abs is necessary (or wanted)… you probably want to distinguish between negative and positive values.</p>
<ul>
<li>
</li>
</ul>
<p>ne (not equal) returns Boolean values which should be simply summed.</p>

aasharma90: <p>I found this to be a better approach (also compares batch_norm layers for their running_mean and running_var params) -</p>
<pre><code class="lang-auto">def compare_models(model_1, model_2):
    models_differ = 0
    for key_item_1, key_item_2 in zip(model_1.state_dict().items(), model_2.state_dict().items()):
        if torch.equal(key_item_1[1], key_item_2[1]):
            pass
        else:
            models_differ += 1
            if (key_item_1[0] == key_item_2[0]):
                print('Mismtach found at', key_item_1[0])
            else:
                raise Exception
    if models_differ == 0:
        print('Models match perfectly! :)')
</code></pre>

pinocchio: <p>why isn’t this or something like this a built in method?</p>

ptrblck: <p>Feature requests and PR are always welcome, if you think this feature is useful and often used. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Andrew_Edmondson: <p>Adding an answer since I spent a 1/2 hour trying to figure out the same thing to check whether a network was really updating (spoiler, it wasn’t).</p>
<p>The easiest way I found to directly compare two networks is to convert the state_dict to a string representation and compare them directly.</p>
<pre><code class="lang-auto">
state_a = network.state_dict().__str__()
loss.backward()
opt.step()
state_b = network.state_dict().__str__()
 
if state_a == state_b:
    print("Network not updating.")
</code></pre>

desaibhargav: <p>Thanks for this!</p>
<p>PS: I think <code>str(network.state_dict())</code> works too!</p>

MaRodriguezB777: <p>This method does not work if one of the models is a superset of the other because the zip could get rid of the differing excess</p>