DropConnect implementation

username: michaelklachko
cooked: <p>Can someone point out what are the advantages of <a href="https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/nn/weight_drop.html" rel="nofollow noopener">this implementation</a> of DropConnect over a simpler method like this:</p>
<pre><code class="lang-auto">for i in range(num_batches):

    orig_params = []
    for n, p in model.named_parameters():
        orig_params.append(p.clone())
        p.data = F.dropout(p.data, p=drop_prob) * (1 - drop_prob)

    output = model(input)

    for orig_p, (n, p) in zip(orig_params, model.named_parameters()):  
        p.data = orig_p.data

    loss = nn.CrossEntropyLoss()(output, label)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>

username: tom
cooked: <p>Some thoughts:</p>
<p>Don’t use <code>.data</code> these days! It’s bad for you, really! <code>with torch.no_grad():</code> will do fine.</p>
<p>In fact, you probably should copy back the original values between the backward and the step. For a reasonably complicated model (more than one layer), your gradients might be off from this and if you had stayed clear of using .data PyTorch would have told you.</p>
<p>You multiply with <code>1- drop_prob</code>, which seems unusual.</p>
<p>The convenience of a wrapper: In any single instance is easily done manually, but now you would want to apply this to some weights rather than all. Is it still as straightforward?</p>
<p>The safety of a wrapper: Having a well-tested wrapper saves you from implementation mistakes. (See above.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: michaelklachko
cooked: <p>I’m forced to use p.data, because if I replace p.data with p, the weights won’t be modified. For example, if you print p values before and after the dropout line, you will see that only p.data method works (zeros out some weights). Unless you mean something else?</p>
<p>Good point about moving weight restore after backward call! Thank you.</p>
<p>I have to multiply by <code>1 - drop_prob</code> because dropout scales its input internally by <code>1 / (1 - drop_prob)</code>, and if I don’t do this the accuracy drops sharply: with drop_prob=0.05 it does not even converge if I don’t scale back the weights. I’m not sure what’s going on, but I suspect it might have something to do with batchnorm. Any ideas?</p>
<p>What do you mean “is it still straightforward?” With my method it’s much easier to apply dropconnect selectively, I don’t have to create wrappers for every single layer type, and I don’t have to modify my model forward function. I agree with your point about safety.</p>

username: michaelklachko
cooked: <p><a class="mention" href="/u/tom">@tom</a>, should open a bug about p not being zeroed out when used in dropout call?</p>

username: tom
cooked: <aside class="quote no-group" data-username="michaelklachko" data-post="4" data-topic="70921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michaelklachko/48/541_2.png" class="avatar"> michaelklachko:</div>
<blockquote>
<p>open a bug about p not being zeroed</p>
</blockquote>
</aside>
<p>No.  <code>p = something</code> just doesn’t overwrite elements of <code>p</code>, but instead assigns a new thing to the name <code>p</code>. That’s inherent in how Python works, you want <code>p.copy_(...)</code>.<br>
There are extremely few reasons to use p.data, and chances are you’re doing it wrong if you’re using it. (And people are getting serious about removing it properly, so hopefully it’ll go away soon.)</p>
<p>For the scaling, I don’t know. From a cursory look at the Gal and Ghahramani paper, maybe they also use the plain Bernoulli. I’d probably multiply with <code>torch.bernoulli(weight, 1-drop_prob)</code> instead of using dropout and scaling.</p>
<p>Best regards</p>
<p>Thomas</p>

username: michaelklachko
cooked: <p>Ok, it makes sense, I replaced all “p.data =” with “p.copy_” and added no_grad() context. No difference in performance that I can see, but if it’s safer, so be it.</p>
<p>I ran a few experiments with scaling, and yes, it seems like scaling is necessary, otherwise batchnorm will screw things up during inference. Recomputing batch statistics during inference also fixes the issue (same good accuracy with or without scaling), but that’s obviously not a solution. Not sure how to use torch.bernoulli did you mean binomial? Tried generating binomial masks, but I don’t see a good way to generate them quickly on GPU. I could only do mask = binomial_distr.sample(p.size).cuda() and this is very slow.</p>

username: tom
cooked: <p>I’ll have to admit that the bernoulli API is more than a little bit awkward (it should be a factory function just like randn, really), but</p>
<pre><code class="lang-python">mask = torch.bernoulli(torch.randn(1, 1, device='cuda').expand(*weight.shape), 1-drop_prob)
</code></pre>
<p>should work.</p>
<p>Best regards</p>
<p>Thomas</p>

username: G.M
cooked: <p>Hi, I have an irrelevant question: is <code>.data</code> equivalent to <code>.detach()</code>?</p>

username: tom
cooked: <p>No <code>.data</code> is more risky because it also skips the versioning count. It <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" rel="nofollow noopener">has long been deprecated</a>. The <a href="https://github.com/pytorch/pytorch/issues/30987" rel="nofollow noopener">remove <code>.data</code> issue</a> has great technical discussion.</p>
<p>Best regards</p>
<p>Thomas</p>

username: bobbyx27
cooked: <p>I believe <a href="https://discuss.pytorch.org/t/is-the-pytorch-nlp-dropconnect-implementation-correct/96282">this post</a> may have a point about proposed dropconnect implementations being wrong (?), when relying on ‘dropout’ method</p>