Is the PyTorch NLP DropConnect implementation correct?

chhablanig: <p>The implementation for basic Weight Drop in the PyTorch NLP source code is as follows:</p>
<pre><code class="lang-auto">def _weight_drop(module, weights, dropout):
    """
    Helper for `WeightDrop`.
    """

    for name_w in weights:
        w = getattr(module, name_w)
        del module._parameters[name_w]
        module.register_parameter(name_w + '_raw', Parameter(w))

    original_module_forward = module.forward

    def forward(*args, **kwargs):
        for name_w in weights:
            raw_w = getattr(module, name_w + '_raw')
            w = torch.nn.functional.dropout(raw_w, p=dropout, training=module.training)
            setattr(module, name_w, w)

        return original_module_forward(*args, **kwargs)

    setattr(module, 'forward', forward)
</code></pre>
<p>This method, clearly, uses the dropout function available in torch.nn.functional to perform the dropping of the weights.</p>
<p>I wasn’t able to find the actual implementation of that dropout function, but I assume it is correct as it is widely used. In the <a href="https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf" rel="noopener nofollow ugc">Dropout Paper</a> the authors mention differences in the two phases of training and testing. During training, they drop the activations randomly, which I’m sure the dropout implementation has worked out correctly, and during testing, they multiply the weights in the connecting layer, by p, so that the expected value of the activations is same in the next layer as it was in training, which makes sense. I assume that Dropout does exactly that when it takes ‘module.training’ as an argument, to decide what to do.</p>
<p>For <a href="http://yann.lecun.com/exdb/publis/pdf/wan-icml-13.pdf" rel="noopener nofollow ugc">DropConnect</a>, however, the authors mention the following algorithms for training and testing:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/199731a62a6994b006d76a2d2742b906c048a998.png" alt="image" data-base62-sha1="3EnRPHcYRiqiCI2K82M3Y7Yl0bS" width="321" height="251"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e29068cb9259e1794a15ddbdbe6b8498d0ad6916.png" alt="image" data-base62-sha1="wkhiQluJHSavJ17wqXeT7nFbNYO" width="323" height="229"></p>
<p>Basically, the training part is the same as Dropout, in that here the weights are dropped, just like Dropout. But, during inference, the process changes. The DropConnect paper describes that ‘averaging’ the values by multiplying the weights with p in case of Dropout, is not justified mathematically as the averaging is done before applying the activation functions.</p>
<p>So, for DropConnect paper does this: take the values of the activations, just before the DropConnect weights. Finds the mean and variance of the next layer values, based on W,p and the inputs. Draws Z (could be a hyperparameter as far as I understand) samples from the distribution, assuming it is a Gaussian, to create lots (Z) of possible values. Applies activations on all of those values, and then finally average over Z to get the input for the next weights/layer (softmax in their case).</p>
<p>Since, the original Dropout doesn’t consider this, and if that is the way the implementation for PyTorch Dropout is, then essentially, the DropConnect implementation linked above must be ‘wrong’ (not what they explained in the paper).</p>
<p>I am not sure why DropConnect hasn’t gained much traction here or in DL research, but can someone please explain whether the above implementation is right or wrong?<br>
And, in case it is indeed wrong, what can I do to implement DropConnect correctly?</p>

henrique: <p>Both DropConnect in EfficientNet and PyTorch Dropout seem to divide by (1 - p) to help maintaining the mean (although this isn’t true if there are negative values, e.g. after batchnorm, or swish)<br>
respectively:</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/utils.py#L149" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/utils.py#L149" target="_blank" rel="noopener nofollow ugc">lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/utils.py#L149</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="139" style="counter-reset: li-counter 138 ;">
<li>        return inputs</li>
<li>
<li>    batch_size = inputs.shape[0]</li>
<li>    keep_prob = 1 - p</li>
<li>
<li>    # generate binary_tensor mask according to probability (p for 0, 1-p for 1)</li>
<li>    random_tensor = keep_prob</li>
<li>    random_tensor += torch.rand([batch_size, 1, 1, 1], dtype=inputs.dtype, device=inputs.device)</li>
<li>    binary_tensor = torch.floor(random_tensor)</li>
<li>
<li class="selected">    output = inputs / keep_prob * binary_tensor</li>
<li>    return output</li>
<li>
<li>
<li>def get_width_and_height_from_size(x):</li>
<li>    """Obtain height and width from x.</li>
<li>
<li>    Args:</li>
<li>        x (int, tuple or list): Data size.</li>
<li>
<li>    Returns:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/v1.8.1/aten/src/ATen/native/Dropout.cpp#L62" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/v1.8.1/aten/src/ATen/native/Dropout.cpp#L62" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/v1.8.1/aten/src/ATen/native/Dropout.cpp#L62</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="52" style="counter-reset: li-counter 51 ;">
<li>
<li>  at::Tensor b; // used for alpha_dropout only</li>
<li>  auto noise = feature_dropout ? make_feature_noise(input) : at::empty_like(input, LEGACY_CONTIGUOUS_MEMORY_FORMAT);</li>
<li>  noise.bernoulli_(1 - p);</li>
<li>  if (alpha_dropout) {</li>
<li>    constexpr double alpha = 1.7580993408473766;</li>
<li>    double a = 1. / std::sqrt((alpha * alpha * p + 1) * (1 - p));</li>
<li>    b = noise.add(-1).mul_(alpha * a).add_(alpha * a * p);</li>
<li>    noise.mul_(a);</li>
<li>  } else {</li>
<li class="selected">    noise.div_(1 - p);</li>
<li>  }</li>
<li>
<li>  if (!alpha_dropout) {</li>
<li>    return multiply&lt;inplace&gt;(input, noise);</li>
<li>  } else {</li>
<li>    return multiply&lt;inplace&gt;(input, noise).add_(b);</li>
<li>  }</li>
<li>}</li>
<li>
<li>#define ALIAS_SPECIALIZATION(ALIAS_NAME, IS_FEATURE, IS_ALPHA)                      \</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The difference between dropoutNd and dropconnect (as in efficientnet) is that the former drops some channels for each batch sample (1st and 2nd tensor dimensions, batch and channel), but dropconnect drops whole samples all together (only the 1st batch dimension).</p>
<p>So as a late answer, I don’t think the NLP implementation above is equivalent to any of two here. I believe dropout1d would be an approximation to the paper’s algorithm, as it tries to maintain the statistics between training and inference, and drops whole feature channels of each sample.</p>

numpee: <p>Is the EfficientNet implementation of DropConnect correct? It seems like they use this function <a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/7e8b0d312162f335785fb5dcfa1df29a75a1783a/efficientnet_pytorch/model.py#L130" rel="noopener nofollow ugc">here</a>, but if you apply this function to the <code>x</code> (activation) then it is not DropConnect, but rather just simple Dropout. Let me know if i’m missing something…</p>

henrique: <p>Dropout2d drops random channels from each image, their drop_connect drops whole images from the batch.<br>
Therefore, every image is only seen by a subset of layers.<br>
I am not sure what is the ‘correct’ way of doing DropConnect, but it seems to work reasonably well.</p>

numpee: <p>Maybe this is a different type of DropConnect that they are referencing. I was under the impression that DropConnect referred to dropping the weights of a layer during the forward pass, as described in this <a href="http://yann.lecun.com/exdb/publis/pdf/wan-icml-13.pdf" rel="noopener nofollow ugc">paper</a>. As far as I understand, there would be no way to implement this by using the activations of a layer’s output - it would only work by masking the weights of the layer itself.</p>

henrique: <p>Yeah, it seems to be a confusion with names here. I believe <a href="https://arxiv.org/pdf/1905.11946.pdf" rel="noopener nofollow ugc">efficientnet</a>’s drop_connect() method linked above is their implementation of <a href="https://arxiv.org/pdf/1603.09382.pdf" rel="noopener nofollow ugc">stochastic depth</a></p>