Understanding cdist() function

promach: <p>What does this <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-594212162" rel="nofollow noopener">new_cdist()</a> function actually do ?</p>
<p>I mean that it seems to be related to a <a href="https://github.com/huawei-noah/AdderNet/issues/5" rel="nofollow noopener">new type of back-propagation equation and adaptive learning rate</a>.</p>
<pre><code class="lang-auto">def new_cdist(p, eta):
    class cdist(torch.autograd.Function):
        @staticmethod
        def forward(ctx, W, X):
            ctx.save_for_backward(W, X)
            out = -torch.cdist(W, X, p)
            return out

        @staticmethod
        def backward(ctx, grad_output):
            W, X = ctx.saved_tensors
            grad_W = grad_X = None
            if ctx.needs_input_grad[0]:
                _temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(W.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                grad_W = torch.matmul(grad_output, _temp)
                # print('before norm: ', torch.norm(grad_W))
                grad_W = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W) * grad_W
                print('after norm: ', torch.norm(grad_W))
            if ctx.needs_input_grad[1]:
                _temp1 = torch.unsqueeze(W, 2).expand(W.shape[0], W.shape[1], X.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(X.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                _temp = torch.nn.functional.hardtanh(_temp, min_val=-1., max_val=1.)
                grad_X = torch.matmul(grad_output.transpose(0, 1), _temp)
            return grad_W, grad_X
    return cdist().apply
</code></pre>

ptrblck: <p>I think the authors of the repository will give you a better answer, but based on the code it seems the backward method was reimplemented for the negated <a href="https://pytorch.org/docs/stable/torch.html#torch.cdist">cdist</a> method.</p>
<p>From the docs of <code>cdist</code>:</p>
<blockquote>
<p>Computes batched the p-norm distance between each pair of the two collections of row vectors.</p>
</blockquote>
<p>I’m not familiar with the implementation of the repository and would recommend to create an issue there.</p>

promach: <p>Yes, see <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-613362421" rel="nofollow noopener">https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-613362421</a></p>
<p>Do you have any idea what <code>_temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)</code> actually does ?</p>
<p>Same for <code>_temp1</code> and <code>_temp2</code></p>
<pre><code class="lang-auto">_temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)

_temp2 = torch.unsqueeze(W.transpose(0, 1), 1)
</code></pre>

ptrblck: <p>Let’s go through all operations separately in the calls:</p>
<ul>
<li>
<code>toch.cdist(a, b, p)</code> calculates the <code>p-norm</code> distance between each pair of the two collections of row vectos, as explained above</li>
<li>
<code>.squeeze()</code> will remove all dimensions of the result tensor where <code>tensor.size(dim) == 1</code>
</li>
<li>
<code>.transpose(0, 1)</code> will permute dim0 and dim1, i.e. it’ll “swap” these dimensions</li>
<li>
<code>torch.unsqueeze(tensor, dim)</code> will add a new dimension specific by <code>dim</code><br>
<code> expand()</code> will manipulate the meta data to create a view with the new shape (no copy of the data)<br>
<code> </code>permute<code>is similar to</code>transpose` for multiple dimensions</li>
</ul>

promach: <ol>
<li>
<p>What is the actual purpose of <code>_temp1</code> and <code>_temp2</code> in the calculation for <a href="https://arxiv.org/pdf/1912.13200.pdf#page=4" rel="nofollow noopener">backward propagation</a> ?</p>
</li>
<li>
<p>How is <code>needs_input_grad[]</code> used ?</p>
</li>
<li>
<p>Why is there <code>sqrt()</code> operation when the AdderNet paper does not use it in backward propagation equation ?</p>
</li>
<li>
<p>How is this <code>new_dist()</code> function different from the implementation of <code>forward()</code> and <code>backward()</code> in <a href="https://pytorch.org/docs/master/notes/extending.html" rel="nofollow noopener">https://pytorch.org/docs/master/notes/extending.html</a> ?</p>
</li>
<li>
<p>How can I eliminate GPU out-of-memory runtime error reported in <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-613362421" rel="nofollow noopener">new_dist()</a> ?</p>
</li>
</ol>

promach: <p>Have a look at <a href="https://stackoverflow.com/a/61229485/6422632" rel="nofollow noopener">https://stackoverflow.com/a/61229485/6422632</a></p>
<p>Some of the questions have been answered.</p>

promach: <p>Any comments on <a href="https://www.reddit.com/r/pytorch/comments/g31hmd/replacing_torchcdist_function_to_eliminate_gpu/" rel="nofollow noopener">https://www.reddit.com/r/pytorch/comments/g31hmd/replacing_torchcdist_function_to_eliminate_gpu/</a> ?</p>

promach: <p>I was told that <code>_temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)</code> seems to use up a lot of GPU memory during <a href="https://github.com/promach/PyTorch-YOLOv3/blob/addernet/adder.py#L29" rel="nofollow noopener">intermediate calculation</a> .</p>
<p>But how to modify this line to use less GPU memory ?</p>

ptrblck: <p>All mentioned operations manipulate the metadata (shape, stride) of the tensor and will not use more memory in this particular line of code.<br>
However, the result will be a non-contiguous tensor, and the next <code>_temp1.contiguous()</code> call (either manually or in a function) will trigger the copy and use more memory.<br>
You cannot avoid this copy, if a method needs a contiguous tensor to operate on.</p>
<p>Here is a small example showing the memory usage:</p>
<pre><code class="lang-python">print(torch.cuda.memory_allocated() / 1024**2)
&gt; 0.0

x = torch.randn(256, 256, device='cuda')
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 0.25

x = x.unsqueeze(2).expand(x.size(0), x.size(1), 256).permute(1, 0, 2)
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 0.25

x = x.contiguous()
print(torch.cuda.memory_allocated() / 1024**2)
&gt; 64.0
</code></pre>

promach: <p>I was told to either</p>
<ol>
<li>
<strong>decrease the tensors dimension, then increase the tensors dimension again</strong> or</li>
<li>
<strong>increase the tensors dimensions and then use depth-wise convolution</strong> or</li>
<li><strong>use mixed-precision training</strong></li>
</ol>
<p>I know I can use <a href="https://github.com/NVIDIA/apex" rel="nofollow noopener">https://github.com/NVIDIA/apex</a> for the mixed precision training.<br>
However, I am not sure how to modify the code for suggestions 1 and 2 above.</p>

promach: <p>So, are you implying that <code>torch.cdist()</code> function itself contains <code>contiguous()</code> call ?</p>
<p>I tried to replace <code>torch.cdist()</code> using <code>fast_cdist()</code> as shown below. However, I still have GPU out-of-memory error.</p>
<pre><code class="lang-auto">def fast_cdist(x1, x2):
    adjustment = x1.mean(-2, keepdim=True)
    x1 = x1 - adjustment
    x2 = x2 - adjustment  # x1 and x2 should be identical in all dims except -2 at this point

    # Compute squared distance matrix using quadratic expansion
    # But be clever and do it with a single matmul call
    x1_norm = x1.pow(2).sum(dim=-1, keepdim=True)
    x1_pad = torch.ones_like(x1_norm)
    x2_norm = x2.pow(2).sum(dim=-1, keepdim=True)
    x2_pad = torch.ones_like(x2_norm)
    x1_ = torch.cat([-2. * x1, x1_norm, x1_pad], dim=-1)
    x2_ = torch.cat([x2, x2_pad, x2_norm], dim=-1)
    res = x1_.matmul(x2_.transpose(-2, -1))

    # Zero out negative values
    res.clamp_min_(1e-30).sqrt_()
    return res
</code></pre>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-615270730" target="_blank" rel="nofollow noopener">github.com/huawei-noah/AdderNet</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-615270730" target="_blank" rel="nofollow noopener">RuntimeError: _cdist_backward requires X2 to be contiguous</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-03-03" data-time="02:19:34" data-timezone="UTC">02:19AM - 03 Mar 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-03-10" data-time="06:20:31" data-timezone="UTC">06:20AM - 10 Mar 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/ranery" target="_blank" rel="nofollow noopener">
          <img alt="ranery" src="https://avatars3.githubusercontent.com/u/18202368?v=4" class="onebox-avatar-inline" width="20" height="20">
          ranery
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Hi, I am trying to train your addernet, but it returns me one runtime error, which I supposed attributes to .continuous()...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <aside class="quote no-group" data-username="promach" data-post="11" data-topic="76296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/promach/48/22571_2.png" class="avatar"> promach:</div>
<blockquote>
<p>So, are you implying that <code>torch.cdist()</code> function itself contains <code>contiguous()</code> call ?</p>
</blockquote>
</aside>
<p>Probably yes, and <a href="https://github.com/pytorch/pytorch/blob/a836c4ca78b72ecc8e0664e1b684af64ce83be42/aten/src/ATen/native/Distance.cpp#L78-L79">these lines</a> are probably the right ones.</p>
<aside class="quote no-group" data-username="promach" data-post="11" data-topic="76296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/promach/48/22571_2.png" class="avatar"> promach:</div>
<blockquote>
<p>I tried to replace <code>torch.cdist()</code> using <code>fast_cdist()</code> as shown below. However, I still have GPU out-of-memory error.</p>
</blockquote>
</aside>
<p>In your code other methods, such as <code>torch.cat</code> will create contiguous tensors as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.randn(1, 1).expand(10, 10)
print(a.is_contiguous())
&gt; False

b = torch.randn(10, 10)
print(b.is_contiguous())
&gt; True

c = torch.cat((a, b), dim=1)
print(c.is_contiguous())
&gt; True
</code></pre>
<p>The main issue, is that your data is too large for the applied operations, as at least some of them work on contiguous tensors, which will create the memory increase.</p>
<p>For mixed-precision training, I would recommend to install the nightly and use native amp as described <a href="https://pytorch.org/docs/stable/amp.html">here</a>.</p>

promach: <p>See the <a href="https://github.com/pytorch/pytorch/issues/15253#issuecomment-618976320" rel="nofollow noopener">runtime error</a> where I try to use <strong>another</strong> <code>my_cdist()</code> function.</p>
<pre><code class="lang-auto">@torch.jit.script
def my_cdist(x1, x2):
    x1_norm = x1.pow(2).sum(dim=-1, keepdim=True)
    x2_norm = x2.pow(2).sum(dim=-1, keepdim=True)
    res = torch.addmm(x2_norm.transpose(-2, -1), x1, x2.transpose(-2, -1), alpha=-2).add_(x1_norm)
    res = res.clamp_min_(1e-30).sqrt_()
    return res
</code></pre>
<p>For your reference, please also see <a href="https://github.com/huawei-noah/AdderNet/issues/6" rel="nofollow noopener">this addernet github issue</a></p>
<p>As for replacing <code>torch.cat()</code>, <a href="https://discuss.pytorch.org/t/concatenate-tensors-without-memory-copying/34609">there is no way to replace the function yet</a>.</p>

promach: <p>As for using pytorch native amp library, I have the following runtime error with <a href="https://paste.ubuntu.com/p/ZWhFbyR7cp/" rel="nofollow noopener">train.py</a></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 112, in &lt;module&gt;
    loss, outputs = model(imgs, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 563, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rog/Downloads/PyTorch-YOLOv3/models.py", line 266, in forward
    x, layer_loss = module[0](x, targets, img_dim)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 563, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rog/Downloads/PyTorch-YOLOv3/models.py", line 203, in forward
    loss_conf_obj = self.bce_loss(pred_conf[obj_mask], tconf[obj_mask])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 563, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 520, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2417, in binary_cross_entropy
    input, target, weight, reduction_enum)
RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to autocast.
Many models use a sigmoid layer right before the binary cross entropy layer.
In this case, combine the two layers using torch.nn.functional.binary_cross_entropy_with_logits
or torch.nn.BCEWithLogitsLoss.  binary_cross_entropy_with_logits and BCEWithLogits are
safe to autocast.

</code></pre>

ptrblck: <p>As the error message stats, you should replace the usage of <code>sigmoid + nn.BCELoss</code> to <code>logits + nn.BCEWithLogitsLoss</code>, as the former approach is unsafe for autocasting.</p>

promach: <p>I have already tried Nvidia Apex, but it still give me OOM issue.</p>
<p>I will re-try pytorch native amp later.</p>
<p>By the way, is there a way to replace torch.cdist() which contains <a href="https://discuss.pytorch.org/t/understanding-cdist-function/76296/12">contiguous() calls</a> ?</p>

promach: <p>See the <a href="https://github.com/pytorch/pytorch/issues/37408#issuecomment-620723535" rel="nofollow noopener">comments about cdist()</a> with regards to <a href="https://discuss.pytorch.org/t/understanding-cdist-function/76296/12">contiguous() call</a></p>

ptrblck: <p>Thanks for following up.<br>
In that case, these calls are no-ops and the method apparently just uses this much memory.</p>

promach: <p>Now back to square one. So, <code>torch.cdist()</code> is not the OOM culprit.</p>
<p>And what do you think about <a href="https://github.com/huawei-noah/AdderNet/issues/16#issuecomment-620971070" rel="nofollow noopener">the dimension of the tensor variables</a> ?</p>

WaterKnight: <p>I am also getting this error, the output of my model uses F.sigmoid. When i compute loss I use BCELoss. If I decide to use autocast, should I just remove sigmoid and use bcelosswithlogits?</p>
<p>I don’t understand the way of solving it.</p>