Most efficient, differentiable Way to get differentiable pair-wise Distance

Andreas_Binder: <p>Hi all!<br>
I look for the most efficient, differentiable way for a 3D PointCloud matrix with shape (1024,3) to find the vector containing the pairwise distances (shape: (1024x1024,1).<br>
Currently, I use this:</p>
<pre><code class="lang-auto">x = torch.ones(1024, 3, requires_grad=True)
pdist = nn.PairwiseDistance(p=2, keepdim=True) 
out = torch.cat ([ pdist(x[n], x[i]) for n in range (len (x)) for i in range (len (x))])
</code></pre>
<p>which is not differentiable:</p>
<pre><code class="lang-auto">torch.autograd.grad(out.sum(), x)[0]

RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_1030593/3913672061.py in &lt;module&gt;
----&gt; 1 torch.autograd.grad(out.sum(), x)[0]

~/miniconda3/envs/test_torch/lib/python3.7/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused, is_grads_batched)
    300         return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    301             t_outputs, grad_outputs_, retain_graph, create_graph, t_inputs,
--&gt; 302             allow_unused, accumulate_grad=False)  # Calls into the C++ engine to run the backward pass
    303 
    304 

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Thank you so much!</p>

ptrblck: <p>Your code works for me:</p>
<pre><code class="lang-python">x = torch.ones(10, 3, requires_grad=True)
pdist = nn.PairwiseDistance(p=2, keepdim=True) 
out = torch.cat ([ pdist(x[n], x[i]) for n in range (len (x)) for i in range (len (x))])
out.mean().backward()
print(x.grad)
# tensor([[ 9.3132e-10,  9.3132e-10,  9.3132e-10],
#         [ 9.3132e-10,  9.3132e-10,  9.3132e-10],
#         [ 9.3132e-10,  9.3132e-10,  9.3132e-10],
#         [ 9.3132e-10,  9.3132e-10,  9.3132e-10],
#         [-9.3132e-10, -9.3132e-10, -9.3132e-10],
#         [-9.3132e-10, -9.3132e-10, -9.3132e-10],
#         [-9.3132e-10, -9.3132e-10, -9.3132e-10],
#         [-9.3132e-10, -9.3132e-10, -9.3132e-10],
#         [-9.3132e-10, -9.3132e-10, -9.3132e-10],
#         [-9.3132e-10, -9.3132e-10, -9.3132e-10]])
</code></pre>

Andreas_Binder: <p>I think I forgot the <code>requires_grad=True</code> line in my original code … ^^<br>
Thank you Patrick!<br>
As a side question, do you know if there is a more efficient (built-in) way to a arrive at this point wise distance for x?</p>

ptrblck: <p>Yes, you could add dimensions and allow broadcasting to avoid the for loop:</p>
<pre><code class="lang-python">x = torch.randn(10, 3, requires_grad=True)
pdist = nn.PairwiseDistance(p=2, keepdim=True) 
ref = torch.cat ([ pdist(x[n], x[i]) for n in range (len (x)) for i in range (len (x))])

out = pdist(x.unsqueeze(1), x.unsqueeze(0))
out = out.view(-1)
print((out - ref).abs().max())
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Andreas_Binder: <p>Oh wow, that is amazing! Does this technique have a specific name or can I find more information on that in the PyTorch documentation? The computation time is now seconds instead of minutes <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I would refer to it as <code>broadcasting</code> and the ability to execute operators on batched inputs. <a href="https://pytorch.org/docs/stable/notes/broadcasting.html">This doc</a> explains it a bit more and also links to the numpy docs, which has more examples.</p>