Find 10 Nearest neighbors for each point on a pointcloud that comprise 6M points

username: miguelG_97
cooked: <p>Greeting to all, I need to verify that the distance between data points meets some insight requirements. The pandas and numpy synergy takes roughly 20min for a set of 10000 points on a CPU device. Now that I 've deployed pytorch library using google colab GPU the amount of time has not been reduced at all, instead it’s taking much time than the former approach. I’m attaching the script I’m currently using. Any thoughts on how to improve the performance?</p>
<p>import pandas as pd</p>
<p>import numpy as np</p>
<p>import torch</p>
<h1>
<a name="determine-the-supported-device-1" class="anchor" href="#determine-the-supported-device-1"></a>determine the supported device</h1>
<p>def get_device():</p>
<pre><code>if torch.cuda.is_available():

    device = torch.device('cuda')

else:

    device = torch.device('cpu') # don't have GPU 

return device
</code></pre>
<h1>
<a name="a-read-the-text-file-2" class="anchor" href="#a-read-the-text-file-2"></a>a). read the text file</h1>
<p>pointCloud=pd.read_csv(r’/content/500K POINTS.txt’,index_col=0)</p>
<h1>
<a name="bordering-the-data-set-pertaining-to-its-x-values-3" class="anchor" href="#bordering-the-data-set-pertaining-to-its-x-values-3"></a>b).ordering the data set pertaining to its X values</h1>
<p>newdfSorted=pointCloud.sort_values(by=[‘Ycoord’], ascending=True,ignore_index=True)</p>
<p>newdfSorted=newdfSorted.iloc[39900:50000]</p>
<h1>
<a name="c-get-the-total-number-of-rows-4" class="anchor" href="#c-get-the-total-number-of-rows-4"></a>c). get the total number of rows</h1>
<p>rows,colums=newdfSorted.shape</p>
<h1>
<a name="d-from-df-to-torch-tensor-5" class="anchor" href="#d-from-df-to-torch-tensor-5"></a>d). from df to torch tensor</h1>
<p>myDevice=get_device()</p>
<p>dfTensor=torch.from_numpy(newdfSorted[[‘Xcoord’,‘Ycoord’,‘Zcoord’]].values).to(device=myDevice,dtype=torch.float)</p>
<h1>
<a name="h1-creating-a-new-empty-torch-for-further-usage-6" class="anchor" href="#h1-creating-a-new-empty-torch-for-further-usage-6"></a>h.1) creating a new empty torch for further usage</h1>
<p>distanceTensor=torch.empty((rows,10),dtype=torch.float,device=myDevice)</p>
<p>for i in range(0,rows,1):</p>
<h1>
<a name="e-pick-a-tensor-row-a-point-data-7" class="anchor" href="#e-pick-a-tensor-row-a-point-data-7"></a>e). Pick a tensor row (a point data)</h1>
<p>testPoint=dfTensor[i]</p>
<h1>
<a name="f-generate-a-new-tensor-without-the-previously-selected-point-on-c-and-that-contains-1000-points-ahead-the-test-point-index-and-1000-points-behind-it-8" class="anchor" href="#f-generate-a-new-tensor-without-the-previously-selected-point-on-c-and-that-contains-1000-points-ahead-the-test-point-index-and-1000-points-behind-it-8"></a>f). generate a new tensor without the previously selected point on (c) and that contains 1000 points ahead the test point index and 1000 points behind it</h1>
<p>if i-1000&lt;= 0:</p>
<pre><code>df1=dfTensor[(i-1000):] 

df2=dfTensor[0:i]

df3=dfTensor[i+1:i+1000]#i+1 will exclude the current point value to avoid a zero distance value

newDataFrame=torch.cat([df1,df2,df3],dim=0).to(device=myDevice)
</code></pre>
<p>else:</p>
<pre><code>df1=dfTensor[(i-1000):i]

df2=dfTensor[i+1:i+1000]#i+1 will exclude the current point value to avoid a zero distance value

newDataFrame=torch.cat([df1,df2],dim=0).to(device=myDevice)
</code></pre>
<h1>
<a name="g-determine-the-near-10-closest-points-to-the-i-loop-point-9" class="anchor" href="#g-determine-the-near-10-closest-points-to-the-i-loop-point-9"></a>g). determine the near 10 closest points to the “i” loop point</h1>
<p>distance1=torch.norm(testPoint-newDataFrame[0],p=‘fro’)</p>
<p>firstV=torch.tensor([distance1*100]).to(device=myDevice)</p>
<p>for j in range(1,newDataFrame.size(dim=0),1):</p>
<pre><code>distance=torch.norm(testPoint-newDataFrame[j],p='fro').to(device=myDevice)

firstV=torch.cat([firstV,torch.tensor([distance*100]).to(device=myDevice)]).to(device=myDevice)
</code></pre>
<p>values,indices=torch.sort(firstV,dim=0,descending=False)</p>
<p>nearDistances=values[0:10]</p>
<h1>
<a name="h2attach-the-10-closest-neighbour-to-the-created-tensor-on-h1-10" class="anchor" href="#h2attach-the-10-closest-neighbour-to-the-created-tensor-on-h1-10"></a>h.2)Attach the 10 closest neighbour to the created tensor on (h.1)</h1>
<p>distanceTensor[i]=nearDistances</p>
<h1>
<a name="ijoin-original-tensor-from-txtfile-and-the-one-that-comprise-the-closest-distance-algon-the-horizontal-axis-11" class="anchor" href="#ijoin-original-tensor-from-txtfile-and-the-one-that-comprise-the-closest-distance-algon-the-horizontal-axis-11"></a>i)Join original tensor from txtfile and the one that comprise the closest distance algon the horizontal axis</h1>
<p>torch.concat([dfTensor,distanceTensor],dim=1).to(device=myDevice)</p>