Error on CUDA Programming

username: shim94kr
cooked: <p>Hello. I’m modifying a CUDA code of deformable convolution to work with circular padding, and the main change is to introduce modulo operation on the translated coordinates by offsets. I tried two versions that do the same thing, but the problem is that one raises an error, but the other works well. I want to ask you the difference or the reason for the error as I think they are the same in functions.</p>
<p>The base code I used is <a href="https://github.com/CHONSPQX/modulated-deform-conv" class="inline-onebox" rel="noopener nofollow ugc">GitHub - CHONSPQX/modulated-deform-conv: deformable convolution 2D 3D DeformableConvolution DeformConv Modulated Pytorch CUDA</a>. You can test the code by simply replacing the trilinear interpolation part in src/deformable_conv3d.cu.</p>
<p>The central part that I changed is the lines 25-27, which involve modulo operation.<br>
Line 27 does not raise an error, and I think this covers the discrepancy between version 1 and version 2.</p>
<p>7 template <br>
8 <strong>device</strong> scalar_t deform_conv3d_im2col_trilinear(<br>
9         const scalar_t *bottom_data, const int data_width,const int data_length,<br>
10         const int height, const int width, const int length,scalar_t h, scalar_t w,scalar_t l)<br>
11 {<br>
12<br>
13   int h_low = floor(h);<br>
14   int w_low = floor(w);<br>
15   int l_low = floor(l);<br>
17   int w_high = w_low + 1;<br>
18   int l_high = l_low + 1;<br>
19<br>
20   scalar_t lh = h - h_low;//dh<br>
21   scalar_t lw = w - w_low;//dw<br>
22   scalar_t ll = l - l_low;//dl<br>
23   scalar_t hh = 1 - lh, hw = 1 - lw, hl = 1 - ll; //1-dh 1-dw 1-dl<br>
24<br>
25   h_low = ((h_low % height) + height) % height;<br>
26   h_high = ((h_high % height) + height) % height;<br>
27   assert(h_low &gt;= 0 &amp;&amp; h_high &lt;= height - 1);</p>
<p>The code of 28-49 lines is the version1 that raises an error of CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling <code>cublasCreate(handle)</code><br>
28 /*<br>
29   scalar_t v1 = 0;<br>
30   scalar_t v5 = 0;<br>
31   if (w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)<br>
32     v1 = bottom_data[h_low * data_width<em>data_length + w_low</em>data_length+ l_low];<br>
33     v5 = bottom_data[h_high * data_width<em>data_length + w_low</em>data_length+ l_low];<br>
34   scalar_t v2 = 0;<br>
35   scalar_t v6 = 0;<br>
36   if (w_low &gt;=0 &amp;&amp; l_high&lt;= length -1)<br>
37     v2 = bottom_data[h_low * data_width<em>data_length + w_low</em>data_length+ l_high];<br>
38     v6 = bottom_data[h_high * data_width<em>data_length + w_low</em>data_length+ l_high];<br>
39   scalar_t v3 = 0;<br>
40   scalar_t v7 = 0;<br>
41   if (w_high &lt;= width - 1 &amp;&amp; l_low  &gt;= 0)<br>
42     v3 = bottom_data[h_low * data_width<em>data_length + w_high</em>data_length+ l_low];<br>
43     v7 = bottom_data[h_high * data_width<em>data_length + w_high</em>data_length+ l_low];<br>
44   scalar_t v4 = 0;<br>
45   scalar_t v8 = 0;<br>
46   if (w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)<br>
47     v4 = bottom_data[h_low * data_width<em>data_length + w_high</em>data_length+ l_high];<br>
48     v8 = bottom_data[h_high * data_width<em>data_length + w_high</em>data_length+ l_high];<br>
49 */</p>
<p>The code of 50-76 lines is version2 which works without error.<br>
50 /*<br>
51   scalar_t v1 = 0;<br>
52   if (h_low &gt;= 0 &amp;&amp; w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)<br>
53     v1 = bottom_data[h_low * data_width<em>data_length + w_low</em>data_length+ l_low];<br>
54   scalar_t v2 = 0;<br>
55   if (h_low &gt;= 0 &amp;&amp; w_low &gt;=0 &amp;&amp; l_high&lt;= length -1)<br>
56     v2 = bottom_data[h_low * data_width<em>data_length + w_low</em>data_length+ l_high];<br>
57   scalar_t v3 = 0;<br>
58   if (h_low &gt;= 0 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_low  &gt;= 0)<br>
59     v3 = bottom_data[h_low * data_width<em>data_length + w_high</em>data_length+ l_low];<br>
60   scalar_t v4 = 0;<br>
61   if (h_low &gt;= 0 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)<br>
62     v4 = bottom_data[h_low * data_width<em>data_length + w_high</em>data_length+ l_high];<br>
63<br>
64   scalar_t v5 = 0;<br>
65   if (h_high &lt;= height -1 &amp;&amp; w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)<br>
66     v5 = bottom_data[h_high * data_width<em>data_length + w_low</em>data_length+ l_low];<br>
67   scalar_t v6 = 0;<br>
68   if (h_high &lt;= height -1 &amp;&amp; w_low &gt;= 0  &amp;&amp; l_high&lt;= length -1)<br>
69     v6 = bottom_data[h_high * data_width<em>data_length + w_low</em>data_length+ l_high];<br>
70   scalar_t v7 = 0;<br>
71   if (h_high &lt;= height -1 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_low &gt;= 0)<br>
72     v7 = bottom_data[h_high * data_width<em>data_length + w_high</em>data_length+ l_low];<br>
73   scalar_t v8 = 0;<br>
74   if (h_high &lt;= height -1 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)<br>
75     v8 = bottom_data[h_high * data_width<em>data_length + w_high</em>data_length+ l_high];<br>
76 */</p>
<p>The rest part performs the interpolation.<br>
77   scalar_t w1 = hh * hw *hl, w2 = hh <em>hw <em>ll, w3 = hh * lw</em>hl, w4 = hh * lw</em> ll;<br>
78   scalar_t w5 = lh * hw *hl, w6 = lh <em>hw <em>ll, w7 = lh * lw</em>hl, w8 = lh * lw</em> ll;<br>
79<br>
80   scalar_t val = (w1 * v1 + w2 * v2 + w3 * v3 + w4 * v4+w5 * v5 + w6 * v6 + w7 * v7 + w8 * v8);<br>
81   return val;<br>
82 }</p>
<p>Any comment or help on this would be appreciated!</p>

username: ptrblck
cooked: <p>Assuming you want to debug a potential indexing error, run the code via <code>compute-sanitizer</code> and see if it would report invalid reads or writes. Additionally <code>cuda-gdb</code> (with the <code>memcheck</code> option set to on) might be useful as it could help narrowing down where the issue is.</p>

username: shim94kr
cooked: <p>I couldn’t install the <code>compute-sanitizer</code> even though the manual said it is installed by default with nvcc.<br>
I tried <code>sudo apt-get install cuda-sanitizer-api-10-2</code> as I installed cuda-gdb with apt-get before.<br>
Can I get a way to install it?</p>
<p>For running memcheck in cuda-gdb, I got a warp illegal address error, but I couldn’t identify where it comes from. It was hard to debug further as the cuda-gdb doesn’t step one-by-one through the cuda kernels as gdb does.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0a9f4b8c4187734102b570534895aed0dec0058.png" data-download-href="https://discuss.pytorch.org/uploads/default/c0a9f4b8c4187734102b570534895aed0dec0058" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0a9f4b8c4187734102b570534895aed0dec0058_2_690x155.png" alt="image" data-base62-sha1="runNNILq8lWhpmgfu9VBVzyyuPm" width="690" height="155" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0a9f4b8c4187734102b570534895aed0dec0058_2_690x155.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0a9f4b8c4187734102b570534895aed0dec0058_2_1035x232.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0a9f4b8c4187734102b570534895aed0dec0058_2_1380x310.png 2x" data-dominant-color="0D0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1404×316 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But, I found the error is raised when the offset is decimal (when the offset is an integer is okay).<br>
Yet, it is weird because a target point’s result is aggregated through interpolating the results of multiple integer points, and there’s nothing to do with the decimal points.</p>
<p>Thank you for answering!</p>

username: ptrblck
cooked: <p>If you are using an older CUDA toolkit you could use <code>cuda-memcheck</code> instead of <code>compute-sanitizer</code>.<br>
In any case, <code>cuda-gdb</code> already points to te warp illegal address in the <code>deform_conv3d_im2col...</code> kernel so you would need to check the indexing logic there. Adding <code>assert</code>s might also be helpful in you have trouble isolating the issue.</p>

username: shim94kr
cooked: <p>I added the asserts to version1 and made it the same code as version2.<br>
What I did was to change if statement to the assert one.</p>
<p>However, running with version 1 raises the same error, but version 2 doesn’t.<br>
I think they are the same code, so further trouble-shooting might be turned out to be a waste of time.<br>
Would you mind confirming this in that they are the same code?<br>
Thank you for your time and efforts.</p>
<p>Version1</p>
<pre><code class="lang-auto">   scalar_t v1 = 0;
   if (w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)
     assert (h_low&gt;=0);
     v1 = bottom_data[h_low * data_width*data_length + w_low*data_length+ l_low];
   scalar_t v2 = 0;
   if (w_low &gt;=0 &amp;&amp; l_high&lt;= length -1)
     assert (h_low&gt;=0);
     v2 = bottom_data[h_low * data_width*data_length + w_low*data_length+ l_high];
   scalar_t v3 = 0;
   if (w_high &lt;= width - 1 &amp;&amp; l_low  &gt;= 0)
     assert (h_low&gt;=0);
     v3 = bottom_data[h_low * data_width*data_length + w_high*data_length+ l_low];
   scalar_t v4 = 0;
   if (w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)
     assert (h_low&gt;=0);
     v4 = bottom_data[h_low * data_width*data_length + w_high*data_length+ l_high];
   scalar_t v5 = 0;
   if (w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)
     assert (h_high&lt;=height - 1);
     v5 = bottom_data[h_high * data_width*data_length + w_low*data_length+ l_low];
   scalar_t v6 = 0;
   if (w_low &gt;=0 &amp;&amp; l_high&lt;= length -1)
     assert (h_high&lt;=height - 1);
     v6 = bottom_data[h_high * data_width*data_length + w_low*data_length+ l_high];
   scalar_t v7 = 0;
   if (w_high &lt;= width - 1 &amp;&amp; l_low  &gt;= 0)
     assert (h_high&lt;=height - 1);
     v7 = bottom_data[h_high * data_width*data_length + w_high*data_length+ l_low];
   scalar_t v8 = 0;
   if (w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)
     assert (h_high&lt;=height - 1);
     v8 = bottom_data[h_high * data_width*data_length + w_high*data_length+ l_high];
</code></pre>
<p>Version2</p>
<pre><code class="lang-auto">   scalar_t v1 = 0;
   if (h_low &gt;= 0 &amp;&amp; w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)
     v1 = bottom_data[h_low * data_width*data_length + w_low*data_length+ l_low];
   scalar_t v2 = 0;
   if (h_low &gt;= 0 &amp;&amp; w_low &gt;=0 &amp;&amp; l_high&lt;= length -1)
     v2 = bottom_data[h_low * data_width*data_length + w_low*data_length+ l_high];
   scalar_t v3 = 0;
   if (h_low &gt;= 0 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_low  &gt;= 0)
     v3 = bottom_data[h_low * data_width*data_length + w_high*data_length+ l_low];
   scalar_t v4 = 0;
   if (h_low &gt;= 0 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)
     v4 = bottom_data[h_low * data_width*data_length + w_high*data_length+ l_high];

   scalar_t v5 = 0;
   if (h_high &lt;= height -1 &amp;&amp; w_low &gt;= 0 &amp;&amp; l_low &gt;= 0)
     v5 = bottom_data[h_high * data_width*data_length + w_low*data_length+ l_low];
   scalar_t v6 = 0;
   if (h_high &lt;= height -1 &amp;&amp; w_low &gt;= 0  &amp;&amp; l_high&lt;= length -1)
     v6 = bottom_data[h_high * data_width*data_length + w_low*data_length+ l_high];
   scalar_t v7 = 0;
   if (h_high &lt;= height -1 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_low &gt;= 0)
     v7 = bottom_data[h_high * data_width*data_length + w_high*data_length+ l_low];
   scalar_t v8 = 0;
   if (h_high &lt;= height -1 &amp;&amp; w_high &lt;= width - 1 &amp;&amp; l_high&lt;= length -1)
     v8 = bottom_data[h_high * data_width*data_length + w_high*data_length+ l_high];
</code></pre>

username: shim94kr
cooked: <p>Just in a case, you can reproduce the problem of my case with following steps.</p>
<p>Step1: clone the following projects<br>
git clone <a href="https://github.com/shim94kr/modulated_deformable_conv.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - shim94kr/modulated_deformable_conv</a></p>
<p>Step2: create virtual env &amp; install pytorch<br>
conda create -n reproduce python=3.8<br>
conda activate reproduce<br>
pip install -r requirements.txt</p>
<p>Step3: install deformable conv<br>
python setup.py install</p>
<p>Step4: execute the test code<br>
python my_test.py</p>
<p>The code is set to version1 and will raise the error.<br>
If you switch version1 to version2 by commenting out version1 in src/deformable_conv3d.cu Line#28-59, the error will be disappeared.</p>