CUDA Exception: Warp Illegal Address

username: Sashika_Himali1
cooked: <p>I  am  try to finetune modified version of Blip2 model using a custom dataset. I am using mixed precision when training. As I found out error occurs here pooled_feature_map = adaptive_pool(reshaped_feature_map)</p>
<pre><code> # Reshape the upsampled feature map to combine the channel and spatial dimensions
    reshaped_feature_map = upsampled_feature_map.view(batch_size, 1280 * 32 * 32).float()
    reshaped_feature_map = reshaped_feature_map.contiguous()

    # Adaptive Pooling to match the target number of elements (257 * 1408)
    adaptive_pool = nn.AdaptiveAvgPool1d(257 * 1408)
    with autocast(enabled=False):  # Force full precision
        #pooled_tensor = adaptive_avg_pooling_layer(input_tensor)
        pooled_feature_map = adaptive_pool(reshaped_feature_map)
    #pooled_feature_map = adaptive_pool(reshaped_feature_map.float())
    pooled_feature_map = pooled_feature_map.half()

    # Final reshape to the target shape (257, 1408)
    image_embeds = pooled_feature_map.view(batch_size, 257, 1408)
</code></pre>
<p>Error found by debugging</p>
<p>Epoch 1, Step 1, Loss: 14.890625<br>
[New Thread 0x7af8253ff640 (LWP 7201)]<br>
Batch 2 shapes:<br>
input_ids: torch.Size([8, 256])<br>
attention_mask: torch.Size([8, 256])<br>
pixel_values: torch.Size([8, 3, 224, 224])<br>
labels: torch.Size([8, 128])</p>
<p>CUDA Exception: Warp Illegal Address<br>
The exception was triggered at PC 0x7af699eb1230</p>
<p>Thread 205 “pt_autograd_0” received signal CUDA_EXCEPTION_14, Warp Illegal Address.<br>
[Switching focus to CUDA kernel 5, grid 5392, block (0,0,0), thread (7,0,0), device 0, sm 0, warp 1, lane 7]<br>
0x00007af699eb1370 in void at::native::(anonymous namespace)::adaptive_average_pool(float const*, float*, int, int, int, int, long, long, long)&lt;&lt;&lt;(8,2,1),(32,8,1)&gt;&gt;&gt; ()</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? If an older one, could you update to the latest stable or nightly release?</p>

username: Sashika_Himali1
cooked: <p>I am using PyTorch 2.4  and error mainly occurs  in following line of code.<br>
pooled_feature_map = adaptive_pool(reshaped_feature_map)</p>
<p>Pls advice.</p>

username: Sashika_Himali1
cooked: <p>I am using PyTorch 2.4 and error mainly occurs in following line of code.<br>
pooled_feature_map = adaptive_pool(reshaped_feature_map)</p>
<p>Pls advice.</p>

username: ptrblck
cooked: <p>Could you post the shape of <code>reshaped_feature_map</code>, please?</p>

username: Sashika_Himali1
cooked: <p>I tried to find the error using the following code. I printed the sizes of all the variables but couldn’t find any issue. batch-size is 4</p>
<pre><code>    reshaped_feature_map = upsampled_feature_map.view(batch_size, 1280 * 32 * 32)
    # Check input size and data type before pooling
    print(f"Reshaped feature map shape: {reshaped_feature_map.shape}, dtype: {reshaped_feature_map.dtype}")

    # Perform the pooling
    # Adaptive Pooling to match the target number of elements (257 * 1408)
    adaptive_pool = nn.AdaptiveAvgPool1d(257 * 1408)
    try:
        pooled_feature_map = adaptive_pool(reshaped_feature_map)
        print("pooling sucessfful ")
    except Exception as e:
        print(f"Error occurred during pooling: {e}")
        raise

    print(f"pooled feature map shape: {  pooled_feature_map.shape}, dtype: {  pooled_feature_map.dtype}")
    #pooled_feature_map = adaptive_pool(reshaped_feature_map)

    # Final reshape to the target shape (257, 1408)
    image_embeds = pooled_feature_map.view(batch_size, 257, 1408)
</code></pre>
<p>Sizes of the   varialbles</p>
<p>Reshaped feature map shape: torch.Size([4, 1310720]), dtype: torch.float16<br>
pooling sucessfful<br>
pooled feature map shape: torch.Size([4, 361856]), dtype: torch.float16<br>
Reshaped image_embeds shape: torch.Size([4, 257, 1408]), dtype: torch.float16<br>
image_attention_mask torch.Size([4, 257])</p>

username: ptrblck
cooked: <p>Does this mean you cannot reproduce the issue in isolation using the input shapes you have shared?</p>

username: Sashika_Himali1
cooked: <p>I used chatgpt to explain the error and it says that adaptive_avg_pooling_layer is having the issue. Furthermore, according to the GPT, it can not perform the pooling operation as I am using float16. But when I  changed the code as above it performed the pooling operation. When I trained the model I could train it for one and a half epoch and after that, I got the same error. Next, I increased the batch size and checked, and then I got the same error before training completed the first epoch. Is it a memory issue?</p>

username: Sashika_Himali1
cooked: <p>I used chatgpt to explain the error and it says that adaptive_avg_pooling_layer is having the issue. Furthermore, according to the GPT, it can not perform the pooling operation as I am using float16. But when I changed the code as above it performed the pooling operation. When I trained the model I could train it for one and a half epoch and after that, I got the same error.</p>
<p>CUDA Exception: Warp Illegal Address The exception was triggered at PC 0x730431eb1200 Thread 205 “pt_autograd_0” received signal CUDA_EXCEPTION_14, Warp Illegal Address. [Switching focus to CUDA kernel 0, grid 5389, block (0,0,0), thread (7,0,0), device 0, sm 0, warp 2, lane 7] 0x0000730431eb1380 in void at::native::(anonymous namespace)::adaptive_average_pool<a>c10::Half</a>(c10::Half const*, c10::Half*, int, int, int, int, long, long, long) &lt;&lt;&lt;(8,2,1),(32,8,1)&gt;&gt;&gt; ()</p>
<p>Next, I increased the batch size and checked, and then I got the same error before training completed the first epoch. Is it a memory issue?</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet that is causing the issue or which input shape was used to trigger the error?</p>

username: Sashika_Himali1
cooked: <p>Error appeared after  completing the first epoch completed.<br>
<strong>Code</strong></p>
<pre><code>    reshaped_feature_map = upsampled_feature_map.view(batch_size, 1280 * 32 * 32)
    # Check input size and data type before pooling
    print(f"Reshaped feature map shape: {reshaped_feature_map.shape}, dtype: {reshaped_feature_map.dtype}")

    # Perform the pooling
    # Adaptive Pooling to match the target number of elements (257 * 1408)
    adaptive_pool = nn.AdaptiveAvgPool1d( 257 * 1408)
    try:
        pooled_feature_map = adaptive_pool(reshaped_feature_map)
        print("pooling sucessfful ")
    except Exception as e:
        print(f"Error occurred during pooling: {e}")
        raise

    print(f"pooled feature map shape: {  pooled_feature_map.shape}, dtype: {  pooled_feature_map.dtype}")
</code></pre>
<p>Error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a1da4ee762a7a675b3fe0a985aa502a7079ab88.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a1da4ee762a7a675b3fe0a985aa502a7079ab88" title="img"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a1da4ee762a7a675b3fe0a985aa502a7079ab88_2_690x370.png" alt="img" data-base62-sha1="60zz6RkCNketTRLerivL4hTNIve" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a1da4ee762a7a675b3fe0a985aa502a7079ab88_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a1da4ee762a7a675b3fe0a985aa502a7079ab88_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a1da4ee762a7a675b3fe0a985aa502a7079ab88_2_1380x740.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img</span><span class="informations">1531×822 46.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>