Segfault / SIGSEGV using torchvision.ops.RoIAlign

Francois: <p>Howdy,</p>
<p>Applying <code>RoIAlign</code> (torchvision.ops.RoIAlign) on <code>rois</code> that is a tensor of shape (K, 5) results in a segfault : <code>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)</code>.</p>
<p>The faulty behavior is <strong>not</strong> observed when :</p>
<ul>
<li>Applying <code>RoIAlign</code> on <code>rois</code> that is a list of tensor of shape (4,)</li>
<li>
<code>RoIAlign</code> input size is small enough (typically LEQ 32,10,10)</li>
</ul>
<p>Here is a code snippet that should allow you to reproduce my observations.</p>
<pre><code class="lang-auto">import argparse

import torch
from torchvision.ops import RoIAlign

NB_ROI_PER_DOC = 5

if __name__ == "__main__":

    torch.manual_seed(0)

    parser = argparse.ArgumentParser(description='minimum reproducible example')
    parser.add_argument('--bug', action='store_true', default=False, help='toggle on the issue')
    parser.add_argument('--batch-size', type=int, default=1)
    parser.add_argument('--input-size', type=str, default="64,10,10")
    args = parser.parse_args()

    align = RoIAlign((3, 3), spatial_scale=14 / 224, sampling_ratio=2)
    batch_size = args.batch_size
    input_ = torch.randn((batch_size, *(int(dim) for dim in args.input_size.split(","))))
    rois = [torch.abs(torch.randn(NB_ROI_PER_DOC, 4)) for _ in input_]

    if args.bug:
        nb_rois = NB_ROI_PER_DOC * batch_size
        boxes = torch.abs(torch.randn(nb_rois, 4))
        roi_ids = torch.arange(nb_rois).view(-1, 1)
        rois = torch.cat((roi_ids, boxes), dim=1)

    output = align(input=input_, rois=rois)
    print(rois)
    print(output.shape)
</code></pre>
<p>Running the code in docker to be freeze packages versions and the like.</p>
<pre><code class="lang-auto">FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
RUN mkdir -p /opt/debug
WORKDIR /opt/debug
ADD mre.py .
ENTRYPOINT ["python", "/opt/debug/mre.py"]
</code></pre>
<hr>
<p>Sample outputs :</p>
<p><strong>Problematic</strong> behavior, to run : <code>build -f Dockerfile.minimal -t debug-torch:v0 . ; docker run  docker run debug-torch:v0 --bug; docker ps -a |grep debug-torch:v0</code>  results with:</p>
<pre><code class="lang-auto">46b40fe6efee   debug-torch:v0 "python /opt/debug/m…"     16 seconds ago   Exited (139) 
</code></pre>
<p><strong>Expected</strong> behavior, to run : <code>docker run  docker run debug-torch:v0</code> results with:</p>
<pre><code class="lang-auto">[tensor([[1.4164, 0.2379, 0.9334, 1.1331],
        [0.3530, 2.0928, 0.6356, 1.5069],
        [0.9527, 1.0599, 0.9549, 1.3355],
        [0.5251, 0.7416, 0.4269, 0.4008],
        [0.7872, 0.0834, 1.1256, 1.5490]])]
torch.Size([5, 64, 3, 3])
</code></pre>
<p>To reduce the input size solves the problem, to run <code>docker run  docker run debug-torch:v0 --bug --input-size 5,5,5</code> results with:</p>
<pre><code class="lang-auto">tensor([[0.0000, 0.3584, 1.5616, 0.3546, 1.0811],
        [1.0000, 0.8760, 0.2871, 1.0216, 0.5111],
        [2.0000, 1.7137, 0.5101, 0.4749, 0.6334],
        [3.0000, 1.2063, 0.6074, 0.5472, 1.1005],
        [4.0000, 0.7201, 0.0119, 0.3398, 0.2635]])
torch.Size([5, 64, 3, 3])
</code></pre>
<hr>
<p>I’m well aware that the most likely reason for the problem is me missing out something obvious in the using of <code>RoIAlign</code>, feel free to simply let me know if this is the case.</p>
<p>Thank for reading down to this point &amp; have a nice day</p>

KFrank: <p>Hi Francois!</p>
<aside class="quote no-group quote-modified" data-username="Francois" data-post="1" data-topic="135639" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/francois/48/43523_2.png" class="avatar"> Francois:</div>
<blockquote>
<p>Applying <code>RoIAlign</code> (torchvision.ops.RoIAlign) on <code>rois</code> that is a tensor of shape (K, 5) results in a segfault : <code>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)</code>.</p>
<p><strong>Expected</strong> behavior, to run : <code>docker run  docker run debug-torch:v0</code> results with:</p>
<pre><code class="lang-auto">[tensor([[1.4164, 0.2379, 0.9334, 1.1331],
        [0.3530, 2.0928, 0.6356, 1.5069],
        [0.9527, 1.0599, 0.9549, 1.3355],
        [0.5251, 0.7416, 0.4269, 0.4008],
        [0.7872, 0.0834, 1.1256, 1.5490]])]
torch.Size([5, 64, 3, 3])
</code></pre>
</blockquote>
</aside>
<p>I can’t reproduce your issue running a tweaked version of your code<br>
by pasting it into a python interpreter running with torch / torchvision<br>
versions 1.9.0 / 0.10.0:</p>
<pre><code class="lang-plaintext">Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import argparse
&gt;&gt;&gt;
&gt;&gt;&gt; import torch
&gt;&gt;&gt; from torchvision.ops import RoIAlign
&gt;&gt;&gt;
&gt;&gt;&gt; NB_ROI_PER_DOC = 5
&gt;&gt;&gt;
&gt;&gt;&gt; if __name__ == "__main__":
...
...     torch.manual_seed(0)
...
...     parser = argparse.ArgumentParser(description='minimum reproducible example')
...     parser.add_argument('--bug', action='store_true', default=False, help='toggle on the issue')
...     parser.add_argument('--batch-size', type=int, default=1)
...     parser.add_argument('--input-size', type=str, default="64,10,10")
...     args = parser.parse_args()
...
...     align = RoIAlign((3, 3), spatial_scale=14 / 224, sampling_ratio=2)
...     batch_size = args.batch_size
...     input_ = torch.randn((batch_size, *(int(dim) for dim in args.input_size.split(","))))
...     rois = [torch.abs(torch.randn(NB_ROI_PER_DOC, 4)) for _ in input_]
...
...     args.bug = True
...     if args.bug:
...         print ('args.bug =', args.bug)
...         nb_rois = NB_ROI_PER_DOC * batch_size
...         boxes = torch.abs(torch.randn(nb_rois, 4))
...         roi_ids = torch.arange(nb_rois).view(-1, 1)
...         rois = torch.cat((roi_ids, boxes), dim=1)
...
...     output = align(input=input_, rois=rois)
...     print(rois)
...     print(output.shape)
...
&lt;torch._C.Generator object at 0x000002C110C51978&gt;
_StoreTrueAction(option_strings=['--bug'], dest='bug', nargs=0, const=True, default=False, type=None, choices=None, help='toggle on the issue', metavar=None)
_StoreAction(option_strings=['--batch-size'], dest='batch_size', nargs=None, const=None, default=1, type=&lt;class 'int'&gt;, choices=None, help=None, metavar=None)
_StoreAction(option_strings=['--input-size'], dest='input_size', nargs=None, const=None, default='64,10,10', type=&lt;class 'str'&gt;, choices=None, help=None, metavar=None)
args.bug = True
tensor([[0.0000, 0.7319, 1.0969, 0.6430, 0.8533],
        [1.0000, 1.2892, 0.2074, 0.4249, 0.0755],
        [2.0000, 1.9148, 2.2872, 0.1998, 0.9350],
        [3.0000, 0.5617, 0.9486, 1.7332, 0.7013],
        [4.0000, 0.1312, 3.0198, 1.7909, 0.4429]])
torch.Size([5, 64, 3, 3])
&gt;&gt;&gt; print (torch.__version__)
1.9.0
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; print (torchvision.__version__)
0.10.0
</code></pre>
<p>Are you expecting this to be some kind of docker / configuration /<br>
hardware issue, or do you think it’s a pure pytorch issue?  I don’t see<br>
it with my test.  What versions of torch / torchvision are you using?</p>
<p>Can you reproduce your issue with a “raw” python / pytorch installation,<br>
or does in only happen when you use docker?</p>
<p>Best.</p>
<p>K. Frank</p>

Francois: <p>Hey Frank, thank you for your attempt to reproduce the issue !</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="135639">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are you expecting this to be some kind of docker / configuration /<br>
hardware issue, or do you think it’s a pure pytorch issue? I don’t see<br>
it with my test. What versions of torch / torchvision are you using?</p>
<p>Can you reproduce your issue with a “raw” python / pytorch installation,<br>
or does in only</p>
</blockquote>
</aside>
<p>Yes, I do reproduce the issue in a virutal env (ie. w/o docker). Using <code>https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py</code> I get the environment specs below.</p>
<pre><code class="lang-auto">python collect_env.py
Collecting environment information...
PyTorch version: 1.10.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Sep 28 2021, 16:10:42)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.3
[pip3] torch==1.10.0
[pip3] torchvision==0.11.1
[conda] Could not collect

</code></pre>

KFrank: <p>Hi Francois!</p>
<aside class="quote no-group" data-username="Francois" data-post="3" data-topic="135639" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/francois/48/43523_2.png" class="avatar"> Francois:</div>
<blockquote>
<pre><code class="lang-auto">[pip3] torch==1.10.0
[pip3] torchvision==0.11.1
</code></pre>
</blockquote>
</aside>
<p>I was also unable to reproduce your issue using a freshly-downloaded<br>
stable version, 1.10.0 / 0.11.1, under conda on ubuntu:</p>
<pre><code class="lang-plaintext">args.bug = True
tensor([[0.0000, 0.7319, 1.0969, 0.6430, 0.8533],
        [1.0000, 1.2892, 0.2074, 0.4249, 0.0755],
        [2.0000, 1.9148, 2.2872, 0.1998, 0.9350],
        [3.0000, 0.5617, 0.9486, 1.7332, 0.7013],
        [4.0000, 0.1312, 3.0198, 1.7909, 0.4429]])
torch.Size([5, 64, 3, 3])
&gt;&gt;&gt; print (torch.__version__)
1.10.0
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; print (torchvision.__version__)
0.11.1
</code></pre>
<p>(Again, I pasted my tweaked version of your code into a python<br>
interpreter.)</p>
<p>Also, for what it’s worth, I could not reproduce your issue on a older<br>
1.10.0 nightly version:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; print (torch.__version__)
1.10.0.dev20210830
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; print (torchvision.__version__)
0.11.0.dev20210830
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Francois: <p>Thank you Frank, this might have something to do with the fact I use Python version: 3.8.10 whereas you use Python 3.6.5 |Anaconda, Inc ?</p>