Expected output.numel() <= std::numeric_limits<int32_t>::max() to be true, but got false

mahesh_bhosale: <p>How to avoid this issue? It appears that it is related to the type of tensor. Does it mean the tensor is float64 instead of float32?<br>
Below is the stack trace.</p>
<blockquote>
<p>File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/csgrad/mbhosale/.vscode-server/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/…/…/debugpy/launcher/…/…/debugpy/<strong>main</strong>.py”, line 39, in <br>
cli.main()<br>
File “/home/csgrad/mbhosale/.vscode-server/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/…/…/debugpy/launcher/…/…/debugpy/…/debugpy/server/cli.py”, line 430, in main<br>
run()<br>
File “/home/csgrad/mbhosale/.vscode-server/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/…/…/debugpy/launcher/…/…/debugpy/…/debugpy/server/cli.py”, line 284, in run_file<br>
runpy.run_path(target, run_name=“<strong>main</strong>”)<br>
File “/home/csgrad/mbhosale/.vscode-server/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py”, line 321, in run_path<br>
return _run_module_code(code, init_globals, run_name,<br>
File “/home/csgrad/mbhosale/.vscode-server/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py”, line 135, in _run_module_code<br>
_run_code(code, mod_globals, init_globals,<br>
File “/home/csgrad/mbhosale/.vscode-server/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py”, line 124, in _run_code<br>
exec(code, run_globals)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/mainvm.py”, line 250, in <br>
mp.spawn(run_parallel,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 240, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 198, in start_processes<br>
while not context.join():<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 160, in join<br>
raise ProcessRaisedException(msg, error_index, failed_process.pid)<br>
torch.multiprocessing.spawn.ProcessRaisedException:</p>
<p>– Process 1 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 69, in _wrap<br>
fn(i, *args)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/mainvm.py”, line 132, in run_parallel<br>
train.run()<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 210, in run<br>
self.train_epoch(optimizer, scheduler, epoch)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 107, in train_epoch<br>
loss, trdice = self.trainIter(fixed, moving, fixed_label, moving_label, fixed_nopad=fixed_nopad)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 34, in trainIter<br>
sim_loss, grad_loss, seg_loss, dice = self.model.forward(fix, moving,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 1008, in forward<br>
output = self._run_ddp_forward(*inputs, **kwargs)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 969, in _run_ddp_forward<br>
return module_to_run(*inputs[0], **kwargs[0])<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/models.py”, line 191, in forward<br>
unet_out = self.unet(x)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/models.py”, line 149, in forward<br>
y = self.upsample(y)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/upsampling.py”, line 153, in forward<br>
return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/functional.py”, line 3912, in interpolate<br>
return torch._C._nn.upsample_nearest3d(input, output_size, scale_factors)<br>
RuntimeError: Expected output.numel() &lt;= std::numeric_limits&lt;int32_t&gt;::max() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
<p>/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 26 leaked semaphore objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’</p>
</blockquote>

mahesh_bhosale: <p>Appears its not the value but the size of the tensor that is causing the issue. But I checked its well below int32 max value.</p>

KFrank: <p>Hi Mahesh!</p>
<aside class="quote no-group quote-modified" data-username="mahesh_bhosale" data-post="1" data-topic="174870" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahesh_bhosale/48/30773_2.png" class="avatar"> mahesh_bhosale:</div>
<blockquote>
<blockquote>
<p>RuntimeError: Expected output.numel() &lt;= std::numeric_limits&lt;int32_t&gt;::max() to be true, but got false.</p>
</blockquote>
</blockquote>
</aside>
<p>I don’t know how to fix your issue, but …</p>
<p>This error message does seem a bit odd.  Could you tell what version<br>
of pytorch you are using?</p>
<p>My understanding is that pytorch supports tensors that are quite large<br>
(assuming adequate memory) and pytorch indexes tensors with <code>int64</code>s<br>
(<code>long</code>s).</p>
<p>Here is an illustration (using python, rather than drilling down into c++<br>
to get <code>std::numeric_limits&lt;int32_t&gt;::max()</code>):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1'
&gt;&gt;&gt; torch.iinfo (torch.int32).max
2147483647
&gt;&gt;&gt; torch.zeros (2**32, dtype = torch.int8).numel()
4294967296
</code></pre>
<p>If your pytorch version doesn’t explain this, could you post a minimal,<br>
fully-self-contained, runnable script that reproduces the error, preferably<br>
without multiprocessing, unless you think that that is part of the cause<br>
of the error?</p>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>Based on the stacktrace it seems <code>F.interpolate</code> fails, but I cannot reproduce it on the CPU/CUDA via:</p>
<pre><code class="lang-python">x = torch.randn(1, 2**10, 2**10, 2**10)
out = F.interpolate(x, size=(2**11+1, 2**11))
out.numel() &gt; 2**32
# True
</code></pre>

mahesh_bhosale: <p>I don’t know how, but the cache error is resolved after emptying.</p>

ptrblck: <p>Could you give me more information which “cache error” was raised and what exactly you were emptying?</p>

mahesh_bhosale: <p>The error is the same as that I posted earlier. In addition, I also sometimes got the memory error -</p>
<blockquote>
<p>CUDA out of memory. Tried to allocate 6.85 GiB (GPU 0; 23.69 GiB total capacity; 9.79 GiB already allocated; 2.73 GiB free; 16.13 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>I cleared the cache with the below,</p>
<blockquote>
<p>torch.cuda.empty_cache()</p>
</blockquote>

ptrblck: <p>Are you seeing this error:</p>
<pre><code class="lang-python">RuntimeError: Expected output.numel() &lt;= std::numeric_limits&lt;int32_t&gt;::max() to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
</code></pre>
<p>when running my code snippet?</p>

nico_nico: <p>I also encounter this problem when I use torch.nn.interpolate function,but why?</p>

ptrblck: <p>I don’t know as I’m not able to reproduce the issue using my provided code snippet and the author did not follow up after my last question.</p>

mahesh_bhosale: <p>Apologies for the delayed reply. Caught up with deadlines.</p>
<p>I ran the script that <a class="mention" href="/u/ptrblck">@ptrblck</a> provided; I don’t see any error and the output is True.</p>

mahesh_bhosale: <p>Okay, so I am getting this error again.</p>
<blockquote>
<p>– Process 1 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 69, in _wrap<br>
fn(i, *args)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/mainvm.py”, line 135, in run_parallel<br>
train.run()<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 159, in run<br>
self.train_epoch(optimizer, scheduler, epoch)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 119, in train_epoch<br>
loss, trdice = self.trainIter(fixed, moving, fixed_label, moving_label, fixed_nopad=fixed_nopad, seg_f=seg_fname)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 43, in trainIter<br>
sim_loss, grad_loss, seg_loss, dice = self.model.forward(fix, moving,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 1008, in forward<br>
output = self._run_ddp_forward(*inputs, **kwargs)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 969, in _run_ddp_forward<br>
return module_to_run(*inputs[0], **kwargs[0])<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/models.py”, line 244, in forward<br>
unet_out, enc_out = self.unet(x)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/models.py”, line 157, in forward<br>
y = self.upsample(y)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/upsampling.py”, line 153, in forward<br>
return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/functional.py”, line 3912, in interpolate<br>
return torch._C._nn.upsample_nearest3d(input, output_size, scale_factors)<br>
RuntimeError: Expected output.numel() &lt;= std::numeric_limits&lt;int32_t&gt;::max() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
</blockquote>
<p>Oddly enough, when I change to the single GPU setup, I rather get out of memory error.</p>
<blockquote>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 69, in _wrap<br>
fn(i, *args)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/mainvm.py”, line 135, in run_parallel<br>
train.run()<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 159, in run<br>
self.train_epoch(optimizer, scheduler, epoch)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 119, in train_epoch<br>
loss, trdice = self.trainIter(fixed, moving, fixed_label, moving_label, fixed_nopad=fixed_nopad, seg_f=seg_fname)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/train.py”, line 43, in trainIter<br>
sim_loss, grad_loss, seg_loss, dice = self.model.forward(fix, moving,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 1008, in forward<br>
output = self._run_ddp_forward(*inputs, **kwargs)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 969, in _run_ddp_forward<br>
return module_to_run(*inputs[0], **kwargs[0])<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/models.py”, line 244, in forward<br>
unet_out, enc_out = self.unet(x)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/Image_registration/registration_copy/registration/models.py”, line 157, in forward<br>
y = self.upsample(y)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/modules/upsampling.py”, line 153, in forward<br>
return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners,<br>
File “/home/csgrad/mbhosale/anaconda3/envs/registration/lib/python3.8/site-packages/torch/nn/functional.py”, line 3912, in interpolate<br>
return torch._C._nn.upsample_nearest3d(input, output_size, scale_factors)<br>
RuntimeError: CUDA out of memory. Tried to allocate 8.59 GiB (GPU 0; 23.69 GiB total capacity; 10.37 GiB already allocated; 1.84 GiB free; 12.23 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>May be this really is the wrong error message in the multi-GPU training case? Also, it should explain why after increasing the memory, I was able to resolve the issue.</p>
<p>Let me know.</p>

Franz: <p>I ran into the same problem (image segmentation, <strong>Pytorch: 1.13.1+cu117</strong>, Ubuntu 20.04).<br>
Basically, I ran a try-except loop with decreasing input sizes to find optimal inference parameters.</p>
<p><strong>I can confirm that this problem occurred (rarely) when I ran out of memory.</strong> I used a single gpu in a subprocess.</p>
<p>More info:</p>
<p>I added this error to my excepted errors and everything worked fine as before. I also monitored ram usage and it was easy to determine the problem. However, using Pytorch 1.9, I did not have this error message.</p>
<p>Here are further related errors I faced over the time, indicating a CUDA out of memory error (strating from Pytorch 1.6 to 1.13):</p>
<pre><code class="lang-auto">NON_INTUITIVE_RAM_ERRORS = [
    'CUDA out of memory',  # some are of course intuitive
    'CUDA error: out of memory',
    'Unable to find a valid cuDNN algorithm to run convolution',
    'cuDNN error: CUDNN_STATUS_NOT_SUPPORTED',
    'DefaultCPUAllocator: not enough memory:',
    'Expected output.numel() &lt;= std::numeric_limits&lt;int32_t&gt;::max() to be true, but got false',
]
</code></pre>
<p>Try-except loop was:</p>
<pre><code class="lang-auto">model, optimizer, loss_func = model_loader.setup_model_optimizer_loss()
device = model_loader.device
channels = model.in_channels
model.eval()
# ram usage with workers seems stable after 3 iterations
iterations = 3
num_samples = iterations * batch_size
data_loader = get_data_loader(model, num_samples, batch_size, channels, input_size, input_depth)
with torch.no_grad():
    with torch.cuda.amp.autocast(enabled=constants.ENABLE_AMP):
        try:
            for i, (input_batch, target_batch) in enumerate(data_loader):
                input_batch = input_batch.to(device)
                out_logits = model(input_batch)
                out_logits = out_logits.float().cpu().numpy()
                ram_used, ram_total, _ = get_memory(device)
        except (RuntimeError, ValueError) as e:
            for non_intuitive_error in constants.NON_INTUITIVE_RAM_ERRORS:
                if non_intuitive_error in e.args[0]:
                    model_loader.wipe_memory()
                    return False, None, None
            return False, None, e.args[0]
model_loader.wipe_memory()
return True, ram_used, None
</code></pre>