Expected nested_tensorlist[0].size() > 0 to be true, but got false

jack_S: <p>I was using a <code>clip_grad_value_</code> in <code>torch.nn.utils.clip_grad</code> to clip the grad of <code>resnet18</code> in <code>torchvision.models</code>. However, I encountered this error and I have no idea about how to debug.</p>
<pre><code class="lang-auto">  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/nn/utils/clip_grad.py", line 122, in clip_grad_value_
    grouped_grads = _group_tensors_by_device_and_dtype([grads])
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/utils/_foreach_utils.py", line 42, in _group_tensors_by_device_and_dtype
    torch._C._group_tensors_by_device_and_dtype(tensorlistlist, with_indices).items()
RuntimeError: Expected nested_tensorlist[0].size() &gt; 0 to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
</code></pre>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the error, please?</p>

jack_S: <p>Sure</p>
<pre><code class="lang-auto">from torchvision.models import resnet18
from torch.nn.utils import clip_grad_value_
from torch import randn


x = randn(10, 3, 32, 32)
model = resnet18()
loss = model(x).sum()
clip_grad_value_(model.parameters(), 1)
</code></pre>

mikaylagawarecki: <p>I could reproduce this locally, one of the problems here is that you are not calling <code>.backward()</code> on the loss so autograd does not run and the <a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/utils/clip_grad.py#L121" rel="noopener nofollow ugc">list of grads in <code>clip_grad_value</code> is empty</a></p>
<p>e.g. This snippet should work</p>
<pre><code class="lang-auto">from torchvision.models import resnet18
from torch.nn.utils import clip_grad_value_
from torch import randn


x = randn(10, 3, 32, 32)
model = resnet18()
loss = model(x).sum().backward()
clip_grad_value_(model.parameters(), 1)
</code></pre>
<p>However <code>clip_grad_value</code> should return without error in the edge case where the all the grads are None, I will submit a PR to fix this.</p>

jack_S: <p>Thanks for reminding me to call <code>backward</code> before clipping. I just forgot when the parameters have grads. <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>As was already explained, you need to compute the gradients before clipping them. However, the error message might be improved.</p>

JuriAbramov: <p>I think I’m running into the same issue (NGC 24.01 container):</p>
<p>Expected !nested_tensorlist[0].empty() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
<p>This is certainly after backward(), and the weird thing: I call<br>
clip_grad_norm_(parameters, max_norm, foreach=True)<br>
clip_grad_value_(parameters, max_value, foreach=True)<br>
The former works. I can also call it twice. But the latter does not. I’d expect that norm_ and value_would behave the same…</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

JuriAbramov: <p>Tricky. But I think it has something to do with model.parameters() become a generated object.<br>
If I convert it to a list, it seems to work.</p>

ptrblck: <p>Yes, this might be the case as explained <a href="https://discuss.pytorch.org/t/parameters-disappear-after-setting-require-grad-true/192488/3">here</a> as well.</p>

Z-MU-Z: <p>I encountered the same problem. Everything was normal when I used torch 2.0.1+cu118. When I used torch 2.1.0+cu118, this error occurred.</p>