SparseAdam occur dimension error when input empty features

Zhu_Kai_hua: <h3>
<a name="describe-the-bug-1" class="anchor" href="#describe-the-bug-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/bug.png?v=12" title=":bug:" class="emoji" alt=":bug:" loading="lazy" width="20" height="20"> Describe the bug</h3>
<pre><code class="lang-auto">from torch import nn,optim
import torch
import torch.nn.functional as F
from torch.autograd import Variable

def cumsum_1d_with_zero(input: torch.Tensor, include_last: bool = False) -&gt; torch.Tensor:
    output = torch.zeros(input.size(0) + 1, dtype=input.dtype)
    torch.cumsum(input, 0, out=output[1:])
    if include_last:
        return output
    else:
        return output[:-1]

class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()
        self.embedding = nn.EmbeddingBag(4, 300, sparse=True, mode='sum')
        self.linear1 = nn.Linear(300, 1)
        self.optimizer = optim.SparseAdam(self.embedding.parameters(), 0.0001)
        self.dense_optimizer = optim.Adam(self.linear1.parameters(), 0.0001)

    def forward(self,inputs):
        rows = [torch.tensor(input, dtype=torch.int64) for input in inputs]
        lengths = torch.tensor([x.size(0) for x in rows], dtype=torch.int64)
        offsets = cumsum_1d_with_zero(lengths)
        indices = torch.cat(rows)
        embedding_res = self.embedding(indices, offsets)
        print(embedding_res)
        logits = F.sigmoid(self.linear1(embedding_res))
        return logits

    def optimize(self, logits):
        labels = Variable(torch.LongTensor([0, 1, 0, 1]).reshape(4,1).float())
        criterion = nn.BCEWithLogitsLoss()
        loss = criterion(logits, labels)
        print(loss.item())
        loss.backward()
        self.optimizer.step()
        self.dense_optimizer.step()
</code></pre>
<p>run with this! but sparseAdam hit bug!</p>
<pre><code class="lang-auto">inputs = [[],[],[],[]]
result = model(inputs)
model.optimize(result)
</code></pre>
<ul>
<li>Thanks! SparseAdam will hit this bug, but SGD optimizer is ok.</li>
</ul>
<p><img width="690" alt="截屏2022-11-21 下午10 26 12" src="https://user-images.githubusercontent.com/90753896/203082184-21e6b516-cc28-4fae-a2fe-b013a1e0dbd9.png" height="364"></p>
<h3>
<a name="versions-2" class="anchor" href="#versions-2"></a>Versions</h3>
<p>Collecting environment information…<br>
PyTorch version: 1.8.2+cu111<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.2 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)<br>
Python platform: Linux-5.15.0-1022-aws-x86_64-with-glibc2.29<br>
Is CUDA available: False<br>
CUDA runtime version: 10.1.243<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration: Could not collect<br>
Nvidia driver version: Could not collect<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.23.0<br>
[pip3] torch==1.8.2+cu111<br>
[conda] Could not collect</p>