Complex recurrent layers produce NaN as grad

username: FunBotan
cooked: <p>Hi all. I am trying to run the most basic single-layer RNN with complex inputs on the nightly build of PyTorch (1.10.0-CPU). The problem is that the gradient always evaluates to NaN. I’ve tried all recurrent layers (RNN, GRU, LTSM) with the same result. Here is the model:</p>
<pre><code class="lang-auto">class CRNN(torch.nn.Module):
    def __init__(
        self,
        input_dim: int,
        output_dim: int,
        **kwargs
    ):
        super().__init__(**kwargs)
        self.model = torch.nn.RNN(input_dim, output_dim,
                                  batch_first=True,
                                  dtype=torch.cfloat)
        self.loss = torch.nn.L1Loss()
        self.optimizer = torch.optim.Adam(self.parameters())

    def forward(self, x):
        x = torch.unsqueeze(x, 0)
        return self.model(x)[0]

    def fit(self, x, y):
        self.optimizer.zero_grad()
        z = torch.squeeze(self(x), 0)
        loss = self.loss(z, y)
        loss.backward()
        self.optimizer.step()
        return loss.item()
</code></pre>
<p>I used <code>torch.autograd.set_detect_anomaly(True)</code> and it gave the following results:</p>
<ul>
<li>For RNN, the first NaN appears in <code>AddmmBackward</code>
</li>
<li>For GRU and LSTM, the first NaN appears in <code>L1LossBackward</code>
</li>
</ul>
<p>Since I have no idea how to interpret this, I am left to hope for help from the community.</p>

username: ptrblck
cooked: <p>Are you able to reproduce this issue using random inputs? If so, could you post the random tensor initialization so that we could try to reproduce it, please?</p>

username: FunBotan
cooked: <p>Ok, I have tried running the same model with the following “dataset”:</p>
<pre><code class="lang-auto">class RandomDataset(torch.utils.data.IterableDataset):
    def __init__(self):
        super(RandomDataset, self).__init__()

    def __iter__(self):
        return repeat(torch.rand(100, dtype=torch.cfloat))
</code></pre>
<p>Results are the same.</p>

username: ptrblck
cooked: <p>Thanks for the update!<br>
I cannot reproduce the issue using:</p>
<pre><code class="lang-python">class CRNN(torch.nn.Module):
    def __init__(
        self,
        input_dim: int,
        output_dim: int,
        **kwargs
    ):
        super().__init__(**kwargs)
        self.model = torch.nn.RNN(input_dim, output_dim,
                                  batch_first=True,
                                  dtype=torch.cfloat)
        self.loss = torch.nn.L1Loss()
        self.optimizer = torch.optim.Adam(self.parameters())

    def forward(self, x):
        x = torch.unsqueeze(x, 0)
        return self.model(x)[0]

    def fit(self, x, y):
        self.optimizer.zero_grad()
        z = torch.squeeze(self(x), 0)
        loss = self.loss(z, y)
        loss.backward()
        self.optimizer.step()
        return loss.item()

class RandomDataset(torch.utils.data.IterableDataset):
    def __init__(self):
        super(RandomDataset, self).__init__()

    def __iter__(self):
        return repeat(torch.rand(100, dtype=torch.cfloat))
    
model = CRNN(100, 100)
dataset = RandomDataset()
loader = torch.utils.data.DataLoader(dataset)

for i, data in enumerate(loader):
    loss = model.fit(data, data)
    print(i, loss)
    if i &gt; 10000:
        break
</code></pre>

username: FunBotan
cooked: <p>Can you please try with GRU/LSTM too?</p>

username: FunBotan
cooked: <p>OK, it seems like I found something interesting.<br>
In your code, try to replace</p>
<pre><code class="lang-auto">loader = torch.utils.data.DataLoader(dataset)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">loader = torch.utils.data.DataLoader(dataset, batch_size=100)
</code></pre>
<p>Whether or not the error appears depends on the <code>batch_size</code>. For me, it breaks when <code>batch_size</code> is ≥30.<br>
Looks like exploding gradients to me.</p>

username: ptrblck
cooked: <p>Yes, this could be the case, but note that your code snippet uses the specified <code>batch_size</code> as the sequence length, since you are calling <code>x = torch.unsqueeze(x, 0)</code> and are thus creating a hard-coded batch size of <code>1</code> while the <code>batch_size</code> specified in the <code>DataLoader</code> is moved to <code>dim1</code> and is thus the sequence length.</p>

username: FunBotan
cooked: <p>Yes, this is exactly why I assumed it to be an issue of exploding gradients: the error appears when the sequence length reaches a critical threshold.<br>
Is this expected behavior or a bug?</p>

username: ptrblck
cooked: <p>I would claim it’s expected as usually gradient clipping is applied for long sequences.</p>

username: FunBotan
cooked: <p>The same behavior does not occur with real numbers for the same model, though. Is there a theoretical reason as to why complex numbers are more prone to exploding gradients?<br>
Also, I’ve tested the same setup with LTSM and GRU and there the cutoff line is around 100, meaning they delay the onset of exploding gradients compared to RNN, but do not prevent it. Same with L2-norm gradient clipping.</p>

username: Nico78
cooked: <p>The initial set of <em>layers</em> for <em>recurrent</em> neural operations universally … in the complexity of the task, we should use more <em>complex</em> models.</p>
<p><a href="https://www.dgcustomerfirst.ltd/" rel="noopener nofollow ugc">DGCustomerFirst.com</a></p>