Weights not updating during training

mhr: <p>Hello,</p>
<p>I’m having a strange issue where not a single one of my model’s parameters is getting updated. I saw another answer that’s similar to the problem I have (<a href="https://discuss.pytorch.org/t/model-parameters-is-none-while-training/6830/2?u=mhr">Model.parameters() is None while training</a>), but the solution that’s given doesn’t quite solve my problem, because notably <strong>every</strong> one of my model’s parameter grads is None.</p>
<pre><code class="lang-python">for p in model.parameters():
    if p.grad is not None:
        print(p.grad.data)
</code></pre>
<p>I.e. the above loop won’t print out anything for my model. And even after calling loss.backward() and optimizer.step(), the parameter grads all remain None.</p>
<p>I’ve verified that a loss is being calculated, it just seems that optimization isn’t taking place, even if the backward() and step() functions are definitely getting run.</p>
<p>This is my optimization code:</p>
<pre><code class="lang-python">def step(batch, model, criterion, optimizer=None):
    # let go of old gradients
    model.zero_grad()
    
    X = batch["X"].to(DEVICE)
    y = batch["y"].to(DEVICE)

    ## Forward Pass ##
    predictions = model(inputs)

    ## Calculate Loss ##
    loss = criterion(predictions, y)

    if optimizer is not None:
        # backward pass + optimize
        loss.backward()
        optimizer.step()
        
    return loss

def train_model(model=None, lr=0.01):
    criterion = nn.CrossEntropyLoss().to(DEVICE)
    params = list(filter(lambda p: p.requires_grad, model.parameters()))
    optimizer = torch.optim.Adam(params=params, lr=lr)

    for epoch in range(1, N_EPOCHS+1):
        for i, batch in enumerate(tqdm(train_loader)):
            loss = step(batch, model, criterion, optimizer=optimizer)

model = NeuralNet()
model.to(DEVICE)
train_model(model=model)
</code></pre>
<p>The only thing I can think of is that it’s that how I’ve split my optimization code into functions, rather than keeping it all in the same scope. I was able to get a toy neural net with toy randn()-generated data to train properly in a single training loop that isn’t spread out across multiple functions (the model was initialized within the same scope it was trained).</p>
<p>I have a couple pre-trained <code>torch.nn</code> modules in NeuralNet(): a pre-trained Embedding layer and a pre-trained ResNet, both of which have frozen weights by setting <code>requires_grad</code> to <code>False</code>.</p>
<p>What could be causing this issue?</p>

justusschock: <p>can you try with this step-function:</p>
<pre><code class="lang-auto">def step(batch, model, criterion, optimizer=None):
    x = batch["X"].to(DEVICE)
    y = batch["y"].to(DEVICE)

    ## Forward Pass ##
    predictions = model(inputs)

    ## Calculate Loss ##
    loss = criterion(predictions, y)

    if optimizer is not None:
        optimizer.zero_grad()
        # backward pass + optimize
        loss.backward()
        optimizer.step()
        
    return loss
</code></pre>
<p>I don’t know whether it is a difference if you zero the grads by the optimizer’s function or the model’s function, but I usually do it with the optimizer and this works for me.</p>

mhr: <p>Thanks for replying! I tried that, and it didn’t make a difference. The parameter <code>grad</code> attributes are all still None.</p>

ptrblck: <p>Where do you get <code>inputs</code> from? Is it a typo and it should be <code>x</code> instead?<br>
Also, remove the if condition on the <code>optimizer</code> and just call <code>loss.backward()</code> and <code>optimizer.step()</code>.</p>

mhr: <p>Oops, yes. That’s a typo. It was supposed to be <code>x</code>, I tried to make my code more generic.</p>
<p>Sorry, I left this out. The reason I have the if condition is because I reuse the same <code>step()</code> function for validation, except with <code>step(optimizer=None)</code>, so that neither <code>loss.backward()</code> nor <code>optimizer.step()</code> get called.</p>

ptrblck: <p>Thanks for the clarification.<br>
Your code seems to work. I’ve created a small dummy example where a very simple model fits some random noise.<br>
Both approaches work, i.e. the plain code and using your functions.<br>
As the code is quite long, I’ve created a <a href="https://gist.github.com/ptrblck/e6a59d2ad9a719ad74e9d84de61b0f30">gist</a>.<br>
Could you think about other issues, where you might have reset your gradients or the optimizer?</p>

mhr: <p>Thank you so much for your help! I also tried training a simple mock network last night and came to a similar conclusion: my optimization code is valid, but my actual (<em>not</em> mock) network definition must be faulty in some way. I’d rather not post my network definition classes (classes that inherit from <code>nn.Module</code>) at the moment. I will create mock <code>nn.Module</code> classes to track down which part of my network is causing the bug, and I’ll post again for future readers once I figure it out.</p>

mhr: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I discovered the source of the issue!</p>
<p>I’ve had a memory leak that has plagued me while using an LSTM, and to fix the issue, my solution was to use <code>copy.copy()</code> (from the Python standard lib) on tensors in a few places. But in using <code>copy.copy()</code>, it appears to have disrupted the end-to-end nature of the network, preventing proper backpropagation. When I removed <code>copy.copy()</code>, I found that backpropagation works now.</p>
<p>But I still have the memory leak problem and can’t complete a single batch if I’m not <code>copy()</code>ing, so this isn’t a perfect solution. Do you have a suggestion for an alternative solution to the memory leak that doesn’t break backpropagation?</p>
<p>A simplified version of my code is in the following gist:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/mhr/2b214985c1bafdbba971590878c83fe3" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/mhr/2b214985c1bafdbba971590878c83fe3" target="_blank" rel="nofollow noopener">https://gist.github.com/mhr/2b214985c1bafdbba971590878c83fe3</a></h4>
<h5>either_backprop_or_leak.py</h5>
<pre><code class="Python">def find_indices(condition, list_):
    return [i for i, x in enumerate(list_) if condition(x)]

class Network(nn.Module):
    def __init__(self, vocab_dim, embed_dim, input_dim, hidden_dim=512):
        self.embedding = nn.Embedding(vocab_dim, embed_dim)
        self.rnn = nn.LSTM(embed_dim+input_dim,
                           hidden_dim,
                           batch_first=True)
        self.out = nn.Linear(hidden_dim, vocab_dim)</code></pre>
This file has been truncated. <a href="https://gist.github.com/mhr/2b214985c1bafdbba971590878c83fe3" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In particular, pay attention to lines 49-52. This is where I’m calling <code>copy.copy()</code>.</p>
<p>Thank you again for your help.</p>

ptrblck: <p>Skimming through your code, I couldn’t find anything obviously wrong.</p>
<p>Could your issue be related to <a href="https://github.com/pytorch/pytorch/issues/1183#issuecomment-408285420">this one</a>?</p>
<p>To investigate it further, could you tell which OS, GPU (if used), CUDA + cuDNN version you are using?</p>

mhr: <p>Right, and I don’t have the memory leak when I run the code on the CPU, only when I switch to the GPU.</p>
<p>I don’t think this is related to that particular bug, because my leak happens in the forward pass (that issue is specifically about <code>.backward()</code>). My memory leak occurs anytime I accumulate tensors. “Accumulate” could mean, e.g.: assigning a tensor at the index of another tensor (lines 49-52), iteratively summing losses with +=, or appending a tensor to a list. Somehow this accumulation prevents the tensors from being garbage collected.</p>
<p>These are the details of my hardware, drivers, and software:</p>
<ul>
<li>OS
<ul>
<li>Ubuntu 16.04.5 LTS</li>
</ul>
</li>
<li>GPU
<ul>
<li>NVIDIA Quadro GP100</li>
</ul>
</li>
<li>CUDA
<ul>
<li>9.0.176</li>
</ul>
</li>
<li>cuDNN
<ul>
<li>6.0.21</li>
</ul>
</li>
<li>PyTorch
<ul>
<li>0.4.0</li>
</ul>
</li>
</ul>
<p>If this problem comes from below the code as you’re suggesting, is there a short-term fix I can use in the code to get around the memory leak without disrupting backprop?</p>
<p>Perhaps <code>copy.copy()</code> is copying too shallowly, leaving out some important attributes that make backprop work. Do you know what attributes those might be?</p>
<p>Given this hypothesis of shallow copying being the culprit for the backprop bug, I’ve tried to use <code>copy.deepcopy()</code> for lines 49-52. But using <code>copy.deepcopy()</code> leads to the following error:</p>
<p><code>RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment</code></p>

ptrblck: <p>Remember that each node you save in a list stores the whole computation graph assigned to it.<br>
This is quite a common issue regarding memory leaks.<br>
E.g. if you save the loss for debugging/printing purposes with <code>losses += loss</code> instead of <code>losses += loss.item()</code>, the whole computation graph will be stored and cannot be freed.<br>
Could you check for these issues?</p>

mhr: <p>That makes perfect sense for appending to lists and summing. But what about with lines 49-52, assigning tensors at an index in another tensor? There shouldn’t be a memory leak. But in that case, we <em>do</em> want to preserve the computation graph.</p>

mhr: <p>For some additional information, when I tried turning off cudnn, the memory leak still occurred:<br>
<code>torch.backends.cudnn.enabled = False</code></p>

catica: <p>I just wanted to add that I also found that the parameters from my model did not update when I did model2 = copy.deepcopy(model).</p>