Error while saving state of optimizer in Pytorch

username: Sumera_Rounaq
cooked: <p>Hi all,<br>
I am saving state of optimizer and getting this error<br>
“state_dict() missing 1 required positional argument: ‘self’”.</p>
<p>Code:</p>
<pre><code class="lang-auto">torch.save({
            'epoch': num_epochs,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': opt_func.state_dict(),
             'lr':lr,
            'loss': val_loss,
            'accuracy': val_acc 
            }, 'CNN-model.pt')
</code></pre>
<p>Any help shall be highly appreciated</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet which would reproduce the issue?<br>
Based on the error it seems as if you’ve overridden the <code>state_dict</code> method and might be missing an argument. However, you’ve also used the <code>torchx</code> category so I’m unsure if that’s specific to <code>torchx</code> or not.</p>

username: Sumera_Rounaq
cooked: <p>I did some search and came up with some solution and applied it in my code but still I am having some confusions and also getting errors. So, I am posting my changed code over here:</p>
<p>Question  : If I am not passing saved optimizer in my fit function then how it is calculating accuracy and loss?<br>
Confusion: When I am passing optimizer3(saved optimizer) in fit function  it is throwing error “Adam is not callable”. (For reference please check attached snapshot).Kindly help me understand the cause of this issue and provide the way to  resolve it.</p>
<p><strong>Model Training</strong></p>
<pre><code class="lang-auto">@torch.no_grad()
def evaluate(model, val_loader):
    model.eval()
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)
#optimizer = torch.optim.Adam(model.parameters(), lr)
def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
#def fit(epochs, lr, model, train_loader, val_loader, optimizer):  
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase 
        model.train()
        train_losses = []
        for batch in train_loader:
            loss = model.training_step(batch)
            train_losses.append(loss)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        result['train_loss'] = torch.stack(train_losses).mean().item()
        model.epoch_end(epoch, result)
        history.append(result)
    return history, optimizer
</code></pre>
<pre><code class="lang-auto">num_epochs = 2
opt_func = torch.optim.Adam
lr = 0.001
history, optimizer = fit(num_epochs, lr, model, train_dl, val_dl, opt_func)
</code></pre>
<p>Output:</p>
<blockquote>
<p>Epoch [0], train_loss: 1.7836, val_loss: 1.4373, val_acc: 0.4659<br>
Epoch [1], train_loss: 1.2700, val_loss: 1.1127, val_acc: 0.6018</p>
</blockquote>
<pre><code class="lang-auto"># Saving the model and optimizer

torch.save({
            'epoch': num_epochs,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict()            
            }, 'CNN-model.pt')

# Load the model and optimizer
model3 = to_device(Cifar10CnnModel(), device) # instantiate model
optimizer3 = torch.optim.Adam(model3.parameters(), lr) # instantiate optimizer
checkpoint = torch.load('CNN-model.pt')
model3.load_state_dict(checkpoint['model_state_dict']) # Load model
optimizer3.load_state_dict(checkpoint['optimizer_state_dict']) # load optimizer
epoch = checkpoint['epoch']
history, optimizer = fit(4, lr, model3, train_dl, val_dl) # no optimizer
</code></pre>
<p><strong>Output:</strong></p>
<blockquote>
<p>Epoch [0], train_loss: 1.0236, val_loss: 1.0559, val_acc: 0.6151<br>
Epoch [1], train_loss: 1.0089, val_loss: 1.0491, val_acc: 0.6178<br>
Epoch [2], train_loss: 1.0020, val_loss: 1.0447, val_acc: 0.6209<br>
Epoch [3], train_loss: 0.9967, val_loss: 1.0421, val_acc: 0.6193</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4959cc6f489242352b282262dfcd612c3e386345.png" data-download-href="https://discuss.pytorch.org/uploads/default/4959cc6f489242352b282262dfcd612c3e386345" title="Error-snapshot"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4959cc6f489242352b282262dfcd612c3e386345.png" alt="Error-snapshot" data-base62-sha1="asTfmV5ESoWyp6u18U5oLfvePJP" width="690" height="278" data-dominant-color="F5F4F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Error-snapshot</span><span class="informations">764×308 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sumera_Rounaq" data-post="3" data-topic="136463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sumera_rounaq/48/30432_2.png" class="avatar"> Sumera_Rounaq:</div>
<blockquote>
<p>If I am not passing saved optimizer in my fit function then how it is calculating accuracy and loss?</p>
</blockquote>
</aside>
<p>The optimizer is not calculating the accuracy or loss and is used to update the passed parameter using the already calculated gradients.</p>
<aside class="quote no-group" data-username="Sumera_Rounaq" data-post="3" data-topic="136463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sumera_rounaq/48/30432_2.png" class="avatar"> Sumera_Rounaq:</div>
<blockquote>
<p>When I am passing optimizer3(saved optimizer) in fit function it is throwing error “Adam is not callable”.</p>
</blockquote>
</aside>
<p>I guess you are passing an optimizer object in this case instead of the function:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)

# works
opt_func = torch.optim.Adam
optimizer = opt_func(model.parameters(), lr=1e-3)

# raises your error
optimizer = optimizer(model.parameters(), lr=1e-3)
&gt; TypeError: 'Adam' object is not callable
</code></pre>

username: Sumera_Rounaq
cooked: <p>Thank you so much for your guidance. I am still getting error while passing saved optimizer.<br>
<code>history, optimizer = fit(4, lr, model3, train_dl, val_dl, optimizer3)</code><br>
Still same error!</p>

username: ptrblck
cooked: <p>This is expected.<br>
Your current method uses the <code>opt_func</code> argument as a function not an object:</p>
<pre><code class="lang-python">def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    [...]
    optimizer = opt_func(model.parameters(), lr) # !!! this is wrong if opt_func is already an optimizer object
</code></pre>
<p>so you would have to change the <code>fit</code> method to accept an optimizer instead of the function.</p>

username: dbish
cooked: <p>Just checking in to confirm this question, <a class="mention" href="/u/sumera_rounaq">@Sumera_Rounaq</a>  you tagged torchx. Is there anything torchx specific you were doing or another reason you tagged this category?</p>

username: Sumera_Rounaq
cooked: <p>Thank you so much. It really helped alot and i figured out what was my mistake.</p>

username: Sumera_Rounaq
cooked: <p>Actually I want to deploy my model in next step so for this reason i tagged torchx. Anyways, as of this in hand problem it has nothing to do with torchx. For better understanding of my problem i removed that specific part of code. Hope I clarified.</p>