Saving and Loading Optimizer Params

username: alic
cooked: <p>Hi,<br>
I’m trying to save and load optimizer params as we do for a model, but although i tried in many different ways, still i couldn’t work it. Here is the code:</p>
<pre><code class="lang-auto">best_model_wts = copy.deepcopy(model.state_dict())
best_optim_pars = copy.deepcopy(optimizer.state_dict())

for epoch in range(num_epochs):
    for phase in ['train', 'val']:
         if phase == 'train':
             model.train()
         else:
             model.eval()

         running_loss = 0.0
         running_corrects = 0

         if epoch &lt; 20:
             error_sigma = 2.0
         elif 19 &lt; epoch &lt; 40:
             error_sigma = 1.5
             if epoch == 20:
                 model.load_state_dict(best_model_wts)
                 optimizer.load_state_dict(best_optim_pars)

         for inputs, labels, _ in data_loaders[phase]:
             batch_size = len(labels)
             converted_inputs = dictionary_to_tensor(inputs, batch_size)
             converted_inputs = numpy2tensor(converted_inputs)
             labels = labels.to(device)

             optimizer.zero_grad()
             with torch.set_grad_enabled(phase == 'train'):
                    outs = model(converted_inputs)
                    loss = criterion(outs, labels)
                    target_err = torch.sign(loss - error_sigma)

                    _, preds_all = torch.max(outs, 1)

                    if phase == 'train':
                        loss.backward()
                        optimizer.step(target_err)

                running_loss += loss.item() * converted_inputs.size(1)
                running_corrects += torch.sum(preds_all == labels.data)

            if phase == 'train':
                scheduler.step()

            data_size = len(data_loaders[phase].dataset)

            epoch_loss = running_loss / data_size
            epoch_acc = running_corrects.double() / data_size

            if phase == 'val':
                val_acc_history.append(epoch_acc)
                if epoch_acc &gt; best_acc:
                    best_acc = epoch_acc
                    best_epoch = epoch
                    best_loss = epoch_loss
                    best_model_wts = copy.deepcopy(model.state_dict())
                    best_optim_pars = copy.deepcopy(optimizer.state_dict()) #this line gives error

</code></pre>
<p>Then, I get the below error:<br>
<strong>raise RuntimeError("Only Tensors created explicitly by the user "</strong><br>
<strong>RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment</strong></p>
<p>Any idea about how to solve it? Thank you very much.</p>

username: Alexey_Demyanchuk
cooked: <p>Hi. Did you try the official pytorch <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="noopener nofollow ugc">tutorial</a>?<br>
Is it work for you?</p>

username: Alexey_Demyanchuk
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn

m = nn.Linear(10, 2)
opt = torch.optim.Adam(m.parameters())
best = {'optimizer_state_dict': opt.state_dict()}

opt.zero_grad()
opt.step()

opt = torch.optim.Adam(m.parameters())
opt.load_state_dict(best['optimizer_state_dict'])
</code></pre>
<p>This dummy example is working fine for me.</p>

username: alic
cooked: <p>Hi, Thank you for the follow up. Yes, I tried the official solution but still it didn’t work. There’s no problem with model save/load. The problem is optimizer state save/load. In my current case, the below code raises an error:</p>
<pre><code class="lang-auto">best_optim_pars = copy.deepcopy(optimizer.state_dict())
</code></pre>
<p>When I update the code by removing copy.deepcopy as below:</p>
<pre><code class="lang-auto">best_optim_pars = optimizer.state_dict()
</code></pre>
<p>Then the line here gives error:</p>
<pre><code class="lang-auto">optimizer.load_state_dict(best_optim_pars)
</code></pre>

username: Alexey_Demyanchuk
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn

X = torch.rand(2, 10)
y = torch.tensor([[0,1], [1,0]], dtype=torch.float32)

m = nn.Linear(10, 2)
opt = torch.optim.Adam(m.parameters())
state = opt.state_dict()
crit = nn.BCEWithLogitsLoss()

for i in range(5):
    if i == 2:
        # load
        opt.load_state_dict(state)
    opt.zero_grad()
    out = m(X)
    loss = crit(y, out)
    loss.backward()
    opt.step()
    # save
    state = opt.state_dict()
</code></pre>
<p>Update to less dummy example. Also works fine.</p>

username: Alexey_Demyanchuk
cooked: <p>If you are running this in notebook, I would suggest to restart kernel so to get rid of <code>best_optim_pars = copy.deepcopy(optimizer.state_dict())</code> variable.</p>

username: Alexey_Demyanchuk
cooked: <p>also, I am doing it in current version of Pytorch</p>

username: alic
cooked: <p>I tried all these different ways actually, the error remains. Optimizer is SGD with momentum:</p>
<pre><code class="lang-auto">optimizer_ft = optim.SGD(model_ft.parameters(), lr=params.lr, momentum=params.momentum)
optimizer_ft.zero_grad()
</code></pre>
<p>I also tried without copy.deepcopy as in your solution:</p>
<pre><code class="lang-auto">best_model_wts = copy.deepcopy(model.state_dict())
best_optim_pars = optimizer.state_dict()
</code></pre>
<p>And then, in the code when I load:</p>
<pre><code class="lang-auto"> if epoch == 20:
     model.load_state_dict(best_model_wts)
     #this is for initialization, not sure if needed?
     optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
     optimizer.zero_grad()
     optimizer.load_state_dict(best_optim_pars)
</code></pre>
<p>For saving the best model and the current optim params:</p>
<pre><code class="lang-auto"> if phase == 'val':
     val_acc_history.append(epoch_acc)
     if epoch_acc &gt; best_acc:
         best_acc = epoch_acc
         best_epoch = epoch
         best_loss = epoch_loss
         best_model_wts = copy.deepcopy(model.state_dict())
         best_optim_pars = optimizer.state_dict()
</code></pre>
<p>Still getting this error:</p>
<p>File “/media/alic/ssdmain/Projects/crandrnn/src/train_multi_level_model.py”, line 83, in train_model<br>
optimizer.load_state_dict(best_optim_pars)<br>
File “/home/alic/anaconda3/envs/crandrnn/lib/python3.7/site-packages/torch/optim/optimizer.py”, line 105, in load_state_dict</p>
<p>File “/home/alic/anaconda3/envs/crandrnn/lib/python3.7/site-packages/torch/tensor.py”, line 23, in <strong>deepcopy</strong><br>
raise RuntimeError("Only Tensors created explicitly by the user "<br>
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment.</p>

username: Alexey_Demyanchuk
cooked: <p>It is strange, I can’t reproduce your error using deepcopy. I am not sure, it may be python version or even system dependent</p>
<pre><code class="lang-auto">import copy
import torch
import torch.nn as nn

X = torch.rand(2, 10)
y = torch.tensor([[0,1], [1,0]], dtype=torch.float32)

m = nn.Linear(10, 2)
opt = torch.optim.SGD(m.parameters(), lr=0.1)

m_state = copy.deepcopy(m.state_dict())
state = opt.state_dict()

crit = nn.BCEWithLogitsLoss()

for i in range(5):
    if i == 2:
        # load
        m.load_state_dict(m_state)
        opt = torch.optim.SGD(m.parameters(), lr=0.1)
        opt.load_state_dict(state)
    opt.zero_grad()
    out = m(X)
    loss = crit(y, out)
    loss.backward()
    opt.step()
    # save
    m_state = copy.deepcopy(m.state_dict())
    state = opt.state_dict()
</code></pre>
<p>Perhaps, as error suggest you can get rid off all deepcopy calls.</p>

username: Alexey_Demyanchuk
cooked: <p>Also, you could take a look <a href="https://discuss.pytorch.org/t/how-does-one-reproduce-the-deepcopy-error-of-tensors-made-by-code/69852/3">in this thread</a>. Perhaps, you are doing something similar with your model or optimizer.</p>

username: alic
cooked: <p>Thank you for your help. I’m actually doing the same as you suggest in your solution:<br>
Initial:</p>
<pre><code class="lang-auto">best_model_wts = copy.deepcopy(model.state_dict())
best_optim_state = optimizer.state_dict()
</code></pre>
<p>Updating in the loop:</p>
<pre><code class="lang-auto">if epoch &lt; 20:
    error_sigma = 2.0
    elif 19 &lt; epoch &lt; 40:
        error_sigma = 1.5
        if epoch == 20:
            model.load_state_dict(best_model_wts)
            optimizer = optim.SGD(model.parameters(), lr=0.001)
            optimizer.load_state_dict(best_optim_state)
</code></pre>
<p>And saving the best models and optimizer state:</p>
<pre><code class="lang-auto">if phase == 'val':
    val_acc_history.append(epoch_acc)
    if epoch_acc &gt; best_acc:
        best_acc = epoch_acc
        best_epoch = epoch
        best_loss = epoch_loss
        best_model_wts = copy.deepcopy(model.state_dict())
        best_optim_state = optimizer.state_dict()
</code></pre>
<p>However, I’m still getting the same error during the update line for optimizer.<br>
I tried your solution and it runs without any problem on my environment (python 3.7.5, pytorch 1.2) but mine is still giving the above-mentioned runtime error.</p>

username: Alexey_Demyanchuk
cooked: <p>Did you try not to use <code>deepcopy</code> at all in your code?<br>
Also, did you check if your code updating or reassigning a model parameters somewhere (as in linked thread I posted before)?<br>
Can you update to a current pytorch version?</p>

username: alic
cooked: <p>Thank you very much. It works after updating to the current pytorch version.</p>
<ul>
<li>I tried not to use any deepcopy at all, but again it failed during optimizer state updating with the same error. No problem for model update.</li>
<li>I checked the link but in their code, they are reproducing the error, I did not get it for my case actually.</li>
</ul>
<p>Anyway, as soon as I updated the pytorch version, it worked without any problem. Still didn’t get it why but it’s working.<br>
Thanks again.</p>

username: Alexey_Demyanchuk
cooked: <p>Glad, I can help. Perhaps some intrinsic of pytorch deepcopy works differently in new version, the only reason I see.  Apparently, now all tensors “support the deepcopy protocol”, not only “graph leaves” <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>