UnboundLocalError: local variable 'values' referenced before assignment in lr_scheduler

111220: <p>This my code:</p>
<pre><code class="lang-auto">optimizer=torch.optim.AdamW(model.parameters(),lr=0.005,weight_decay=0.01)
scheduler=torch.optim.lr_scheduler.OneCycleLR(optimizer,max_lr=0.0005,total_steps=total_steps,epochs=30)

</code></pre>
<p>when I use scheduler.step(), this error appear:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3ec81fb5bde7354a1881b64ad1e6d29d408512a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3ec81fb5bde7354a1881b64ad1e6d29d408512a" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3ec81fb5bde7354a1881b64ad1e6d29d408512a.png" alt="捕获" data-base62-sha1="no8xFZFATf6aTfhAwNVyWpVBJYC" width="690" height="74" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">1071×116 5.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>how to solve this error?</p>

ptrblck: <p>Could you post a code snippet to reproduce this issue, please?<br>
This dummy example runs fine:</p>
<pre><code class="lang-python">model = models.resnet18()
optimizer=torch.optim.AdamW(model.parameters(),lr=0.005,weight_decay=0.01)
scheduler=torch.optim.lr_scheduler.OneCycleLR(optimizer,max_lr=0.0005,total_steps=10,epochs=30)

output = model(torch.randn(1, 3, 224, 224))
loss = F.cross_entropy(output, torch.randint(0, 1000, (1,)))
loss.backward()
optimizer.step()
scheduler.step()
</code></pre>

111220: <pre><code class="lang-auto">criterion=smp.utils.losses.DiceLoss()
for epoch in range(epochs):
        model.train()
        epoch_loss=0
        for i,(image,mask) in enumerate(train_dl):
                 optimizer.zero_grad()
                 image = image.to(device).float()
                 output=model(image)
                 output=output.to('cpu')
                 output=torch.sigmoid(output)
                 loss=criterion(output,mask)
                 epoch_loss+=loss
                 loss.backward()
                 optimizer.step()
                 scheduler.step()
</code></pre>

111220: <p>Need any other snippet code? I’m not sure how much complete code you need</p>

ptrblck: <p>A minimal and executable code snippet would be great.<br>
Could you try to remove unnecessary functions and use some random inputs, so that we can reproduce this issue locally?</p>

111220: <pre><code class="lang-auto">criterion=smp.utils.losses.DiceLoss()
for epoch in range(epochs):
        model.train()
        epoch_loss=0
        for i,(image,mask) in enumerate(train_dl):
                 data=torch.randn(8,3,128,128)
                 target=torch.randn(8,4,128,128)
                 optimizer.zero_grad()
                 data=data.to(device).float()
                 output=model(data)
                 output=torch.sigmoid(output)
                 output=output.to('cpu')
                 loss=criterion(output,target)
                 epoch_loss += loss
                 loss.backward()
                 optimizer.step()
                 scheduler.step()
           
        epoch_loss = epoch_loss / total_steps
        print('epoch:{},epoch_loss:{}'.format(epoch,epoch_loss))
</code></pre>
<p>I’m sorry for my slow response。This code might suit your needs。</p>
<p>This is the error：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc50e0b783e6f43c139f0c2b76757e1c9c85c1e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc50e0b783e6f43c139f0c2b76757e1c9c85c1e1" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc50e0b783e6f43c139f0c2b76757e1c9c85c1e1.png" alt="捕获" data-base62-sha1="vr0k3658KF42esLO2BwIpem37Ff" width="690" height="94" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">1114×152 7.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You can see that the first call to scheduler. Step should have been fault-free because it printed the following print statement, as well as eval for the model. But an error should have occurred on the second call</p>

111220: <p>I forgot the code that came out and this is the following code</p>
<pre><code class="lang-auto"> model.eval()
 metric,metric2,valid_loss=evalue(model,valid_dl)
 if metric2&gt;best_score:
            state={'state':model.state_dict(),'best_score':metric2}
            torch.save(state,checkpoint_path)
            best_score=metric2
logging.basicConfig(filename='cloud4.log', level=logging.DEBUG, format='%(asctime)s-%(message)s')
logging.warning('epoch_loss:{},metric1:{},metric2:{}'.format(epoch_loss,metric,metric2))
</code></pre>

111220: <p>Now it turns out that if you use annotated code, you get an error, while unannotated code works fine</p>
<pre><code class="lang-auto">  # for image,mask in train_dl:
        for i in range(5):
                 data=torch.randn(8,3,128,128)
                 target=torch.randn(8,4,128,128)
                 optimizer.zero_grad()
                 data=data.to(device).float()
                 output=model(data)
                 output=torch.sigmoid(output)
                 output=output.to('cpu')
                 loss=criterion(output,target)
                 epoch_loss += loss
                 loss.backward()
                 optimizer.step()
                 scheduler.step()
                 # image = image.to(device).float()
                 # optimizer.zero_grad()
                 # pre_mask=model(image)
                 # pre_mask=pre_mask.to('cpu')
                 # pre_mask=torch.sigmoid(pre_mask)
                 # loss=criterion(pre_mask,mask)
                 # epoch_loss+=loss
                 # loss.backward()
                 # optimizer.step()
                 # scheduler.step()
</code></pre>
<p>I really don’t know why, is it the data problem that caused the error</p>

ptrblck: <p>Thanks for the code so far.<br>
Could you also post the code you are using to initialize the model, optimizer, and scheduler?<br>
Also, could you try to run the code on the CPU only and check, if you see the same error?<br>
If not, could you rerun the GPU code using <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace again?</p>

111220: <p>This is my code:</p>
<pre><code class="lang-auto">model=smp.Unet('efficientnet-b3',encoder_weights=None,classes=4)
# # #
for m in model.modules():
    weights_init_kaiming(m)
# checkpoint=torch.load(checkpoint_path)
# model.load_state_dict(checkpoint['state'])
device=torch.device('cuda:0')
model.to(device)

total_steps=len(train_dl)
optimizer=torch.optim.AdamW(model.parameters(),lr=0.05,weight_decay=0.01)
scheduler=torch.optim.lr_scheduler.OneCycleLR(optimizer,max_lr=0.0005,total_steps=total_steps,epochs=30)

</code></pre>
<p>It was very embarrassing that I could not debug with CPU because of the machine, but if I used GPU to debug, the error result did not change</p>

ptrblck: <p>Could you call <code>scheudler.get_lr()</code> before the error is thrown and check the return value, please?</p>

111220: <p>I am very sorry for replying to you a few days later, because I am a sophomore student in university and I have a lot of things to do recently, so I didn’t deal with this problem for a few days. As you requested, I added this line of code. The problem is that it returned the value successfully without any problems during the first epoch. But after the second epoch, it reported an error</p>

ptrblck: <p>Are you recreating or manipulating the scheduler or optimizer in each epoch somehow?</p>

111220: <p>29/5000<br>
Thank you for answering my question so patiently. My code should not have this problem。<br>
If I use following code,the error will not appear( it will appear when using the annotating code):</p>
<pre><code class="lang-auto">for epoch in epochs:
     for batch in train_loader:
          #scheduler.step()
     scheduler.step()
</code></pre>
<p>Here is my complete training code(When you put scheduler. Step () into each batch iteration,error will apear)：</p>
<pre><code class="lang-auto">for epoch in range(epochs):
        model.train()
        epoch_loss=0
        epoch_mask_loss=0
        epoch_label_loss=0
        for image,mask,label in train_dl:
            optimizer.zero_grad()
            r=np.random.rand(1)

            #cutmix transform
            if r&gt;threshold:
                lam=np.random.beta(50,50)
                image,mask,cutmix_label=make_cutmix(image,mask,lam)
                image=image.to(device).float()
                mask_prediction,label_prediction=model(image)
                label_prediction=label_prediction.to('cpu')
                label_loss=lam*label_criterion(label_prediction,label)+(1-lam)*label_criterion(label_prediction,cutmix_label)
            else:
                image = image.to(device).float()
                mask_prediction,label_prediction=model(image)
                label_prediction=label_prediction.to('cpu')
                label_loss=label_criterion(label_prediction,label)

            mask_prediction=torch.sigmoid(mask_prediction)
            mask_prediction = mask_prediction.to('cpu')
            mask_loss=mask_criterion(mask_prediction,mask)
            epoch_mask_loss+=mask_loss
            epoch_label_loss+=label_loss
            loss=label_loss+mask_loss
            epoch_loss+=loss
            loss.backward()
            optimizer.step()



        epoch_loss = epoch_loss / total_steps
        epoch_label_loss=epoch_label_loss/total_steps
        epoch_mask_loss=epoch_mask_loss/total_steps
        print('epoch:{},epoch_loss:{},epoch_label_loss:{},epoch_mask_loss:{}'.format(epoch,epoch_loss,epoch_label_loss,epoch_mask_loss))
        model.eval()
        metric,metric2,valid_loss=evalue(model,valid_dl)
        if metric2&gt;best_score:
            state={'state':model.state_dict(),'best_score':metric2}
            torch.save(state,checkpoint_path)
            best_score=metric2

        logging.warning('epoch_loss:{},metric1:{},metric2:{}'.format(epoch_loss,metric,metric2))
        scheduler.step()
</code></pre>

kradkahaddi: <p>Hi,</p>
<p>I had the same error that I think has been fixed.</p>
<p>In the end it seems like the number of epochs you had mentioned in your scheduler was less than the number of epochs you tried training for.<br>
I went into %debug in the notebook and tried calling self.get_lr() as suggested.<br>
I got this message:<br>
*** ValueError: Tried to step 3752 times. The specified number of total steps is 3750</p>
<p>Then with some basic math and a lot of code search I realised that I had specified 5 epochs in my scheduler but called for 10 epochs in my fit function.</p>
<p>Hope this helps.</p>

Sanjayvarma11: <p>There is an error with total_steps.i am also getting same error but i rectified it</p>

lizhitwo: <p>Related to this issue.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/32639" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/32639" target="_blank" rel="nofollow noopener">Unbound local variable in LR scheduler</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-01-27" data-time="01:49:24" data-timezone="UTC">01:49AM - 27 Jan 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-02-11" data-time="17:29:15" data-timezone="UTC">05:29PM - 11 Feb 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/vadimkantorov" target="_blank" rel="nofollow noopener">
          <img alt="vadimkantorov" src="https://avatars0.githubusercontent.com/u/1041752?v=4" class="onebox-avatar-inline" width="20" height="20">
          vadimkantorov
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">This is about a mysterious error. I have the following code that runs ok:
import torch
class MultiStepLR(torch.optim.lr_scheduler._LRScheduler):
 def __init__(self, optimizer, gamma, milestones,...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: optimizer</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
If get_lr() throws an error, pytorch suppresses it but will later encounter this “values” unbounded bug. Fix the get_lr() error and this bug will go away.</p>

warrenfelsh: <p>Python has lexical scoping by default, which means that although an enclosed scope can access values in its enclosing scope, it cannot modify them (unless they’re declared global with the global keyword). A closure binds values in the enclosing environment to names in the local environment. The local environment can then use the bound value, and even reassign that name to something else, but it can’t modify the binding in the enclosing environment. UnboundLocalError happend because when python sees an assignment inside a function then it considers that variable as local variable and will not fetch its value from enclosing or global scope when we execute the function. To modify a <a href="http://net-informations.com/python/iq/global.htm" rel="nofollow noopener">global variable</a> inside a function, you must use the global keyword.</p>

xiaochaosui: <p>Hello,I had the same error that I can’t solve it.<br>
I want to ask you some question about it.Thank you.<br>
What is  the ‘The specified number of total steps is 3750’?<br>
How to change the number of steps?<br>
Thank you.</p>

kradkahaddi: <p>Hi make sure that your dataloader and the scheduler have the same number of iterations. If I remember correctly I got this error when using the OneCycle LR scheduler which needs you to specify the max number of steps as init parameter. Hope this helps! If this isn’t the error you have, then please provide code and try to see what your scheduler.get_lr() method returns.</p>