RuntimeError : element 0 of tensors does not require grad and does not have a grad_fn

username: christian12
cooked: <p>Hi,</p>
<p>I’m trying to use the “Artificial Trust-Based Task Allocation (ATTA)” code from this github: <a href="https://github.com/arshaali/artificial-trust-task-allocation/tree/main" rel="noopener nofollow ugc">Link</a><br>
I am using Python 3.8.3 and PyTorch 1.7.1 as stated on github. All other packages were installed with pip and are the current available versions.</p>
<p>I’ve been attempting to debug the issue, but I’m unsure what the problem is and how to solve it.</p>
<p>Running file “<em>atta_caseII.py</em>” gives me an error in Line 527.<br>
The code runs for some time until the error occurs.<br>
Adding the full code here exceeds the character limit.<br>
Full code of “atta_caseII.py”: <a href="https://github.com/arshaali/artificial-trust-task-allocation/blob/main/code/atta_caseII.py" rel="noopener nofollow ugc">https://github.com/arshaali/artificial-trust-task-allocation/blob/main/code/atta_caseII.py</a></p>
<p>Line 527:</p>
<pre><code class="lang-auto">loss.backward() #take deriv of loss function wrt the model parameters
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError 
element 0 of tensors does not require grad and does not have a grad_fn 
  File "C:\[PATH_TO_FILE]\code\atta_caseII.py", line 527, in closure      
     loss.backward() #take deriv of loss function wrt the model parameters    
  File "C:\[PATH_TO_FILE]\code\atta_caseII.py", line 538, in &lt;module&gt;      
     optimizer.step(closure) #optimizer calculates the gradient and adjusts the parameters that will minimize the loss function  
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>The code around Line 527:</p>
<pre><code class="lang-auto">def closure(): #closure function must be defined for pytorch
    #this will calculate the gradients. this runs everytime.
    #diff1 = model(bin_c, obs_probs_idxs)
                            
    #diff = torch.tensor(model(bin_c, obs_probs_idxs) - obs_probs_vect, requires_grad=True)
    diff = model(bin_c, obs_probs_idxs) - obs_probs_vect #the diff between trust estimated from artificial trust model and trust approximation
    #print("model diff = ", diff1)
    #print("obs_probs_vect = ", obs_probs_vect)
    #print("diff = ", diff)
    #diff.retain_grad()

    # loss = torch.tensor(torch.mean( torch.pow(diff, 2.0) ), requires_grad=True) #loss needs to be defined in pytorch
                            
    loss = torch.mean( torch.pow(diff, 2.0) ) #calculate the current loss
    #loss.retain_grad() #Something to try if current implementation doesnt work
    #loss = torch.mean( torch.pow( (model(bin_c, obs_probs_idxs) - obs_probs_vect), 2.0 ) )
    #print("loss = ", loss)
    #print("loss.grad_fn = ", loss.grad_fn)
    optimizer.zero_grad() #standard command to give. Sets the gradients all to 0.
    #print("ran zero grad")
    loss.backward() #take deriv of loss function wrt the model parameters 
    #pytorch lets you choose a function to minimize. We are minimizing the loss function defined above.
    #print("ran loss backward")
    return loss

    #print("_l1 = ", model.sigm( model.pre_l_1))
    #print("_u1 = ", model.sigm( model.pre_u_1))
    #print("_l2 = ", model.sigm( model.pre_l_2))
    #print("_u2 = ", model.sigm( model.pre_u_2))
                        
                        
optimizer.step(closure) #optimizer calculates the gradient and adjusts the parameters that will minimize the loss function
#running the optimizer to update the parameters below
</code></pre>

username: ptrblck
cooked: <p>Based on the stacktrace it seems as it <code>loss</code> is detached from the computation graph. Could you check if the model output tensor has a valid <code>.grad_fn</code> or is also already detached?</p>

username: christian12
cooked: <p>It’s <code>grad_fn: None</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccd45309ab82b534d3920a28722c0ce425e0ab0a.png" data-download-href="https://discuss.pytorch.org/uploads/default/ccd45309ab82b534d3920a28722c0ce425e0ab0a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccd45309ab82b534d3920a28722c0ce425e0ab0a.png" alt="image" data-base62-sha1="te0j1rNOCI99xCZviyQmwZPy3iy" width="176" height="500" data-dominant-color="211F21"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">275×778 14.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried different things and the code works now. What ultimately lead to the solution was a warning section containing <code>requires_grad_()</code> in the PyTorch documentation about torch.tensor.<br>
This is the code which works now:</p>
<pre><code class="lang-auto">def closure():
    loss = torch.mean( torch.pow( (model(bin_c, obs_probs_idxs) - obs_probs_vect), 2.0 ) ) 
    loss.requires_grad_(True)
    optimizer.zero_grad()
    loss.backward() 
    return loss
optimizer.step(closure)
</code></pre>
<p>I think this code is ok?<br>
Thanks</p>

username: ptrblck
cooked: <p>No, I don’t think your code fixes the actual issue and just masks it by explicitly calling <code>.requires_grad_()</code> on the already detached loss. Could you post the model definition so that I could check if the activation tensors are detached there?</p>

username: christian12
cooked: <p>Thank you for your quick reply.</p>
<p>I just started with PyTorch but I think you ask about these lines, right?</p>
<pre><code class="lang-auto">class RobotTrustModel(torch.nn.Module):

    def __init__(self):
        super(RobotTrustModel, self).__init__()

        #beta is not relevant for artificial trust model (see BTM paper for natural trust)
        self.pre_beta_1 = Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)
        self.pre_beta_2 = Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)

        #instead of using lower bounds of 0 and upper bounds of 1, the code is more
        #stable when using the range [-10,10] and then converting back later to [0,1].
        #requires_grad=True means it is part of the gradient computation (i.e., the value should be updated when being optimized)
        self.pre_l_1 = Parameter(dtype(-10.0 * np.ones(1)), requires_grad=True)
        self.pre_u_1 = Parameter(dtype( 10.0 * np.ones(1)), requires_grad=True)
        self.pre_l_2 = Parameter(dtype(-10.0 * np.ones(1)), requires_grad=True)
        self.pre_u_2 = Parameter(dtype( 10.0 * np.ones(1)), requires_grad=True)

</code></pre>
<p>The whole code can be found here:<br>
<a href="https://github.com/arshaali/artificial-trust-task-allocation/blob/main/code/atta_caseII.py" rel="noopener nofollow ugc">https://github.com/arshaali/artificial-trust-task-allocation/blob/main/code/atta_caseII.py</a></p>

username: christian12
cooked: <p>After some more digging I think I found the problem.</p>
<p><strong>What I didn’t know was this:</strong><br>
If any operations are applied to two tensors, where at least one of them has requires_grad=True, the resulting tensor will also have requires_grad=True .</p>
<p>And this is what happens here. It happen that the returned tensor is a product of tensors without requires_grad=True.</p>
<p>That’s the problem right?</p>
<p>Here is my detailed Explanation:<br>
Here we set requires_grad=True, that’s fine.<br>
Code A:</p>
<pre><code class="lang-auto">...
from torch.nn import Parameter
...
dtype = torch.DoubleTensor
...
class RobotTrustModel(torch.nn.Module):
    def __init__(self):
       super(RobotTrustModel, self).__init__()
           self.pre_beta_1 = Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)
           self.pre_beta_2 = Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)

           self.pre_l_1 = Parameter(dtype(-10.0 * np.ones(1)), requires_grad=True) 
           self.pre_u_1 = Parameter(dtype( 10.0 * np.ones(1)), requires_grad=True) 
           self.pre_l_2 = Parameter(dtype(-10.0 * np.ones(1)), requires_grad=True) 
           self.pre_u_2 = Parameter(dtype( 10.0 * np.ones(1)), requires_grad=True) 
...
</code></pre>
<p>Coming from here:</p>
<pre><code class="lang-auto">...
def closure(): 
    diff = model(bin_c, obs_probs_idxs) - obs_probs_vect 
    loss = torch.mean( torch.pow(diff, 2.0) )  
    optimizer.zero_grad()  
    loss.backward() 
    return loss 

optimizer.step(closure)
...
</code></pre>
<p>the function “forward” which contains this among other things is called:<br>
Code B:</p>
<pre><code class="lang-auto">def forward(self, bin_centers, obs_probs_idxs):
   ...
   for i in range(n_diffs): #loop over the number of the [row,col] indexes (max is nbins x nbins (625))
        # get the index for the row and column
        bin_center_idx_1 = obs_probs_idxs[i, 0] #get the ith cell row
        bin_center_idx_2 = obs_probs_idxs[i, 1] #get the ith cell col
       # pass l, u, beta &amp; the bin to compute_trust for 1 &amp; 2
        trust[i] = self.compute_trust(l_1, u_1, beta_1, bin_centers[bin_center_idx_1]) * self.compute_trust(l_2, u_2, beta_2, bin_centers[bin_center_idx_2])
   ...
   return trust 
</code></pre>
<p>In order to get tensor trust[i] the function “compute_trust” has to be called twice in each iteration of the loop.<br>
l_1, u_1 &amp; beta_1 and those for _2 all have requires_grad=True because they are all a result of operations of tensors (defined in Code A) where requires_grad=True.</p>
<p>Function “compute_trust”:<br>
Code C:</p>
<pre><code class="lang-auto">def compute_trust(self, l, u, b, p):
            trust = 1.0 - 1.0 / (b * (u - l)) * torch.log( (1.0 + torch.exp(b * (p - l))) / (1.0 + torch.exp(b * (p - u))) )

        else: 
            if p &lt;= l: 
                trust = torch.tensor([1.0], requires_grad=True) #assign a trust of 1
            elif p &gt; u: 
                trust = torch.tensor([0.0], requires_grad=True) #assign a trust of 0
            else:
                trust = (u - p) / (u - l + 0.0001) 
        
        return trust 

</code></pre>
<p>Here is the problem.  <code>trust = torch.tensor([1.0] </code> and <code>trust = torch.tensor([0.0] </code> don’t have requires_grad=True set.<br>
In the case that both function calls of “compute_trust” within the loop in Code B always get either <code>trust = torch.tensor([1.0] </code> or <code>trust = torch.tensor([0.0] </code> as a return the returned trust in Code B also won’t have a required_trust= True which results in an error.</p>
<p>Changing <code>trust = torch.tensor([1.0] </code> and <code>trust = torch.tensor([0.0] </code> in Code C to this and the code works:</p>
<pre><code class="lang-auto">       if p &lt;= l:  
                trust = torch.tensor([1.0] , requires_grad=True) 
       elif p &gt; u:  
                trust = torch.tensor([0.0] , requires_grad=True) 
</code></pre>
<p>.<br>
.<br>
.<br>
.<br>
.<br>
The complete relevant code:</p>
<pre><code class="lang-auto">...
def closure(): 
    diff = model(bin_c, obs_probs_idxs) - obs_probs_vect 
    loss = torch.mean( torch.pow(diff, 2.0) )  
    optimizer.zero_grad()  
    loss.backward() 
    return loss 

optimizer.step(closure)
...
</code></pre>
<pre><code class="lang-auto">class RobotTrustModel(torch.nn.Module):

    def __init__(self):
        super(RobotTrustModel, self).__init__()

        #beta is not relevant for artificial trust model (see BTM paper for natural trust)
        self.pre_beta_1 = Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)
        self.pre_beta_2 = Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)

        #instead of using lower bounds of 0 and upper bounds of 1, the code is more
        #stable when using the range [-10,10] and then converting back later to [0,1].
        #requires_grad=True means it is part of the gradient computation (i.e., the value should be updated when being optimized)
        self.pre_l_1 = Parameter(dtype(-10.0 * np.ones(1)), requires_grad=True)
        self.pre_u_1 = Parameter(dtype( 10.0 * np.ones(1)), requires_grad=True)
        self.pre_l_2 = Parameter(dtype(-10.0 * np.ones(1)), requires_grad=True)
        self.pre_u_2 = Parameter(dtype( 10.0 * np.ones(1)), requires_grad=True)


    def forward(self, bin_centers, obs_probs_idxs):
        # the bins and hols [row,col] indexes which are not nan
        #bin_centers and obs_probs_idxs are passed in
        
        n_diffs = obs_probs_idxs.shape[0] #the number of [row,col] indexes (max is nbins x nbins (625))
        trust = torch.zeros(n_diffs) #create a 1xn_diffs array of 0s

        if(self.pre_l_1 &gt; self.pre_u_1): #if the lower bound is greater than the upper bound
            buf = self.pre_l_1 #switch the l_1 and u_1 values
            self.pre_l_1 = self.pre_u_1
            self.pre_u_1 = buf

        if(self.pre_l_2 &gt; self.pre_u_2): #if the lower bound is greater than the upper bound
            buf = self.pre_l_2 #switch the l_2 and u_2 values
            self.pre_l_2 = self.pre_u_2
            self.pre_u_2 = buf

        l_1 = self.sigm(self.pre_l_1) #convert to [0,1] range
        u_1 = self.sigm(self.pre_u_1)
        beta_1 = self.pre_beta_1 * self.pre_beta_1 #want beta to be positive to compute trust using the artificial trust model

        l_2 = self.sigm(self.pre_l_2)
        u_2 = self.sigm(self.pre_u_2)
        beta_2 = self.pre_beta_2 * self.pre_beta_2

        for i in range(n_diffs): #loop over the number of the [row,col] indexes (max is nbins x nbins (625))
            # get the index for the row and column
            bin_center_idx_1 = obs_probs_idxs[i, 0] #get the ith cell row
            bin_center_idx_2 = obs_probs_idxs[i, 1] #get the ith cell col
            # pass l, u, beta &amp; the bin to compute_trust for 1 &amp; 2
            trust[i] = self.compute_trust(l_1, u_1, beta_1, bin_centers[bin_center_idx_1]) * self.compute_trust(l_2, u_2, beta_2, bin_centers[bin_center_idx_2])
            #computing the trust estimate for each cell based on the current lower and upper bounds (basically the 3d trust plot)

        if usecuda:
            trust = trust.cuda()

        return trust

    def compute_trust(self, l, u, b, p):
        #passing in lower bound capability belief, upper bound capability belief, beta, task requirement lambdabar
        # p is the value from one bin
        if b &lt; -50: #this is for natural trust. This never happens for the artificial trust model.
            trust = 1.0 - 1.0 / (b * (u - l)) * torch.log( (1.0 + torch.exp(b * (p - l))) / (1.0 + torch.exp(b * (p - u))) )
            print(f"b {trust}")
        else: #as long as you pass in a positive beta, we will be calculating artificial trust which doesnt depend on beta
            if p &lt;= l: #if lambdabar is less than the lower bound capability belief
                #trust = torch.tensor([1.0])    # OLD
                trust = torch.tensor([1.0], requires_grad=True) #assign a trust of 1
            elif p &gt; u: #if lambdabar is greater than the upper bound capability belief
                #trust = torch.tensor([0.0])    # OLD
                trust = torch.tensor([0.0], requires_grad=True) #assign a trust of 0
            else:
                trust = (u - p) / (u - l + 0.0001) #assign trust as a constant slope between u and l. 0.0001 is to not divide by 0.

        if usecuda:
            trust = trust.cuda()
        
        return trust #returns the trust in human agent given lower bound l, upper bound u, beta term b, and task requirement lambdabar p 

    def sigm(self, x): #sigmoid function to convert [-10,10] (really [-inf,inf]) to [0,1]
        return 1 / (1 + torch.exp(-x))

    def sigmoid(self, lambdabar, agent_c):
        #takes in task requirement for one dimension and the agent's actual capability for that dimension
        #calculates true trust to determine the stochastic task outcome

        #if lambdabar == agent_c, the sigmoid output is 0.5
        eta = 1/50.0 #is a good value through testing for good capability updating
        #eta = 1/5.0
        #eta = 1/500.0
        return 1 / (1 + math.exp((lambdabar - agent_c)/eta))
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="christian12" data-post="6" data-topic="179771">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/f17d59/48.png" class="avatar"> christian12:</div>
<blockquote>
<p>Changing <code>trust = torch.tensor([1.0] </code> and <code>trust = torch.tensor([1.0] </code> in Code C to this and the code works:</p>
</blockquote>
</aside>
<p>You explanation is mostly correct, but as already explained setting <code>requires_grad=True</code> on an already detached tensor will not fix the issue.</p>
<p>E.g. look at this example:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)
x = torch.randn(1, 1)

out = model(x)

# explicitly detach the tensor from the computation graph
out_detached = out.detach()

# this will raise an error
out_detached.mean().backward()
# RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

# try to "fix" it
out_detached.requires_grad_()

# backward won't fail anymore
out_detached.mean().backward()

# but also no gradiets are calculated
print(model.weight.grad)
# None
</code></pre>

username: christian12
cooked: <p>Thanks.</p>
<p>I think I have it now, have I?</p>
<p>I deleted the <code>, requires_grad=True</code> from the <code>self.pre_</code> variables.<br>
Eg. What used to be<br>
<code>Parameter(dtype(4.0 * np.ones(1)), requires_grad=True)</code> is now<br>
<code>Parameter(dtype(4.0 * np.ones(1)))</code>.</p>
<pre><code class="lang-auto">...
from torch.nn import Parameter
...
dtype = torch.DoubleTensor
...
class RobotTrustModel(torch.nn.Module):

    def __init__(self):
        super(RobotTrustModel, self).__init__()

        #beta is not relevant for artificial trust model (see BTM paper for natural trust)
        self.pre_beta_1 = Parameter(dtype(4.0 * np.ones(1)))
        self.pre_beta_2 = Parameter(dtype(4.0 * np.ones(1)))

        #instead of using lower bounds of 0 and upper bounds of 1, the code is more
        #stable when using the range [-10,10] and then converting back later to [0,1].
        #requires_grad=True means it is part of the gradient computation (i.e., the value should be updated when being optimized)
        self.pre_l_1 = Parameter(dtype(-10.0 * np.ones(1)))
        self.pre_u_1 = Parameter(dtype( 10.0 * np.ones(1)))
        self.pre_l_2 = Parameter(dtype(-10.0 * np.ones(1)))
        self.pre_u_2 = Parameter(dtype( 10.0 * np.ones(1)))
...
</code></pre>
<p>And what used to be:<br>
<code>torch.tensor([1.0], requires_grad=True)</code> is now <code>Parameter(dtype(np.ones(1)))</code> &amp;<br>
<code>torch.tensor([0.0], requires_grad=True)</code> is now <code>Parameter(dtype(np.zeros(1)))</code></p>
<pre><code class="lang-auto">def compute_trust(self, l, u, b, p):
        if b &lt; -50:
            trust = 1.0 - 1.0 / (b * (u - l)) * torch.log( (1.0 + torch.exp(b * (p - l))) / (1.0 + torch.exp(b * (p - u))) )
        else: 
            if p &lt;= l: 
                trust = Parameter(dtype(np.ones(1))) #assign a trust of 1 
            elif p &gt; u: 
                trust = Parameter(dtype(np.zeros(1))) #assign a trust of 0
            else:
                trust = (u - p) / (u - l + 0.0001) 
        
        return trust 
</code></pre>
<p>With this code <code>requires_grad=True</code> isn’t used anymore and hopefully no tensor gets detached. The whole code runs through without an error.</p>