Error regarding calling numpy() on Tensor with requires grad

username: Mat_Hopps
cooked: <p>Hi all,</p>
<p>I am implementing  part of the code for Federated Matched Averaging (full code found here: <a href="https://github.com/IBM/FedMA/blob/master/language_modeling/language_fedma.py" rel="noopener nofollow ugc">https://github.com/IBM/FedMA/blob/master/language_modeling/language_fedma.py</a>)</p>
<p>An error occurs when the following line runs:</p>
<pre><code class="lang-auto">batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)]
</code></pre>
<p>And, the runtime error is</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
c:\Users\65967\Desktop\Federated Learning\Python\PyT_FedMA.py in &lt;module&gt;
     345                                                                     it=it,
     346                                                                     n_layers=NUM_LAYERS,
---&gt; 347                                                                     matching_shapes=matching_shapes)
     348     matching_shapes.append(next_layer_shape)
     349     assignments_list.append(assignments)

c:\Users\65967\Desktop\Papers\battery\Federated Learning\Python\language_fedma.py in layerwise_fedma(batch_weights, layer_index, sigma_layers, sigma0_layers, gamma_layers, it, n_layers, matching_shapes)
    283     ########################################
    284     assignment_c, global_weights_c, global_sigmas_c, popularity_counts = match_layer(weights_bias, sigma_inv_layer, mean_prior,
--&gt; 285                                                                   sigma_inv_prior, gamma, it)
    286 
    287     ########################################

c:\Users\65967\Desktop\Federated Learning\Python\language_fedma.py in match_layer(weights_bias, sigma_inv_layer, mean_prior, sigma_inv_prior, gamma, it)
    146     #AA: On how to use built-in sorted() function with key attribute https://www.programiz.com/python-programming/methods/built-in/sorted
    147     group_order = sorted(range(J), key=lambda x: -weights_bias[x].shape[0]) #AA: This sorts index of J from one with largest negative number (hence smallest value) to lowest negative number (hence largest value)?? Does it??
--&gt; 148     batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)] #AA: RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
    149     #batch_weights_norm = [w.detach().numpy() * s.detach().numpy() for w, s in zip(weights_bias, sigma_inv_layer)]
    150     #batch_weights_norm = [torch.Tensor.cpu(w) * torch.Tensor.cpu(s) for w, s in zip(weights_bias, sigma_inv_layer)]

c:\Users\65967\Desktop\Federated Learning\Python\language_fedma.py in &lt;listcomp&gt;(.0)
    146     #AA: On how to use built-in sorted() function with key attribute https://www.programiz.com/python-programming/methods/built-in/sorted
    147     group_order = sorted(range(J), key=lambda x: -weights_bias[x].shape[0]) #AA: This sorts index of J from one with largest negative number (hence smallest value) to lowest negative number (hence largest value)?? Does it??
--&gt; 148     batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)] #AA: RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
    149     #batch_weights_norm = [w.detach().numpy() * s.detach().numpy() for w, s in zip(weights_bias, sigma_inv_layer)]
    150     #batch_weights_norm = [torch.Tensor.cpu(w) * torch.Tensor.cpu(s) for w, s in zip(weights_bias, sigma_inv_layer)]

~\anaconda3\envs\torch17cuda11\lib\site-packages\torch\_tensor.py in __array__(self, dtype)
    676             return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
    677         if dtype is None:
--&gt; 678             return self.numpy()
    679         else:
    680             return self.numpy().astype(dtype, copy=False)

RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
</code></pre>
<p>I have tried using <code>w.detach().numpy()</code> and <code>torch.Tensor.cpu(w)</code> to rectify the error but I got a message saying that <code>w</code> is already a numpy array.</p>
<p>Could anyone please enlighten me on what is triggering this error?</p>
<p>Thank you very much.</p>

username: ptrblck
cooked: <p>Calling <code>.detach().numpy()</code> should work, since the error message explains that an explicit detaching is needed as seen here:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)
x.numpy()
# &gt; RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
print(x.detach().numpy())
# &gt; array([-0.13592759], dtype=float32)
</code></pre>

username: Mat_Hopps
cooked: <p>Thank you for the reply!</p>
<p>I had tried <code>w.detach().numpy()</code> previously. Specifically, this was how I revised line 148:</p>
<pre><code class="lang-auto"> batch_weights_norm = [w.detach().numpy() * s.detach().numpy() for w, s in zip(weights_bias, sigma_inv_layer)]
</code></pre>
<p>I get the following attribute error message which indicates <code>w</code> is a numpy array already. Hence, my confusion.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
c:\Users\65967\Desktop\Federated Learning\Python\PyT_FedMA.py in &lt;module&gt;
     345                                                                     it=it,
     346                                                                     n_layers=NUM_LAYERS,
---&gt; 347                                                                     matching_shapes=matching_shapes)
     348     matching_shapes.append(next_layer_shape)
     349     assignments_list.append(assignments)

c:\Users\65967\Desktop\Federated Learning\Python\language_fedma.py in layerwise_fedma(batch_weights, layer_index, sigma_layers, sigma0_layers, gamma_layers, it, n_layers, matching_shapes)
    283     ########################################
    284     assignment_c, global_weights_c, global_sigmas_c, popularity_counts = match_layer(weights_bias, sigma_inv_layer, mean_prior,
--&gt; 285                                                                   sigma_inv_prior, gamma, it)
    286 
    287     ########################################

c:\Users\65967\Desktop\Federated Learning\Python\language_fedma.py in match_layer(weights_bias, sigma_inv_layer, mean_prior, sigma_inv_prior, gamma, it)
    147     group_order = sorted(range(J), key=lambda x: -weights_bias[x].shape[0]) #AA: This sorts index of J from one with largest negative number (hence smallest value) to lowest negative number (hence largest value)?? Does it??
    148     #batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)] #AA: RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
--&gt; 149     batch_weights_norm = [w.detach().numpy() * s.detach().numpy() for w, s in zip(weights_bias, sigma_inv_layer)]
    150     #batch_weights_norm = [torch.Tensor.cpu(w) * torch.Tensor.cpu(s) for w, s in zip(weights_bias, sigma_inv_layer)]
    151 

c:\Users\65967\Desktop\Federated Learning\Python\language_fedma.py in &lt;listcomp&gt;(.0)
    147     group_order = sorted(range(J), key=lambda x: -weights_bias[x].shape[0]) #AA: This sorts index of J from one with largest negative number (hence smallest value) to lowest negative number (hence largest value)?? Does it??
    148     #batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)] #AA: RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
--&gt; 149     batch_weights_norm = [w.detach().numpy() * s.detach().numpy() for w, s in zip(weights_bias, sigma_inv_layer)]
    150     #batch_weights_norm = [torch.Tensor.cpu(w) * torch.Tensor.cpu(s) for w, s in zip(weights_bias, sigma_inv_layer)]
    151 

AttributeError: 'numpy.ndarray' object has no attribute 'detach'
</code></pre>

username: ptrblck
cooked: <p>I guess <code>weights_bias</code> might be a tensor sometimes and a numpy array other times?<br>
If so, you could add an <code>if</code> condition to check for its <code>type</code>.</p>

username: Mat_Hopps
cooked: <p>Thank you for the reply and spending time on my question! I appreciate it.</p>
<p>It is an interesting thought that <code>weights_bias</code> can be tensor or a numpy array at sometimes. I’d like to clarify that I am working on a Federated learning problem with 3 local clients. <code>weights_bias</code> is actually a list containing the weights of the first layer  of each client’s model. Therefore length <code>weights_bias</code> is 3 (for 3 clients). With this in mind, I checked the data type of each element of the list <code>weights_bias</code> and the list <code>sigma_inv_layer</code>:</p>
<pre><code class="lang-auto">    logger.info("weights bias: {}".format(weights_bias[0].shape)) #AA: weights_bias[0] refers to first client
    logger.info("weights bias: {}".format(weights_bias[1].shape))
    logger.info("weights bias: {}".format(weights_bias[2].shape))
    logger.info("sigma_inv_layer: {}".format(sigma_inv_layer[0].shape))
    logger.info("sigma_inv_layer: {}".format(sigma_inv_layer[1].shape))
    logger.info("sigma_inv_layer: {}".format(sigma_inv_layer[2].shape))
</code></pre>
<p>The output below shows that each element of the  <code>weights_bias</code>  is a tensor but elements of <code>sigma_inv_layer</code> are not. Hence, <code>w * s</code> is resulting in errors as you mentioned.</p>
<pre><code class="lang-auto">INFO:root:weights bias: torch.Size([3, 800])
INFO:root:weights bias: torch.Size([3, 800])
INFO:root:weights bias: torch.Size([3, 800])
INFO:root:sigma_inv_layer: (800,)
INFO:root:sigma_inv_layer: (800,)
INFO:root:sigma_inv_layer: (800,)
</code></pre>
<p>As I need <code>w</code> to be a tensor, one solution I can think of is to change <code>s</code> to be a tensor too. The following revised code eliminates the original error message but I am afraid this tweak is throwing up further errors downstream:</p>
<pre><code class="lang-auto">batch_weights_norm = [w * torch.tensor(s, requires_grad=True) for w, s in zip(weights_bias, sigma_inv_layer)]
</code></pre>
<p>Thus, may I check if there are any alternative approaches to making the original line of code work? (It seems to have worked for the authors of the github repo). If it’s not possible, it’s okay. I am new to Pytorch.</p>
<pre><code class="lang-auto">batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)]
</code></pre>
<p>Thank you once again!</p>

username: ptrblck
cooked: <p>Ah, good catch!<br>
Based on your original code snippet you don’t want <code>w</code> or <code>s</code> to track the gradient history, since you are explicitly detaching both and are using the numpy arrays (which Autograd won’t be able to track).<br>
In this case, keep the <code>w.detach().numpy()</code> usage and just multiply it with <code>s</code> directly (without trying to <code>detach()</code> it).</p>

username: Mat_Hopps
cooked: <p>Thank you for your reply! I was only able to catch it thanks to your astute observation.</p>
<p>Sorry, to clarify, the original code snippet  I was referring to is below (sorry, wasn’t clearer in my question).</p>
<pre><code class="lang-auto">batch_weights_norm = [w * s for w, s in zip(weights_bias, sigma_inv_layer)]
</code></pre>
<p><code>w</code> needs to track the gradient as it is the weights. It looks like <code>s</code> does not need to as it is not a tensor in the first place. I only used <code>w.detach().numpy()</code> due to the initial runtime error.</p>
<p>So, in conclusion, may I check that it is not possible to perform the <code>w * s</code> elementwise multiplication  without changing <code>s</code> to a tensor?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Yes, if <code>s</code> is a numpy array create the tensor via <code>torch.from_numpy(s)</code>. Otherwise <code>w * s</code> will try to convert <code>w</code> to a numpy array and thus the error message will be raised.</p>

username: Mat_Hopps
cooked: <p>Thank you! I understand it now. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:"></p>