Different size for tensor grad compared to the tensor itself

username: Mahdi_Zaferanchi
cooked: <p>I’m implementing an algorithm with a custom <code>autograd.Function</code>. In this algorithm, some variables have gradients with a different size from the tensor itself. I understand this is unusual, but it makes sense for my use case. But this leads to the following error:</p>
<p>RuntimeError: attempting to assign a gradient of size ‘…’ to a tensor of size ‘…’. Please ensure that the gradient and the tensor are the same size</p>
<p>Is there anyway to bypass this check?</p>

username: KFrank
cooked: <p>Hi Mahdi!</p>
<aside class="quote no-group" data-username="Mahdi_Zaferanchi" data-post="1" data-topic="199611" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_zaferanchi/48/56560_2.png" class="avatar"> Mahdi_Zaferanchi:</div>
<blockquote>
<p>some variables have gradients with a different size from the tensor itself. I understand this is unusual</p>
</blockquote>
</aside>
<p>Indeed.</p>
<aside class="quote no-group">
<blockquote>
<p>but it makes sense for my use case.</p>
</blockquote>
</aside>
<p>Could you explain your use case?  What you propose is quite odd.</p>
<aside class="quote no-group">
<blockquote>
<p>But this leads to the following error:</p>
<p>RuntimeError: attempting to assign a gradient of size ‘…’ to a tensor of size ‘…’. Please ensure that the gradient and the tensor are the same size</p>
</blockquote>
</aside>
<p>This is certainly what one would expect mathematically.  The gradient with<br>
respect to a tensor is the set of the partial derivatives of a scalar function of<br>
that tensor with respect to the elements of that tensor.  So the elements of<br>
the gradient should be in one-to-one correspondence with the elements of<br>
the tensor.</p>
<p>Pytorch normally uses such gradients to perform optimization steps on the<br>
tensors that are the weights of a model being trained.  The simplest case<br>
(namely plain-vanilla <code>SGD</code>) subtracts the learning rate times the gradient<br>
from the weight tensor itself – an operation that requires the gradient and<br>
tensor to have the same size.</p>
<p>So what is your use case and how are you hoping it might fit into pytorch’s<br>
overall framework?</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mahdi_Zaferanchi
cooked: <p>Thanks for the reply.</p>
<p>My algorithm uses extreme quantization. So in the forward pass, each bit of a tensor element is its own “unit” and needs a separate gradient. For example, an intermediate representation of shape (10,) with float32 data actually represents 320 independent bits and not 10 float numbers. But each bit still has a float gradient so I need <code>.grad</code> to be of shape (320,). (Batch size is one in this example.)</p>
<p>My custom C++ and CUDA code takes care of handling the forward pass with this unusual setup. This is much faster and memory efficient compared to using Boolean data for my use case.</p>
<p>If I can’t set <code>.grad</code> to a tensor of the size that I need, the only solution I can think of is to initialize a separate variable of the correct size just so it can hold the gradients. This will probably work, but the memory allocated for the tensor is wasted since I’m only using its <code>.grad</code> and not <code>.data</code>.</p>
<p>What I’m doing is inspired by <a href="https://arxiv.org/abs/2210.08277" rel="noopener nofollow ugc">this paper</a> but the training and backward pass is handled differently there.</p>
<p>I had a similar problem because pytorch doesn’t allow integer tensors to have gradients. I changed my C++ functions to expect floats which I cast to integers that can be shifted and manipulated to reveal each individual bit and handle binary operations (AND, OR, etc.). But that effort is in vain now if I can’t work around this shape limitation.</p>
<p>You might be thinking that bits can’t have gradients because they don’t change smoothly. I can’t fully explain everything here but suffice to say that (I think <img src="https://discuss.pytorch.org/images/emoji/apple/crossed_fingers.png?v=12" title=":crossed_fingers:" class="emoji" alt=":crossed_fingers:" loading="lazy" width="20" height="20">) I know what I’m doing and the algorithm has been validated in python already (in a very inefficient way by using a float for each bit).</p>

username: KFrank
cooked: <p>Hi Mahdi!</p>
<aside class="quote no-group" data-username="Mahdi_Zaferanchi" data-post="3" data-topic="199611" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_zaferanchi/48/56560_2.png" class="avatar"> Mahdi_Zaferanchi:</div>
<blockquote>
<p>This will probably work, but the memory allocated for the tensor is wasted since I’m only using its <code>.grad</code> and not <code>.data</code>.</p>
</blockquote>
</aside>
<p>Why not then just allocate a single tensor of the appropriate size for your<br>
(weird) gradient.  The elements of <code>.grad</code> and <code>.data</code> won’t match up<br>
one-to-one anyway, so what do you gain by having your gradient tensor<br>
be attached to your “data” tensor as its <code>.grad</code> property?</p>
<aside class="quote no-group">
<blockquote>
<p>You might be thinking that bits can’t have gradients because they don’t change smoothly.</p>
</blockquote>
</aside>
<p>Yes, that is exactly what I am thinking …</p>
<p>You haven’t told us what you will be using your “gradient” for.  I can hardly<br>
imagine that you will be able to hook it into pytorch’s autograd framework<br>
in any sensible way.  So it would seem like creating a separate “gradient”<br>
tensor – not attached as a <code>.grad</code> property to anything else – would be the<br>
way to go.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mahdi_Zaferanchi
cooked: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="199611">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Why not then just allocate a single tensor of the appropriate size for your<br>
(weird) gradient.</p>
</blockquote>
</aside>
<p>That would solve my problem. At least in terms of not wasting memory.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="199611">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I can hardly imagine that you will be able to hook it into pytorch’s autograd framework<br>
in any sensible way.</p>
</blockquote>
</aside>
<p>I <em>am</em> using the autograd engine. That is why I need to store the gradients in a <code>.grad</code>.</p>
<p>Seems like there really isn’t a straightforward solution for me and my current workaround might be the best I can get. Perhaps apart from compiling pytorch from source and removing the shape mismatch check which I’m not even sure would work. Maybe later when I’m more confident about my algorithm and trying to optimize for performance.</p>
<p>Thank you for your help.</p>