Allow size mis-match in autograd forward vs. backward

A_A: <p>Hello,</p>
<p>I have a slightly less conventional scenario involving my custom autograd function (M). Given target of shape: [*, N, *, *], The computational graph looks like this:</p>
<p><code>&gt; INPUT -&gt; NN -&gt; M.forward (out shape: [*, C, *, *]) -&gt; Zero pad to match C = N -&gt; Loss on all N channels</code></p>
<p>During backward I want the gradients to the module M be of shape [*, N, *, *], even though its output was having less channels. I know how to handle the extra padded dimensions in the gradient.</p>
<p>An option can be that I pad inside the module M, however I do not know the shape of target before-hand.</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>Would you have a dummy code sample that shows what you’re trying to do by any change (20 lines that we can run on colab for example)?<br>
In particular, I am wondering what you expect to happen with the NN part of the backward, will they get gradients with different shape than the forward?</p>

A_A: <p>Thanks for you reply.</p>
<p>NN part of the backward would be oblivious to what is going on outside its box <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:">. Not sure, if I can make a toy colab example, but perhaps I can elaborate more about the computational graph. Denoting modules by capital letters and variable in small letters:</p>
<pre><code class="lang-auto">input -&gt; NN -&gt; x -&gt; M -&gt; z (z has C channels)
z, z_g -&gt; ZERO PADDING -&gt; z_p (Now z has changed to N channels by knowing z_g the ground-truth)
(z_p, z_g) -&gt; LOSS
</code></pre>
<p>Module M needs to send back the gradients just w.r.t x, but z is the variable which I am using to calculate how bad x is. In particular I would need have d Loss / dz to have N channels instead of C.</p>

albanD: <p>Ok,</p>
<p>But from you schema, I would expect that the backward of the zero padding will be doing just that. And so by the time you get to M, your grads only have C channels again.</p>

A_A: <p>No, but I would need grad_z to have N channels coming into backward() for it to compute grad_x.</p>

A_A: <p>To clarify further, this feature request actually breaks what I want to do:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/17893" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/17893" target="_blank" rel="noopener nofollow ugc">Check that outputs and grad_outputs have same shape in torch.autograd.grad() </a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-03-11" data-time="22:28:58" data-timezone="UTC">10:28PM - 11 Mar 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-08-30" data-time="14:31:27" data-timezone="UTC">02:31PM - 30 Aug 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/zplizzi" target="_blank" rel="noopener nofollow ugc">
          <img alt="zplizzi" src="https://avatars.githubusercontent.com/u/5598968?v=4" class="onebox-avatar-inline" width="20" height="20">
          zplizzi
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
If I'm understanding things right, for every item in outputs, the corresponding item in grad_outputs should have the same shape...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: autograd</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: bootcamp</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: error checking</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Although, I believe it is good to have these size checks for normal situations. I would be glad if there is some ‘hack’ to by-pass the size consistency checks. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <aside class="quote no-group" data-username="A_A" data-post="6" data-topic="111506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7ea924/48.png" class="avatar"> A_A:</div>
<blockquote>
<p>To clarify further, this feature request actually breaks what I want to do:</p>
</blockquote>
</aside>
<p>Well all the formulas make that assumption so that would most likely break anyways <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>No, but I would need grad_z to have N channels coming into backward() for it to compute grad_x.</p>
</blockquote>
<p>Then why not fold the padding into your custom function M (making z_g (or just N) an input to the custom Function).<br>
That way the custom function will get the gradient of the right size and can handle all of that whichever wya it wants.</p>

A_A: <p>I see. Then I think I have following two options:</p>
<ol>
<li>
<p>As you suggested. Making the zero padding an autograd function, whose backward would actually implement the necessary computations for M.backward(). M.backward() would possibly not do much itself.</p>
</li>
<li>
<p>Return from M an output z padded with ‘dummy’ channels to make C &gt; N (assuming I know an upper bound on N). The zero padding module is then not required and I can just compute the loss on valid channels of z. That would hurt me in terms of memory, but perhaps I can store the dummy channels through some sparse/empty tensor hack <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> ?</p>
</li>
</ol>
<p>Thanks.</p>

albanD: <p>I think that merging the two together (M and padding) will be the best indeed. That makes sure you don’t need any hack in between the two</p>

A_A: <p>True, but the memory would be an issue. Is there anyway to concatenate ‘empty’ tensor to a valid tensor to inflate number of channels of valid tensor while keeping the memory similar? Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <blockquote>
<p>True, but the memory would be an issue.</p>
</blockquote>
<p>I’m not sure to see why.<br>
Currently, you already have a x → M → z → PADDING → z_p<br>
I think you want (x, z_g) → M_AND_PADDING → z_p</p>
<p>And in that new custom Function, you don’t need to do anything beyond what the padding is currently doing.</p>

A_A: <p>Yes, but making it generic would require me to know the upper bound on N. But, on hind-sight you are right, that upper-bound might still become tight for some samples in the batch anyway.</p>
<p>Thanks for advices <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I will give a try. Have a nice day!</p>