Dealing with 2D tensors of different shape in a batch size greater than 1 without using zero-filling or sampling

Mona_Jalal: <p>Apart from zero-filling and sampling, is there a less aggressive way to handle 2D tensors in which the first dimension is of variable size between 11 to 8000 and the second dimension is constantly 512 in a batch size greater than 1? Ideally, batch size of 64 in PyTorch?</p>
<p>For example, if the batch size is 4? I could have a list of such 2D tensors in a batch:</p>
<p>[200 * 512, 1000 * 512, 23 * 512, 7000 * 512]</p>

KFrank: <p>Hi Mona!</p>
<aside class="quote no-group quote-modified" data-username="Mona_Jalal" data-post="1" data-topic="152162" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>Apart from zero-filling and sampling, is there a less aggressive way to handle 2D tensors in which the first dimension is of variable size<br>
…<br>
Ideally, batch size of 64 in PyTorch?</p>
</blockquote>
</aside>
<p>I assume that your use case is to pass a batch of tensors of different sizes<br>
through your model, calculate a loss, backpropagate, and then optimize<br>
your model’s parameters.</p>
<p>You will not be able to pass your batch of tensors through your model as<br>
a single batch (unless you zero-fill or sample, etc.) because pytorch does<br>
not support such “ragged” tensors.</p>
<p>So you will have to pass your individual tensors through your model one by<br>
by one* (probably giving them a leading singleton batch dimension), at the<br>
possible detriment of not making fully efficient use of your gpu (and / or cpu)<br>
pipelines.</p>
<p>You could either loop over the individual samples in your batch, passing<br>
them through your model, and accumulate the per-sample losses<br>
together into a <code>batch_loss</code> and then backpropagate once by calling<br>
<code>batch_loss.backward()</code>, and then call <code>opt.step()</code>.</p>
<p>Or you could call <code>loss.backward()</code> for each sample separately (<code>.backward()</code><br>
accumulates the gradients until you call <code>.zero_grad()</code>) and then call<br>
<code>opt.step()</code>, which will act on the accumulated gradients.</p>
<p>Or you could backpropagate and call <code>opt.step()</code> for each individual sample<br>
in the “batch.”  (The extent to which batches of samples help the optimization<br>
process is a nuanced question, but batches do help keep your gpu pipelines<br>
full.)</p>
<p>In general, I would recommend the first approach (where you loop over<br>
the samples for the forward passes, but then backpropagate and optimize<br>
just once per batch).</p>
<p>*)  If some of the samples in your training set have the same size – let’s<br>
say that you can find four samples in your training set that all have shape<br>
<code>[200, 512]</code> – you could package like-sized samples together into non-ragged<br>
batch tensors – in this example case, a batch of shape <code>[4, 200, 512]</code> – and<br>
pass them as single tensors through your model.  You might not get the full<br>
benefit of having a batch size of 64, but you will still likely make better use<br>
of your gpu than if you had passed the four samples through your model<br>
separately.</p>
<p>Best.</p>
<p>K. Frank</p>

Mona_Jalal: <p>Hi Frank, thanks for your response, is there a code snippet that does this batch size of one and doing the opt step and accumulated gradient when tensors in a batch are not of the same size you could point me to ?</p>

KFrank: <p>Hi Mona!</p>
<aside class="quote no-group" data-username="Mona_Jalal" data-post="3" data-topic="152162" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>is there a code snippet that does this batch size of one and doing the opt step and accumulated gradient</p>
</blockquote>
</aside>
<p>I don’t know of an example off-hand, but it could be a simple as something<br>
like this:</p>
<pre data-code-wrap="python"><code class="lang-python">for  i in range (nBatches):   # for example, this could be the number of batches in an epoch
    input, target = get_next_batch_list():   # a batch is a list of single samples (and labels)
    opt.zero_grad()
    for  j in range (len (input)):   # loop over samples in the batch
        pred = model (input)
        loss = loss_fn (pred, target)
        loss.backward()   # accumulates gradient for this single sample
    opt.step()   # takes step using gradient accumulated over batch
</code></pre>
<p>Just to reiterate:</p>
<p><code>input</code> can’t be a single tensor that contains a batch of samples, because,<br>
according to the problem at hand, the samples have differing shapes (so<br>
such a batch tensor would be “ragged”).  Instead, <code>input</code> is a <code>list</code> of<br>
individual sample tensors that are passed through <code>model</code> one by one<br>
as we loop over the <code>input</code> <code>list</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Mona_Jalal: <p>Thanks a lot for your response.</p>
<p>I have two technical questions for you:</p>
<ol>
<li>Do you call this method “gradient accumulation”?</li>
<li>Are there built-in mechanism for “gradient accumulation” in PyTorch or PyTorch Beta?</li>
</ol>

KFrank: <p>Hi Mona!</p>
<aside class="quote no-group" data-username="Mona_Jalal" data-post="5" data-topic="152162" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<ol>
<li>Do you call this method “gradient accumulation”?</li>
</ol>
</blockquote>
</aside>
<p>I don’t know whether “gradient accumulation” is an officially-defined<br>
term.  What I’m referring to is the fact that <code>loss.backward()</code> adds<br>
the gradients it computed into the <code>.grad</code> properties of the various<br>
<code>Parameter</code>s.</p>
<p>(This is the built-in behavior of <code>.backward()</code>, which is why we normally<br>
call something like <code>opt.zero_grad()</code> before we call <code>loss.backward()</code>.<br>
Normally we don’t want to accumulate the newly-computed gradient into<br>
some previously-computed gradient that hadn’t been cleared out yet.)</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>Are there built-in mechanism for “gradient accumulation” in PyTorch or PyTorch Beta?</li>
</ol>
</blockquote>
</aside>
<p>Yes, in the sense that <code>.backward()</code> accumulates the gradients, as<br>
described above.</p>
<p>Best.</p>
<p>K. Frank</p>