Drop row from tensor in cuda

Augusto_Maillo: <p>I’m using Whisper by transformers library to transcribe some audios. In my application I need to transcribe near real time, so I use batching to speed up. However whisper generation is sequential, and some samples in batch may (and probably will) finished later. I don’t want to hold the whole batch while a lazy sample is not finished. I’m using a <a href="https://huggingface.co/docs/transformers/main/internal/generation_utils#transformers.TextStreamer" rel="noopener nofollow ugc">Streamer</a> to early return completed predictions, but I’m wondering if drop them from the batch for the next predictions should speed my inference.</p>
<p>For example, lets say a N size batch. If I identify that sample <em>i</em> finished (by end of string token), I will return the transcription for the requester, but for the next whisper model predictions I want a n inference over a N-1 sized batch.</p>
<p>Is there a faster way that I can do this with a cuda Tensor? Do you think it’s worthy?</p>

ptrblck: <aside class="quote no-group" data-username="Augusto_Maillo" data-post="1" data-topic="209489">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/augusto_maillo/48/72708_2.png" class="avatar"> Augusto_Maillo:</div>
<blockquote>
<p>For example, lets say a N size batch. If I identify that sample <em>i</em> finished (by end of string token), I will return the transcription for the requester, but for the next whisper model predictions I want a n inference over a N-1 sized batch.</p>
</blockquote>
</aside>
<p>I don’t fully understand this concept, so could you explain how a single sample can “finish” before the entire batch?</p>
<p>In standard PyTorch layers, the entire batch is processed at once and thus the returned output is also containing the result for all samples.<br>
However, it seems your use case processes the samples inside a batch independently?</p>

Augusto_Maillo: <p>In Whisper generation there are multiple inference steps in order to predict the next sentence token.</p>
<p>For a batch with 3 samples, it will predict at each stop a new token for the sentence. However, one sample in the batch may finish at step <em>i</em>, where the predicted token for its is <code>end_of_string_token</code>. Whisper will keep this sample in the batch for the next iterations, even if the prediction for it is over.</p>
<p>I’m wondering if I can drop this sample from the batch, so the next inference step will be compute with N-1 samples. It may seem worthless since remove a single sample should not improve speed a lot, but if I drop continously, in a <code>batch_size=64</code> sample where half of them finished before the others, the last inferences will be computed over 32 samples, and that will make a lot of difference.</p>
<p>That said, in order to do this, I need to modify a torch tensor in GPU very quickly.</p>
<pre><code class="lang-auto">input_tensor = torch.rand((64, 1024)).to('cuda')
inference_finished:
while not inference_finished:
  result = predict(input_tensor)
  input_tensor = drop_concluded_predictions(input_tensor) 
  inference_finished = check_predictions(result)
</code></pre>
<p>In the <code>ith</code> iteration, input_tensor.shape[0] will be lower then 64.</p>
<p>There is a way to do this very quickly?</p>

ptrblck: <p>You could profile different approaches of slicing and creating a new tensor using the remaining inputs via <code>torch.cat</code> or by moving the remaining inputs. Both approaches would trigger a copy so you could then profile how much performance you are gaining by reducing the workload vs. triggering the copies.</p>