LSTMs for real-time prediction and different batch_size in training

eagomez: <p>Hi all,</p>
<p>I came across this post and I’m facing a similar issue with PyTorch where I don’t know what would be the best approach:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://machinelearningmastery.com/wp-content/uploads/2019/09/icon-16x16.png" class="site-icon" width="16" height="16">
      <a href="https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/" target="_blank" rel="noopener nofollow ugc" title="07:00PM - 14 May 2017">Machine Learning Mastery – 14 May 17</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:640/423;"><img src="https://machinelearningmastery.com/wp-content/uploads/2017/05/How-to-use-Different-Batch-Sizes-for-Training-and-Predicting-in-Python-with-Keras.jpg" class="thumbnail" width="640" height="423"></div>

<h3><a href="https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/" target="_blank" rel="noopener nofollow ugc">How to use Different Batch Sizes when Training and Predicting with LSTMs</a></h3>

<p>Keras uses fast symbolic mathematical libraries as a backend, such as TensorFlow and Theano. A downside of using these libraries […]</p>

  <p><span class="label1">Est. reading time: 13 minutes</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I am working on a network that is supposed to run on real-time in CPU during inference, taking a noisy audio chunk of 512 samples as input to produce a clean version of the same audio chunk of 512 samples as output. The network I am using involves LSTM layers that according to the documentation require a known batch size during training of dimensions <code>(seq_len, batch_size, input_size)</code> which in my case would be <code>(1, 1, 512)</code>:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html</a></p>
<p>I would ideally like to train the network on batches bigger than 1 (e.g. <code>batch_size=32</code>) but use the model during inference with <code>batch_size=1</code> since it will be running on real time taking single input frames of 512 samples. Also, the model has to be stateful and as far as I understand, this can be implemented by detaching the output hidden and cell states in order to feed them back in as the inicial hidden and cell states of the next batch, is that correct? Along with this one, my other questions would be:</p>
<p><strong>1.</strong> Is it possible in PyTorch to train LSTM on <code>batch_size=32</code> and then use the same saved model to do inference in single steps of <code>batch_size=1</code>? (i.e.  train on <code>(1, 32, 512)</code> and inference in <code>(1, 1, 512)</code> assuming <code>batch_first=False</code>) if so, how? and, may the predictions be affected by choosing a different <code>batch_size</code> during inference?</p>
<p><strong>2.</strong> Would it be okay to just do train and inference on <code>batch_size=1</code> and use the model like that or would it be expected to be less time efficient than <strong>(1)</strong> during training? Overall, my main concert is to obtain the least CPU intensive model during inference.</p>
<p>Sorry if some questions may have an easy answer, but I couldn’t find much information about similar scenarios. Any guidelines are highly appreciated.<br>
Esteban</p>

vdw: <p>You can train your model with a batch size of, say, 32 and make individual inferences (i.e., batch size of 1). Why should this cause any problem?</p>
<p>The only thing you have to consider is that you have to initialize the initial hidden state with the correct batch size: 32 for training, and 1 for inferencing.</p>

eagomez: <p>Hi <a class="mention" href="/u/vdw">@vdw</a> and thanks for your help!</p>
<p>What would happen in such case if I were to do my initial hidden states learnable as suggested <a href="https://r2rt.com/non-zero-initial-states-for-recurrent-neural-networks.html" rel="noopener nofollow ugc">here</a>? I am confused in the sense that now a whole bunch of time steps during training are on the same batch. I assume I should just copy the hidden states values to the new instance, but I am not sure if the ones to be copied in such case are the <em>topmost</em> (i.e. <code>[:, 0, :]</code>) <em>bottommost</em> or (i.e. <code>[:, batch_size - 1 :]</code>).</p>

vdw: <p>If the sequences within and between batches are independent, then it shouldn’t really matter which  one of the 32 hidden states you should pick, being the topmost, bottommost, or any other one. Arguably, if you train a large amount of epoch with enough shuffling, the 32 hidden states should be kind of similar.</p>
<p>Or you simple make 32 inferences with for the same input sequence but with all 32 available hidden states and see if it effects the result in any way :).</p>

eagomez: <p>Thanks again <a class="mention" href="/u/vdw">@vdw</a> ! what could be a sound approach in the case of dependent sequences within a batch but not between batches? (basically a single audio file chopped in smaller sequential frames per batch)</p>

aaaaab: <p>Hi <a class="mention" href="/u/eagomez">@eagomez</a><br>
Did you figure out how to solve it? I have the same problem.<br>
I have a solution, but not sure whether it is true or not. What about copying the input 32 times, making an input with 32 batches that all of them are the same, and finally, taking an average of the output of each batch after feeding to the model? Do you think it works?</p>

stellarpower: <p><a class="mention" href="/u/eagomez">@eagomez</a> Just wondering, was the model you created ever uploaded at all? I am performing a similar process of realtime filtering a noisy signal to get a clean one. I have found many of the examples online don’t really apply so well to this paradigm. Would be interesting to see what you did, if the code is available at all.</p>
<p>Thanks</p>