How does the batch normalization work for sequence data?

username: pp18
cooked: <p>I have sequence data going in for RNN type architecture with batch first i.e. my input data to the model will be of dimension 64x256x16 (64 is the batch size, 256 is the sequence length and 16 features) and coming output is 64x256x1024 (again 64 is the batch size, 256 is the sequence length and 1024 features). Now, if I want to apply batch normalization should it not be on output features which is 1024 but the problem is pytorch does not allow this or I do not completely understand.  From this <a href="https://pytorch.org/docs/stable/nn.html#normalization-layers" rel="nofollow noopener">doc</a> it is clear from the following statement</p>
<pre><code class="lang-auto">`torch.nn.BatchNorm1d`: 
Parameters: num_features – C from an expected input of size (N,C,L) or
                           L from input of size (N,L)
</code></pre>
<p>it is clear for 2D data that batch-normalization is executed on L for input size(N, L) as N is incoming features to the layer and L is outgoing features but it is confusing for 3D data which I believe should also be L.</p>
<p>Please someone who has used batch-normalization for 3D data.</p>
<p>Any help is very much appreciated.</p>
<p>Thank you for all the help.</p>

username: ptrblck
cooked: <p>If you would like to use the feature dimension in batch norm, you could simply permute your input:</p>
<pre><code class="lang-python">bn = nn.BatchNorm1d(1024)
x = torch.randn(64, 256, 1024)
x = x.permute(0, 2, 1)
output = bn(x)
</code></pre>
<p>The <code>BatchNorm1d</code> layer will now have 1024 running estimates.</p>

username: pp18
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  thanks a lot. I feel so stupid for not thinking that way.</p>
<p>Since we can always permute the data to fit according to our need but still I would like to know what is the idea for this default setting in case od 3D data shape to normalize on the middle dimension.</p>

username: ptrblck
cooked: <p>I guess it’s just the choice to provide a similar API for 1D, 2D and 3D cases, which all use dim1.</p>

username: Yasser
cooked: <p>Hello, I just came across this topic because I’m actually trying to do batch normalization for multivariate time series data, and I did it on the features following exactly the same method you described :<br>
<code>bn = nn.BatchNorm1d(1024)</code><br>
<code>x = torch.randn(64, 256, 1024)</code><br>
<code>x = x.permute(0, 2, 1)</code><br>
<code>output = bn(x)</code></p>
<p>However I applied this in the forward method of the LSTM class I created, and therefore I give it as an input to the LSTM, the results are just perfect for me, but I still have one question :</p>
<ul>
<li>Since Batch normalization applies to each layer in the LSTM I have the feeling it is not the case following what I just did, because I just add a few line in the forward method of the LSTM, and I don’t know it really applies to each layer or just the input, because it looks like I only apply it to the input.<br>
Here’s the part I added in the forward method :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd79db99434b01d6d37712acb250b197c76fb9b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd79db99434b01d6d37712acb250b197c76fb9b7" title="Capture d’écran 2020-05-11 à 16.17.30"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd79db99434b01d6d37712acb250b197c76fb9b7.png" alt="Capture d’écran 2020-05-11 à 16.17.30" data-base62-sha1="AalSvxfHZnpYUT5gMttyGk3GT43" width="690" height="254" data-dominant-color="313233"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2020-05-11 à 16.17.30</span><span class="informations">950×350 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>

username: ptrblck
cooked: <p>In your code snippet the batchnorm layer will be applied to <code>features</code> only before they are passed to <code>self.lstm</code>.<br>
What is your use case? Would you like to apply batchnorm layers after each layer in a multi-layer LSTM?</p>

username: Yasser
cooked: <p>In my case I am working with multivariate time series data, and just like you said, I want to use batchnorm layers after each layer in the multi-layer LSTM.<br>
I’ve did some research to find how to do it, but it says it’s not possible with the normal LSTM.<br>
The only way to do it is to modify the LSTM so that a recurrent batchnorm (<a href="https://arxiv.org/pdf/1603.09025.pdf" rel="nofollow noopener">Article</a>) can be applied, the implementation of the modified LSTM has been given in this <a href="https://github.com/jihunchoi/recurrent-batch-normalization-pytorch/blob/master/bnlstm.py" rel="nofollow noopener">Github repo</a>.<br>
Do you confirm what I’ve just said or there’s a simpler way to do it ?<br>
Thank you ! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ptrblck
cooked: <p>The repository looks alright and I also wanted to suggest to use <code>LSTMCell</code> instead. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Yasser
cooked: <p>Ok, Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Maghoumi
cooked: <p>If the output of the LSTM is a <code>PackedSquence</code>, what’s the correct way of applying <code>BatchNorm1d</code>?</p>

username: ptrblck
cooked: <p>I think you would have to apply the batchnorm layer on each input separately or pad the input sequences to the same length. <code>PackedSquence</code> would contain inputs with different lengths, so the batchnorm layer wouldn’t be able to process this input together.</p>

username: Maghoumi
cooked: <blockquote>
<p>or pad the input sequences to the same length</p>
</blockquote>
<p>You mean use <code>pad_packed_sequence</code> on the <code>PackedSquence</code>, then apply the BatchNorm?<br>
Something like this?</p>
<pre><code class="lang-auto"># Assuming 'input' is a PackedSquence...
output, _ = lstm(input)
output, _ = pad_packed_sequence(output, batch_first=True)
output = bn(output.permute(0, 2, 1)).permute(0, 2, 1)
</code></pre>

username: Maghoumi
cooked: <p>Hmm… Following <a href="https://discuss.pytorch.org/t/how-to-use-pack-sequence-if-we-are-going-to-use-word-embedding-and-bilstm/28184/4">this discussion</a>, I think the correct way of applying BatchNorm is this (please correct me <a class="mention" href="/u/ptrblck">@ptrblck</a> if I’m wrong):</p>
<pre><code class="lang-auto">def simple_elementwise_apply(fn, packed_sequence):
    """applies a pointwise function fn to each element in packed_sequence"""
    return torch.nn.utils.rnn.PackedSequence(fn(packed_sequence.data), packed_sequence.batch_sizes)

# Assuming 'input' is a PackedSquence and bn = BatchNorm1d(....)
output, _ = lstm(input)
output = simple_elementwise_apply(bn, output)
</code></pre>

username: ptrblck
cooked: <p>That looks correct, yes. (I also always trust <a class="mention" href="/u/tom">@tom</a>’s solutions <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> )</p>