Batch processing in Linear layers

superjunnis: <p>Hi,</p>
<p>I am trying to understand how to process batches in an nn.Linear layer. Since the nn.Linear function is defined using (in_features, out_features) I am not sure how I should handle them when I have batches of data. I am currently processing all batches at once in the forward pass, using</p>
<pre><code class="lang-auto"># input_for_linear has the shape [nr_of_observations, batch_size, in_features]
input_for_linear.view(-1, batch_size * in_features)                     
</code></pre>
<p>as my input - i.e. flattening all the batches out. My linear layer is defined as:</p>
<pre><code class="lang-auto">linear = nn.Linear(batch_size * in_features, out_features)
</code></pre>
<p>This process however saves an unnecessary amount of parameters in the linear layer as it differentiates between observations in each batch. With lots of data and small batch sizes it averages out over many epochs so it is maybe not so crucial to change? (right?)</p>
<p>Should I instead do a for-loop over my batches in the input_for_linear and use these in a layer defined with:</p>
<pre><code class="lang-auto">linear = nn.Linear(in_features, out_features)
</code></pre>
<p>And then lastly combine each of the outputs back to a batch? The results from the linear layers are used in a Variational Auto Encoder with LSTM’s which are capable of handling batched data.</p>
<p>Perhaps there is a smarter solution?</p>
<p>Kind Regards,<br>
Jonas</p>

ptrblck: <p>You don’t need to change anything, as <code>nn.Linear</code> (and all other layers) are accepting batched data.<br>
Just pass your input as <code>[batch_size, nb_features]</code> to the module and the output will be <code>[batch_size, out_features]</code>.</p>

superjunnis: <p>I am not quite sure how to pass my input as <code>[batch_size, nb_features]</code> as I have a time-series of 500 observations, with 4 variables - with 4 of these series in a batch. (These parameters can change as my project progresses.)</p>
<p>How would i pass <code>[batch_size, nb_features]</code> to the linear input layer and how should the linear layer be defined? Also, what is <code>nb_features</code> in this case?</p>
<p>I am currently passing this:</p>
<pre><code class="lang-auto"># input_for_linear has the shape [nr_of_observations, batch_size, in_features]
input_for_linear.view(-1, batch_size * in_features)       
</code></pre>
<p>Thank you for your help.</p>

ptrblck: <p>It depends how you would like to process this input.<br>
<code>nn.Linear</code> uses a fully-connected weight matrix, such that each input feature will be used to create an output value.<br>
Since you are using a temporal signal, you could create a tensor in the shape <code>[batch_size, sequence_length * features]</code> or alternatively you could also let the linear layer process each “time stamp” separately by passing an input of <code>[batch_size, sequence_length, features]</code>.</p>
<p>For temporal singals you should also have a look at e.g. <code>nn.Conv1d</code> etc. as they might be beneficial for your use case.</p>
<aside class="quote no-group" data-username="superjunnis" data-post="3" data-topic="77527">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8dc957/48.png" class="avatar"> superjunnis:</div>
<blockquote>
<p>Also, what is <code>nb_features</code> in this case?</p>
</blockquote>
</aside>
<p><code>in_features</code> defined the number of input features for the linear layer.</p>

GabbyChan: <p>Hi ! I am a little confused with this part too…<br>
Let’s say if I used the input of Linear Layer as <code>input = [batch_size, sequence_length, feature_dim]</code><br>
So, does this mean that the size of the last dimension  (in this case it is <code>feature_dim</code>) has to match the input dimension of the Linear Layer is that right?</p>

ptrblck: <p>Yes, that’s correct as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 4)
lin = nn.Linear(4, 10)
out = lin(x)
print(out.shape)
&gt; torch.Size([2, 3, 10])
</code></pre>