Conv2d for time series transformation

Kimonili: <p>Hi all,</p>
<p>I am working on a time series dataset where 30 days of data are given to the model during training for each batch. The shape of the tensor is  [5,<strong>9</strong>,30,10] (batch_size, <strong>time_steps</strong>, <span class="hashtag">#days</span>, features).</p>
<p>I want to pass this tensor to a conv2d layer to transform it to a shape [5,<strong>1</strong>,30,10] and do squeeze(1) to finally get a shape of [5,30,10]. What I want to do is to summarise the time steps for each day.</p>
<p>I am having a hard time understanding if the order of the tensor’s dimensions is correct, and how they should be passed to the conv2d function.</p>
<p>The RGB channels equivalent for my case is the time_steps or the <span class="hashtag">#days</span> dimension?</p>

ptrblck: <p>The “RGB channels” would be located in <code>dim1</code> so would correspond to <code>time_steps</code>.<br>
This would mean that each conv kernel would use <em>all</em> <code>time_steps</code> and would then convolve over the <code>days</code> and <code>features</code> dimensions using its spatial size (in the default setup).<br>
I.e. since you would like to get a single output channel, you could use e.g.:</p>
<pre><code class="lang-python">x = torch.randn(5, 9, 30, 10)
conv = nn.Conv2d(9, 1, 3, 1, 1)
out = conv(x)
print(out.shape)
&gt; torch.Size([5, 1, 30, 10])
</code></pre>
<p>which would use a single kernel with <code>in_channel=9</code>, a spatial kernel size of <code>3x3</code>, a <code>stride</code> of 1, and <code>padding</code> of 1 to keep the “spatial size” of the output equal.</p>

Kimonili: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for the quick response. At first I planned to do it this way, but a part of me said that it was not the correct way. I basically wanted to summarise the <code>time_series</code> that belong to each one of the 30 <code>days</code> for each <code>feature</code>.</p>
<p>What I ended up doing is this:</p>
<pre><code class="lang-auto">x = torch.randn(5, 30, 9, 10) #[batch, #days, time_steps, features]

from torch import nn

class ConvNet(nn.Module):

    def __init__(self, channels, features):
        super().__init__()
        
        self.layer1 = nn.Sequential(
            nn.Conv1d(in_channels=channels,
                      out_channels=channels,
                      kernel_size=(5,1)),
            nn.BatchNorm2d(num_features=channels),
            nn.ReLU(),
            nn.AdaptiveMaxPool2d(output_size=(7, features)))

        self.layer2 = nn.Sequential(
           nn.Conv1d(in_channels=channels,
                     out_channels=channels,
                     kernel_size=(3,1)),
            nn.BatchNorm2d(num_features=channels),
            nn.ReLU(),
            nn.AdaptiveMaxPool2d(output_size=(1, features)))

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)

        return out
</code></pre>
<ol>
<li>
<code>dim1</code> (<span class="hashtag">#days</span>): maintained the same size across the network</li>
<li>
<code>dim2</code> (time_steps): decreased its size in the first and second convolution operation, and enforced an ouput of 1 in the last <code>AdaptiveMaxPool2d</code>.</li>
<li>
<code>dim3</code> (features): maintained the same across the network</li>
</ol>
<p>My tensor ended up being of shape <code>[5, 30, 10]</code> #[batch_size, <span class="hashtag">#days</span>, features]</p>
<p>I used the 1D convolution operation because I found out that it is the most common to use when you have time series data (although I have 4 dimensions in total) because the filter only moves to one direction.</p>
<p>I would really appreciate your feedback on my approach!</p>

ptrblck: <p>Your approach sounds also valid. It would be interesting to hear from other experiments in case you are changing the architecture in the future and see which approach worked best. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kimonili: <p>Great, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! Will give you a heads-up <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=10" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>