Efficient Time Distributed Dense

negreanu: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/234d9ae11578fee538d091d78187444d05461aaa.png" data-download-href="https://discuss.pytorch.org/uploads/default/234d9ae11578fee538d091d78187444d05461aaa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234d9ae11578fee538d091d78187444d05461aaa_2_690x258.png" alt="image" data-base62-sha1="52iYIPD9eMh6qfmogKAKODsaiym" width="690" height="258" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234d9ae11578fee538d091d78187444d05461aaa_2_690x258.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234d9ae11578fee538d091d78187444d05461aaa_2_1035x387.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234d9ae11578fee538d091d78187444d05461aaa_2_1380x516.png 2x" data-dominant-color="9D9D9D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2792×1048 96.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Let’s see if anyone can help me with this particular case. I have an input of dimension 1600x240 (1600 time steps and 240 features for each time step) and I want to apply a linear layer independently for each time step. I know that the pytorch nn.Linear module works with 2 dimensional inputs, but it doesn’t do exactly what I want. If I apply nn.Linear(240,100) on the input, we are only considering a 240x100 weight matrix. I have tried these four alternatives:</p>
<pre><code class="lang-auto">import torch.nn as nn

class Linear_only(nn.Module):
    def __init__(self,):
        super(Linear_only, self).__init__()
        self.linear = nn.Linear(240,100)
    def forward(self,x):
        return self.linear(x)

class TimeDistributedLinear(nn.Module):
    def __init__(self,):
        super(TimeDistributedLinear, self).__init__()
        self.linear = nn.Linear(240,100)
    def forward(self,x):
        x = x.view(-1,240)
        return self.linear(x)

class UsingModulelist(nn.Module):
    def __init__(self,):
        super(UsingModulelist, self).__init__()
        self.modulelistLinear = nn.ModuleList([nn.Linear(240,100) for i in range(1600)])
    def forward(self,x):
        output_list = []
        for i in range(1600):
            linear = self.modulelistLinear[i]
            x_i = linear(x[:,i,:].view(16,240))
            output_list.append(x_i)
        x = torch.stack(output_list)
        x = x.permute(1,0,2)
        return x    

class UsingParameter(nn.Module):
    def __init__(self,):
        super(UsingParameter, self).__init__()
        self.weight_last_linear = nn.Parameter(torch.empty((1600, 240,100), requires_grad = True))
        self.bias_last_linear = nn.Parameter(torch.empty(1600,100),requires_grad = True)

        stdv = 1. / math.sqrt(self.weight_last_linear.size(1))
        self.weight_last_linear.data.uniform_(-stdv, stdv)
        if self.bias_last_linear is not None:
            self.bias_last_linear.data.uniform_(-stdv, stdv)
            
    def forward(self,x):
        x = torch.einsum('ijl, jlm -&gt; ijm',x,self.weight_last_linear)+self.bias_last_linear
        x = x.view(16,1600,100)
        return x


batch_size = 16
x = torch.rand(batch_size,1600, 240)
net_linear = Linear_only()
print(net_linear(x).size()) #torch.Size([16, 1600, 100]) Total Trainable Params: 24100
net_td_linear = TimeDistributedLinear()
print(net_td_linear(x).size()) #torch.Size([16, 1600, 100]) Total Trainable Params: 24100
net_modulelist = UsingModulelist()
print(net_modulelist(x).size()) #torch.Size([16, 1600, 100]) Total Trainable Params: 38560000
net_parameter = UsingParameter()
print(net_parameter(x).size()) #torch.Size([16, 1600, 100]) Total Trainable Params: 38560000
</code></pre>
<p>Of these alternatives, the first two do not do what I want. The third one does exactly what I want, but the 1600 loop makes it take too long, so I’m not interested. And the last one, I think it should do what I really want but it doesn’t work properly, so I prefer not to use nn.Parameter(). Can you think of any other way?</p>

ID56: <aside class="quote no-group" data-username="negreanu" data-post="1" data-topic="153741">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e47c2d/48.png" class="avatar"> negreanu:</div>
<blockquote>
<p>And the last one, I think it should do what I really want but it doesn’t work properly, so I prefer not to use nn.Parameter()</p>
</blockquote>
</aside>
<p>Can you elaborate more on why doesn’t it work properly?</p>

negreanu: <p>Hi <a class="mention" href="/u/id56">@ID56</a>,<br>
I can’t give much information about the use case I am working on but it is a time-series anomaly detection case. The code I have attached is just a small part of a model that returns the reconstructions of the input time-series. When I use nn.Parameter(), I get strange spikes in the reconstructions, like these (in the lilac-colored series):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c976a014a74f463ae4fc5d2ce7de42ef25f2541.png" alt="image" data-base62-sha1="6mtpxfhs29N7GuejaPkhEmNj585" width="309" height="112"><br>
Something that with nn.Linear() does not happen…</p>