Different results caused by CatBackward

username: Einsturing
cooked: <p>I tried to convolve my input twice, then connect the input and the convolved vector(all 3 dimensional), send it to the model and pass an LSTM. But I found that when the order of my connection is different, as the training progresses and the loss decreases, my model finally achieves completely different results.<br>
For example</p>
<pre><code class="lang-auto">    for idx, (inputs, tlabels) in enumerate(train_loader):
        optimizer.zero_grad()

        inputs = inputs.float().permute(1, 0, 2, 3).to(device)
        hinputs = model.conv1(inputs)  # 2D-Conv, padding=same
        hinputs2 = model.conv2(hinputs)  # 2D-Conv, padding=same

        tlabels = tlabels.float().to(device)
        hlabels = model.conv1(tlabels)
        hlabels2 = model.conv2(hlabels)

        linputs = torch.cat((hinputs, hinputs2, inputs), 1)  # order firstConv, secondConv, raw
        llabels = torch.cat((hlabels, hlabels2, tlabels), 1) 

        # linputs = torch.cat((inputs, hinputs, hinputs2), 1)  # order raw, onceConv, secondConv
        # llabels = torch.cat((tlabels, hlabels, hlabels2), 1)

        lpreds = model.low_level(linputs)  # LSTM

        loss = criterion(lpreds, llabels.reshape(1, 3, -1).permute(1, 0, 2))

        loss.backward()
        optimizer.step()
</code></pre>
<p>In the code, I ‘cat’ the three vectors,and the other ‘cat’ method was commented. These two methods lead to completely different prediction results for the model. In general, it is closer to the result of using the first dimension to train alone.<br>
The only difference between them is the order of ‘next_functions’ in ‘grad_fn’.At first I thought that the calculation order in the calculation graph would cause the gradient of the leaf nodes to be different, but after testing with a few custom vectors, I found that it was not.<br>
How to explain this problem?I would be grateful if anyone could answer me.</p>

username: ptrblck
cooked: <p>Based on the code snippet it seems that the model architecture is equal, but the <code>torch.cat</code> call uses a different order of inputs. Could you explain how <code>dim1</code> is treated in <code>model.low_level</code>? Would it be the temporal dimension and thus the LSTM would see a different sequence?</p>

username: Einsturing
cooked: <p>Thanks for your reply.</p>
<pre><code class="lang-auto">class LSTMNet(nn.Module):
    def __init__(self):
        super(LSTMNet, self).__init__()
        self.lstm = nn.LSTM(n_steps, 64, 2)
        self.out = nn.Sequential(
            nn.Linear(64, 256),
            nn.Linear(256, 16),
            nn.Linear(16, 1)
        )

    def forward(self, x):
        x, _ = self.lstm(x)
        x = self.out(x).permute(0, 2, 1)
        return x
</code></pre>
<p>This is my LSTM architecture, and the <code>n_steps</code> is the temporal dimension.<br>
This is the function low_level</p>
<pre><code class="lang-auto">    def low_level(self, x):
        x = self.lstmnet(x.permute(1, 0, 2, 3).reshape(3, n_steps, -1).permute(0, 2, 1))  # C2D + LSTM
        return x
</code></pre>
<p>For example, the shape of my inputs is (1, T, H, W), and I permute it to (T, 1, H, W), which lets T be the batch and variate be the channel, to do the Conv2D(padding=same) twice. Then <code>torch.cat(dim=1)</code> them to get the shape (T, 3, H, W) and run the <code>model.low_level()</code>.Before LSTM, do some <code>permute</code> and <code>reshape</code>, (T, 3, H, W) → (3, T, H, W) → (3, T, H * W) → (3, H * W, T), and my class LSTM within FC makes the shape change to (3, 1(also represent T), H * W), which is my output.</p>

username: ptrblck
cooked: <p>In the default setup <code>nn.LSTM</code> expects an input in the shape <code>[L, N, H_in]</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html?highlight=lstm#torch.nn.LSTM">docs</a>.<br>
Based on your description you are feeding inputs as <code>[3, H*W, T]</code>. Would this correspond to <code>[N=batch_size, H_in, L]</code>? If so, you would be mixing the dimensions and might want to check the inputs again.</p>

username: Einsturing
cooked: <p>Thanks for your reply!<br>
Is the <code>L - sequence length</code> a concept in NLP? The task of my LSTM is the prediction for the time series. ‘3’ means the three Tensors <code>torch.cat()</code> by me. ‘H*W’ means the all points in my dataset. ‘T’ means the timesteps in the train window. I think <code>H*W</code> correspond to <code>N</code> because the prediction is for every single point. And ‘T’ correspond to ‘H_in’ because I want to map it to the hidden state. How should I understand the L? <code>3</code> is just the demension <code>torch.cat()</code> by three Tensors, it can be any number as I add some Conv2D(I also have plans to try this operation in the future).<br>
In fact, my code has achieved the best results I have ever gotten, and it has already met my expectations.</p>
<pre><code class="lang-auto">        linputs = torch.cat((hinputs, hinputs2, inputs), 1)
        llabels = torch.cat((hlabels, hlabels2, tlabels), 1)
</code></pre>
<p>is better than</p>
<pre><code class="lang-auto">        linputs = torch.cat((hinputs2, hinputs, inputs), 1)
        llabels = torch.cat((hlabels2, hlabels, tlabels), 1)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">        linputs = torch.cat((inputs, hinputs, hinputs2), 1)
        llabels = torch.cat((tlabels, hlabels, hlabels2), 1)
        #or this
        #linputs = torch.cat((inputs, hinputs2, hinputs), 1)
        #llabels = torch.cat((tlabels, hlabels2, hlabels), 1)
</code></pre>
<p>In the absence of any changes in other codes, poor results were obtained that were completely out of line with expectations.<br>
Therefore, the possible reason is that the dimension by <code>torch.cat()</code> is used as L, which causes these three Tensors to play different roles in the calculation of LSTM, which leads to the different predictions, right?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Einsturing" data-post="5" data-topic="139637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/einsturing/48/45019_2.png" class="avatar"> Einsturing:</div>
<blockquote>
<p>Is the <code>L - sequence length</code> a concept in NLP?</p>
</blockquote>
</aside>
<p>Yes, a sequence length is often used in NLP (e.g. you can see words as a sequence forming a sentence) but is not limited to NLP (e.g. EEG readings can be processed in a similar way).</p>
<aside class="quote no-group" data-username="Einsturing" data-post="5" data-topic="139637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/einsturing/48/45019_2.png" class="avatar"> Einsturing:</div>
<blockquote>
<p>Therefore, the possible reason is that the dimension by <code>torch.cat()</code> is used as L, which causes these three Tensors to play different roles in the calculation of LSTM, which leads to the different predictions, right?</p>
</blockquote>
</aside>
<p>Yes, that could be the case. <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">This post</a> explains it in more detail. While the processing would be different, I wouldn’t claim that only a specific sequence input would work, but you would need to run some experiments.</p>