Shuffle a tensor a long a certain dimension

username: Atila1
cooked: <p>Dear all,</p>
<p>I have a 4D tensor [batch_size, temporal_dimension, data[0], data[1]], the 3d tensor of [temporal_dimension, data[0], data[1]] is actually my input data to the network. I would shuffle the tensor along the second dimension, which is my temporal dimension to check if the network is learning something from the temporal dimension or not. Will be glad if this shuffling is kind of reproducible .</p>
<p>Is there a certain way to do that?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>If I understand your use case correctly, you would like to be able to revert the shuffling?<br>
If so, this should work:</p>
<pre><code class="lang-python"># setup
N, M, K = 2, 5, 2
x = torch.arange(N*M*K).view(N, M, K)
print(x)
&gt; tensor([[[ 0,  1],
           [ 2,  3],
           [ 4,  5],
           [ 6,  7],
           [ 8,  9]],

          [[10, 11],
           [12, 13],
           [14, 15],
           [16, 17],
           [18, 19]]])

# shuffle
idx = torch.randperm(x.size(1))
y = x[:, idx]
print(y)
&gt; tensor([[[ 8,  9],
           [ 2,  3],
           [ 0,  1],
           [ 4,  5],
           [ 6,  7]],

          [[18, 19],
           [12, 13],
           [10, 11],
           [14, 15],
           [16, 17]]])

# reverse the shuffling
idx_inv = torch.sort(idx).indices
print(y[:, idx_inv])
&gt; tensor([[[ 0,  1],
           [ 2,  3],
           [ 4,  5],
           [ 6,  7],
           [ 8,  9]],

          [[10, 11],
           [12, 13],
           [14, 15],
           [16, 17],
           [18, 19]]])
</code></pre>

username: Atila1
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thanks a lot for your response. I am not really willing to revert the shuffling.</p>
<p>I have a tensor coming out of my training_loader. It is of the size of 4D <code>[batch_size, num_steps, data_0, data_1]</code>. What I want to do before feeding the data to the model is to shuffle the data along my temporal dimension which is <code>num_steps</code>. So I willing to shuffle this 4D tensor along the 2nd dimension <code>num_steps</code> and afterward forwarding it to the model, just to check if my model learning something from the temporal dimension or not.</p>

username: ptrblck
cooked: <p>In that case the indexing with <code>idx</code> created by <code>randperm</code> should work and you could skip the last part. This would shuffle the <code>x</code> tensor in <code>dim1</code>.</p>

username: Atila1
cooked: <p>Thanks a lot, <a class="mention" href="/u/ptrblck">@ptrblck</a> .</p>
<p>Well, I think what you are doing should be exactly the same as:</p>
<pre><code class="lang-auto">tensor = torch.arange(N*M*K).view(N, M, K)
dim = 1
idx = torch.randperm(tensor.shape[dim])
t_shuffled = tensor[:,idx]
</code></pre>
<p>Am I right? I tried your approach and this, however, the output is not the same. <code>t_shuffle</code> is not as <code>y</code></p>
<p>I just need to be sure that when the data is shuffled along the <code>num_steps</code> dimension the <code>data_0</code> and <code> data_1</code> corresponding to the <code>num_step</code> being shuffled as well. Such that the 3d block of the data keeps being logical.</p>

username: ptrblck
cooked: <p>Yes, the codes should be equal as your code just replaces some variable names.<br>
Could you post an example of the input data and the desired output, please?</p>

username: Atila1
cooked: <p>Now, I have another general question. I am setting <code>shuffle=True</code> for my training dataset loader. But anyhow what comes out of the training loader will be of the 4D size <code>[batch_size, num_steps, data_0, data_1]</code>. I need to shuffle this tensor along the 2nd dimension as mentioned before. Then I will unsqueeze and add extra dimension so the tensor will be <code>[batch_size, num_steps, 1, data_0, data_1]</code>. Now I will send this tensor to my model which is a CNN and I will iterate over the <code>num_steps</code> step by step in the forward path such that the input to the CNN is just 1 channel. I stack the output for each step in a way that the output of the forward path will be ``[num_steps, batch_size, number_of_classes]. For the loss calculation I will then do as well loss for each num_step some up thier losses then do backpropagation.  So. from my point of view as I am calculating the loss for each step the final loss should also be the same whether I shuffled or not, and the parameters should be updated in the same way.</p>
<p>My aim is then to find that the CNN without shuffeling and with shuffeling should give me the same final accuracies as it should not be remebering any kind of memory. That is why I need to shuffle and check. But I made one approach now and I found that when I shuffled a long the <code>num_steps</code> the final accuracy changed as well? does that make any sense to you?</p>

username: ptrblck
cooked: <p>I’m not sure I understand the concern. Since you are creating the <code>idx</code> in both cases randomly, a different result would be expected.<br>
If you rerun the code, you should see different results unless you seed the code.</p>
<aside class="quote no-group" data-username="Atila1" data-post="8" data-topic="129798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/41988e/48.png" class="avatar"> Atila1:</div>
<blockquote>
<p>So. from my point of view as I am calculating the loss for each step the final loss should also be the same whether I shuffled or not, and the parameters should be updated in the same way.</p>
</blockquote>
</aside>
<p>Assuming you don’t have any randomness in the model (e.g. dropout) or any layers, which are using the shuffled dimension in a sequential manner, then your assumption might be correct. To verify it you could create a single ordered and shuffled batch, calculate the loss as well as the gradients, and compare both approaches.</p>

username: Atila1
cooked: <p>I have to admit it will not be easy to figure out what I am trying to say <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:">  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>
<p>Anyhow, if you can just confirm for me that they way I am doing the shuffeling above, when I just added the batch size to make the tensor 4D as it is in my case. is correct, I will be glad to try it out and come back if I have further questions <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Atila1
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I tried now to shuffle the data but I figured out why the final loss coming out of the CNN is different when I apply the shuffling to the data as I showed above. I find that even the very first few samples in my batch are not shuffled, however, the floats are not exact in the loss and that is why the final loss and the weights will be updated differently.</p>
<p>So here is the loss when I have no shuffle applied</p>
<pre><code class="lang-auto">[214.9400634765625, 205.25238037109375, 204.9016571044922, 207.81027221679688, 204.00399780273438, 203.37063598632812]
</code></pre>
<p>And here is how it looks like when I apply shuffle</p>
<pre><code class="lang-auto">[214.94007873535156, 205.2523956298828, 204.90162658691406, 207.81028747558594, 204.0040740966797, 203.37075805664062]
</code></pre>
<p>You can notice that the decimal points are different which will result in a smooth increase in the final loss but epoch after epoch they will accumulate and accordingly the final accuracy will differ.</p>
<p>Do you have any clue why this occurs?</p>
<p>Also, a weird behavior when I use torch.round on the losses. So the losses in the case where I apply no shuffle should be:</p>
<pre><code class="lang-auto">[214.9400634765625, 205.25238037109375, 204.9016571044922, 207.81027221679688, 204.00399780273438, 203.37063598632812]
</code></pre>
<p>When I apply rounding the output come out to be:</p>
<pre><code class="lang-auto">[215, 208, 211, 216, 212, 216, 209, 225, 207, 212, 220]
</code></pre>
<p>You can see they are completely different from their float representation. I do not know what is going on <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:">  However, in this case shuffeling and not shuffeling will be the same <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Really thanks and looking forward to your response.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Atila1" data-post="11" data-topic="129798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/41988e/48.png" class="avatar"> Atila1:</div>
<blockquote>
<p>Do you have any clue why this occurs?</p>
</blockquote>
</aside>
<p>These small errors are most likely caused by the limited floating point precision and a different order of operation as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10, 10, 10)
sum1 = x.sum()
sum2 = x.sum(0).sum(0).sum(0)
print((sum1 - sum2).abs().max())
&gt; tensor(4.7684e-07)
</code></pre>
<aside class="quote no-group" data-username="Atila1" data-post="11" data-topic="129798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/41988e/48.png" class="avatar"> Atila1:</div>
<blockquote>
<p>You can see they are completely different from their float representation.</p>
</blockquote>
</aside>
<p>I cannot reproduce this issue:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([214.9400634765625, 205.25238037109375, 204.9016571044922, 207.81027221679688, 204.00399780273438, 203.37063598632812])
y = torch.round(x)
print(y)
&gt; tensor([215., 205., 205., 208., 204., 203.])
</code></pre>
<p>and get the expected results. Could you check, if my code snippet reproduces the issue in your setup?</p>

username: Atila1
cooked: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<blockquote>
<p>These small errors are most likely caused by the limited floating-point precision and a different order of operation</p>
</blockquote>
<p>So as far as I can understand that this is normal and we do not have anything to do with it.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Atila1" data-post="13" data-topic="129798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/41988e/48.png" class="avatar"> Atila1:</div>
<blockquote>
<p>So as far as I can understand that this is normal and we do not have anything to do with it.</p>
</blockquote>
</aside>
<p>If you need more numerical precision, you could use <code>float64</code>, but note that the performance would be worse.</p>
<p>Based on the new code snippet you are accumulating the loss so I’m not sure which loss value you are printing.<br>
In any case, in case the rounding issue is reproducible for you, please post an executable code snippet, so that we could debug it.</p>

username: mathDR
cooked: <p>I understand that <a class="mention" href="/u/ptrblck">@ptrblck</a> has solutions to your technical problem in this thread.  I wanted to ask you though:  why do you think shuffling the temporal dimension is a viable thing to do?  Do you assume that the instances are independent along the time dimension?  If not, then shuffling is absolutely <em>not</em> the thing to do!</p>
<p>If your model has time dependence, then you do not want your model learning the past from the future.  You need to take care in only training the model on timesteps that occur <em>before</em> the test data (in time).  You could think of this similarly to text encoding, wherein you mask later words in a sentence because you don’t want later words to affect which words to predict next.</p>
<p>Does this make sense?  (although if your data/model are time independent, disregard this entire post)</p>

username: Atila1
cooked: <p><a class="mention" href="/u/mathdr">@mathDR</a>  Thanks for your reply.</p>
<p>I totally agree with all of what you have said.</p>
<p>I only used it as a test to check that the model which does not have a temporal dimension will not be affected with shuffling compared to the model with a temporal dimension. Nothing more <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: martimatics5
cooked: <p>Please I have a question?</p>
<p>I have a tensor of 2D 4x4 and I wish to create an output in a zigzag order using PyTorch.</p>
<p>Any help…That will be highly recommendable.</p>