DataLoader - Problems

username: chaslie
cooked: <p>Afternoon,</p>
<p>I’m hoping someone can help me.</p>
<p>I am using Dataloader to load 16 1x2048 vectors into pytorch. The problem is that when i view what DataLoader is loading I find that the values have all been shuffled up. I have a tensor which is 16x1x2048 and all the values are present, just not in the correct order (to quote the late, great Eric Morcombe :-)…)</p>
<p>Does anybody know how I can extract 16 1x2048 vectors which maintain in the orginal file please?</p>
<p>Thanks for your help</p>
<p>Chaslie</p>

username: Prerna_Dhareshwar
cooked: <p>Theres an argument to <code>Dataloader()</code> called <code>shuffle</code>, just set <code>shuffle = False</code>. Hope this helps.</p>

username: chaslie
cooked: <p>Hi Prerna,</p>
<p>I Tried this and this doesn’t affect the make up of the vector. I would the order of the vector to be maintained ie:</p>
<p>(1 2 3 4 5 6 7 8)</p>
<p>but pytorch dataloader is loading the vector as:</p>
<p>(1 4 2 6 7 3 8 5)</p>

username: Prerna_Dhareshwar
cooked: <p>Thats strange, it shouldn’t shuffle along the feature dimension like that. Can you post some code?<br>
Edit : Follow up question, along what dimension is the dataloader shuffling the values? Batch or feature?</p>

username: chaslie
cooked: <p>Hi Prerna,</p>
<p>I am loading the data from a 42000x2048 array, and then creating a 16x1x2048 tensor called data, when i view this it has lost the order from the 42000x2048 array</p>
<pre><code class="lang-auto">train_dataset = TensorDataset(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=BATCHSIZE, shuffle=False)
</code></pre>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for data, target in train_loader:
#        print("data=",data.shape)
        np.savetxt(f,data.numpy())
        np.savetxt(f2,target.numpy())
        data=data.unsqueeze(1)
        print("data_sq=",data.shape)
        data = data.cuda()
</code></pre>
<p>I think its shuffling along the feature, though looking at the data i would guess that its generating a 16x1x2048 tensor of numbers randomly taken from x_train array</p>

username: Prerna_Dhareshwar
cooked: <p>I think the issue might be when you are reshaping it that it is shuffling, because the data loader itself never shuffles along the feature dimension.</p>

username: chaslie
cooked: <p>is there a way I can view the contents of train_loader and train_dataset?</p>

username: Prerna_Dhareshwar
cooked: <p>So was TensorDataset()  a custom dataset that you defined?</p>
<p>Edit: Looks like its a predefined instance of the Dataset() class, I think you can find the source code online for it.</p>

username: chaslie
cooked: <p>yes, X_Train is the 42000x2048 input data and y_train is the labels 1x42000 array numbering from 1 t0 42000</p>

username: Prerna_Dhareshwar
cooked: <p>Here - <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#TensorDataset" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#TensorDataset</a></p>

username: chaslie
cooked: <p>ok so being a bit slow (Friday night over here!) is the problem with the y_train array calling numbers 1 to 42000, and if so whats the best way to solve it? should I leave y_train empty?</p>

username: Prerna_Dhareshwar
cooked: <p>Okay can you try this in the for loop-</p>
<pre><code class="lang-auto">ctr = 0
for idx, (batch, target) in enumerate(train_loader):
    if ctr ==0:
        print(batch)
    batch += 1
</code></pre>
<p>Now check if it shuffles it?<br>
I did this and it didn’t shuffle along feature dimension for me</p>

username: Prerna_Dhareshwar
cooked: <p>Actually even when I do</p>
<pre><code class="lang-auto">for (batch, target) in train_loader:
</code></pre>
<p>It doesn’t shuffle it.</p>
<p>I’m not sure what the issue might be in your case.</p>

username: chaslie
cooked: <p>its still shuffling, i wish i caould view the contents of:</p>
<p>train_dataset = TensorDataset(X_train, y_train)</p>
<p>and</p>
<p>train_loader = DataLoader(train_dataset, batch_size=BATCHSIZE, shuffle=True)</p>

username: ptrblck
cooked: <p>You can index the <code>Dataset</code> directly and check for equal values:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(42, 2048)
y = torch.randn(42, 1)

dataset = TensorDataset(x, y)

for idx in range(x.size(0)):
    data, target = dataset[idx]
    assert (data==x[idx]).all(), "data shuffled"
    assert (target==y[idx]).all(), "target shuffled"
</code></pre>
<aside class="quote no-group" data-username="chaslie" data-post="9" data-topic="50451">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>X_Train is the 42000x2048 input data and y_train is the labels 1x42000 array numbering from 1 t0 42000</p>
</blockquote>
</aside>
<p>That would raise an exception, since the number of samples is different for <code>X_Train</code> and <code>y_train</code> (42000 vs. 1). Could you check it again and post the code how you are creating the <code>[16, 1, 2048]</code>-shaped tensor?</p>

username: chaslie
cooked: <p>Hi Ptrblck,</p>
<p>apologies for the delay in getting back to you. I ran your code on a sample deck and compared the values in data to the input array X and i can see that the tensor data doesn’t get shuffled by view the output in the  print data statement to the print X statement…</p>
<pre><code class="lang-auto">from __future__ import print_function
import torch
import torch.nn.parallel
from torch.utils.data import DataLoader
from torch.utils.data import TensorDataset

x = torch.randn(42, 2048)
y = torch.randn(42, 1)
print("x=",x)
#print("y=",y)
dataset = TensorDataset(x, y)
train_loader = DataLoader(dataset, batch_size=1, shuffle=True)
for idx in range(x.size(0)):
    data, target = dataset[idx]
    assert (data==x[idx]).all(), "data shuffled"
    assert (target==y[idx]).all(), "target shuffled"   
for epoch in range(1):
    for (data, target) in train_loader:
        #print("data=",data.shape)
        print("data",data)
        #np.savetxt(f2,target.numpy())
</code></pre>
<p>however when I repeat the same exercise with my dataset the contents of data are not in the same order as the input. The code is:</p>
<pre><code class="lang-auto">X_train=train_array.transpose([1,0,2]).reshape(42000,2048)  #the input array size is [42000,2048,1]
X_train=torch.Tensor(X_train)
y_train=np.arange(1,42001,1)
y_train=y_train.reshape(42000,1)
y_train=torch.from_numpy(y_train)

train_dataset = TensorDataset(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=BATCHSIZE, shuffle=True)

for idx in range(X_train.size(0)):
    data, target = train_dataset[idx]
    assert (data==X_train[idx]).all(), "data shuffled"
    assert (target==y_train[idx]).all(), "target shuffled"

for epoch in range(num_epochs):
    for (data, target) in train_loader:
        #print("data=",data.shape)
        np.savetxt(f,data.numpy())
        np.savetxt(f2,target.numpy())
        data=data.unsqueeze(1)
        print("data_sq=",data.shape)
</code></pre>
<p>I’m begining to think that i have done something silly but i really can’t see what I have done <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Chaslie</p>

username: chaslie
cooked: <p>All Fixed, thanks for all your help</p>

username: ptrblck
cooked: <p>What was the reason for the shuffled data? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: chaslie
cooked: <p>to be honest, I do not know. I just kept changing things until it work <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>