Test Loader seems to be loading train data

Jordan_Howell: <p>Hello,</p>
<p>I have a weird issue and I can’t seem to find a typo.  I have a training data set of shape <code>(3719,100)</code> and a testing data set of <code>(6635,100)</code>.  When I transform all data to tensors, the custom dataset and loader for the test data seems to be loading train data according to the <code>.size(0</code> outputs.  Below is the code I’m working with:</p>
<pre><code class="lang-auto">class tensor_data(Dataset):
    def __init__(self, num, cats, labels, images):
        self.num = num_tensor
        self.cats = cats
        self.labels = labels
        self.images = images
    
    def __len__(self):
        return len(self.num)
    
    def __getitem__(self, idx):
        return self.num[idx], self.cats[idx], self.labels[idx], self.images[idx]
</code></pre>
<p>Here is how I derive the data to use the above:</p>
<pre><code class="lang-auto">###########################train tensors#####################################
num_tensor = torch.tensor(np.asarray(train_sample.loc[:, numerical_columns]), dtype = torch.float)
col_tensor = torch.tensor(np.asarray(train_sample.loc[:, non_loca_cat_columns]), dtype = torch.long)
label_tensor = torch.tensor(np.asarray(train_sample.loc[:, 'target']), dtype = torch.long)
image_tensor = []
for i, row in train_sample.iterrows():
    image_tensor.append(train_transform(Image.open(Path(row['location']))))
images = torch.stack(image_tensor)
print("train: ", num_tensor.size(), col_tensor.size(), label_tensor.size(), images.size())
output: train:  torch.Size([3719, 6]) torch.Size([3719, 8]) torch.Size([3719]) torch.Size([3719, 3, 224, 224])

train_data = tensor_data(num_tensor, col_tensor, label_tensor, images)
print('train_data:', len(train_data))
output: train_data: 3719


train_loader = DataLoader(train_data, batch_size = 10, shuffle = True)
print('train_loader: ', len(train_loader))
output: train_loader:  372
</code></pre>
<p>So all of the above makes sense.  This is what happens when I do the same with the test data.</p>
<pre><code class="lang-auto">test_num_tensor = torch.tensor(np.asarray(test.loc[:, numerical_columns]), dtype = torch.float)
test_col_tensor = torch.tensor(np.asarray(test.loc[:, non_loca_cat_columns]), dtype = torch.long)
test_label_tensor = torch.tensor(np.asarray(test.loc[:, 'target']), dtype = torch.long)
test_image_tensor = []
for i, row in test.iterrows():
    test_image_tensor.append(test_transform(Image.open(Path(row['location']))))
test_images = torch.stack(test_image_tensor)
print("test: ", test_num_tensor.size(),  test_col_tensor.size(),  test_label_tensor.size(),  test_images.size())
output: test:  torch.Size([6635, 6]) torch.Size([6635, 8]) torch.Size([6635]) torch.Size([6635, 3, 224, 224])

test_data = tensor_data(test_num_tensor, test_col_tensor, test_label_tensor, test_images)
print('test_data:', len(test_data))
 output: test_data: 3719

test_loader = DataLoader(test_data, batch_size = 10, shuffle = False)
print('test_loader: ', len(test_loader)) 
output: test_loader:  372
</code></pre>
<p>The above shows the test tensors the correct size of <code>(6635, x)</code> but when bringing them in through the custom dataset then loader, I get the lengths of the train loader.  Am I missing something simple?</p>

harsha_g: <aside class="quote no-group" data-username="Jordan_Howell" data-post="1" data-topic="87296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jordan_howell/48/16421_2.png" class="avatar"> Jordan_Howell:</div>
<blockquote>
<p><code>self.num = num_tensor</code></p>
</blockquote>
</aside>
<p>That’s the problem. Should be <code>self.num = num</code></p>

Jordan_Howell: <p>Wow.  Yep. That was it.  Thank you!</p>