How to do transforms, and DataLoading from a CSV file (MNIST Digit Recognition)

akib62: <p>Hi,<br>
I am trying to learn <code>PyTorch</code>. So, I started with <code>MNIST Digit Recognition</code>. I am using the <code>CSV</code>  dataset file.</p>
<p>I wanted to write my program using functions. Like, I call to <code>def load_test_data</code> and the output of this function must be the <code>processed train images and labels</code>.</p>
<p>After, loading the <code>CSV</code> file into a <code>panda framework</code> I converted it into <code>NumPy and tensor</code> respectively. But, as a newcomer, when I am trying to use <code>transforms</code> it is giving different types of errors. I have searched <code>Google, StackOverflow, and PyTorch Forum</code> and found that I have to create <code>own class</code> to <code>transform</code>. It seems difficult to me!</p>
<p>Is there any way to do <code>transforms and DataLoader</code> for my code?</p>
<pre><code class="lang-auto">def load_train_data():
    # Loding CSV file in a dataframe with values
    train_img_df = pd.read_csv(os.path.join(config.dataset_path(), "mnist_train.csv"))
    
    #Getting image pixels values and labels from dataset 
    train_images = train_img_df.iloc[:, 1:]
    train_images = train_images.to_numpy()
    train_images = train_images.reshape((-1, 1, 28, 28))
    train_images = torch.tensor(train_images)
    print(train_images.shape) #torch.Size([42000, 1, 28, 28])
    
    train_labels = train_img_df.iloc[:, 0]
    train_labels = train_labels.to_numpy()
    train_labels = train_labels.reshape(train_labels.shape[0], 1)
    train_labels = torch.tensor(train_labels)
    print(train_labels.shape) #torch.Size([42000, 1])
    
    
    transformations=torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
    #What I have to write in here to transform and for DataLoader
    train_images = ???
    
    #Trying to creat  Dataloder
    train_img_loader = torch.utils.data.DataLoader(train_images, batch_size= config.batch_size)
    train_lab_loader = torch.utils.data.DataLoader(train_labels, batch_size= config.batch_size)
    
 
</code></pre>

ptrblck: <p>Image transformations in <code>torchvision.transforms</code> work mostly on <code>PIL.Image</code>s, so you would need to transform the tensors in <code>train_images</code> to <code>Image</code>s first, apply the transformations, and transform the <code>Image</code>s back to tensors.<br>
E.g. this might work:</p>
<pre><code class="lang-python">trans = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomCrop(24),
    transforms.ToTensor()])
</code></pre>

angiro: <p><a class="mention" href="/u/akib62">@akib62</a>  I am also trying to learn pytorch on MNIST. Could you please show, where u got MNIST in csv format. Thanks</p>

akib62: <p><a class="mention" href="/u/angiro">@angiro</a> you can download the dataset from Kaggle!</p>

akib62: <p><a class="mention" href="/u/angiro">@angiro</a> Yes</p>
<p>Link <a href="https://www.kaggle.com/c/digit-recognizer/data" rel="noopener nofollow ugc">https://www.kaggle.com/c/digit-recognizer/data</a></p>

akib62: <p>Hello, <a class="mention" href="/u/ptrblck">@ptrblck</a> Sorry, I am asking you another question! When I am trying to print some images from the <code>test loader</code> but it is giving me an error.</p>
<p><code>TypeError: Invalid shape (1, 28, 8) for image data</code></p>
<p>However, my <code>train loader</code> is working fine. Cold you tell me, where I am doing wrong?</p>
<pre><code class="lang-auto">test_images = (test_img_df.iloc[:,:].values).astype('float32')
test_images = test_images.reshape(test_images.shape[0], 28, 28)

#Converting Images to tensor
test_tensor = torch.tensor(test_images)/255.0

#Train DataLoder Generator
test_loader = DataLoader(test_tensor, batch_size= config.batch_size, 
                              num_workers = 2,shuffle= True)

#Plot some sample images using the data generator
for data in test_tensor:
    img_grid = make_grid(data[0:8,].unsqueeze(1), nrow=8)
    break

plt.imshow(img_grid.numpy().transpose((1,2,0)))
plt.rcParams['figure.figsize'] = (10, 2)
plt.show()
</code></pre>

ptrblck: <p>The error is most likely raised by <code>matplotlib</code> in <code>plt.imshow</code>, which expects an image array in the shape <code>[height, width, channels]</code> or <code>[height, width]</code>, while you seem to pass a single channel image array containing the batch dimension to this call.<br>
Remove the batch dimension e.g. via <code>tensor.squeeze(0)</code>.</p>