Dataloader for sequential data using PyTorch deep learning framework — Part 2 | by Alleviate and Elevate (A & E) | May, 2021 | Medium | Towards Data Science

Deep learning and machine learning algorithms are ruling the world these days. PyTorch is one of the most commonly used deep learning framework used for implementing various deep learning algorithms. On the other hand, the learning-based method essentially requires some annotated training dataset which can be used by the model to extract the relation between input data and labels. In order to feed data to the neural network, we define a data loader. In this blog, we will see how to write a data loader for different datasets in the PyTorch framework.
We will be working on dog vs cat image classification problem. We have to classify whether the given image is of cat or dog and the dataset can be downloaded from here. The training dataset contains a total of 25,000 images. As this is a classification problem, the label for dog is “0” and the label for cat is “1”.
Let’s start by importing all the required libraries.
import osfrom PIL import Imageimport torchfrom torch.utils.data import DataLoader, Datasetimport torchvision.transforms as transformsimport torch.nn as nn
The dataset class for PyTorch framework is defined as a class, whose basic structure is as follows
class data(Dataset):   def __init__(self, param1, param2):        #the function is initialised here           def __len__(self):        #the function returns length of data       def __getitem__(self, index):        #gives one item at a time
The ultimate aim of this class is to provide one data point at a time using function __getitem__. This is done using index which is passed to the function internally, using a sampler function defined in the Dataloader (will be discussed in the coming blog).
The function __init__ is called when you initialize the object for the dataset. Here, you can pass multiple arguments that will be useful for writing __getitem__.
The __len__ function is used to return the total length of the dataset. Based on this, index will be generated which will then be provided to __getitem__ .
The format of the dog vs cat dataset is as follows -:
data/   - dog_1.jpg   - dog_2.jpg    ...    ...    ...   - cat_1.jpg   - cat_2.jpg    ...    ...    ...
Now that we understand the components required for writing our data loader, let’s delve deeper into our use case.
class data(Dataset):      def __init__(self, path, transform):        self.files = os.listdir(path)        self.transform = transform        self.path = path   def __len__(self):        return len(self.files)   def __getitem__(self, index):       filename = self.files[index]       input = Image.open(os.path.join(self.path, filename))       label = 0 if filename.find("dog")>=0 else 1       img_as_tensor = self.transform(input)       return img_as_tensor, labeltransformations = transforms.Compose(         [transforms.Resize((224,224)),transforms.ToTensor()]                 )path = "./data"train_dataset = data(path, transformations)dataloader = DataLoader(train_dataset, batch_size=Train_Batch_Size, shuffle=True)
Let us first understand the function __init__. The class data is initialized with two arguments, path and transform which are passed as arguments to __init__. When we declare an object of this class it internally calls __init__ which stores the list of names of files self.files, the transform in self.transform and the path in self.path.
Since the __len__ is used to return the length of the whole dataset, I have used len(self.files) to return the same.
The function __getitem__ is the most crucial, it loads the image, then resizes it, and then converts it into a tensor. An important thing to note here is that the data provided to the neural network should always be normalized. We take care of normalization using transforms.ToTensor(). Finally, __getitem__ returns two things, image_as_tensor and label for the corresponding data point.
After initializing the class data, we use a DataLoader function which automatically batches the whole data into a defined batch size. So if your original datapoint size is (3, 224, 224) (that you get from __getitem__), each item of the dataloader will have size (batch_size, 3, 224, 224) i.e. it automatically samples batch_size number of datapoints. This is possible in our case because the size of images is constant so the DataLoader function is able to create batches automatically. However, in cases like Natural language processing when the size is not constant, we need to write our own batching function.
Let’s deal with sequential dataset i.e. sentences, time series, audio, etc. now. Here __getitem__ will no longer give us data points with the same size. For example, consider the task of sentiment classification (explained here), then one sentence can be “The flight service was very good” and other could be “I did not get my baggage on the belt, pathetic service.” Here the length of both sentences is different.
To tackle this, let’s answer three questions first.
What is batching? — Batching means to merge tensors of multiple data points into a single tensorWhy do we need batching? — Batching is used to make computation faster, because of batching we can process multiple data points together instead of just one at a time.How is batching done? — Since we are merging multiple tensors here, the size along each dimension of the tensor needs to be the same. Since out data points are of varied sizes we have a problem in our hands.
What is batching? — Batching means to merge tensors of multiple data points into a single tensor
Why do we need batching? — Batching is used to make computation faster, because of batching we can process multiple data points together instead of just one at a time.
How is batching done? — Since we are merging multiple tensors here, the size along each dimension of the tensor needs to be the same. Since out data points are of varied sizes we have a problem in our hands.
Mainly we need to now tackle problem of batching.
For the purpose of our discussion here, we will use the IMDB dataset which is a review dataset and can be downloaded from here. Since we are dealing with sentences here, the way to process the dataset will vary a bit. Since the neural networks only understand numbers, not words, we will have to convert each word into a number. In order to do that we have to build a vocabulary as explained in the following code.
The function reader is used to read the whole data and it returns a list of all sentences and labels “0” for negative review and “1” for positive review.
The function build_vocab takes data and minimum word count as input and gives as output a mapping (named “word2id”) of each word to a unique number. For every unknown word going forward the corresponding number will be 1.
Moving on to writing a dataset class for the sequential dataset. Our objective here is to output one item at a time, given an index.
As the functionality of different functions is already discussed above, I will briefly recap.
The function __init__ takes word2id mapping and train_path. Then __init__ calls reader to get data and labels corresponding to the sentences.
The function __len__ returns the length of the whole dataset i.e. self.data.
The function preprocess converts the input sentence into a tensor of numbers, where each number corresponds to the word in the sentence.
The function __getitem__ is used to output one processed data point at a time with the help of the index.
Coming on to the solution of our problem of each datapoint having different size. The following code defines collate_fn. This function is used to take care of batching data points of different sizes.
train_dataset = Dataset_seq(word2id, train_path)train_dataloader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True,collate_fn=collate_fn)
A point to note here is that in a list of tuple, each tuple can have different sizes but in a tensor, the size along all the dimensions need to be same in order to merge them.
The collate_fn is automatically given one input named data, which is a list of tuples with length equal to batch size. Each tuple contains tensors of numbers (seq returned from __getitem__) and their corresponding label. For the sake of simplicity, we will call them sequence and label respectively. So ultimately we have to convert each sequence in such a manner that their size becomes constant. To achieve that we perform zero-padding as shown in the code above. Since zero padding is uniformly used for the whole dataset, the model learns that it is not of much use and that it just represents waste values.
We have reached a solution for sure but the question remains, is it an optimal solution? In cases when the original sizes of all the sequence vary with a huge difference or in other words with a huge variance, we will end up wasting a lot of GPU memory with zero padding, which is ultimately of no use. There has to be a better way that minimizes the requirement of zero padding! This problem and its solution is discussed here.
Become a Medium member to unlock and read many other stories on medium. Follow us on Medium for reading more such blog posts.