How to Solve Data Loading Bottlenecks in Your Deep Learning Training | by Gorkem Polat | Towards Data Science

For each iteration in deep learning training, you form a batch of data and transfer it to the model. Reading data from the HDD is a very time-consuming process because a mechanical actuator arm moves through the HDD plates to find the target chunk of data points. And you know, we generally shuffle the dataset and form the batches from this shuffled dataset. This means your data points in the batches are randomly distributed among the HDD plates. This is called random read and it is very slow compared to the sequential read. If you are going to train a large dataset (like ImageNet), you should avoid these kinds of bottlenecks. Here is an example: with RTX2080 GPU and 7200 RPM HDD, one iteration of the ResNet18 Model on ImageNet data with 32 batch size takes 0.44 seconds. For 100 epochs, it takes 20 days! When we measure the timing of the functions, data loading+preprocessing takes 0.38 seconds (where 90% of this time belongs to the data loading part) while the optimization (forward+backward pass) time takes only 0.055 seconds. If the data loading time is reduced to a reasonable time, full training can be easily reduced to 2.5 days!
If your hardware system supports an SSD, buying an NVMe M2 type SSD will immediately solve your problem. Why? Let’s check the read speeds of some storage devices.
7200 RPM HDD: 100MB/sec
SSD with SATA speed: 600 MB/sec
SSD (NVMe type) via M2: 3500 MB/sec
So between NVMe type SSD and your 7200 RPM HDD, there is 35 times the difference. Therefore, buying and NVMe type SSD is the most basic and fastest solution.
What if an SSD is not an option for you? Then you have to continue with your HDD. Here are some receipts for you to make data loading faster with your HDD.
Load (read) all dataset into RAM at the beginning of the training so that there will be no time consumed reading from the disk in each iteration. You have to write a custom Dataset class for that (For PyTorch, check here: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html).
If you are working on a large dataset such as ImageNet (around 150 GB), you can not load all your data into RAM. So, you have to read the data batch by batch from the HDD. Yet, how can we make this faster? The hint is in the first paragraph of this article: We should perform sequential read instead of random read.
You shuffle your dataset and write the shuffled dataset again into a disk so that if you read each batch sequentially, you will get shuffled batches. You can use the Tensorpack library for this (https://tensorpack.readthedocs.io/tutorial/efficient-dataflow.html).
Now, let’s compare these three different data loading methods.
HDD, random read is the slowest as expected. Yet there is a surprise here! HDD, sequential read, is a way faster than the SSD, random read. So, instead of spending money on SSD, can’t we just use an HDD?
Unfortunately no.
First, reading and writing the shuffling data takes time. For ImageNet 2012 data (150 GB), it takes 4 hours to read, shuffle, and write. Therefore, you can not perform this operation for each epoch. This is only life-saver if you do not have an SSD.
Second, reducing data loading time from 0.370 sec to 0.018 sec is important but reducing it from 0.018 to 0.003 is unnecessary for the training because there is an additional optimization time (forward pass + backward pass). Once data loading time is reduced, that optimization time becomes the main bottleneck. In addition, many frameworks provide asynchronous data loading; therefore, that difference will not significantly affect overall training performance.
Basic operations such as cropping, flipping, and normalizing do not affect training time significantly; therefore, most of the time, you can perform these operations on the fly. Yet, if you have a long sequence of preprocessing functions or custom preprocessing functions that slow down the process, you can perform these operations on the original dataset once and write it to disk, then you can read the preprocessed files and directly feed it to your model.You can use GPU tensors to perform preprocessing. NVIDIA DALI [2] is designed to do preprocessing on a GPU, it also has fast CPU implementation.You can use a low-level language (such as C++ ) to perform preprocessing and fully utilize the threading mechanism.
Basic operations such as cropping, flipping, and normalizing do not affect training time significantly; therefore, most of the time, you can perform these operations on the fly. Yet, if you have a long sequence of preprocessing functions or custom preprocessing functions that slow down the process, you can perform these operations on the original dataset once and write it to disk, then you can read the preprocessed files and directly feed it to your model.
You can use GPU tensors to perform preprocessing. NVIDIA DALI [2] is designed to do preprocessing on a GPU, it also has fast CPU implementation.
You can use a low-level language (such as C++ ) to perform preprocessing and fully utilize the threading mechanism.
Frameworks such as PyTorch and Tensorflow isolate the complex operations and GPU, CPU data transfers from users. It is one of the beauties of these libraries. Yet, it can be harmful. For example, you want to get a value of a tensor in each iteration, you can get it easily with the following line (in PyTorch):
aValue = aTensor.item()
Although it seems a very innocent line, every time this line is executed, tensors from GPU RAM are copied to the main RAM, which is a time-consuming operation. It is like a print() function, we can not use it whenever we want (especially in loops). Therefore, we have to know what is going on and how functions operate in the background. Knowing this will help us write more effective codes.
Here is a humble flowchart of the methods mentioned in this article:
Found this article useful? Check out my other articles below:
medium.com
towardsdatascience.com
[1] Wu, Yuxin. “Tensorpack.” (2016). (https://github.com/tensorpack/tensorpack)