Storing Training Data on the Cloud | by Connor Shorten | Towards Data Science

When you are building deep learning models, you will likely benefit from having more data to train with. You may run out of space on your local machine and therefore need to find ways to store datasets on the cloud and then access them while training your models.
I recently came across on an obstacle in an Image Classification project where I did not have enough storage on my laptop to increase the size of my training dataset. When doing multi-output, (more than 2 classes), image recognition problems, storing 10–100,000s of images for each class, (especially at medium+ resolution), can result in a lot of GBs!
In this article, I will present a couple of tips that helped me use cloud storage in order to formulate batches during model training.
How much is 1 GB of storage?
The image above is 54 KB compressed, (png), at a size of 266x168x3, (resolution x RGB color channels).
Therefore, you could store 18,518 of these images with 1 GB.
The beauty of cloud storage is that we can store many GBs online for free and even TBs, (1,000 GBs), for fairly low costs... Whereas storing several GBs on our local machines can be quite problematic/outright impossible.
How can I access data hosted on the cloud while training my model?
The main idea I wanted to communicate with this article is how a naming convention for my dataset helped me retrieve batches of images from the URL. This is important because when training models, we want to select some random batch of images to feed into the model at each iteration. The dataset is too big to load the entire dataset into memory and select random instances from there. Therefore, I used a naming convention that would let me partition the batches with indices and then fetch the images from the bucket URL accordingly.
You will be given a URL to your bucket and you can use a random number generator to append desired indices to the URL request. This will help you parse the bucket to formulate these batches for training a model.
For example, if you name each item in the dataset: SHOE_1, SHOE_2,... SHOE_9998, SHOE_9999. You will be able to formulate a URL fetch string with something like:
#PSEUDOCODEi = randomnumber()fetch_item = url + ‘SHOE_’ + inew_item = httprequest(fetch_item)batch.append(new_item)
By using a consistent naming convention for your dataset, you can easily parse and fetch random batches for training, without loading the dataset into memory.
Please check out this stack overflow post on exactly how to access s3 files such as images with python:
stackoverflow.com
Conclusion
In Conclusion, I hope this article inspired thought about how you plan on storing your datasets once they grow too large to be hosted on your local machine. I think this is great to do as soon as possible, especially if you are using the google colab environment to train your models. This way you can easily access your data from google colab. Please share any thoughts or strategies that you have used to host custom machine learning training datasets on the cloud. Thanks for reading!
Connor Shorten is a Computer Science student at Florida Atlantic University. Research interests in software economics, deep learning, and software engineering.