Package: webdataset-latch
WebDatasetWebDataset is a PyTorch Dataset (IterableDataset) implementation providing
efficient access to datasets stored in POSIX tar archives and uses only sequential/streaming
data access. This brings substantial performance advantage in many compute environments, and it
is essential for very large scale training.While WebDataset scales to very large problems, it also works well with smaller datasets and simplifies
creation, management, and distribution of training data for deep learning.WebDataset implements standard PyTorchIterableDatasetinterface and works with the PyTorchDataLoader.
Access to datasets is as simple as:importwebdatasetaswdsdataset=wds.WebDataset(url).shuffle(1000).decode("torchrgb").to_tuple("jpg;png","json")dataloader=torch.utils.data.DataLoader(dataset,num_workers=4,batch_size=16)forinputs,outputsindataloader:...In that code snippet,urlcan refer to a local file, a local HTTP server, a cloud storage object, an object
on an object store, or even the output of arbitrary command pipelines.WebDataset fulfills a similar function to Tensorflow's TFRecord/tf.Example
classes, but it is much easier to adopt because it does not actually
require any kind of data conversion: data is stored in exactly the same
format inside tar files as it is on disk, and all preprocessing and data
augmentation code remains unchanged.DocumentationGetting Started-- a simple demonstration of how to use WebDatasetHow It Works-- WebDataset is just a collection of PyTorch IterableDataset implementationsDecoding-- WebDataset uses file-based representations internally and its Decoder makes turning these into PyTorch tensors easyDesktop Usage and Caching-- WebDataset is great for desktop development and supports built-in cachingCreating Datasets in WebDataset Format-- there are a variety of tools for creating datasets in WebDataset formatWriting WebDataset from Python-- you can write WebDataset datasets directly from PythonMultinode Training-- what you need to know to handle distributed, multi-node training with WebDatasetData Sources-- accessing data on local disks, on web servers, or in cloud storage bucketsInstallation$ pip install webdatasetFor the Github version:$ pip install git+https://github.com/tmbdev/webdataset.gitDocumentation:ReadTheDocsIntroductory VideosHere are some videos talking about WebDataset and large scale deep learning:Introduction to Large Scale Deep LearningLoading Training Data with WebDatasetCreating Datasets in WebDataset FormatTools for Working with Large DatasetsMore Examplesloading videossplitting raw videos into clips for trainingconverting the Falling Things datasetRelated Libraries and SoftwareTheAIStoreserver provides an efficient backend for WebDataset; it functions like a combination of web server, content distribution network, P2P network, and distributed file system. Together, AIStore and WebDataset can serve input data from rotational drives distributed across many servers at the speed of local SSDs to many GPUs, at a fraction of the cost. We can easily achieve hundreds of MBytes/s of I/O per GPU even in large, distributed training jobs.Thetarprocutilities provide command line manipulation and processing of webdatasets and other tar files, including splitting, concatenation, andxargs-like functionality.Thetensorcomlibrary provides fast three-tiered I/O; it can be inserted betweenAIStoreandWebDatasetto permit distributed data augmentation and I/O. It is particularly useful when data augmentation requires more CPU than the GPU server has available.You can find the full PyTorch ImageNet sample code converted to WebDataset attmbdev/pytorch-imagenet-wds