Package: ppwebdataset
Paddle-WebDataset, This is a Paddle version of WebDatasetRef:https://github.com/webdataset/webdataset/tree/main%matplotlibinlineimportmatplotlib.pyplotaspltimportpaddle.nnfromrandomimportrandrangeimportosos.environ["WDS_VERBOSE_CACHE"]="1"os.environ["GOPEN_VERBOSE"]="0"The WebDataset FormatWebDataset format files are tar files, with two conventions:within each tar file, files that belong together and make up a training sample share the same basename when stripped of all filename extensionsthe shards of a tar file are numbered likesomething-000000.tartosomething-012345.tar, usually specified using brace notationsomething-{000000..012345}.tarWebDataset can read files from local disk or from any pipe, which allows it to access files using common cloud object stores. WebDataset can also read concatenated MsgPack and CBORs sources.The WebDataset representation allows writing purely sequential I/O pipelines for large scale deep learning. This is important for achieving high I/O rates from local storage (3x-10x for local drives compared to random access) and for using object stores and cloud storage for training.The WebDataset format represents images, movies, audio, etc. in their native file formats, making the creation of WebDataset format data as easy as just creating a tar archive. Because of the way data is aligned, WebDataset works well with block deduplication as well and aligns data on predictable boundaries.Standard tools can be used for accessing and processing WebDataset-format files.bucket="https://storage.googleapis.com/webdataset/testdata/"dataset="publaynet-train-{000000..000009}.tar"url=bucket+dataset!curl-s{url}|tartf-|sed10qPMC4991227_00003.json
PMC4991227_00003.png
PMC4537884_00002.json
PMC4537884_00002.png
PMC4323233_00003.json
PMC4323233_00003.png
PMC5429906_00004.json
PMC5429906_00004.png
PMC5592712_00002.json
PMC5592712_00002.png
tar: stdout: write errorNote that in these.tarfiles, we have pairs of.jsonand.pngfiles; each such pair makes up a training sample.PP-WebDataset LibrariesThere are several libraries supporting the WebDataset format:ppwebdatasetfor Python3 (includes thewidslibrary), this repositoryWebdataset.jla Julia implementationtarp, a Golang implementation and command line toolRay Data sources and sinksTheppwebdatasetlibrary can be used with Paddle.TheppwebdatasetLibraryTheppwebdatasetlibrary is an implementation of PaddleIterableDataset(or a mock implementation thereof if you aren't using Paddle). It implements as form of stream processing. Some of its features are:large scale parallel data access through shardinghigh performance disk I/O due to purely sequential readslatency insensitive due to big fat pipesno local storage requiredinstant startup for training jobsonly requires reading from file descriptors/network streams, no special APIsits API encourages high performance I/O pipelinesscalable from tiny desktop datasets to petascale datasetsprovides local caching if desiredrequires no dataset metadata; any collection of shards can be read and used instantlyThe main limitations people run into are related to the fact thatIterableDatasetis less commonly used in Paddle and some existing code may not support it as well, and that achieving an exactly balanced number of training samples across many compute nodes for a fixed epoch size is tricky; for multinode training,webdatasetis usually used with shard resampling.There are two interfaces, the concise "fluid" interface and a longer "pipeline" interface. We'll show examples using the fluid interface, which is usually what you want.importppwebdatasetaswdspil_dataset=wds.WebDataset(url).shuffle(1000).decode("pil").to_tuple("png","json")The resulting datasets are standard PaddleIterableDatasetinstances.isinstance(pil_dataset,paddle.io.IterableDataset)Trueforimage,jsoninpil_dataset:breakplt.imshow(image)<matplotlib.image.AxesImage at 0x7f73806db970>We can add onto the existing pipeline for augmentation and data preparation.importpaddle.vision.transformsastransformsfromPILimportImagepreproc=transforms.Compose([transforms.Resize((224,224)),transforms.ToTensor(),lambdax:1-x,])defpreprocess(sample):image,json=sampletry:label=json["annotations"][0]["category_id"]except:label=0returnpreproc(image),labeldataset=pil_dataset.map(preprocess)forimage,labelindataset:breakplt.imshow(image.numpy().transpose(1,2,0))<matplotlib.image.AxesImage at 0x7f7375fc2230>WebDatasetis just an instance of a standardIterableDataset. It's a single-threaded way of iterating over a dataset. Since image decompression and data augmentation can be compute intensive, Paddle usually uses theDataLoaderclass to parallelize data loading and preprocessing.WebDatasetis fully compatible with the standardDataLoader.Here are a number of notebooks showing how to use WebDataset for image classification and LLM training:train-resnet50-wds-- simple, single GPU training from Imagenettrain-resnet50-multiray-wds-- multinode training using webdatasetgenerate-text-dataset-- initial dataset generationtesseract-wds-- shard-to-shard transformations, here for OCR running over large datasetstrain-ocr-errors-hf-- an example of LLM fine tuning using a dataset in webdataset formatThewds-notesnotebook contains some additional documentation and information about the library.ThewebdatasetPipeline APIThewds.WebDatasetfluid interface is just a convenient shorthand for writing down pipelines. The underlying pipeline is an instance of thewds.DataPipelineclass, and you can construct data pipelines explicitly, similar to the way you usenn.Sequentialinside models.dataset=wds.DataPipeline(wds.SimpleShardList(url),# at this point we have an iterator over all the shardswds.shuffle(100),# add wds.split_by_node here if you are using multiple nodeswds.split_by_worker,# at this point, we have an iterator over the shards assigned to each workerwds.tarfile_to_samples(),# this shuffles the samples in memorywds.shuffle(1000),# this decodes the images and jsonwds.decode("pil"),wds.to_tuple("png","json"),wds.map(preprocess),wds.shuffle(1000),wds.batched(16))batch=next(iter(dataset))batch[0].shape,batch[1].shape([16, 3, 224, 224], (16,))TheppwidsLibrary for Indexed WebDatasetsInstalling theppwebdatasetlibrary installs a second library calledppwids. This library provides fully indexed/random access to the same datasets thatppwebdatasetaccesses using iterators/streaming.Like theppebdatasetlibrary,ppwidsis high scalable and provides efficient access to very large datasets. Being indexed, it is easily backwards compatible with existing data pipelines based on indexed dataset, including precise epochs for multinode training. The library comes with its ownChunkedSamplerandDistributedChunkedSamplerclasses, which provided shuffling accross nodes while still preserving enough locality of reference for efficient training.Internally, the library uses ammap-basedtarfile reader implementation; this allows very fast access without precomputed indexes, and it also means that shard and the equivalet of "shuffle buffers" are shared in memory between workers on the same machine.This additional power comes at some cost: the library requires a small metadata file that lists all the shards in a dataset and the number of samples contained in each, the library requires local storage for as many shards as there are I/O workers on a node, it uses shared memory andmmap, and the availability of indexing makes it easy to accidentally use inefficient access patterns.Generally, the recommendation is to usewebdatasetfor all data generation, data transformation, and training code, and to useppwidsonly if you need fully random access to datasets (e.g., for browing or sparse sampling), need an indexed-based sampler, or are converting tricky legacy code.importppwidstrain_url="https://storage.googleapis.com/webdataset/fake-imagenet/imagenet-train.json"dataset=ppwids.ShardListDataset(train_url)sample=dataset[1900]print(sample.keys())print(sample[".txt"])plt.imshow(sample[".jpg"])dict_keys(['.cls', '.jpg', '.txt', '__key__', '__dataset__', '__index__', '__shard__', '__shardindex__'])
a high quality color photograph of a dog


https://storage.googleapis.com/webdataset/fake-ima base: https://storage.googleapis.com/webdataset/fake-imagenet name: imagenet-train nfiles: 1282 nbytes: 31242280960 samples: 128200 cache: /tmp/_wids_cache





<matplotlib.image.AxesImage at 0x7f7373669e70>There are several examples of how to useppwidsin theexamplesdirectory.train-resnet50-widsshows how to train a ResNet-50 model on ImageNet usingppwidstrain-resnet50-multiray-widsshows how to train a ResNet-50 model on ImageNet using multiple nodesNote that the APIs betweenwebdatasetandwidsare not fully consistent:ppwidskeeps the extension's "." in the keys, whilewebdatasetremoves it (".txt" vs "txt")ppwidsdoesn't have a fully fluid interface, andadd_transformationjust adds to a list of transformationsppwebdatasetcurrently can't read theppwidsJSON specificationsInstallation and Documentation$ pip install ppwebdatasetFor the Github version:$ pip install git+https://github.com/JunnYu/ppwebdataset.gitHere are some videos talking about WebDataset and large scale deep learning:Introduction to Large Scale Deep LearningLoading Training Data with WebDatasetCreating Datasets in WebDataset FormatTools for Working with Large DatasetsDependenciesThe WebDataset library only requires Paddle, NumPy, and a small library calledbraceexpand.WebDataset loads a few additional libraries dynamically only when they are actually needed and only in the decoder:PIL/Pillow for image decodingpaddle.visionfor image decodingmsgpackfor MessagePack decodingthecurlcommand line tool for accessing HTTP serversthe Google/Amazon/Azure command line tools for accessing cloud storage bucketsLoading of one of these libraries is triggered by configuring a decoder that attempts to decode content in the given format and encountering a file in that format during decoding. (Eventually, the paddle... dependencies will be refactored into those libraries.)