Package: torchdata
TorchData (see note below on current status)Why TorchData?|Install guide|What are DataPipes?|Beta Usage and Feedback|Contributing|Future Plans:warning: As of July 2023, we have paused active development on TorchData and have paused new releases. We have learnt a lot from building it and hearing from users, but also believe we need to re-evaluate the technical design and approach given how much the industry has changed since we began the project. During the rest of 2023 we will be re-evaluating our plans in this space. Please reach out if you suggestions or comments (please use#1196for feedback).torchdatais a library of common modular data loading primitives for easily constructing flexible and performant data
pipelines.This library introduces composable Iterable-style and Map-style building blocks calledDataPipesthat work well out of the box with the PyTorch'sDataLoader. These built-inDataPipeshave the necessary functionalities to reproduce many different datasets in TorchVision and TorchText, namely
loading files (from local or cloud), parsing, caching, transforming, filtering, and many more utilities. To understand
the basic structure ofDataPipes, please seeWhat are DataPipes?below, and to see howDataPipescan be practically composed together into datasets, please see ourexamples.On top ofDataPipes, this library provides a newDataLoader2that allows the execution of these data pipelines in
various settings and execution backends (ReadingService). You can learn more about the new version ofDataLoader2in
ourfull DataLoader2 documentation. Additional features
are work in progres, such as checkpointing and advanced control of randomness and determinism.Note that because many features of the original DataLoader have been modularized into DataPipes, their source codes live
asstandard DataPipes in pytorch/pytorchrather than torchdata to preserve backward-compatibility support and functional parity withintorch. Regardless, you
can to them by importing them fromtorchdata.Why composable data loading?Over many years of feedback and organic community usage of the PyTorchDataLoaderandDataset, we've found that:The originalDataLoaderbundled too many features together, making them difficult to extend, manipulate, or
replace. This has created a proliferation of use-case specificDataLoadervariants in the community rather than an
ecosystem of interoperable elements.Many libraries, including each of the PyTorch domain libraries, have rewritten the same data loading utilities over
and over again. We can save OSS maintainers time and effort rewriting, debugging, and maintaining these commonly used
elements.These reasons inspired the creation ofDataPipeandDataLoader2, with a goal to make data loading components more
flexible and reusable.InstallationVersion CompatibilityThe following is the correspondingtorchdataversions and supported Python versions.torchtorchdatapythonmaster/nightlymain/nightly>=3.8,<=3.112.0.00.6.0>=3.8,<=3.111.13.10.5.1>=3.7,<=3.101.12.10.4.1>=3.7,<=3.101.12.00.4.0>=3.7,<=3.101.11.00.3.0>=3.7,<=3.10ColabFollow the instructionsin this Colab notebook. The notebook also
contains a simple usage example.Local pip or condaFirst, set up an environment. We will be installing a PyTorch binary as well as torchdata. If you're using conda, create
a conda environment:condacreate--nametorchdata
condaactivatetorchdataIf you wish to usevenvinstead:python-mvenvtorchdata-envsourcetorchdata-env/bin/activateInstall torchdata:Using pip:pipinstalltorchdataUsing conda:condainstall-cpytorchtorchdataYou can then proceed to runour examples, such asthe IMDb one.From sourcepipinstall.If you'd like to include the S3 IO datapipes and aws-sdk-cpp, you may also followthe instructions hereIn case building TorchData from source fails, install the nightly version of PyTorch following the linked guide on thecontributing page.From nightlyThe nightly version of TorchData is also provided and updated daily from main branch.Using pip:pipinstall--pretorchdata--extra-index-urlhttps://download.pytorch.org/whl/nightly/cpuUsing conda:condainstalltorchdata-cpytorch-nightlyWhat are DataPipes?Early on, we observed widespread confusion between the PyTorchDatasetwhich represented reusable loading tooling
(e.g.TorchVision'sImageFolder),
and those that represented pre-built iterators/accessors over actual data corpora (e.g. TorchVision'sImageNet). This led to an
unfortunate pattern of siloed inheritance of data tooling rather than composition.DataPipeis simply a renaming and repurposing of the PyTorchDatasetfor composed usage. ADataPipetakes in some
access function over Python data structures,__iter__forIterDataPipesand__getitem__forMapDataPipes, and
returns a new access function with a slight transformation applied. For example, take a look at thisJsonParser, which
accepts an IterDataPipe over file names and raw streams, and produces a new iterator over the filenames and deserialized
data:importjsonclassJsonParserIterDataPipe(IterDataPipe):def__init__(self,source_datapipe,**kwargs)->None:self.source_datapipe=source_datapipeself.kwargs=kwargsdef__iter__(self):forfile_name,streaminself.source_datapipe:data=stream.read()yieldfile_name,json.loads(data,**self.kwargs)def__len__(self):returnlen(self.source_datapipe)You can see in this example how DataPipes can be easily chained together to compose graphs of transformations that
reproduce sophisticated data pipelines, with streamed operation as a first-class citizen.Under this naming convention,Datasetsimply refers to a graph ofDataPipes, and a dataset module likeImageNetcan be rebuilt as a factory function returning the requisite composedDataPipes. Note that the vast majority of
built-in features are implemented asIterDataPipes, we encourage the usage of built-inIterDataPipeas much as
possible and convert them toMapDataPipeonly when necessary.DataLoader2A new, light-weight DataLoader2 is introduced to decouple the overloaded data-manipulation functionalities fromtorch.utils.data.DataLoadertoDataPipeoperations. Besides, certain features can only be achieved withDataLoader2, such as like checkpointing/snapshotting and switching backend services to perform high-performant
operations.Please read thefull documentation here.TutorialA tutorial of this library isavailable here on the documentation site. It covers four topics:using DataPipes,working with DataLoader,implementing DataPipes, andworking with Cloud Storage Providers.There is also a tutorial available onhow to work with the new DataLoader2.Usage ExamplesWe provide a simple usage example in thisColab notebook. It can also be downloaded
and executed locally as a Jupyter notebook.In addition, there are several data loading implementations of popular datasets across different research domains that
useDataPipes. You can find a fewselected examples here.Frequently Asked Questions (FAQ)What should I do if the existing set of DataPipes does not do what I need?You canimplement your own custom DataPipe. If
you believe your use case is common enough such that the community can benefit from having your custom DataPipe added to
this library, feel free to open a GitHub issue. We will be happy to discuss!What happens when theShufflerDataPipe is used with DataLoader?In order to enable shuffling, you need to add aShufflerto your DataPipe line. Then, by default, shuffling will
happen at the point where you specified as long as you do not setshuffle=Falsewithin DataLoader.What happens when theBatcherDataPipe is used with DataLoader?If you choose to useBatcherwhile settingbatch_size > 1for DataLoader, your samples will be batched more than
once. You should choose one or the other.Why are there fewer built-inMapDataPipesthanIterDataPipes?By design, there are fewerMapDataPipesthanIterDataPipesto avoid duplicate implementations of the same
functionalities asMapDataPipe. We encourage users to use the built-inIterDataPipefor various functionalities, and
convert it toMapDataPipeas needed.How is multiprocessing handled with DataPipes?Multi-process data loading is still handled by theDataLoader, see theDataLoader documentation for more details.
As of PyTorch version >= 1.12.0 (TorchData version >= 0.4.0), data sharding is automatically done for DataPipes within
theDataLoaderas long as aShardingFilterDataPipe exists in your pipeline. Please see thetutorialfor an example.What is the upcoming plan for DataLoader?DataLoader2is in the prototype phase and more features are actively being developed. Please see theREADME file intorchdata/dataloader2. If
you would like to experiment with it (or other prototype features), we encourage you to install the nightly version of
this library.Why is there an Error saying the specified DLL could not be found at the time of importingportalocker?It only happens for people who runstorchdataon Windows OS as a common problem withpywin32. And, you can find the
reason and the solution for it in thelink.ContributingWe welcome PRs! See theCONTRIBUTINGfile.Beta Usage and FeedbackWe'd love to hear from and work with early adopters to shape our designs. Please reach out by raising an issue if you're
interested in using this tooling for your project.LicenseTorchData is BSD licensed, as found in theLICENSEfile.