Package: torchdatasets
Usemap,apply,reduceorfilterdirectly onDatasetobjectscachedata in RAM/disk or via your own method (partial caching supported)Full PyTorch'sDatasetandIterableDatasetsupportGeneraltorchdata.mapslikeFlattenorSelectExtensible interface (your own cache methods, cache modifiers, maps etc.)Usefultorchdata.datasetsclasses designed for general tasks (e.g. file reading)Support fortorchvisiondatasets (e.g.ImageFolder,MNIST,CIFAR10) viatd.datasets.WrapDatasetMinimal overhead (single call tosuper().__init__())VersionDocsTestsCoverageStylePyPIPythonPyTorchDockerRoadmap:bulb: ExamplesCheck documentation here:https://szymonmaszke.github.io/torchdataGeneral exampleCreate image dataset, convert it to Tensors, cache and concatenate with smoothed labels:importtorchdataastdimporttorchvisionclassImages(td.Dataset):# Different inheritancedef__init__(self,path:str):super().__init__()# This is the only changeself.files=[fileforfileinpathlib.Path(path).glob("*")]def__getitem__(self,index):returnImage.open(self.files[index])def__len__(self):returnlen(self.files)images=Images("./data").map(torchvision.transforms.ToTensor()).cache()You can concatenate above dataset with another (saylabels) and iterate over them as per usual:fordata,labelinimages|labels:# Do whatever you want with your dataCache first1000samples in memory, save the rest on disk in folder./cache:images=(ImageDataset.from_folder("./data").map(torchvision.transforms.ToTensor())# First 1000 samples in memory.cache(td.modifiers.UpToIndex(1000,td.cachers.Memory()))# Sample from 1000 to the end saved with Pickle on disk.cache(td.modifiers.FromIndex(1000,td.cachers.Pickle("./cache")))# You can define your own cachers, modifiers, see docs)To see what else you can do please checktorchdata documentationIntegration withtorchvisionUsingtorchdatayou can easily splittorchvisiondatasets and apply augmentation
only to the training part of data without any troubles:importtorchvisionimporttorchdataastd# Wrap torchvision dataset with WrapDatasetdataset=td.datasets.WrapDataset(torchvision.datasets.ImageFolder("./images"))# Split datasettrain_dataset,validation_dataset,test_dataset=torch.utils.data.random_split(model_dataset,(int(0.6*len(dataset)),int(0.2*len(dataset)),int(0.2*len(dataset))),)# Apply torchvision mappings ONLY to train datasettrain_dataset.map(td.maps.To(torchvision.transforms.Compose([torchvision.transforms.RandomResizedCrop(224),torchvision.transforms.RandomHorizontalFlip(),torchvision.transforms.ToTensor(),torchvision.transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]),])),# Apply this transformation to zeroth sample# First sample is the label0,)Please notice you can usetd.datasets.WrapDatasetwith any existingtorch.utils.data.Datasetinstance to give it additionalcachingandmappingpowers!:wrench: Installation:snake:pipLatest release:pipinstall--usertorchdataNightly:pipinstall--usertorchdata-nightly:whale2:DockerCPU standaloneand various versions ofGPU enabledimages are available
atdockerhub.For CPU quickstart, issue:dockerpullszymonmaszke/torchdata:18.04Nightly builds are also available, just prefix tag withnightly_. If you are going forGPUimage make sure you havenvidia/dockerinstalled and it's runtime set.:question: ContributingIf you find any issue or you think some functionality may be useful to others and fits this library, pleaseopen new Issueorcreate Pull Request.To get an overview of thins one can do to help this project, seeRoadmap