Package: nonechucks
nonechucksnonechucksis a library that provides wrappers for PyTorch's datasets, samplers, and transforms to allow for dropping unwanted or invalid samples dynamically.IntroductionExamplesInstallationContributingLicensingIntroductionWhat if you have a dataset of 1000s of images, out of which a few dozen images are unreadable because the image files are corrupted? Or what if your dataset is a folder full of scanned PDFs that you have to OCRize, and then run a language detector on the resulting text, because you want only the ones that are in English? Or maybe you have anAlternateIndexSampler, and you want to be able to move todataset[6]afterdataset[4]fails while attempting to load!PyTorch's data processing module expects you to rid your dataset of any unwanted or invalid samples before you feed them into its pipeline, and provides no easy way to define a "fallback policy" in case such samples are encountered during dataset iteration.Why do I need it?You might be wondering why this is such a big deal when you could simplyfilterout samples before sending it to your PyTorch dataset or sampler! Well, it turns out that it can be a huge deal in many cases:When you have a small fraction of undesirable samples in a large dataset, orWhen your sample-loading operation is expensive, orWhen you want to let downstream consumers know that a sample is undesirable (with nonechucks, transforms are not restricted to modifying samples; they can drop them as well),When you want your dataset and sampler to be decoupled.In such cases, it's either simply too expensive to have a separate step to weed out bad samples, or it's just plain impossible because you don't even know what constitutes as "bad", or worse - both!nonechucksallows you to wrap your existing datasets and samplers with "safe" versions of them, which can fix all these problems for you.Examples1. Dealing with bad samplesLet's start with the simplest use case, which involves wrapping an existingDatasetinstance withSafeDataset.Create a dataset (the usual way)Using something like torchvision'sImageFolderdataset class, we can load an entire folder of labelled images for a typical supervised classification task.importtorchvision.datasetsasdatasetsfruits_dataset=datasets.ImageFolder('fruits/')Without nonechucksNow, if you have a sneakyfruits/apple/143.jpg(that is corrupted) sitting in yourfruits/folder, to avoid the entire pipeline from surprise-failing, you would have to resort to something like this:importrandom# Shuffle datasetindices=list(range(len(fruits_dataset))random.shuffle(indices)batch_size=4foriinrange(0,len(indices),batch_size):try:batch=[fruits_dataset[idx]foridxinindices[i:i+batch_size]]# Do something with itpassexceptIOError:# Skip the entire batchcontinueNot only do you have to put your code inside an extratry-exceptblock, but you are also forced to use a for-loop, depriving yourself of PyTorch's built-inDataLoader, which means you can't use features like batching, shuffling, multiprocessing, and custom samplers for your dataset.I don't know about you, but not being able to do that kind of defeats the whole point of using a data processing module for me.With nonechucksYou can transform your dataset into aSafeDatasetwith a single line of code.importnonechucksasncfruits_dataset=nc.SafeDataset(fruits_dataset)That's it! Seriously.And that's not all. You can also use aDataLoaderon top of this.dataloader=nc.SafeDataLoader(fruits_dataset,batch_size=4,shuffle=True)fori_batch,sample_batchedinenumerate(dataloader):# Do something with itpassIn this case,SafeDatasetwill skip the erroneous image, and use the next one in the place of it (as opposed to dropping the entire batch).2. Use Transforms as Filters!The function of transorms in PyTorch is restricted tomodifyingsamples. With nonechucks, you can simply returnNone(or raise an exception) from the transform's__call__method, and nonechucks will drop the sample from the dataset for you, allowing you to use transforms as filters!For the example, we'll assume aPDFDocumentsDataset, which reads PDF files from a folder, aPlainTextTransform, which transforms the files into raw text, and aLanguageFilter, which retains only documents of a particular language.classLanguageFilter:def__init__(self,language):self.language=languagedef__call__(self,sample):# Do machine learning magicdocument_language=detect_language(sample)ifdocument_language!=self.language:returnNonereturnsampletransforms=transforms.Compose([PlainTextTransform(),LanguageFilter('en')])en_documents=PDFDocumentsDataset(data_dir='pdf_files/',transform=transforms)en_documents=nc.SafeDataset(en_documents)InstallationTo install nonechucks, simply use pip:$ pip install nonechucksor clone this repo, and build from source with:$ python setup.py install.ContributingAll PRs are welcome.LicensingnonechucksisMIT licensed.