Package: kappadata
KappaDataUtilities fordatasets and dataloadingwithpytorchmodular datasetscaching datasets in-memoryadditional transformsallow deterministic augmentations (e.g. for calculating the test loss on augmented samples)RandAugmentpatchwise augmentationsvarious dataset filters and other dataset manipulationfilter by classlimit size to a %MixupCutmixlabel smoothing...repeated augmentationsModular datasetspytorch datasetsload all data in the__getitem__.
KappaData decouples the__getitem__such that single properties of the dataset can be loaded independently.Image classification dataset exampleLet's take an image classification dataset as an example. A sample consists of an image with an associated class label.class ImageClassificationDataset(torch.utils.data.Dataset):
    def __init__(self, image_paths):
        super().__init__()
        self.image_paths = image_paths
    def __len__(self):
        return len(self.image_paths)
    
    def __getitem__(self, idx):
        img = load_image(self.image_paths[idx])
        class_label = image_path_to_class_label(self.image_paths[idx])
        return img, class_labelIf your training process contains something that only requires the class labels, the dataset has to additionally load
all the images which can take a long time (whereas loading only labels is very fast). With KappaData the__getitem__method is split into subparts:# inherit from kappadata.KDDataset
class ImageClassificationDataset(kappadata.KDDataset):
    def __init__(self, image_paths):
        super().__init__()
        self.image_paths = image_paths
    def __len__(self):
        return len(self.image_paths)
    
    # replace __getitem__ with getitem_x and getitem_y
    def getitem_x(self, idx, ctx=None):
        return load_image(self.image_paths[idx])
    def getitem_y(self, idx, ctx=None):
        return image_path_to_class_label(self.image_paths[idx])Now each subpart of the dataset can be retrieved by wrapping the dataset into aModeWrapper:ds = ImageClassificationDataset(image_paths=...)
for y in kappadata.ModeWrapper(ds, mode="y"):
    ...
for x, y in kappadata.ModeWrapper(ds, mode="x y"):
    ...torch.utils.data.Subset/torch.utils.data.ConcatDatasetcan be used by simply replacing them withkappadata.KDSubset/kappadata.KDConcatDataset.Wrappers"Dataset Wrappers"KappaData implements various ways to manipulate datasets (kappadata.wrappers.dataset_wrappers).Filter by classkappadata.ClassFilterWrapper(ds, valid_classes=[0, 1])kappadata.ClassFilterWrapper(ds, invalid_classes=[0, 1])Balance data by oversampling underrepresented classeskappadata.OversamplingWrapper(ds)Subset by specifying percentageskappadata.PercentFilterWrapper(ds, from_percent=0.25)kappadata.PercentFilterWrapper(ds, to_percent=0.75)kappadata.PercentFilterWrapper(ds, from_percent=0.25, to_percent=0.75)Repeat the whole datasetrepeat twice:kappadata.RepeatWrapper(ds, repetitions=2)repeat until size is > 100kappadata.RepeatWrapper(ds, min_size=100)Shuffle datasetkappadata.ShuffleWrapper(ds, seed=5)"Sample Wrappers"KappaData implements various ways to manipulate how samples are sampled from the underlying dataset
(kappadata.wrappers.sample_wrappers). "Sample Wrappers" are similar to transforms in that they transform the sample in
some way, but "Sample Wrappers" are more powerful because they have full access to the underlying dataset whereas normal
transforms only have access to a single sample.class Transform:
  def forward(x):
    # only x can be manipulated (e.g. normalized, image-transforms, ...)class SampleWrapper(kd.KDWrapper):
  def getitem_x(idx, ctx=None):
    # access to the underlying dataset via self.dataset
    # e.g. return the sum of two different samples
    idx2 = np.random.randint(len(self))
    return self.dataset.getitem_x(idx, ctx) + self.dataset.getitem_x(idx2, ctx)This allows implementing more complex transformations. KappaData implements the following SampleWrappers:Mixupkappadata.MixupWrapper(dataset=ds, alpha=1., p=1.)Cutmixkappadata.CutmixWrapper(dataset=ds, alpha=1., p=1.)MixuporCutmixkappadata.MixWrapper(dataset=ds, cutmix_alpha=1., mixup_alpha=1., p=1., cutmix_p=0.5)TODO sampling multiple viewslabel smoothingkappadata.LabelSmoothingWrapper(dataset=ds, smoothing=.1)Augmentation parametersWith KappaData you can also retrieve various properties of your data prepocessing (e.g. augmentation parameters). The
following example shows how you can retrieve the parameters
oftorchvision.transforms.RandomResizedCrop.import torchvision.transforms.functional as F
class MyRandomResizedCrop(torchvision.transforms.RandomResizedCrop):
    def forward(self, img, ctx=None):
        # make random resized crop
        i, j, h, w = self.get_params(img, self.scale, self.ratio)
        cropped = F.resized_crop(img, i, j, h, w, self.size, self.interpolation)
        # store parameters
        if ctx is not None:
          ctx["crop_parameters"] = (i, j, h, w)
        return cropped
  
class ImageClassificationDataset(kappadata.KDDataset):
    def __init__(self, ...):
      ...
      self.random_resized_crop = MyRandomResizedCrop()
    ...
    def getitem_x(self, idx, ctx=None):
        img = load_image(self.image_paths[idx])
        return self.random_resized_crop(img, ctx=ctx)When you want to access the parameters simply passreturn_ctx=Trueto theModeWrapper:ds = ImageClassificationDataset(image_paths=...)
for x, ctx in kappadata.ModeWrapper(ds, mode="x", return_ctx=True):
    print(ctx["crop_parameters"])
for (x, y), ctx in kappadata.ModeWrapper(ds, mode="x y", return_ctx=True):
    ...Caching datasets in-memorySharedDictDatasetkappadata.SharedDictDatasetprovides a wrapper to store arbitrary datasets in-memory via a dictionary shared between
all worker processes (usingpython multiprocessingdata
structures). The shared memory part is important
fordataloadingwithnum_workers > 0. Small and medium sized datasets can be cached in-memory to avoid bottlenecks when loading data
from a disk. For example even the fullImageNetcan be cached on many servers as it has ~
130GB and its not too uncommon for GPU servers to have more RAM than that.Caching image datasetsNaively caching image datasets can lead to high memory consumption because image data is usually stored in a compressed
format and decompressed during loading. To reduce memory, the raw uncompressed data needs to be cached.Example caching
atorchvision.datasets.ImageFolder:from kappadata.loading.image_folder import raw_image_loader, raw_image_folder_sample_to_pil_sample 
class CachedImageFolder(kappadata.KDDataset):
    def __init__(self, ...):
        # modify ImageFolder to load raw samples (NOTE: can't apply transforms onto raw data)
        self.ds = torchvision.datasets.ImageFolder(..., transform=None, loader=raw_image_loader)
        # initialize cached dataset that decompresses the raw data into a PIL image
        self.cached_ds = kappadata.SharedDictDataset(self.ds, transform=raw_image_folder_sample_to_pil_sample)
        # store transforms to apply after decompression
        self.transform = ...
    def getitem_x(self, idx, ctx=None):
        x, _ = self.cached_ds[idx]
        if self.transform is not None:
            x = self.transform(x)
        return xAutomatically copy datasets to a local (fast) diskDatasets are often stored on a global (slow) storage and before training moved to a local (fast) disk.kappadata.copy_folder_from_global_to_localprovides an utility function to do this automatically:local path doesn't exist -> automatically copy from global to locallocal path exists -> do nothinglocal path exists but is incomplete -> clear directory and copy againfrom pathlib import Path
from kappadata import copy_folder_from_global_to_local
global_path = Path("/system/data/ImageNet")
local_path = Path("/local/data")
# /system/data/ImageNet contains a 'train' and a 'val' folder -> copy whole dataset
copy_folder_from_global_to_local(global_path, local_path)
# copy only "train"
copy_folder_from_global_to_local(global_path, local_path, relative_path="train")The above code will also work (without modification) if/system/data/ImageNetcontains only 2 zip filestrain.zipandval.zipMiscellaneousall datasets derived fromkappadata.KDDatasetautomatically support python slicingall_class_labels = ModeWrapper(ds, mode="y")[:]all_class_labels = ModeWrapper(ds, mode="y")[5:-3:2]all datasets derived fromkappadata.KDDatasetimplementiterfor y in ModeWrapper(ds, mode="y"):
    ...retrieve the original dataset without wrappersds.root_dataset