Package: clloader
Continual Loader (CLLoader)A library for PyTorch's loading of datasets in the field of Continual LearningAka Continual Learning, Lifelong-Learning, Incremental Learning, etc.Example:Install from and PyPi:pip3installclloaderAnd run!fromtorch.utils.dataimportDataLoaderfromclloaderimportCLLoaderfromclloader.datasetsimportMNISTclloader=CLLoader(MNIST("my/data/path",download=True),increment=1,initial_increment=5)print(f"Number of classes:{clloader.nb_classes}.")print(f"Number of tasks:{clloader.nb_tasks}.")fortask_id,(train_dataset,test_dataset)inenumerate(clloader):train_loader=DataLoader(train_dataset)test_loader=DataLoader(test_dataset)# Do your cool stuff hereSupported ScenariosNameAcronymSupportedNew InstancesNI:x:New ClassesNC:white_check_mark:New Instances & ClassesNIC:x:Supported Datasets:Note that the task sizes are fully customizable.NameNb classesImage SizeAutomatic DownloadMNIST1028x28x1:white_check_mark:Fashion MNIST1028x28x1:white_check_mark:KMNIST1028x28x1:white_check_mark:EMNIST1028x28x1:white_check_mark:QMNIST1028x28x1:white_check_mark:MNIST Fellowship3028x28x1:white_check_mark:CIFAR101032x32x3:white_check_mark:CIFAR10010032x32x3:white_check_mark:CIFAR Fellowship11032x32x3:white_check_mark:ImageNet100100224x224x3:x:ImageNet10001000224x224x3:x:Permuted MNIST1028x28x1:white_check_mark:Rotated MNIST1028x28x1:white_check_mark:Furthermore some "Meta"-datasets are available:InMemoryDataset, for in-memory numpy array:x_train,y_train=gen_numpy_array()x_test,y_test=gen_numpy_array()clloader=CLLoader(InMemoryDataset(x_train,y_train,x_test,y_test),increment=10,)PyTorchDataset,for any dataset defined in torchvision:clloader=CLLoader(PyTorchDataset("/my/data/path",dataset_type=torchvision.datasets.CIFAR10),increment=10,)ImageFolderDataset, for datasets having a tree-like structure, with one folder per class:clloader=CLLoader(ImageFolderDataset("/my/train/folder","/my/test/folder"),increment=10,)Fellowship, to combine several continual datasets.:clloader=CLLoader(Fellowship("/my/data/path",dataset_list=[CIFAR10,CIFAR100]),increment=10,)Some datasets cannot provide an automatic download of the data for miscealleneous reasons. For example for ImageNet, you'll need to download the data from theofficial page. Then load it likewise:clloader=CLLoader(ImageNet1000("/my/train/folder","/my/test/folder"),increment=10,)Some papers use a subset, called ImageNet100 or ImageNetSubset. You'll need to get the subset ids. It's either a file in the following format:my/path/to/image0.JPEG target0
my/path/to/image1.JPEG target1Or a list of tuple[("my/path/to/image0.JPEG", target0), ...]. Then loading the continual loader is very simple:clloader=CLLoader(ImageNet100("/my/train/folder","/my/test/folder",train_subset=...# My subset idstest_subset=...# My subset ids),increment=10,)Continual LoaderThe Continual LoaderCLLoaderloads the data and batch it in several tasks. See there some example arguments:clloader=CLLoader(my_continual_dataset,increment=10,initial_increment=2,train_transformations=[transforms.RandomHorizontalFlip()],common_transformations=[transforms.ToTensor(),transforms.Normalize((0.4914,0.4822,0.4465),(0.2023,0.1994,0.2010))],evaluate_on="seen")Here the first task is made of 2 classes, then all following tasks of 10 classes. You can have a more finegrained increment by providing a list of ìncrement=[2, 10, 5, 10]`.Thetrain_transformationsis applied only on the training data, while thecommon_transformationson both the training and testing data.By default, we evaluate our model after each task onseenclasses. But you can evalute only oncurrentclasses, or even onallclasses.Sample ImagesMNIST:Task 0Task 1Task 2Task 3Task 4FashionMNIST:Task 0Task 1Task 2Task 3Task 4CIFAR10:Task 0Task 1Task 2Task 3Task 4MNIST Fellowship (MNIST + FashionMNIST + KMNIST):Task 0Task 1Task 2PermutedMNIST:Task 0Task 1Task 2Task 3Task 4