Repository: lightly-ai/lightly

Issue:
Why are there different backbones across SSL methods in CIFAR10 benchmark?
Hi, thank you so much for developing this great library, I found it extremely useful for my work!

I noticed that in the [CIFAR10 benchmark script](https://github.com/lightly-ai/lightly/blob/master/docs/source/getting_started/benchmarks/cifar10_benchmark.py), there are different backbones across the SSL methods. 

The two differences I've noticed are:
1. some use `torchvision.models.resnet18()`, others use `lightly.models.ResNetGenerator('resnet-18')`
    * I understand that for the CIFAR datasets, we want to use a variant that's slightly different than the ones in torchvision (which is typically used for ImageNet)
    * what I don't understand is why you did not use one variant consistently across all SSL methods 
2. some have an additional `nn.AdaptiveAvgPool2d(1)` layer at the end, some don't. 
    * my guess would be that this has something to do with the respective projection heads
    * to be safe, I hope you don't mind me asking

I'd highly appreciate any help!


Discussion:
Dear @JeanKaddour thanks for bringing this up! It's most likely an oversight and we'll have to fix it and make it consistent.
The `nn.AdaptiveAvgPool2d(1)` ensures that the output of the backbone is compatible with the projection heads.

thank you very much! 

Resolving patch:
diff --git a/docs/source/getting_started/benchmarks/cifar10_benchmark.py b/docs/source/getting_started/benchmarks/cifar10_benchmark.py
--- a/docs/source/getting_started/benchmarks/cifar10_benchmark.py
+++ b/docs/source/getting_started/benchmarks/cifar10_benchmark.py
@@ -65,7 +65,6 @@
 
 # use a GPU if available
 gpus = -1 if torch.cuda.is_available() else 0
-distributed_backend = 'ddp' if torch.cuda.device_count() > 1 else None
 
 # Adapted from our MoCo Tutorial on CIFAR-10
 #
@@ -103,6 +102,7 @@
 swav_collate_fn = lightly.data.SwaVCollateFunction(
     crop_sizes=[32],
     crop_counts=[2], # 2 crops @ 32x32px
+    crop_min_scales=[0.14]
 )
 
 # No additional augmentations for the test set
@@ -231,7 +231,7 @@ def __init__(self, dataloader_kNN, num_classes):
         resnet = lightly.models.ResNetGenerator('resnet-18')
         self.backbone = nn.Sequential(
             *list(resnet.children())[:-1],
-            nn.AdaptiveAvgPool2d(1),
+            nn.AdaptiveAvgPool2d(1)
         )
         # create a simclr model based on ResNet
         self.resnet_simclr = \
@@ -262,7 +262,7 @@ def __init__(self, dataloader_kNN, num_classes):
         resnet = lightly.models.ResNetGenerator('resnet-18')
         self.backbone = nn.Sequential(
             *list(resnet.children())[:-1],
-            nn.AdaptiveAvgPool2d(1),
+            nn.AdaptiveAvgPool2d(1)
         )
         # create a simsiam model based on ResNet
         self.resnet_simsiam = \
@@ -304,10 +304,11 @@ class BarlowTwinsModel(BenchmarkModule):
     def __init__(self, dataloader_kNN, num_classes):
         super().__init__(dataloader_kNN, num_classes)
         # create a ResNet backbone and remove the classification head
-        resnet = torchvision.models.resnet18()
+        resnet = lightly.models.ResNetGenerator('resnet-18')
         last_conv_channels = list(resnet.children())[-1].in_features
         self.backbone = nn.Sequential(
             *list(resnet.children())[:-1],
+            nn.AdaptiveAvgPool2d(1)
         )
         # create a barlow twins model based on ResNet
         self.resnet_barlowtwins = \
@@ -354,7 +355,7 @@ class BYOLModel(BenchmarkModule):
     def __init__(self, dataloader_kNN, num_classes):
         super().__init__(dataloader_kNN, num_classes)
         # create a ResNet backbone and remove the classification head
-        resnet = torchvision.models.resnet18()
+        resnet = lightly.models.ResNetGenerator('resnet-18')
         self.backbone = nn.Sequential(
             *list(resnet.children())[:-1],
             nn.AdaptiveAvgPool2d(1)
@@ -412,7 +413,7 @@ class SwaVModel(BenchmarkModule):
     def __init__(self, dataloader_kNN, num_classes):
         super().__init__(dataloader_kNN, num_classes)
         # create a ResNet backbone and remove the classification head
-        resnet = torchvision.models.resnet18()
+        resnet = lightly.models.ResNetGenerator('resnet-18')
         self.backbone = nn.Sequential(
             *list(resnet.children())[:-1],
             nn.AdaptiveAvgPool2d(1)
@@ -483,12 +484,10 @@ def configure_optimizers(self):
 
             trainer = pl.Trainer(max_epochs=max_epochs, 
                                 gpus=gpus,
-                                progress_bar_refresh_rate=100,
-                                distributed_backend=distributed_backend,
                                 default_root_dir=logs_root_dir)
             trainer.fit(
                 benchmark_model,
-                train_dataloader=dataloader_train_ssl,
+                train_dataloaders=dataloader_train_ssl,
                 val_dataloaders=dataloader_test
             )
             gpu_memory_usage.append(torch.cuda.max_memory_allocated())


Test patch: