first model commit

README.md

@@ -15,9 +15,10 @@ tags:
 - deep learning
 ---
 
-# Self-supervised ViT-S/16 (small-sized Vision Transformer with patch size 16) model with SimPool.
+# Self-supervised ViT-S/16 (small-sized Vision Transformer with patch size 16) model with SimPool
+
+ViT-S model with SimPool (no gamma) trained on ImageNet-1k for 100 epochs. Self-supervision with [DINO](https://arxiv.org/abs/2104.14294).
 
-ViT-S model with SimPool (no gamma) trained on ImageNet-1k for 100 epochs. Self-supervision with DINO.
 SimPool is a simple attention-based pooling method at the end of network, introduced on this ICCV 2023 [paper](https://arxiv.org/pdf/2309.06891.pdf) and released in this [repository](https://github.com/billpsomas/simpool/).
 Disclaimer: This model card is written by the author of SimPool, i.e. [Bill Psomas](http://users.ntua.gr/psomasbill/).
 
@@ -32,6 +33,15 @@ SimPool is a simple attention-based pooling mechanism as a replacement of the de
 Interestingly, we find that, whether supervised or self-supervised, SimPool improves performance on pre-training and downstream tasks and provides attention maps delineating object boundaries in all cases.
 One could thus call SimPool universal.
 
+## Evaluation with k-NN
+
+| k       | top1    | top5    |
+| ------- | ------- | ------- |
+| 10      | 69.778  | 85.91   |
+| 20      | 69.602  | 87.54   |
+| 100     | 67.318  | 88.674  |
+| 200     | 65.966  | 88.404  |
+
 ## BibTeX entry and citation info
 
 ```

configs.yaml

@@ -0,0 +1,45 @@
+arch: vit_small
+backend: nccl
+batch_size_per_gpu: 100
+clip_grad: 0.0
+data_path: /path/to/imagenet/
+dist_url: env://
+drop_path_rate: 0.1
+epochs: 100
+eval_every: 30
+freeze_last_layer: 1
+global_crops_scale:
+- 0.25
+- 1.0
+local_crops_number: 6
+local_crops_scale:
+- 0.05
+- 0.25
+local_rank: 0
+lr: 0.0005
+min_lr: 1.0e-05
+mode: simpool
+momentum_teacher: 0.996
+nb_knn:
+- 10
+- 20
+- 100
+- 200
+norm_last_layer: false
+num_workers: 10
+optimizer: adamw
+out_dim: 65536
+output_dir: /path/to/output/
+patch_size: 16
+saveckp_freq: 20
+seed: 0
+subset: -1
+teacher_temp: 0.07
+temperature: 0.07
+use_bn_in_head: false
+use_fp16: false
+warmup_epochs: 10
+warmup_teacher_temp: 0.04
+warmup_teacher_temp_epochs: 30
+weight_decay: 0.04
+weight_decay_end: 0.4