Package: fastervit
FasterViT: Fast Vision Transformers with Hierarchical AttentionFasterViT achieves a new SOTA Pareto-front in
terms of accuracy vs. image throughput without extra training data !Note: Please use thelatest NVIDIA TensorRT releaseto enjoy the benefits of optimized FasterViT ops.Quick StartWe can import pre-trained FasterViT models with1 line of code. First, FasterViT can be simply installed by:pipinstallfastervitA pretrained FasterViT model with default hyper-parameters can be created as in the following:>>>fromfastervitimportcreate_model# Define fastervit-0 model with 224 x 224 resolution>>>model=create_model('faster_vit_0_224',pretrained=True,model_path="/tmp/faster_vit_0.pth.tar")model_pathis used to set the directory to download the model.We can also simply test the model by passing a dummy input image. The output is the logits:>>>importtorch>>>image=torch.rand(1,3,224,224)>>>output=model(image)# torch.Size([1, 1000])We can also use the any-resolution FasterViT model to accommodate arbitrary image resolutions. In the following, we define an any-resolution FasterViT-0
model with input resolution of 576 x 960, window sizes of 12 and 6 in 3rd and 4th stages, carrier token size of 2 and embedding dimension of
64:>>>fromfastervitimportcreate_model# Define any-resolution FasterViT-0 model with 576 x 960 resolution>>>model=create_model('faster_vit_0_any_res',resolution=[576,960],window_size=[7,7,12,6],ct_size=2,dim=64,pretrained=True)Note that the above model is intiliazed from the original ImageNet pre-trained FasterViT with original resolution of 224 x 224. As a result, missing keys and mis-matches could be expected since we are addign new layers (e.g. addition of new carrier tokens, etc.)We can simply test the model by passing a dummy input image. The output is the logits:>>>importtorch>>>image=torch.rand(1,3,576,960)>>>output=model(image)# torch.Size([1, 1000])Results + Pretrained ModelsImageNet-1KFasterViT ImageNet-1K Pretrained ModelsNameAcc@1(%)Acc@5(%)Throughput(Img/Sec)Resolution#Params(M)FLOPs(G)DownloadFasterViT-082.195.95802224x22431.43.3modelFasterViT-183.296.54188224x22453.45.3modelFasterViT-284.296.83161224x22475.98.7modelFasterViT-384.997.21780224x224159.518.2modelFasterViT-485.497.3849224x224424.636.6modelFasterViT-585.697.4449224x224975.5113.0modelFasterViT-685.897.4352224x2241360.0142.0modelImageNet-21KFasterViT ImageNet-21K Pretrained Models (ImageNet-1K Fine-tuned)NameAcc@1(%)Acc@5(%)Resolution#Params(M)FLOPs(G)DownloadFasterViT-4-21K-22486.697.8224x224271.940.8modelFasterViT-4-21K-38487.698.3384x384271.9120.1modelFasterViT-4-21K-51287.898.4512x512271.9213.5modelFasterViT-4-21K-76887.998.5768x768271.9480.4modelRobustness (ImageNet-A - ImageNet-R - ImageNet-V2)All models usecrop_pct=0.875. Results are obtained by running inference on ImageNet-1K pretrained models without finetuning.NameA-Acc@1(%)A-Acc@5(%)R-Acc@1(%)R-Acc@5(%)V2-Acc@1(%)V2-Acc@5(%)FasterViT-023.957.645.960.470.990.0FasterViT-131.263.347.561.972.691.0FasterViT-238.268.949.663.473.791.6FasterViT-344.273.051.965.675.092.2FasterViT-449.075.456.069.675.792.7FasterViT-552.777.656.970.076.093.0FasterViT-653.778.457.170.176.193.0A, R and V2 denote ImageNet-A, ImageNet-R and ImageNet-V2 respectively.CitationPlease consider citing FasterViT if this repository is useful for your work.@article{hatamizadeh2023fastervit,
  title={FasterViT: Fast Vision Transformers with Hierarchical Attention},
  author={Hatamizadeh, Ali and Heinrich, Greg and Yin, Hongxu and Tao, Andrew and Alvarez, Jose M and Kautz, Jan and Molchanov, Pavlo},
  journal={arXiv preprint arXiv:2306.06189},
  year={2023}
}LicensesCopyright © 2023, NVIDIA Corporation. All rights reserved.This work is made available under the NVIDIA Source Code License-NC. Clickhereto view a copy of this license.For license information regarding the timm repository, please refer to itsrepository.For license information regarding the ImageNet dataset, please see theImageNet official website.AcknowledgementThis repository is built on top of thetimmrepository. We thankRoss Wrightmanfor creating and maintaining this high-quality library.