Package: gcvit
GCViT: Global Context Vision TransformerTensorflow 2.0 Implementation of GCViTThis library implementsGCViTusing Tensorflow 2.0 specifically intf.keras.Modelmanner to get PyTorch flavor.Update15 Jan 2023:GCViTLargemodel added with ckpt.3 Sept 2022: Annotatedkaggle-notebookbased on this project wonKaggle ML Research Spotlight: August 2022.19 Aug 2022: This project got acknowledged byOfficialrepoherePaper Implementation & Explanation **I have explained the GCViT paper in a Kaggle notebookGCViT: Global Context Vision Transformer, which also includes a detailed implementation of the model from scratch. The notebook provides a comprehensive explanation of each part of the model, with intuition.Do check it out, especially if you are interested in learning more about GCViT or implementing it yourself. Note that this notebook has won theKaggle ML Research Award 2022.ModelArchitecture:Local Vs Global Attention:ResultOfficial codebase had some issue which has been fixed recently (12 August 2022). Here's the result of ported weights onImageNetV2-Testdata,ModelAcc@1Acc@5#ParamsGCViT-XXTiny0.6630.87312MGCViT-XTiny0.6850.88520MGCViT-Tiny0.7080.89928MGCViT-Small0.7200.90151MGCViT-Base0.7310.90790MGCViT-Large0.7340.913202MInstallationpipinstall-Ugcvit# or# pip install -U git+https://github.com/awsaf49/gcvit-tfUsageLoad model using following codes,fromgcvitimportGCViTTinymodel=GCViTTiny(pretrain=True)Any input size other than224x224,fromgcvitimportGCViTTinymodel=GCViTTiny(input_shape=(512,512,3),pretrain=True,resize_query=True)Simple code to check model's prediction,fromskimage.dataimportchelseaimg=tf.keras.applications.imagenet_utils.preprocess_input(chelsea(),mode='torch')# Chelsea the catimg=tf.image.resize(img,(224,224))[None,]# resize & create batchpred=model(img).numpy()print(tf.keras.applications.imagenet_utils.decode_predictions(pred)[0])Prediction:[('n02124075','Egyptian_cat',0.9194835),('n02123045','tabby',0.009686623),('n02123159','tiger_cat',0.0061576385),('n02127052','lynx',0.0011503297),('n02883205','bow_tie',0.00042479983)]For feature extraction:model=GCViTTiny(pretrain=True)# when pretrain=True, num_classes must be 1000model.reset_classifier(num_classes=0,head_act=None)feature=model(img)print(feature.shape)Feature:(None,512)For feature map:model=GCViTTiny(pretrain=True)# when pretrain=True, num_classes must be 1000feature=model.forward_features(img)print(feature.shape)Feature map:(None,7,7,512)Kaggle ModelsThese pre-trained models can also be loaded usingKaggle Models. Settingfrom_kaggle=Truewill enforce model to load weights from Kaggle Models without downloading, thus can be used without internet in Kaggle.fromgcvitimportGCViTTinymodel=GCViTTiny(pretrain=True,from_kaggle=True)Live-DemoFor live demo on Image Classification & Grad-CAM, withImageNetweights, clickpowered by 🤗 Space and Gradio. here's an example,ExampleFor working training example checkout these notebooks onGoogle Colab&Kaggle.Here is grad-cam result after training on Flower Classification Dataset,To DoConvert it to multi-backendKeras 3.0Segmentation PipelineSupport forKaggle ModelsRemovetensorflow_addonsNew updated weights have been added.Working training example in Colab & Kaggle.GradCAM showcase.Gradio Demo.Build model withtf.keras.Model.Port weights from official repo.Support forTPU.AcknowledgementGCVit(Official)Swin-Transformer-TFtfgcvitkeras_cv_attention_modelsCitation@article{hatamizadeh2022global,title={Global Context Vision Transformers},author={Hatamizadeh, Ali and Yin, Hongxu and Kautz, Jan and Molchanov, Pavlo},journal={arXiv preprint arXiv:2206.09959},year={2022}}