| tags: | |
| - image-to-text | |
| - image-captioning | |
| license: apache-2.0 | |
| metrics: | |
| - rouge | |
| datasets: | |
| - nlphuji/flickr30k | |
| widget: | |
| - src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/savanna.jpg | |
| example_title: Savanna | |
| - src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/football-match.jpg | |
| example_title: Football Match | |
| - src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/airport.jpg | |
| example_title: Airport | |
| base_model: | |
| - google/vit-base-patch16-224-in21k | |
| model-index: | |
| - name: mozilla/distilvit | |
| results: | |
| - task: | |
| type: image-to-text | |
| name: Image To Text | |
| dataset: | |
| name: nlphuji/flickr30k | |
| type: nlphuji/flickr30k | |
| metrics: | |
| - name: ROUGE-1 | |
| type: rouge | |
| value: 43.006 | |
| verified: true | |
| - name: ROUGE-2 | |
| type: rouge | |
| value: 16.9939 | |
| verified: true | |
| - name: ROUGE-L | |
| type: rouge | |
| value: 38.8923 | |
| verified: true | |
| - name: ROUGE-LSUM | |
| type: rouge | |
| value: 38.8877 | |
| verified: true | |
| - name: loss | |
| type: loss | |
| value: 0.19939416646957397 | |
| - name: gen_len | |
| type: gen_len | |
| value: 11.327256736227712 | |
| verified: true | |
| # distilvit | |
| This model is a work in progress. Fine-tuned version of those base models: | |
| - a VIT model for the image encoder: https://huggingface.co/google/vit-base-patch16-224-in21k | |
| - a Distilled GPT-2 model for the text decoder: https://huggingface.co/distilbert/distilgpt2 | |
| This model was trained on: | |
| - Flickr30k : https://huggingface.co/datasets/nlphuji/flickr30k | |
| - COCO 2017: https://cocodataset.org | |
| You can get that checkpoint using the 3083a3cef6e3c8dd90df3f088074bbe836b0f403 commit. | |
| It was then further fine-tuned on : | |
| - Flickr30k debiased: https://huggingface.co/datasets/Mozilla/flickr30k-transformed-captions | |
| - DocOrNot: https://huggingface.co/datasets/Mozilla/docornot | |
| You can find the code used to create the model here: https://github.com/mozilla/distilvit | |
| ### Framework versions | |
| - Transformers 4.40.2 | |
| - Pytorch 2.3.0+cu121 | |
| - Datasets 2.19.1 | |
| - Tokenizers 0.19.1 | |