File size: 2,919 Bytes
73ec027 39232a8 73ec027 39232a8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
tags:
- mgp-str
- image-to-text
widget:
- src: https://github.com/AlibabaResearch/AdvancedLiterateMachinery/blob/main/OCR/MGP-STR/demo_imgs/IIIT5k_HOUSE.png
example_title: Example 1
- src: https://github.com/AlibabaResearch/AdvancedLiterateMachinery/blob/main/OCR/MGP-STR/demo_imgs/IIT5k_EVERYONE.png
example_title: Example 2
- src: https://github.com/AlibabaResearch/AdvancedLiterateMachinery/blob/main/OCR/MGP-STR/demo_imgs/CUTE80_KINGDOM.png
example_title: Example 3
---
# MGP-STR (base-sized model)
MGP-STR base-sized model is trained on MJSynth and SynthText. It was introduced in the paper [Multi-Granularity Prediction for Scene Text Recognition](https://arxiv.org/abs/2209.03592) and first released in [this repository](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).
## Model description
MGP-STR is pure vision STR model, consisting of ViT and specially designed A^3 modules. The ViT module was initialized from the weights of DeiT-base, except the patch embedding model, due to the inconsistent input size.
Images (32x128) are presented to the model as a sequence of fixed-size patches (resolution 4x4), which are linearly embedded. One also adds absolute position embeddings before feeding the sequence to the layers of the ViT module. Next, A^3 module selects a meaningful combination from the tokens of ViT output and integrates them into one output token corresponding to a specific character. Moreover, subword classification heads based on BPE A^3 module and WordPiece A^3 module are devised for subword predictions, so that the language information can be implicitly modeled. Finally, these multi-granularity predictions (character, subword and even word) are merged via a simple and effective fusion strategy.
## Intended uses & limitations
You can use the raw model for optical character recognition (OCR) on text images. See the [model hub](https://huggingface.co/models?search=alibaba-damo/mgp-str) to look for fine-tuned versions on a task that interests you.
### How to use
Here is how to use this model in PyTorch:
```python
from transformers import MGPSTRProcessor, MGPSTRModel
import requests
from PIL import Image
processor = MGPSTRProcessor.from_pretrained('alibaba-damo/mgp-str-base')
model = MGPSTRModel.from_pretrained('alibaba-damo/mgp-str-base')
# load image from the IIIT-5k dataset
url = "https://i.postimg.cc/ZKwLg2Gw/367-14.png"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
pixel_values = processor(image, return_tensors="pt").pixel_values
generated_ids, attens = model(pixel_values)
generated_text = processor.batch_decode(generated_ids)['generated_text']
```
### BibTeX entry and citation info
```bibtex
@inproceedings{ECCV2022mgp_str,
title={Multi-Granularity Prediction for Scene Text Recognition},
author={Peng Wang, Cheng Da, and Cong Yao},
booktitle = {ECCV},
year={2022}
}
``` |