JustinLin610 commited on
Commit
24d4292
1 Parent(s): f7b9c28

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -4
README.md CHANGED
@@ -2,10 +2,18 @@
2
  license: apache-2.0
3
  ---
4
 
5
- # OFA-Medium
6
  This is the **medium** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
7
 
8
- To use it in Transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers and download the directory of transformers. After installation, you can use it as shown below:
 
 
 
 
 
 
 
 
9
 
10
  ```
11
  >>> from PIL import Image
@@ -29,6 +37,6 @@ To use it in Transformers, please refer to https://github.com/OFA-Sys/OFA/tree/f
29
  >>> img = Image.open(path_to_image)
30
  >>> patch_img = patch_resize_transform(img).unsqueeze(0)
31
 
32
- >>> gen = model.generate(inputs, patch_img=patch_img, num_beams=4)
33
- >>> print(tokenizer.decode(gen, skip_special_tokens=True, clean_up_tokenization_spaces=False))
34
  ```
 
2
  license: apache-2.0
3
  ---
4
 
5
+ # OFA-tiny
6
  This is the **medium** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
7
 
8
+ The directory includes 4 files, namely `config.json` which consists of model configuration, `vocab.json` and `merge.txt` for our OFA tokenizer, and lastly `pytorch_model.bin` which consists of model weights. There is no need to worry about the mismatch between Fairseq and transformers, since we have addressed the issue yet.
9
+
10
+ To use it in transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers. Install the transformers and download the models as shown below.
11
+ ```
12
+ git clone --single-branch --branch feature/add_transformers https://github.com/OFA-Sys/OFA.git
13
+ pip install OFA/transformers/
14
+ it clone https://huggingface.co/OFA-Sys/OFA-medium
15
+ ```
16
+ After, refer the path to OFA-medium to `ckpt_dir`, and prepare an image for the testing example below. Also, ensure that you have pillow and torchvision in your environment.
17
 
18
  ```
19
  >>> from PIL import Image
 
37
  >>> img = Image.open(path_to_image)
38
  >>> patch_img = patch_resize_transform(img).unsqueeze(0)
39
 
40
+ >>> gen = model.generate(inputs, patch_images=patch_img, num_beams=4)
41
+ >>> print(tokenizer.batch_decode(gen, skip_special_tokens=True))
42
  ```