|
--- |
|
license: mit |
|
--- |
|
|
|
### Simple running code is based on [MoAI-Github](https://github.com/ByungKwanLee/MoAI). |
|
|
|
You need only the following seven steps. |
|
|
|
### [0] Download Github Code of MoAI, install the required libraries, set the necessary environment variable (README.md explains in detail! Don't Worry!). |
|
|
|
```bash |
|
git clone https://github.com/ByungKwanLee/MoAI |
|
bash install |
|
``` |
|
|
|
### [1] Loading Image |
|
|
|
```python |
|
from PIL import Image |
|
from torchvision.transforms import Resize |
|
from torchvision.transforms.functional import pil_to_tensor |
|
image_path = "figures/moai_mystery.png" |
|
image = Resize(size=(490, 490), antialias=False)(pil_to_tensor(Image.open(image_path))) |
|
``` |
|
|
|
### [2] Instruction Prompt |
|
|
|
```python |
|
prompt = "Describe this image in detail." |
|
``` |
|
|
|
### [3] Loading MoAI |
|
```python |
|
from moai.load_moai import prepare_moai |
|
moai_model, moai_processor, seg_model, seg_processor, od_model, od_processor, sgg_model, ocr_model \ |
|
= prepare_moai(moai_path='BK-Lee/MoAI-7B', bits=4, grad_ckpt=False, lora=False, dtype='fp16') |
|
``` |
|
|
|
### [4] Pre-processing for MoAI |
|
```python |
|
moai_inputs = moai_model.demo_process(image=image, |
|
prompt=prompt, |
|
processor=moai_processor, |
|
seg_model=seg_model, |
|
seg_processor=seg_processor, |
|
od_model=od_model, |
|
od_processor=od_processor, |
|
sgg_model=sgg_model, |
|
ocr_model=ocr_model, |
|
device='cuda:0') |
|
``` |
|
|
|
### [5] Generate |
|
```python |
|
import torch |
|
with torch.inference_mode(): |
|
generate_ids = moai_model.generate(**moai_inputs, do_sample=True, temperature=0.9, top_p=0.95, max_new_tokens=256, use_cache=True) |
|
``` |
|
|
|
### [6] Decoding |
|
```python |
|
answer = moai_processor.batch_decode(generate_ids, skip_special_tokens=True)[0].split('[U')[0] |
|
print(answer) |
|
``` |
