周回遅れでLLaVAをDatabricksで動かしてみる

マルチモーダルすごい。

# LLaVA: Large Language and Vision Assistant

https://github.com/haotian-liu/LLaVA/tree/main

https://llava-vl.github.io/

https://huggingface.co/liuhaotian/llava-v1.5-7b

# Databricksでの動作確認

先人達の肩に乗ります。

https://qiita.com/isanakamishiro2/items/f5419f8a031cca0dde41

https://qiita.com/fuyu_quant/items/2692198b65d9763b45a2

https://note.com/npaka/n/n3656c32f71af

https://note.com/owlet_notes/n/nfb8ba6bc0a06

結構大きめのGPUクラスターを準備しました。そうでないと[GPUに収まらないエラー](https://note.com/owlet_notes/n/nfb8ba6bc0a06#cbe35bf5-ffca-450b-a939-2134b3448cb0)になりました。
![Screenshot 2023-11-28 at 18.30.47.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1168882/48e09416-94c4-3a0d-61fb-f18c63c5732d.png)


`ERROR: Project xxxx has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.
`を回避するために`touch setup.cfg`を実行しています。[こちら](https://stackoverflow.com/questions/69711606/how-to-install-a-package-using-pip-in-editable-mode-with-pyproject-toml)を参考にしました。

```sh
%sh
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
touch setup.cfg
pip install -U -qq transformers accelerate
pip install -e .
```

Pythonカーネルをリスタート。

```py
dbutils.library.restartPython()
```

モデルのダウンロード。

```py
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
from llava.eval.run_llava import eval_model

model_path = "liuhaotian/llava-v1.5-7b"

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)
```

推論します。

```py
model_path = "liuhaotian/llava-v1.5-7b"
prompt = "What are the things I should be cautious about when I visit here?"
image_file = "https://llava-vl.github.io/static/images/view.jpg"

args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": prompt,
    "conv_mode": None,
    "image_file": image_file,
    "sep": ",",
    "temperature": 0.2,
    "top_p": 0.8,
    "num_beams": 1,
    "min_new_tokens": None,
    "max_new_tokens": None
})()

displayHTML(f"<img src=\"{image_file}\">")

eval_model(args)
```

動きました！
![Screenshot 2023-11-28 at 18.35.25.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1168882/1992b0e1-c088-7746-460b-2a9df9a9e59e.png)

日本語ではどうでしょう。

```py
prompt = "この場所を訪れた際に注意すべきことはなんですか？"
image_file = "https://llava-vl.github.io/static/images/view.jpg"

args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": prompt,
    "conv_mode": None,
    "image_file": image_file,
    "sep": ",",
    "temperature": 0.2,
    "top_p": 0.8,
    "num_beams": 1,
    "min_new_tokens": None,
    "max_new_tokens": None
})()

displayHTML(f"<img src=\"{image_file}\">")

eval_model(args)
```

す、すごい。
![Screenshot 2023-11-28 at 18.36.24.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1168882/6115afdd-e43c-c137-3e93-0fd0e143cdab.png)

関数化します。

```py
def evaluate_image(image_file, prompt):

  args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": prompt,
    "conv_mode": None,
    "image_file": image_file,
    "sep": ",",
    "temperature": 0.2,
    "top_p": 0.8,
    "num_beams": 1,
    "min_new_tokens": None,
    "max_new_tokens": None
  })()

  displayHTML(f"<img src=\"{image_file}\">")
  eval_model(args)
```

こちらの画像を試します。

https://www.irasutoya.com/2021/01/blog-post_906.html

```py
evaluate_image("https://1.bp.blogspot.com/-pzkUACogq0E/X5OcHr5ZnSI/AAAAAAABb5Q/xb-j2PQXgu03_vypUL1XNOYv4bhpWEFgQCNcBGAsYHQ/s400/bird_mameruriha_inko_blue.png", "何が描かれていますか？")
```

うひー。マルチモーダルすごい。
![Screenshot 2023-11-28 at 18.37.56.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1168882/62399d2b-77cb-97c0-c139-4757e5c65bf3.png)

### Databricksクイックスタートガイド

[Databricksクイックスタートガイド](https://www.amazon.co.jp/dp/B09V1YXFVQ/)


### Databricks無料トライアル

[Databricks無料トライアル](https://databricks.com/jp/try-databricks)