---
license: apache-2.0
tags:
- multi-modal
- language model
- LLaMA
base_model:
- meta-llama/Llama-3.2-11B-Vision
library_name: transformers
pipeline_tag: image-text-to-text
datasets:
- PKU-Alignment/align-anything
widget:
- example_title: Lake Travel
messages:
- role: user
content:
- type: text
text: What are the things I should be cautious about when I visit here?
- type: image_url
image_url:
url: >-
https://huggingface.co/PKU-Alignment/Pony-11B-Vision/resolve/main/example/001.jpg
- role: assistant
content: >
When visiting a place like this, which appears to be a serene lake
with a dock and surrounded by forested mountains, you should be
cautious of several factors:
1. **Weather Conditions**: Be aware of changing weather conditions
such as sudden rainstorms or extreme temperatures, which can make
activities dangerous.
2. **Water Activities**: If engaging in water activities like swimming
or fishing, ensure you're safe around the water, especially if there
are strong currents or deep areas.
3. **Wildlife**: The surrounding forest might have wildlife; stay
informed about potential encounters and know how to react safely.
4. **Injuries on the Dock**: Walking on docks can be slippery, so
watch your step to avoid falls or injuries.
5. **Environmental Protection**: Respect nature by not littering and
staying on designated paths to protect local flora and fauna.
6. **Physical Activity**: Engage in physical activity responsibly,
ensuring you’re prepared for outdoor exertion and have necessary
supplies like food, water, and first aid.
7. **Isolation**: If venturing into remote areas, be mindful of
isolation risks—let someone know your plans and expected return time.
8. **Equipment Check**: When using equipment such as kayaks or camping
gear, ensure everything is in good condition before use.
9. **Local Regulations**: Follow any posted signs or regulations meant
to preserve the environment and enhance safety for visitors.
10. **Emergency Preparedness**: Know basic emergency procedures and
carry essential items like a map, compass, flashlight, and first-aid
kit.By being cautious of these factors, you can enjoy your visit more
safely.
- example_title: Unusual Man
messages:
- role: user
content:
- type: text
text: What is unusual about this image?
- type: image_url
image_url:
url: >-
https://huggingface.co/PKU-Alignment/Pony-11B-Vision/resolve/main/example/002.jpg
- role: assistant
content: >
The unusual aspect of this image is that a person is ironing clothes
on the back of a moving taxi. This is an unconventional and unsafe
activity, as it involves using an electric iron in close proximity to
other vehicles and potentially causing harm or damage. Additionally,
the presence of the ironing board and iron on the side of a speeding
vehicle defies common sense and safety protocols.
- example_title: Mario Themed Mugs
messages:
- role: user
content:
- type: text
text: Write an attractive product description for this.
- type: image_url
image_url:
url: >-
https://huggingface.co/PKU-Alignment/Pony-11B-Vision/resolve/main/example/003.jpg
- role: assistant
content: >
**Classic Mario Themed Mugs**
Step into the world of nostalgia with these vibrant Classic Mario
themed mugs! Featuring the beloved character from the iconic Super
Mario video game series, these mugs are a perfect blend of retro charm
and modern appeal.
- **Vibrant Graphics**: Each mug is adorned with high-quality prints
of the classic Mario character, capturing his playful spirit and
adventurous personality.
- **High-Quality Material**: Made from durable ceramic, these mugs
ensure longevity and resistance to wear and tear, making them ideal
for both daily use and decorative purposes.
- **Perfect Gift**: Whether you're a long-time fan of the Mario
franchise or looking for a fun gift idea, these mugs are sure to
delight gamers of all ages.
- **Versatile Use**: Ideal for coffee, tea, hot chocolate, or even as
a quirky storage container for pens and pencils, these mugs add a
touch of gaming joy to any setting.Bring home a piece of gaming
history with these Classic Mario themed mugs and let your morning
routine be as exciting as rescuing a princess from a towering castle!
---
# 🦫 Beaver-Vision-11B
Beaver-Vision-11B is an Image-Text-to-Text chat assistant trained based on the [LLaMA-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision) (**pretrained version**) using the [Align-Anything-Instruct](https://huggingface.co/datasets/PKU-Alignment/Align-Anything) dataset and [Align-Anything](https://github.com/PKU-Alignment/align-anything) framework.
Beaver-Vision-11B aims to enhance the instruction-following abilities of MLLMs (Multi-modal Large Language Models).
Compared with [LLaMA-3.2-11B-Vision-Instruct](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct), Beaver-Vision-11B uses [Align-Anything-Instruct](https://huggingface.co/datasets/PKU-Alignment/Align-Anything) dataset and post-training alignment method, achieving better performance. More importantly, Beaver-Vision-7B has open-sourced all of its training data, code, and evaluation scripts, providing greater convenience for the community and researchers.
- **Developed by:** the [PKU-Alignment](https://github.com/PKU-Alignment) Team.
- **Model Type:** An auto-regressive multi-modal (Image-Text-to-Text) language model based on the transformer architecture.
- **Fine-tuned from model:** [LLaMA-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision).
## Model Sources
- **Repository:**
- **Dataset:**
## Evaluation Results
| | Metrics | LLaMA-3.2-11B-Vision-Instruct | Beaver-Vision-11B |
|-----------------------------|-------------|-------------------------------|-----------------|
| **LLaVA-Bench-Coco** | Rating | 102.36 | 104.67 |
| **LLaVA-Bench-in-the-Wild** | Rating | 92.76 | 99.13 |
| **POPE** | Accuracy | 80.71 | 87.07 |
| | Recall | 70.95 | 83.02 |
| | Precision | 88.35 | 90.46 |
| | F1 Score | 78.65 | 86.55 |
## Example
### Comparison
|Role|Content|
|---|---|
|User|What's the name of the restaurant serving these dishes?|
|| 
|
|LLaMA-3.2-11B-Vision-Instruct|The dish in the image is from a ramen restaurant called \"Ippudo.\" Ippudo is known for its high-quality, authentic Japanese ramen. The bowl with the distinctive design and rich broth is characteristic of their signature Ramen.|
|GPT-4o|The restaurant serving these dishes is ICHIRAN.|
|Beaver-Vision-11B|The image shows that the restaurant is ICHIRAN. Because it appears to be a ramen dish served in a traditional Japanese-style bowl with various accompaniments such as green onions and possibly some kind of meat or broth flavoring. Additionally, the presence of the ICHIRAN logo on the dishware helps identify the restaurant.|
In this example, LLaMA-3.2-11B-Vision-Instruct failed to recognize the logo text in the bowl and fell into hallucination; GPT-4o recognized accurately but did not provide more understandable information, while Beaver-Vision-11B was able to offer both accurate and detailed answers.
### Further Examples
| 
| 
| 
|
|---|---|---|
| prompt
What are the things I should be cautious about when I visit here?
prompt
What is unusual about this image?
prompt
Write an attractive product description for this.
response
When visiting a place like this, which appears to be a serene lake with a dock and surrounded by forested mountains, you should be cautious of several factors:
1. **Weather Conditions**: Be aware of changing weather conditions such as sudden rainstorms or extreme temperatures, which can make activities dangerous.
2. **Water Activities**: If engaging in water activities like swimming or fishing, ensure you're safe around the water, especially if there are strong currents or deep areas.
3. **Wildlife**: The surrounding forest might have wildlife; stay informed about potential encounters and know how to react safely.
4. **Injuries on the Dock**: Walking on docks can be slippery, so watch your step to avoid falls or injuries.
5. **Environmental Protection**: Respect nature by not littering and staying on designated paths to protect local flora and fauna.
6. **Physical Activity**: Engage in physical activity responsibly, ensuring you’re prepared for outdoor exertion and have necessary supplies like food, water, and first aid.
7. **Isolation**: If venturing into remote areas, be mindful of isolation risks—let someone know your plans and expected return time.
8. **Equipment Check**: When using equipment such as kayaks or camping gear, ensure everything is in good condition before use.
9. **Local Regulations**: Follow any posted signs or regulations meant to preserve the environment and enhance safety for visitors.
10. **Emergency Preparedness**: Know basic emergency procedures and carry essential items like a map, compass, flashlight, and first-aid kit.
By being cautious of these factors, you can enjoy your visit more safely.
response
The unusual aspect of this image is that a person is ironing clothes on the back of a moving taxi. This is an unconventional and unsafe activity, as it involves using an electric iron in close proximity to other vehicles and potentially causing harm or damage. Additionally, the presence of the ironing board and iron on the side of a speeding vehicle defies common sense and safety protocols.
response
**Classic Mario Themed Mugs**
Step into the world of nostalgia with these vibrant Classic Mario themed mugs! Featuring the beloved character from the iconic Super Mario video game series, these mugs are a perfect blend of retro charm and modern appeal.
- **Vibrant Graphics**: Each mug is adorned with high-quality prints of the classic Mario character, capturing his playful spirit and adventurous personality.
- **High-Quality Material**: Made from durable ceramic, these mugs ensure longevity and resistance to wear and tear, making them ideal for both daily use and decorative purposes.
- **Perfect Gift**: Whether you're a long-time fan of the Mario franchise or looking for a fun gift idea, these mugs are sure to delight gamers of all ages.
- **Versatile Use**: Ideal for coffee, tea, hot chocolate, or even as a quirky storage container for pens and pencils, these mugs add a touch of gaming joy to any setting.
Bring home a piece of gaming history with these Classic Mario themed mugs and let your morning routine be as exciting as rescuing a princess from a towering castle!