Update README.md
Browse files
README.md
CHANGED
@@ -19,16 +19,19 @@ llava-Qwen2-7B-Instruct-CLIP-ZH = Qwen/Qwen2-7B-Instruct + multi_modal_projector
|
|
19 |
- vision_tower和language_model的q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj部分进行lora训练</br>
|
20 |
- mmp层全量训练</br>
|
21 |
|
22 |
-
3.
|
23 |
-
lora_r=32,lora_alpha=64,num_train_epochs=5,per_device_train_batch_size=1,gradient_accumulation_steps=8,high_lr=1e-3,low_lr=2e-5,model_max_length=2048
|
|
|
|
|
24 |
|
25 |
-
|
|
|
26 |
使用gemini-1.5-pro, gemini-1.5-flash, yi-vision, gpt4o,claude-3.5-sonnet模型描述emo-visual-data和ChineseBQB数据集。</br>
|
27 |
文本描述信息通过[text-description-of-the-meme](https://huggingface.co/datasets/REILX/text-description-of-the-meme) 下载</br>
|
28 |
图像可通过[emo-visual-data](https://github.com/LLM-Red-Team/emo-visual-data), [ChineseBQB](https://github.com/zhaoolee/ChineseBQB)下载</br>
|
29 |
图片数据总量1.8G,约10835张中文表情包图片。文字总量42Mb,约24332个图像文本对描述信息。
|
30 |
|
31 |
-
|
32 |
![](./images/llava-qwen2-lora-01.JPG)
|
33 |
![](./images/llava-qwen2-lora-02.JPG)
|
34 |
![](./images/llava-qwen2-lora-03.JPG)
|
|
|
19 |
- vision_tower和language_model的q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj部分进行lora训练</br>
|
20 |
- mmp层全量训练</br>
|
21 |
|
22 |
+
3. 微调</br>
|
23 |
+
参数:lora_r=32,lora_alpha=64,num_train_epochs=5,per_device_train_batch_size=1,gradient_accumulation_steps=8,high_lr=1e-3,low_lr=2e-5,model_max_length=2048.</br>
|
24 |
+
设备:8*A800</br>
|
25 |
+
训练时长:5小时12分钟
|
26 |
|
27 |
+
|
28 |
+
5. 数据集</br>
|
29 |
使用gemini-1.5-pro, gemini-1.5-flash, yi-vision, gpt4o,claude-3.5-sonnet模型描述emo-visual-data和ChineseBQB数据集。</br>
|
30 |
文本描述信息通过[text-description-of-the-meme](https://huggingface.co/datasets/REILX/text-description-of-the-meme) 下载</br>
|
31 |
图像可通过[emo-visual-data](https://github.com/LLM-Red-Team/emo-visual-data), [ChineseBQB](https://github.com/zhaoolee/ChineseBQB)下载</br>
|
32 |
图片数据总量1.8G,约10835张中文表情包图片。文字总量42Mb,约24332个图像文本对描述信息。
|
33 |
|
34 |
+
6. 效果展示
|
35 |
![](./images/llava-qwen2-lora-01.JPG)
|
36 |
![](./images/llava-qwen2-lora-02.JPG)
|
37 |
![](./images/llava-qwen2-lora-03.JPG)
|