StarCycle commited on
Commit
552f007
β€’
1 Parent(s): d1dcf2d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +241 -0
README.md CHANGED
@@ -1,3 +1,244 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ datasets:
4
+ - liuhaotian/LLaVA-Pretrain
5
+ - liuhaotian/LLaVA-Instruct-150K
6
+ pipeline_tag: visual-question-answering
7
  ---
8
+
9
+ ## Model
10
+ llava-clip-internlm2-1_8b-pretrain-v1 is a LLaVA checkpoint finetuned from [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) and [CLIP-ViT-Large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) with [LLaVA-Pretrain](liuhaotian/LLaVA-Pretrain) and [LLaVA-Instruct-150K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) by [Xtuner](https://github.com/InternLM/xtuner). The pretraining phase took 16 hours on a single Nvidia A6000 ada GPU (see this [intermediate checkpoint](https://huggingface.co/StarCycle/llava-clip-internlm2-1_8b-pretrain-v1/)). The finetuning phase took 16 hours on 4 Nvidia A6000 GPU.
11
+
12
+ The total size of the model is around 2.2B, which is suitable for embedded applications like robotics.
13
+
14
+ I have not carefully tune the hyperparameters during training. If you have any idea to improve it, please open an issue or just send an email to zhuohengli@foxmail.com. You are welcomed!
15
+
16
+ ## Example
17
+ ![5bb2f23dd595d389e6a9a0aadebd87c.png](https://cdn-uploads.huggingface.co/production/uploads/642a298ae5f33939cf3ee600/iOFZOwLGfEByCQ_2EkR7y.png)
18
+ Explain the photo in English and Chinese:
19
+ ![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/642a298ae5f33939cf3ee600/mbrdKMw-lAHRt2dFHWoGW.jpeg)
20
+
21
+ ## Rank
22
+ In submission...
23
+
24
+ ## Results
25
+ Model | MMBench Test (EN) | MMBench Dev (EN) | MMBench Test (CN) | MMBench Dev (CN) | CCBench Dev
26
+ ------------- | ------------- | ------------- | ------------- | ------------- | -------------
27
+ LLaVA-v1.5-7B | 67.7 | 69.2 | 61.0 | 59.7 | 28.4
28
+ LLaVA-InternLM-7B | 69.0 | 68.5 | 66.7 | 63.8 | 37.3
29
+ LLaVA-InternLM2-7B | 73.3 | 74.6 | 71.7 | 72.0 | 42.5
30
+ Bunny-3B | 69.2 | 68.6 | - | - | -
31
+ MiniCPM-V | 64.1 | 67.9 | 62.6 | 65.3 | 41.4
32
+ llava-clip-internlm2-1_8b-v1 | 63.3 | 63.1 | 63.6 | 61.7 | 35.3
33
+
34
+ ## Installation
35
+ ```
36
+ git clone https://github.com/InternLM/xtuner
37
+ pip install -e ./xtuner[deepspeed]
38
+ apt install git-lfs
39
+ cd ./xtuner
40
+ # Now replace the source code files with the modifed version in modified_xtuner_code directory
41
+ ```
42
+
43
+ ## Chat
44
+ ```
45
+ xtuner chat internlm/internlm2-chat-1_8b \
46
+ --visual-encoder openai/clip-vit-large-patch14-336 \
47
+ --llava ./lora_and_projector \
48
+ --prompt-template internlm2_chat \
49
+ --image $IMAGE_PATH
50
+ ```
51
+
52
+ ## Common Errors
53
+ 1.
54
+ ```
55
+ command error: 'libGL.so.1: cannot open shared object file: No such file or directory'!
56
+ ```
57
+ You can solve it by
58
+ ```
59
+ # For Ubuntu
60
+ sudo apt-get update
61
+ sudo apt-get install libgl1-mesa-glx
62
+
63
+ # For CentOS and Fedora
64
+ sudo yum install mesa-libGL
65
+ ```
66
+
67
+ 2.
68
+ ```
69
+ Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
70
+ Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
71
+ ```
72
+ You can solve it by reinstall numpy.
73
+
74
+ 3.
75
+ ```
76
+ ImportError:
77
+ InternLM2Converter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
78
+ ```
79
+ You just need
80
+ ```
81
+ pip install protobuf
82
+ ```
83
+ 4.
84
+ To use tensorboard to visualize the training loss curve:
85
+ ```
86
+ pip install future tensorboard
87
+ ```
88
+
89
+ 5. If your training process is killed during data preprocessing, you can modify the `map_num_proc` in xtuner/xtuner/dataset
90
+ /huggingface.py
91
+ ```
92
+ def process(dataset,
93
+ do_dataset_tokenization=True,
94
+ tokenizer=None,
95
+ max_length=None,
96
+ dataset_map_fn=None,
97
+ template_map_fn=None,
98
+ max_dataset_length=None,
99
+ split='train',
100
+ remove_unused_columns=False,
101
+ rename_maps=[],
102
+ shuffle_before_pack=True,
103
+ pack_to_max_length=True,
104
+ use_varlen_attn=False,
105
+ input_ids_with_output=True,
106
+ with_image_token=False,
107
+ map_num_proc=32): # modify it to a smaller number, e.g., 4
108
+ ```
109
+
110
+ 6. If you fail to load the model, check whether you installed git-lfs and actually downloaded the model file.
111
+
112
+ ## Data prepration
113
+ 1. File structure
114
+
115
+ ```
116
+ # . means the llava-dinov2-internlm2-7b-v1 folder you clone
117
+ ./data/llava_data
118
+ β”œβ”€β”€ LLaVA-Pretrain
119
+ β”‚Β Β  β”œβ”€β”€ blip_laion_cc_sbu_558k.json
120
+ β”‚Β Β  β”œβ”€β”€ blip_laion_cc_sbu_558k_meta.json
121
+ β”‚Β Β  └── images
122
+ β”œβ”€β”€ LLaVA-Instruct-150K
123
+ β”‚Β Β  └── llava_v1_5_mix665k.json
124
+ └── llava_images
125
+ Β Β  β”œβ”€β”€ coco
126
+ Β Β  β”‚ └── train2017
127
+ Β Β  β”œβ”€β”€ gqa
128
+ Β Β  β”‚ └── images
129
+ Β Β  β”œβ”€β”€ ocr_vqa
130
+ Β Β  β”‚ └── images
131
+ Β Β  β”œβ”€β”€ textvqa
132
+ Β Β  β”‚ └── train_images
133
+ Β Β  └── vg
134
+ Β Β  Β Β  β”œβ”€β”€ VG_100K
135
+ Β Β  └── VG_100K_2
136
+ ```
137
+
138
+ 2. Pretrain Data
139
+
140
+ LLaVA-Pretrain
141
+
142
+ ```shell
143
+ # Make sure you have git-lfs installed (https://git-lfs.com)
144
+ git lfs install
145
+ git clone https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain --depth=1
146
+ ```
147
+
148
+ 3. Finetune Data
149
+
150
+ 3.1 Text data
151
+
152
+ LLaVA-Instruct-150K
153
+
154
+ ```shell
155
+ # Make sure you have git-lfs installed (https://git-lfs.com)
156
+ git lfs install
157
+ git clone https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K --depth=1
158
+ ```
159
+
160
+ 3.2 Image data
161
+
162
+ 3.2.1 COCO (coco): [train2017](http://images.cocodataset.org/zips/train2017.zip)
163
+
164
+ 3.2.2 GQA (gqa): [images](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)
165
+
166
+ 3.2.3 OCR-VQA (ocr_vqa): [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing)
167
+
168
+ ⚠️⚠️⚠️ Modify the name of OCR-VQA's images to keep the extension as `.jpg`!
169
+
170
+ ```shell
171
+ #!/bin/bash
172
+ ocr_vqa_path="<your-directory-path>"
173
+
174
+ find "$target_dir" -type f | while read file; do
175
+ extension="${file##*.}"
176
+ if [ "$extension" != "jpg" ]
177
+ then
178
+ cp -- "$file" "${file%.*}.jpg"
179
+ fi
180
+ done
181
+ ```
182
+
183
+ 3.2.4 TextVQA (textvqa): [train_val_images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip)
184
+
185
+ 3.2.5 VisualGenome (VG): [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip)
186
+
187
+ ## Cheers! Now train your own model!
188
+ 1. Alignment module pretraining
189
+ ```
190
+ # single GPU
191
+ xtuner train ./llava_internlm2_chat_1_8b_clip_vit_large_p14_336_e1_gpu1_pretrain.py --deepspeed deepspeed_zero2
192
+
193
+ # multiple GPU
194
+ NPROC_PER_NODE=4 xtuner train ./llava_internlm2_chat_1_8b_clip_vit_large_p14_336_e1_gpu1_pretrain.py --deepspeed deepspeed_zero2
195
+ ```
196
+
197
+ #### Remember to change the batch size and gradient accumulation parameters to fit your hardware. So your GPU_num * batch_size * gradient_accumulation is roughly equal to mine to reproduce the result.
198
+
199
+ The checkpoint and tensorboard logs are saved by default in ./work_dirs/. I only train it for 1 epoch to be same as the original LLaVA paper. Some researches also report that training for multiple epochs will make the model overfit the training dataset and perform worse in other domains.
200
+
201
+ This is my loss curve for llava-clip-internlm2-1_8b-pretrain-v1:
202
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/642a298ae5f33939cf3ee600/iNxPxfOvSJq8ZPz8uP_sP.png)
203
+
204
+ And the learning rate curve:
205
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/642a298ae5f33939cf3ee600/U1U9Kapcd6AIEUySvt2RS.png)
206
+
207
+ 2. Instruction following fine-tuning
208
+ ```
209
+ NPROC_PER_NODE=4 xtuner train ./llava_internlm2_chat_1_8b_clip_vit_large_p14_336_e1_gpu4_finetune.py --deepspeed deepspeed_zero2
210
+ ```
211
+ Here is my loss curve (the curve fluctuates strongly because the batch size is small, and I only record batch loss instead of epoch loss):
212
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/642a298ae5f33939cf3ee600/kby2Y1dixeTaALliZ4pJa.png)
213
+
214
+ And the learning rate curve:
215
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/642a298ae5f33939cf3ee600/7ue98bikCOU7ub2jEHrom.png)
216
+
217
+ ## Transfer the checkpoints to Huggingface safetensor format
218
+ ```
219
+ xtuner convert pth_to_hf ./llava_internlm2_chat_1_8b_clip_vit_large_p14_336_e1_gpu4_finetune.py ./work_dirs/iter_xxx.pth ./my_lora_and_projector
220
+ ```
221
+ The adapter still need to be used with the internlm/internlm2-chat-7b and facebook/dinov2-large models. I have not tried to merge them yet but it is possible with Xtuner, see this [tutorial](https://github.com/InternLM/xtuner/blob/f63859b3d0cb39cbac709e3850f3fe01de1023aa/xtuner/configs/llava/README.md#L4).
222
+
223
+ ## MMBench Evaluation
224
+ You can first download the MMBench data:
225
+ ```
226
+ wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_DEV_EN.tsv
227
+ wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_TEST_EN.tsv
228
+ wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_DEV_CN.tsv
229
+ wget https://opencompass.openxlab.space/utils/VLMEval/MMBench_TEST_CN.tsv
230
+ wget https://opencompass.openxlab.space/utils/VLMEval/CCBench.tsv
231
+ ```
232
+ Then run:
233
+ ```
234
+ NPROC_PER_NODE=8 xtuner mmbench internlm/internlm2-chat-7b \
235
+ --visual-encoder facebook/dinov2-large \
236
+ --llava ./my_lora_and_projector \
237
+ --prompt-template internlm2_chat \
238
+ --data-path $MMBENCH_DATA_PATH \
239
+ --work-dir $RESULT_PATH
240
+ ```
241
+ You can also use [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) to evaluate it on other benckmarks.
242
+
243
+ ## Deployment
244
+ Xtuner team is developing HF chatbot (based on Huggingface transformers) and LMDeploy chatbot (based on TurboMind). I am waiting for their final version of API.