update v1

README.md

@@ -5,9 +5,9 @@ pipeline_tag: visual-question-answering
 # 😈Imp
 
 
-The Imp project aims to provide a family of  a strong multimodal `small` language models (MSLMs). Our `imp-v0-3b` is a strong MSLM with only **3B** parameters, which is build upon a small yet powerful SLM [Phi-2 ](https://huggingface.co/microsoft/phi-2)(2.7B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set.  
+The Imp project aims to provide a family of  a strong multimodal `small` language models (MSLMs). Our `imp-v1-3b` is a strong MSLM with only **3B** parameters, which is build upon a small yet powerful SLM [Phi-2 ](https://huggingface.co/microsoft/phi-2)(2.7B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set.  
 
-As shown in the Table below, `imp-v0-3b` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks. 
+As shown in the Table below, `imp-v1-3b` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks. 
 
 We release our model weights and provide an example below to run our model . Detailed technical report and corresponding training/evaluation code will be released soon on our [GitHub repo](https://github.com/MILVLG/imp). We will persistently improve our model and release the next versions to further improve model performance :) 
 
@@ -25,11 +25,11 @@ torch.set_default_device("cuda")
 
 #Create model
 model = AutoModelForCausalLM.from_pretrained(
-    "MILVLG/imp-v0-3b", 
+    "MILVLG/imp-v1-3b", 
     torch_dtype=torch.float16, 
     device_map="auto",
     trust_remote_code=True)
-tokenizer = AutoTokenizer.from_pretrained("MILVLG/imp-v0-3b", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("MILVLG/imp-v1-3b", trust_remote_code=True)
 
 #Set inputs
 text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:"
@@ -57,9 +57,10 @@ We conduct evaluation on 9 commonly-used benchmarks, including 5 academic VQA be
 | [LLaVA-Phi](https://github.com/zhuyiche/llava-phi) | 3B | 71.40  | - | 35.90 |    68.40   |    48.60  | 85.00 | 1335.1 | 59.80 |28.9|
 | [MobileVLM](https://huggingface.co/mtgv/MobileVLM-3B) | 3B | - | 59.00  | - |    61.00   |    47.50   | 84.90 | 1288.9 | 59.60  |-|
 | [MC-LLaVA-3b](https://huggingface.co/visheratin/MC-LLaVA-3b) | 3B | 64.24 | 49.60  | 24.88 |    -   |    38.59   | 80.59 | - | -  |-|
-| **Imp-v0 (ours)** | 3B | **79.45**  | 58.55 | **50.09** |**69.96**| **59.38** | **88.02**| 1434.0 | **66.49**  |**33.1**|
+| **Imp-v1 (ours)** | 3B | **79.45**  | 58.55 | **50.09** |**69.96**| **59.38** | **88.02**| 1434.0 | **66.49**  |**33.1**|
 
 ### Examples
+
 ![example1](images/example1.png)
 
 ## License

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "MILVLG/Imp-v0-3b",
+  "_name_or_path": "MILVLG/Imp-v1-3b",
   "activation_function": "gelu_new",
   "architectures": [
     "ImpForCausalLM"

configuration_imp.py

@@ -176,7 +176,7 @@ class ImpConfig(PhiConfig):
     @property
     def vision_tower_cfg(self):
         cfg = SiglipVisionConfig.from_dict(self.vision_tower_config)
-        # imp-v0 only supports `patch` feature for now w/o cls token
+        # imp-v1 only supports `patch` feature for now w/o cls token
         # cfg.mm_vision_select_feature = self.mm_vision_select_feature
         cfg.mm_vision_select_layer = self.mm_vision_select_layer
         cfg.mm_vision_tower = self.mm_vision_tower

requirements.txt

@@ -1,4 +0,0 @@
-torch>=2.0.1
-transformers>=4.31.0
-ftfy>=6.1.3
-timm>=0.9.12