Text model not being loaded with Flash Attention 2

README.md

@@ -18,8 +18,6 @@ datasets:
 - camel-ai/math
 - AtlasUnified/atlas-math-sets
 - tiedong/goat
-- Lin-Chen/ShareGPT4V
-- jxu124/llava_conversation_58k
 language:
 - en
 tags:
@@ -41,7 +39,7 @@ Idefics2 is an open multimodal model that accepts arbitrary sequences of image a
 We release under the Apache 2.0 license 2 checkpoints:
 - [idefics2-8b-base](https://huggingface.co/HuggingFaceM4/idefics2-8b-base): the base model
 - [idefics2-8b](https://huggingface.co/HuggingFaceM4/idefics2-8b): the base model fine-tuned on a mixture of supervised and instruction datasets (text-only and multimodal datasets)
-- [idefics2-8b-chatty](https://huggingface.co/HuggingFaceM4/idefics2-8b-chatty): `idefics2-8b` further fine-tuned on long conservation
+- idefics2-8b-chatty (coming soon): `idefics2-8b` further fine-tuned on long conservation
 
 # Model Summary
 
@@ -53,8 +51,7 @@ We release under the Apache 2.0 license 2 checkpoints:
 - **Resources for more information:**
     - Description of [OBELICS](https://huggingface.co/datasets/HuggingFaceM4/OBELICS): [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
 ](https://huggingface.co/papers/2306.16527)
-    - Paper: [What matters when building vision-language models?
-](https://huggingface.co/papers/2405.02246)
+    - Paper: Coming soon
 
 
 # Uses
@@ -218,39 +215,6 @@ print(generated_texts)
 
 </details>
 
-**Text generation inference**
-
-Idefics2 is integrated into [TGI](https://github.com/huggingface/text-generation-inference) and we host API endpoints for both `idefics2-8b` and `idefics2-8b-chatty`.
-
-Multiple images can be passed on with the markdown syntax (`![](IMAGE_URL)`) and no spaces are required before and after. The dialogue utterances can be separated with `<end_of_utterance>\n` followed by `User:` or `Assistant:`. `User:` is followed by a space if the following characters are real text (no space if followed by an image).
-
-<details><summary>Click to expand.</summary>
-
-```python
-from text_generation import Client
-
-API_TOKEN="<YOUR_API_TOKEN>"
-API_URL = "https://api-inference.huggingface.co/models/HuggingFaceM4/idefics2-8b-chatty"
-
-# System prompt used in the playground for `idefics2-8b-chatty`
-SYSTEM_PROMPT = "System: The following is a conversation between Idefics2, a highly knowledgeable and intelligent visual AI assistant created by Hugging Face, referred to as Assistant, and a human user called User. In the following interactions, User and Assistant will converse in natural language, and Assistant will do its best to answer User’s questions. Assistant has the ability to perceive images and reason about them, but it cannot generate images. Assistant was built to be respectful, polite and inclusive. It knows a lot, and always tells the truth. When prompted with an image, it does not make up facts.<end_of_utterance>\nAssistant: Hello, I'm Idefics2, Huggingface's latest multimodal assistant. How can I help you?<end_of_utterance>\n"
-QUERY = "User:![](https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg)Describe this image.<end_of_utterance>\nAssistant:"
-
-client = Client(
-    base_url=API_URL,
-    headers={"x-use-cache": "0", "Authorization": f"Bearer {API_TOKEN}"},
-)
-generation_args = {
-    "max_new_tokens": 512,
-    "repetition_penalty": 1.1,
-    "do_sample": False,
-}
-generated_text = client.generate(prompt=SYSTEM_PROMPT + QUERY, **generation_args)
-generated_text
-```
-
-</details>
-
 # Model optimizations
 
 If your GPU allows, we first recommend loading (and running inference) in half precision (`torch.float16` or `torch.bfloat16`).
@@ -327,7 +291,7 @@ Fusing can be de-activated by removing `quantization_config` in the call to `fro
 It is also possible to load Idefics2 in 4bits with `bitsandbytes`. To do so, make sure that you have `accelerate` and `bitsandbytes` installed.
 
 ```diff
-+ from transformers import BitsAndBytesConfig
++ from transformer import BitsAndBytesConfig
 
 quantization_config = BitsAndBytesConfig(
     load_in_4bit=True,
@@ -419,27 +383,6 @@ Alongside this evaluation, we also computed the classification accuracy on FairF
 - Despite our efforts in filtering the training data, we found a small proportion of content that is not suitable for all audiences. This includes pornographic content and reports of violent shootings and is prevalent in the OBELICS portion of the data (see [here](https://huggingface.co/datasets/HuggingFaceM4/OBELICS#content-warnings) for more details). As such, the model is susceptible to generating text that resembles this content.
 - We note that we know relatively little about the composition of the pre-trained LM backbone, which makes it difficult to link inherited limitations or problematic behaviors to their data.
 
-**Red-teaming**
-
-In the context of a **[Red-Teaming](https://huggingface.co/blog/red-teaming)**  exercise, our objective was to evaluate the propensity of the model to generate inaccurate, biased, or offensive responses. We evaluated [idefics2-8b-chatty](https://huggingface.co/HuggingFaceM4/idefics2-8b-chatty).
-
-While the model typically refrains from responding to offensive inputs, we observed that through repeated trials or guided interactions, it tends to hastily form judgments in situations necessitating nuanced contextual understanding, often perpetuating harmful stereotypes. Noteworthy instances include:
-- Speculating or passing judgments, or perpetuating historical disparities on individuals' professions, social status, or insurance eligibility based solely on visual cues (e.g., age, attire, gender, facial expressions).
-- Generating content that promotes online harassment or offensive memes reinforcing harmful associations from a portrait, or from a benign image.
-- Assuming emotional states or mental conditions based on outward appearances.
-- Evaluating individuals' attractiveness solely based on their visual appearance.
-
-Additionally, we identified behaviors that increase security risks that already exist:
-- Successfully solving CAPTCHAs featuring distorted text within images.
-- Developing phishing schemes from screenshots of legitimate websites to deceive users into divulging their credentials.
-- Crafting step-by-step guides on constructing small-scale explosives using readily available chemicals from common supermarkets or manipulating firearms to do maximum damage.
-
-It's important to note that these security concerns are currently limited by the model's occasional inability to accurately read text within images.
-
-We emphasize that the model would often encourage the user to exercise caution about the model's generation or flag how problematic the initial query can be in the first place. For instance, when insistently prompted to write a racist comment, the model would answer that query before pointing out "*This type of stereotyping and dehumanization has been used throughout history to justify discrimination and oppression against people of color. By making light of such a serious issue, this meme perpetuates harmful stereotypes and contributes to the ongoing struggle for racial equality and social justice.*". 
-
-However, certain formulations can circumvent (i.e. "jail-break") these cautionary prompts, emphasizing the need for critical thinking and discretion when engaging with the model's outputs. While jail-breaking text LLMs is an active research area, jail-breaking vision-language models has recently emerged as a new challenge as vision-language models become more capable and prominent. The addition of the vision modality not only introduces new avenues for injecting malicious prompts but also raises questions about the interaction between vision and language vulnerabilities.
-
 
 # Misuse and Out-of-scope use
 
@@ -475,17 +418,4 @@ The model is built on top of two pre-trained models: [google/siglip-so400m-patch
       archivePrefix={arXiv},
       primaryClass={cs.IR}
 }
-
-@misc{laurençon2024matters,
-      title={What matters when building vision-language models?}, 
-      author={Hugo Laurençon and Léo Tronchon and Matthieu Cord and Victor Sanh},
-      year={2024},
-      eprint={2405.02246},
-      archivePrefix={arXiv},
-      primaryClass={cs.CV}
-}
 ```
-
-# Acknowledgements
-
-We thank @yjernite, @sasha, @meg, @giadap, @jack-kumar, and @frimelle, who provided help to red-team the model.

processor_config.json

@@ -1,5 +1,4 @@
 {
-  "chat_template": "{% for message in messages %}{{message['role'].capitalize()}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>\n{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}",
   "image_seq_len": 64,
   "processor_class": "Idefics2Processor"
 }