Upload folder using huggingface_hub

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 text_encoder/gemma/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

model_index.json

@@ -33,5 +33,6 @@
     "diffusers",
     "LTX2Vocoder"
   ],
-  "fastvideo_refine_lora_path": "FastVideo/LTX2-Distilled-LoRA"
+  "fastvideo_refine_lora_path": "FastVideo/LTX2-Distilled-LoRA",
+  "gemma_model_path": "text_encoder/gemma"
 }

text_encoder/gemma/.gitattributes

@@ -33,10 +33,10 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
-tokenizer.model filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
 model-00001-of-00005.safetensors filter=lfs diff=lfs merge=lfs -text
 model-00002-of-00005.safetensors filter=lfs diff=lfs merge=lfs -text
 model-00003-of-00005.safetensors filter=lfs diff=lfs merge=lfs -text
 model-00004-of-00005.safetensors filter=lfs diff=lfs merge=lfs -text
 model-00005-of-00005.safetensors filter=lfs diff=lfs merge=lfs -text
-tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.model filter=lfs diff=lfs merge=lfs -text

text_encoder/gemma/README.md

@@ -1,33 +1,19 @@
 ---
-base_model: google/gemma-3-12b-it
 license: gemma
-tags:
-- gemma3
-- gemma
-- google
-pipeline_tag: image-text-to-text
 library_name: transformers
+pipeline_tag: image-text-to-text
 extra_gated_heading: Access Gemma on Hugging Face
-extra_gated_prompt: >-
-  To access Gemma on Hugging Face, you’re required to review and agree to
-  Google’s usage license. To do this, please ensure you’re logged in to Hugging
+extra_gated_prompt: To access Gemma on Hugging Face, you’re required to review and
+  agree to Google’s usage license. To do this, please ensure you’re logged in to Hugging
   Face and click below. Requests are processed immediately.
 extra_gated_button_content: Acknowledge license
+base_model: google/gemma-3-12b-pt
 ---
 
 # Gemma 3 model card
 
 **Model Page**: [Gemma](https://ai.google.dev/gemma/docs/core)
 
-> [!Note]
-> This repository corresponds to the 12B **instruction-tuned** version of the Gemma 3 model using Quantization Aware Training (QAT).
->
-> **The checkpoint in this repository is unquantized, please make sure to quantize with Q4_0 with your favorite tool**
-> 
-> Thanks to QAT, the model is able to preserve similar quality as `bfloat16` while significantly reducing the memory requirements
-> to load the model.
-
-
 **Resources and Technical Documentation**:
 
 * [Gemma 3 Technical Report][g3-tech-report]
@@ -72,6 +58,107 @@ for everyone.
         question, analysis of image content, or a summary of a document
     -   Total output context of 8192 tokens
 
+### Usage
+
+Below, there are some code snippets on how to get quickly started with running the model. First, install the Transformers library. Gemma 3 is supported starting from transformers 4.50.0. 
+
+```sh
+$ pip install -U transformers
+```
+
+Then, copy the snippet from the section that is relevant for your use case.
+
+#### Running with the `pipeline` API
+
+You can initialize the model and processor for inference with `pipeline` as follows.
+
+```python
+from transformers import pipeline
+import torch
+
+pipe = pipeline(
+    "image-text-to-text",
+    model="google/gemma-3-12b-it",
+    device="cuda",
+    torch_dtype=torch.bfloat16
+)
+```
+
+With instruction-tuned models, you need to use chat templates to process our inputs first. Then, you can pass it to the pipeline.
+
+```python
+messages = [
+    {
+        "role": "system",
+        "content": [{"type": "text", "text": "You are a helpful assistant."}]
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
+            {"type": "text", "text": "What animal is on the candy?"}
+        ]
+    }
+]
+
+output = pipe(text=messages, max_new_tokens=200)
+print(output[0]["generated_text"][-1]["content"])
+# Okay, let's take a look! 
+# Based on the image, the animal on the candy is a **turtle**. 
+# You can see the shell shape and the head and legs.
+```
+
+#### Running the model on a single / multi GPU
+
+```python
+# pip install accelerate
+
+from transformers import AutoProcessor, Gemma3ForConditionalGeneration
+from PIL import Image
+import requests
+import torch
+
+model_id = "google/gemma-3-12b-it"
+
+model = Gemma3ForConditionalGeneration.from_pretrained(
+    model_id, device_map="auto"
+).eval()
+
+processor = AutoProcessor.from_pretrained(model_id)
+
+messages = [
+    {
+        "role": "system",
+        "content": [{"type": "text", "text": "You are a helpful assistant."}]
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
+            {"type": "text", "text": "Describe this image in detail."}
+        ]
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt"
+).to(model.device, dtype=torch.bfloat16)
+
+input_len = inputs["input_ids"].shape[-1]
+
+with torch.inference_mode():
+    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
+    generation = generation[0][input_len:]
+
+decoded = processor.decode(generation, skip_special_tokens=True)
+print(decoded)
+
+# **Overall Impression:** The image is a close-up shot of a vibrant garden scene, 
+# focusing on a cluster of pink cosmos flowers and a busy bumblebee. 
+# It has a slightly soft, natural feel, likely captured in daylight.
+```
+
 ### Citation
 
 ```none
@@ -170,10 +257,6 @@ development workflow."*
 
 ## Evaluation
 
-> [!Note]
-> The evaluation in this section correspond to the original checkpoint, not the QAT checkpoint.
->
-
 Model evaluation metrics and results.
 
 ### Benchmark Results

text_encoder/gemma/config.json

@@ -13,50 +13,28 @@
   "mm_tokens_per_image": 256,
   "model_type": "gemma3",
   "text_config": {
-    "attention_bias": false,
-    "attention_dropout": 0.0,
-    "attn_logit_softcapping": null,
-    "cache_implementation": "hybrid",
-    "final_logit_softcapping": null,
-    "head_dim": 256,
-    "hidden_activation": "gelu_pytorch_tanh",
     "hidden_size": 3840,
-    "initializer_range": 0.02,
     "intermediate_size": 15360,
-    "max_position_embeddings": 131072,
     "model_type": "gemma3_text",
     "num_attention_heads": 16,
     "num_hidden_layers": 48,
     "num_key_value_heads": 8,
-    "query_pre_attn_scalar": 256,
-    "rms_norm_eps": 1e-06,
-    "rope_local_base_freq": 10000,
     "rope_scaling": {
       "factor": 8.0,
       "rope_type": "linear"
     },
-    "rope_theta": 1000000,
-    "sliding_window": 1024,
-    "sliding_window_pattern": 6,
-    "torch_dtype": "bfloat16",
-    "use_cache": true,
-    "vocab_size": 262208
+    "sliding_window": 1024
   },
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.52.0.dev0",
+  "transformers_version": "4.50.0.dev0",
   "vision_config": {
-    "attention_dropout": 0.0,
-    "hidden_act": "gelu_pytorch_tanh",
     "hidden_size": 1152,
     "image_size": 896,
     "intermediate_size": 4304,
-    "layer_norm_eps": 1e-06,
     "model_type": "siglip_vision_model",
     "num_attention_heads": 16,
-    "num_channels": 3,
     "num_hidden_layers": 27,
     "patch_size": 14,
-    "torch_dtype": "bfloat16",
     "vision_use_head": false
   }
 }

text_encoder/gemma/generation_config.json

@@ -1,11 +1,13 @@
 {
+  "bos_token_id": 2,
   "cache_implementation": "hybrid",
   "do_sample": true,
   "eos_token_id": [
     1,
     106
   ],
+  "pad_token_id": 0,
   "top_k": 64,
   "top_p": 0.95,
-  "transformers_version": "4.52.0.dev0"
+  "transformers_version": "4.50.0.dev0"
 }

text_encoder/gemma/model-00001-of-00005.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e6fb899db428481aafb45a20130457df6e247e7cb03b7d9f01ee4bc2a9a08138
+oid sha256:4847447e92599833e8dbaa3067cd201c3bb5c052efa91f11ba891e43234f7832
 size 4979902192

text_encoder/gemma/model-00002-of-00005.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d251e7fe9799d529405ddb61705a44cd700bd30a8b66a8d44ae26ddf8365dbc6
+oid sha256:891bd54eed03cba9ee1e705533a02a8217fcc29f356e4a1f53e5fd0d178883ad
 size 4931296592

text_encoder/gemma/model-00003-of-00005.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0684ef801385f0669a0b3e4ab160c50877efdbfa40eb97788595985de2743e78
+oid sha256:7cee411d9d57324e50ce064a192cc5a858276d508611b12fc599e0c9767112e0
 size 4931296656

text_encoder/gemma/model-00004-of-00005.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4b964e6526f81ccfa625c900b72ce92d5e0fd2debb75998763038ad06b9c541
+oid sha256:8bc75a29a730c9e743cad013feda3b0991a913fafe787c58a1c6e20afad97723
 size 4931296656

text_encoder/gemma/model-00005-of-00005.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4ef2de8f93e165b4e02425769fc566000b0674256ef0c3a27b23a0d45eb12088
+oid sha256:ed14bd4908c98fed9f61e8cd410167e0846de9abd78e0452ab092072e5d9252d
 size 4601000928

text_encoder/gemma/tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7d4046bf0505a327dd5a0abbb427ecd4fc82f99c2ceaa170bc61ecde12809b0c
-size 33384570
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568

text_encoder/gemma/tokenizer_config.json

@@ -2160,7 +2160,7 @@
       "normalized": false,
       "rstrip": false,
       "single_word": false,
-      "special": false
+      "special": true
     },
     "256000": {
       "content": "<end_of_image>",
@@ -2168,7 +2168,7 @@
       "normalized": false,
       "rstrip": false,
       "single_word": false,
-      "special": false
+      "special": true
     },
     "256001": {
       "content": "<unused99>",

tokenizer/added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<image_soft_token>": 262144
+}

tokenizer/special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568

tokenizer/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074