Initial import of Qari-OCR files

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,197 @@
+---
+base_model:
+- unsloth/Qwen2-VL-2B-Instruct-unsloth-bnb-4bit
+language:
+- ar
+library_name: peft
+license: apache-2.0
+metrics:
+- bleu
+- wer
+- cer
+pipeline_tag: image-text-to-text
+tags:
+- transformers
+- unsloth
+- qwen2_vl
+- trl
+- ocr
+---
+
+# Qari-OCR-Arabic-0.2.2.1-VL-2B-Instruct Model
+
+This is the model described in the paper [QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation](https://huggingface.co/papers/2506.02295).
+
+## Model Overview
+
+This model is a fine-tuned version of [unsloth/Qwen2-VL-2B-Instruct](https://huggingface.co/unsloth/Qwen2-VL-2B-Instruct-unsloth-bnb-4bit) on an Arabic OCR dataset. It is optimized to perform Arabic Optical Character Recognition (OCR) for full-page text.
+
+Code can be found at this url .
+
+## Key Features
+
+- **Superior Accuracy**: Achieves state-of-the-art performance metrics for Arabic OCR
+- **Diacritics Support**: Full recognition of Arabic diacritical marks (tashkeel) including fatḥah, kasrah, ḍammah, sukūn, shadda, and tanwin forms - a strength confirmed by evaluation on a primarily diacritical text dataset
+- **Multiple Font Support**: Works across a variety of Arabic font styles
+- **Layout Flexibility**: Handles different document layouts and formats
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/628f7a71dd993507cfcbe587/HuUcfziXcDT_2kwDoz5qH.png)
+
+## Model Details
+- **Base Model**: Qwen2 VL
+- **Fine-tuning Dataset**: Arabic OCR dataset
+- **Objective**: Extract full-page Arabic text with high accuracy
+- **Languages**: Arabic
+- **Tasks**: OCR (Optical Character Recognition)
+- **Dataset size**: 50,000 records
+- **Epochs**: 1
+
+## Evaluation Metrics
+Performance is evaluated using three standard metrics:
+- **Word Error Rate (WER)**: Measures word-level accuracy (lower is better)
+- **Character Error Rate (CER)**: Measures character-level accuracy (lower is better)
+- **BLEU Score**: Measures overall translation quality (higher is better)
+
+### Results
+
+| Model | WER ↓ | CER ↓ | BLEU ↑ |
+|-------|-------|-------|--------|
+| **Qari-OCR-0.2.2.1-VL-2B-Instruct** | **0.221** | **0.059** | **0.597** |
+| AIN 8B | 0.757 | 0.309 | 0.103 |
+| Qari-OCR-0.1-VL-2B-Instruct | 1.294 | 0.770 | 0.022 |
+| easyOCR | 1.004 | 0.648 | 0.005 |
+| pytesseract | 0.990 | 0.911 | <0.001 |
+
+### WER Comparison
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/630535e0c7fed54edfaa1a75/Artnw-bVJuSaO_vnLeupE.png" height="600px"/>
+
+### CER Comparison
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/630535e0c7fed54edfaa1a75/GihjVBk32SCyFCpJ81AEX.png" height="600px"/>
+
+### BLEU Score Comparison
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/630535e0c7fed54edfaa1a75/HOOlFw5l_Os3dyyKmTXUs.png" height="600px"/>
+
+## Model Details
+
+### Training Data
+The model was trained using the following specifications:
+
+- **Font Sizes**: 14, 16, 18, 20, 24, 32, 40 pt
+- **Page Layouts**:
+  - A4 (210mm × 297mm)
+  - Letter (216mm × 279mm)
+  - Small (105mm × 148mm)
+  - Square (1080px × 1080px)
+  - OneLine (210mm × 10mm)
+- **Arabic Fonts Used**:
+  - IBM Plex Sans Arabic
+  - KFGQPCUthman Taha Naskh
+  - Scheherazade New
+  - Amiri
+  - Madina
+  - Diwani Letter
+  - Tajawal
+  - Cairo
+  - Lateef
+  - Almarai
+  - AlQalam Quran
+  - Noto Naskh Arabic
+
+### Limitations
+Based on the training specifications, the model has the following limitations:
+
+1. **Font Size Constraints**: May have reduced accuracy with very small (< 14pt) or very large (> 40pt) text
+2. **Font Coverage**: Performance may degrade on uncommon Arabic fonts not represented in the training data
+3. **Diacritics Complexity**: While the model supports diacritics (tashkeel), extremely dense or unconventional diacritical mark combinations may reduce accuracy
+4. **Layout Sensitivity**: May have difficulty with complex multi-column layouts or unconventional page formats
+4. **Handwriting Recognition**: Limited capability with handwritten text as training focused on digital fonts
+5. **Decorative Text**: May struggle with highly stylized or decorative Arabic calligraphy
+6. **Background Complexity**: Optimized for clear backgrounds; performance may degrade with complex or textured backgrounds
+7. **Text Degradation**: May have challenges with severely degraded, blurry, or low-resolution text
+8. **Non-standard Orientations**: Primarily designed for horizontally oriented text; may struggle with vertical or diagonal text
+
+### Evaluation Method
+Evaluation was performed on a diverse dataset of Arabic text images, **primarily featuring diacritical marks (tashkeel)**, measuring:
+- **Word Error Rate (WER)**: The percentage of incorrectly recognized words
+- **Character Error Rate (CER)**: The percentage of incorrectly recognized characters
+- **BLEU Score**: A measure of translation quality, higher scores indicate better overall text recognition
+
+## How to Use
+
+[Try Qari v0.2.2.1 - Google Colab](https://colab.research.google.com/github/NAMAA-ORG/public-notebooks/blob/main/Qari_V0_2_2_1_Free_Colab_updated.ipynb)
+
+You can load this model using the `transformers` and `qwen_vl_utils` library:
+```
+!pip install transformers qwen_vl_utils accelerate>=0.26.0 PEFT -U
+!pip install -U bitsandbytes
+```
+
+```python
+from PIL import Image
+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
+import torch
+import os
+from qwen_vl_utils import process_vision_info
+
+model_name = "NAMAA-Space/Qari-OCR-0.2.2.1-Arabic-2B-Instruct"
+model = Qwen2VLForConditionalGeneration.from_pretrained(
+                model_name,
+                torch_dtype="auto",
+                device_map="auto"
+            )
+processor = AutoProcessor.from_pretrained(model_name)
+max_tokens = 2000
+
+prompt = "Below is the image of one page of a document, as well as some raw textual content that was previously extracted for it. Just return the plain text representation of this document as if you were reading it naturally. Do not hallucinate."
+image.save("image.png")
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": f"file://{src}"},
+            {"type": "text", "text": prompt},
+        ],
+    }
+]
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda")
+generated_ids = model.generate(**inputs, max_new_tokens=max_tokens)
+generated_ids_trimmed = [
+    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)[0]
+os.remove(src)
+print(output_text)
+```
+
+## License
+This model follows the licensing terms of the original Qwen2 VL model. Please review the terms before using it commercially.
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+@article{wasfy2025qari,
+  title={QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation},
+  author={Wasfy, Ahmed and Nacar, Omer and Elkhateb, Abdelakreem and Reda, Mahmoud and Elshehy, Omar and Ammar, Adel and Boulila, Wadii},
+  journal={arXiv preprint arXiv:2506.02295},
+  year={2025}
+}
+```

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/qwen2-vl-2b-instruct-unsloth-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:vision|image|visual|patch|language|text).*?(?:self_attn|attention|attn|mlp|feed_forward|ffn|dense).*?(?:qkv|proj|fc1|fc2|q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj).*?)|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mlp|feed_forward|ffn|dense)\\.(?:(?:qkv|proj|fc1|fc2|q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)))",
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48c56052788cf3568708553918584d4a6e0c264dd860ddb8949b27d0953987e2
+size 115886968

added_tokens.json

@@ -0,0 +1,16 @@
+{
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "Qwen2VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 12845056,
+    "shortest_edge": 3136
+  },
+  "temporal_patch_size": 2
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|vision_pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:948c45c29a91dd2e6ae77d6f5a324a3d408bcca6ad443365b2e79986f1422771
+size 11420540

tokenizer_config.json

@@ -0,0 +1,145 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|vision_pad|>",
+  "padding_side": "right",
+  "processor_class": "Qwen2VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17fbfb7a9acefafd4f5ff7e2dd0a83c6914db946c80d6d26cf93b12237853b17
+size 5624