Update README.md

README.md

@@ -1,10 +1,10 @@
 ---
 base_model: unsloth/Qwen2-VL-2B-Instruct
 tags:
-- text-generation-inference
+- text-generation-inference,text-extraction
 - transformers
-- unsloth
-- qwen2_vl
+- unsloth/Qwen2-VL-2B-Instruct-16Bit
+Base Model: unsloth/Qwen2-VL-2B-Instruct-16Bit
 license: apache-2.0
 language:
 - en
@@ -14,8 +14,140 @@ language:
 
 - **Developed by:** JackChew
 - **License:** apache-2.0
-- **Finetuned from model :** unsloth/Qwen2-VL-2B-Instruct
+- **Finetuned from model :** unsloth/Qwen2-VL-2B-Instruct-16Bit
 
 This qwen2_vl model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
 
 [<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+
+
+## Model Description
+**通义千问 QWEN OCR** is a proprietary model focused on text extraction, specifically designed for extracting text from images of documents, tables, and payslips. The primary goal of this model is to extract COMPLETE/FULL text from images while ensuring that no information is missed.
+
+Qwen2-VL-2B-OCR is a fine-tuned variant of unsloth/Qwen2-VL-2B-Instruct, optimized specifically for Optical Character Recognition (OCR). This model is trained to extract full and complete text from images, with a focus on documents such as payslips, invoices, and tables. The model aims to provide accurate text extraction with minimal loss of information, ensuring that every detail is captured.
+This model uses cutting-edge techniques for text-to-text generation from images and works seamlessly for various OCR tasks, including text from complex documents with structured layouts.
+
+## Intended Use
+Intended Use
+The primary purpose of the model is to extract data from images or documents, especially from payslips and tables, without missing any critical details. It can be applied in various domains such as payroll systems, finance, legal document analysis, and any field where document extraction is required.
+Prompt Example:
+- **text**: The model will BEST WORK to this `"Extract all text from image/payslip without miss anything"`.
+
+
+## Model Benchmark
+
+After fine-tuning, this model has significantly improved in extracting all relevant sections from the payslip, including the previously missing **Deductions** section.
+
+### Example Output Comparison
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/676ed40d25c39d8bd5d6f759/KOAZouqb1qH7toZO6YZsO.png)
+
+#### Fine-tuned Model:
+Here is the extracted data from the payslip:
+
+**Employee Information:**
+- Date of Joining: 2018-06-23
+- Pay Period: August 2021
+- Employee Name: Sally Harley
+- Designation: Marketing Executive
+- Department: Marketing
+
+**Earnings:**
+| Earnings         | Amount | Deductions        | Amount |
+|------------------|--------|-------------------|--------|
+| Basic            | 10000  | Provident Fund    | 1200   |
+| Incentive        | 1000   | Professional Tax  | 500    |
+| House Rent       | 400    | Loan              | 400    |
+| Meal Allowance   | 200    |                   | 9500   |
+
+**Total Earnings:** $11,600  
+**Total Deductions:** $2,100  
+**Net Pay:** $9,500
+
+**Employer Signature**  
+**Employee Signature**
+
+---
+#### Original Model:
+The original model extracted the following data but missed the **Deductions** section:
+
+- **Date of Joining**: 2018-06-23
+- **Pay Period**: August 2021
+- **Employee Name**: Sally Harley
+- **Designation**: Marketing Executive
+- **Department**: Marketing
+- **Earnings**:
+  - Basic: $10,000
+  - Incentive Pay: $1,000
+  - House Rent Allowance: $400
+  - Meal Allowance: $200
+- **Total Earnings**: $11,600
+- **Total Deductions**: $2,100
+- **Net Pay**: $9,500
+- **Employer Signature**: [Signature]
+- **Employee Signature**: [Signature]
+- **This is system-generated payslip**
+
+
+## Quick Start
+Here’s an example code snippet to get started with this model:
+
+### Loading the Model and Processor
+```python
+from transformers import AutoProcessor, AutoModelForImageTextToText
+
+processor = AutoProcessor.from_pretrained("JackChew/Qwen2-VL-2B-OCR")
+model = AutoModelForImageTextToText.from_pretrained("JackChew/Qwen2-VL-2B-OCR")
+
+```
+### Loading an Image
+```python
+# Load your image
+from PIL import Image
+image_path = "xxxxx"  # Replace with your image path
+image = Image.open(image_path)
+```
+### Preparing the Model, Preprocessing Inputs, and Performing Inference
+```python
+import requests
+import torch
+from torchvision import io
+from typing import Dict
+
+model = model.to("cuda")
+conversation = [
+    {
+        "role":"user",
+        "content":[
+            {
+                "type":"image",
+            },
+            {
+                "type":"text",
+                "text":"extract all data from this payslip without miss anything"
+            }
+        ]
+    }
+]
+
+
+# Preprocess the inputs
+text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
+# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\n<|im_start|>assistant\n'
+
+inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")
+inputs = inputs.to('cuda')
+
+# Inference: Generation of the output
+output_ids = model.generate(**inputs, max_new_tokens=2048)
+generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
+output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
+print(output_text)
+```
+
+## Model Fine-Tuning Details
+The model was fine-tuned using the Unsloth framework, which accelerated training by 2x using Huggingface's TRL (Training Reinforcement Learning) library. LoRA (Low-Rank Adaptation) was applied to fine-tune only a small subset of the parameters, which significantly reduces training time and computational resources. Fine-tuning focused on both vision and language layers, ensuring that the model could handle complex OCR tasks efficiently.
+
+Total Trainable Parameters: 57,901,056
+