README.md

---
language:
- fa
- en
license: mit
library_name: transformers
tags:
- Tokenizer
---

# Improved LLaMA 2 Tokenizer with Persian Language Support

## Model Description

This tokenizer is an improved version of the LLaMA 2 tokenizer, specifically enhanced to provide better support for the Persian language. It combines the original LLaMA 2 tokenizer with a custom tokenizer trained on the Persian Wikipedia corpus, resulting in improved tokenization for Persian text while maintaining support for other languages.

### Key Features

- Enhanced support for Persian language tokenization
- Maintained multilingual capabilities of the original LLaMA 2 tokenizer
- Improved handling of Persian-specific characters and word structures
- Larger vocabulary size to accommodate Persian tokens

## Training Data

The tokenizer was created using the following steps:

1. A separate tokenizer with 5000 merges was trained on the Persian Wikipedia corpus to capture Persian-specific tokenization patterns.
2. This Persian-specific tokenizer was then merged with the original LLaMA 2 tokenizer.

## Training Procedure

1. Persian Wikipedia Tokenizer Training:
   - Corpus: Persian Wikipedia dump (specify date if available)
   - Tokenization algorithm: BPE
   - Vocabulary size: 5000

2. Merging with LLaMA 2 Tokenizer:
   - Base tokenizer: LLaMA 2 tokenizer
   - Final vocabulary size: 36954

## Usage

To use this tokenizer with the Hugging Face Transformers library:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("your-username/llama2-persian-tokenizer")

# Example usage
text = "این یک مثال به زبان فارسی است."
tokens = tokenizer(text)
print(tokens)