|
--- |
|
license: mit |
|
language: |
|
- ar |
|
library_name: tokenizers |
|
pipeline_tag: summarization |
|
tags: |
|
- arabic |
|
- summarization |
|
- tokenizers |
|
- BPE |
|
--- |
|
|
|
## Byte Level (BPE) Tokenizer for Arabic |
|
|
|
Byte Level Tokenizer for Arabic, a robust tokenizer designed to handle Arabic text with precision and efficiency. |
|
This tokenizer utilizes a `Byte-Pair Encoding (BPE)` approach to create a vocabulary of `50,000` tokens, catering specifically to the intricacies of the Arabic language. |
|
|
|
### Goal |
|
|
|
This tokenizer was created as part of the development of an Arabic BART transformer model for summarization from scratch using `PyTorch`. |
|
In adherence to the configurations outlined in the official [BART](https://arxiv.org/abs/1910.13461) paper, which specifies the use of BPE tokenization, I sought a BPE tokenizer specifically tailored for Arabic. |
|
While there are Arabic-only tokenizers and multilingual BPE tokenizers, a dedicated Arabic BPE tokenizer was not available. This gap inspired the creation of a `BPE` tokenizer focused solely on Arabic, ensuring alignment with BART's recommended configurations and enhancing the effectiveness of Arabic NLP tasks. |
|
|
|
### Checkpoint Information |
|
|
|
- **Name**: `IsmaelMousa/arabic-bpe-tokenizer` |
|
- **Vocabulary Size**: `50,000` |
|
|
|
### Overview |
|
|
|
The Byte Level Tokenizer is optimized to manage Arabic text, which often includes a range of diacritics, different forms of the same word, and various prefixes and suffixes. This tokenizer addresses these challenges by breaking down text into byte-level tokens, ensuring that it can effectively process and understand the nuances of the Arabic language. |
|
|
|
### Features |
|
|
|
- **Byte-Pair Encoding (BPE)**: Efficiently manages a large vocabulary size while maintaining accuracy. |
|
- **Comprehensive Coverage**: Handles Arabic script, including diacritics and various word forms. |
|
- **Flexible Integration**: Easily integrates with the `tokenizers` library for seamless tokenization. |
|
|
|
### Installation |
|
|
|
To use this tokenizer, you need to install the `tokenizers` library. If you haven’t installed it yet, you can do so using pip: |
|
|
|
```bash |
|
pip install tokenizers |
|
``` |
|
|
|
### Example Usage |
|
Here is an example of how to use the Byte Level Tokenizer with the `tokenizers` library. |
|
|
|
|
|
This example demonstrates tokenization of the Arabic sentence "لاشيء يعجبني, أريد أن أبكي": |
|
|
|
```python |
|
from tokenizers import Tokenizer |
|
|
|
tokenizer = Tokenizer.from_pretrained("IsmaelMousa/arabic-bpe-tokenizer") |
|
|
|
text = "لاشيء يعجبني, أريد أن أبكي" |
|
|
|
encoded = tokenizer.encode(text) |
|
decoded = tokenizer.decode(encoded.ids) |
|
|
|
print("Encoded Tokens:", encoded.tokens) |
|
print("Token IDs:", encoded.ids) |
|
print("Decoded Text:", decoded) |
|
|
|
``` |
|
|
|
output: |
|
|
|
```bash |
|
Encoded Tokens: ['<s>', 'ÙĦا', 'ĠØ´ÙĬØ¡', 'ĠÙĬع', 'جب', 'ÙĨÙĬ', ',', 'ĠأرÙĬد', 'ĠØ£ÙĨ', 'Ġأب', 'ÙĥÙĬ', '</s>'] |
|
|
|
Token IDs: [0, 419, 1773, 667, 2281, 489, 16, 7578, 331, 985, 1344, 2] |
|
|
|
Decoded Text: لا شيء يعجبني, أريد أن أبكي |
|
``` |
|
|
|
### Tokenizer Details |
|
- **Byte-Level Tokenization**: This method ensures that every byte of input text is considered, making it suitable for languages with complex scripts. |
|
- **Adaptability**: Can be fine-tuned or used as-is, depending on your specific needs and application scenarios. |
|
|
|
### License |
|
This project is licensed under the `MIT` License. |
