|
--- |
|
license: bigscience-bloom-rail-1.0 |
|
language: |
|
- ar |
|
- en |
|
--- |
|
|
|
### **Noon - a 7-billion parameter Arabic Large Language Model** |
|
|
|
We present the 7-billion parameter variant of **Noon**, an Arabic Large Language model based on **BLOOM**, a foundation model released by the [bigscience](https://huggingface.co/bigscience) workshop. |
|
|
|
Noon was trained with the main focus of having a model that responds to various types of instructions and questions (text generation, code generation, mathematical problems, closed/open-book questions, etc.) |
|
|
|
We trained the model using the ColossalAI framework which fully supports the HuggingFace library models, and implements different optimization and quantization techniques for billion-scale LLMs. |
|
|
|
The training data is a combination of Arabic datasets covering multiple tasks, more details are provided in the dataset section. |
|
|
|
### **Usage** |
|
|
|
The usage of our model only requires the Transformers library, and can be loaded as follows: |
|
|
|
```python |
|
from transformers import BloomTokenizerFast, BloomForCausalLM, pipeline |
|
|
|
|
|
text="اكتب مقالا من عدة أسطر عن الذكاء الصناعي وتطوراته" |
|
prompt = f'Instruction:\n{text}\n\nResponse:' |
|
|
|
model = BloomForCausalLM.from_pretrained('Naseej/noon-7b') |
|
|
|
# We use the pretrained BLOOM tokenizer as it provides full support for the Arabic vocabulary |
|
tokenizer = BloomTokenizerFast.from_pretrained('bigscience/bloomz-7b1-mt') |
|
|
|
generation_pipeline = pipeline("text-generation", model=model, tokenizer=tokenizer) |
|
|
|
# We recommend the provided hyperparameters for generation |
|
# But encourage you to try different values |
|
response = generation_pipeline(prompt, |
|
pad_token_id=tokenizer.eos_token_id, |
|
num_beams=4, |
|
max_length=500, |
|
top_p=0.9, |
|
repetition_penalty = 3.0, |
|
no_repeat_ngram_size = 3)[0]['generated_text'] |
|
|
|
print(response) |
|
``` |
|
|
|
### **Training's computational requirements** |
|
|
|
### **Dataset** |
|
|
|
To ensure the diversity of data points and satisfy our purpose of instruction-tuning, we collected, labeled, filtered, and reviewed a set of datasets, each tailored to specific instruction types. |
|
Noting that all the datasets are in Arabic, they comprise: |
|
|
|
- [Second version of the Alpaca dataset](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM), generated using GPT4. |
|
- Self-instruct records, split between samples generated by us using the [self-instruct](https://github.com/yizhongw/self-instruct) framework, and further translated ones. |
|
- The instructional dataset released by [Databricks](https://github.com/databrickslabs/dolly), which comprises high quality human-generated instructions and responses. |
|
- [TruthfulQA](https://huggingface.co/datasets/truthful_qa) dataset, to further guide the model on how to truthfully respond to factoid-based questions. |
|
- [Grade School Math](https://huggingface.co/datasets/gsm8k) dataset, to enhance the model's performance using chain-of-thought mathematical problems. |
|
- Arabic arithmetic problems, generated by us using ChatGPT for further improvement of the model's ability to solve mathematical problems. |
|
|
|
The full dataset adds up to over **110K** records. |
|
|
|
### **Disclaimer** |
|
|
|
The generated responses from this AI model are purely algorithmic and should be interpreted with caution. The model's outputs may occasionally exhibit bias, offensive language, or potentially harmful content. It is important to note that these responses do not reflect the personal preferences or viewpoints of the authors or the organization of Naseej. |
|
|
|
While every effort is made to mitigate the harmfulness of the model's outputs, it is impossible to guarantee complete elimination of biases or offensive content. The model learns from vast amounts of data and may inadvertently replicate or amplify existing societal biases present in the training data. |
|
|
|
Users are advised to critically evaluate and verify the information provided by the model. Exercise discretion when utilizing the model's responses, particularly in sensitive or controversial topics. |
|
|
|
We are committed to ongoing research and development to improve the model's performance, minimize biases, and reduce harmful outputs. Your feedback and insights are valuable in helping us achieve these goals. |