Gothef/MAX-Neural-80M-Arabic

MAX Neural (Arabic Models)

This repository contains the MAX Neural family of Arabic language models, built completely from scratch using raw PyTorch.

📦 Available Models

1. MAX Neural 80M (Base)

• Path: max_v2_best.pt & max_v2_tokenizer.json
• Type: Base Model (Text Completion)
• Architecture: 11 layers, 768 hidden size, 16K vocab.

2. MAX Neural 500M (Instruct)

• Path: v3_500M/max_v3_500m_instruct.pt & v3_500M/max_v3_tokenizer_32k.json
• Type: Instruct Model (Question Answering / Chat)
• Architecture: 24 layers, 1024 hidden size, 32K vocab, GQA (4 KV heads).
• Training: Trained on 30,000 Arabic instruction-following examples (Alpaca-GPT4).

🏗️ Architecture (Both Models)

• Implemented directly in raw PyTorch.
• Positional Encoding: RoPE (Rotary Position Embeddings)
• Feed Forward: SwiGLU
• Normalization: RMSNorm
• Attention: Grouped Query Attention (GQA)

⚠️ Limitations

While the 500M Instruct model follows instructions and generates coherent Arabic, it is still a relatively small model. It may hallucinate facts and should not be used for critical decision-making without external verification (RAG).

💻 Usage

To use the 500M Instruct model, download the weights from the v3_500M folder and load them using the custom PyTorch architecture provided in the training script.