Edit model card

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Model Card for Komodo-7B-Base

Komodo-7B-Base is a large language model that is developed through incremental pretraining and vocabulary expansion on top of Llama-2-7B-Base. This model can handle Indonesian, English and 11 regional languages of Indonesia.

Disclaimer : This is not an instruction-tuned model, further fine-tuning is needed for downstream tasks. For example, people usually utilize the Alpaca dataset for further fine-tuning on top of Llama-2-7B-Base model. Hence, there is no prompt template for this model.

Model Details

Model Description

More details can be found in our paper: https://arxiv.org/abs/2403.09362

  • Developed by: Yellow.ai
  • Model type: Decoder
  • Languages: English, Indonesian, Acehnese, Balinese, Banjarese, Buginese, Madurese, Minangkabau, Javanese, Dayak Ngaju, Sundanese, Toba Batak, Lampungnese
  • License: llama2

Usage Example

Since this is a gated model, you need to logged in to your HF account before using the model. Below is one way to do this. You can get the HF Token from your profile (Profile -> Settings -> Access Tokens)

import huggingface_hub
huggingface_hub.login("YOUR_HF_TOKEN")

Once you are logged in, you can start download and load the model & tokenizer. We wrote a custom decoding function for Komodo-7B, that's why we need to pass the trust_remote_code=True. The code also works without this parameter, but decoding process will not work as expected.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda:0" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("Yellow-AI-NLP/komodo-7b-base",trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Yellow-AI-NLP/komodo-7b-base",trust_remote_code=True)
model = model.to(device)

Then, you can try using the model.

full_prompt = "Candi borobudur adalah"

tokens = tokenizer(full_prompt, return_tensors="pt").to(device)
output = model.generate(tokens["input_ids"], eos_token_id=tokenizer.eos_token_id)

print(tokenizer.decode(output[0], skip_special_tokens=True))
# Candi borobudur adalah candi yang terletak di Magelang, Jawa Tengah.

Technical Specifications

Model Architecture and Objective

Komodo-7B is a decoder model using the Llama-2 architecture.

Parameter Komodo-7B
Layers 32
d_model 4096
head_dim 32
Vocabulary 35008
Sequence Length 4096

Tokenizer Details

Recognizing the importance of linguistic diversity, we focused on enhancing our language model's proficiency in both Indonesian and regional languages. To achieve this, we systematically expanded the tokenizer's vocabulary by identifying and incorporating approximately 2,000 frequently used words specific to Indonesian and 1,000 words for regional languages that were absent in the Llama-2 model.

The standard method for enhancing a vocabulary typically involves developing a new tokenizer and integrating it with the existing one. This technique has shown impressive results in projects like Chinese-LLaMA and Open-Hathi. The effectiveness of this strategy can be attributed to the significant linguistic distinctions between languages such as Chinese and Hindi when compared to English. In contrast, the Indonesian language employs the same Latin script as English, which presents a different set of challenges.

We tested the traditional method, as well as a new approach where we included the top n words (not tokens) from the Indonesian vocabulary. We discovered that with the new approach, we could achieve better fertility scores by adding around 3000 new vocabulary words. Adding more than 3000 words did not significantly improve the fertility score further, but it increased the size of the embedding matrix, leading to longer training times.

More details can be found in our paper: https://arxiv.org/abs/2403.09362

Training Data

More details can be found in our paper: https://arxiv.org/abs/2403.09362

Training Procedure

More details can be found in our paper: https://arxiv.org/abs/2403.09362

Preprocessing

More details can be found in our paper: https://arxiv.org/abs/2403.09362

Evaluation & Results

Please note that the benchmarking values below are based on our SFT Model, Komodo-7B-Instruct, while here we only release the base model, Komodo-7B-base.

Organization Model Name Indo MMLU ID-EN XCOPA-ID Intent Classification Colloquial Detection NusaX-Senti ID-Hate Speech TydiQA-ID Indosum Average
OpenAI GPT-3.5-turbo-0301 51.3 64.5 70.0 82.0 64.1 47.2 68.0 85.3 41.0 63.7
OpenAI GPT-3.5-turbo-0613 52.7 66.8 88.2 84.0 75.1 63.3 63.7 86.4 40.0 68.9
OpenAI GPT-3.5-turbo-1106 53.3 69.7 89.3 84.0 64.2 59.8 56.6 88.0 42.0 67.4
OpenAI GPT-4-preview-1106 69.8 78.0 98.3 89.0 92.7 66.1 73.4 72.0 33.0 74.7
Meta Llama-2-7B-Chat 30.4 45.6 41.5 57.0 31.4 2.9 41.3 11.7 34.0 32.9
Meta Llama-2-13B-Chat 32.0 61.7 38.0 59.0 31.1 58.7 57.2 71.9 40.0 50.0
Google Gemma-7B-it 37.4 73.6 57.7 77.1 18.8 44.2 54.8 73.3 44.0 53.4
Mistral Mixtral-8x7B-v0.1-Instruct 45.2 57.8 88.7 86.0 41.1 52.8 68.8 90.3 14.0 60.5
AISingapore Sealion-7B-Instruct-NC 23.9 26.9 41.3 37.0 41.8 30.7 57.3 65.3 26.0 38.9
Cohere Aya-101-13B 47.7 47.3 84.0 64.0 18.9 74.6 72.7 81.3 39.0 58.8
MBZUAI Bactrian-X-Llama-7B 23.6 43.2 45.3 42.0 50.3 44.5 42.4 65.0 15.0 41.3
Alibaba Qwen-1.5-7B-chat 40.0 56.0 29.5 85.0 41.8 58.7 63.9 51.22 29.0 50.6
Yellow.ai Komodo-7B-Instruct 43.2 90.5 79.6 84.0 73.6 79.3 56.2 90.3 43.0 71.1

More details can be found in our paper: https://arxiv.org/abs/2403.09362

Infrastructure

Training Details Komodo-7B
AWS EC2 p4d.24xlarge 1 instances
Nvidia A100 40GB GPU 8
Training Duration 300 hours

Citation

@misc{owen2024komodo,
      title={Komodo: A Linguistic Expedition into Indonesia's Regional Languages}, 
      author={Louis Owen and Vishesh Tripathi and Abhay Kumar and Biddwan Ahmed},
      year={2024},
      eprint={2403.09362},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Model Card Authors

Louis Owen
Vishesh Tripathi
Abhay Kumar
Biddwan Ahmed

Downloads last month
618
Safetensors
Model size
6.76B params
Tensor type
F32
·

Spaces using Yellow-AI-NLP/komodo-7b-base 3