JBalwaySUS/deepseekv3_150m_multilingual_eng-mal-snd

Language Models and Agents

Mini Project

The goals of the mini project is to build a small autoregressive language model (LM) for multiple languages from scratch.

Assigned Languages

• L1: English (LID1)
• L2: Malayalam (LID6)
• L3: Sindhi (LID25)

Assigned Base Model

• DeepseekV3 (MID1)

Assigned Fine Tuning Tasks

• Keyword Extraction (FT48)
• Common Object Location (FT5)

Phase 1: Data Collection and Preprocessing

Corpora Chosen for Each Language

• English: FineWeb-Edu
• Malayalam: FineWeb-2
• Sindhi: FineWeb-2

Preprocessing details

The fineweb data has been extensively cleaned to remove harmful content and duplicate entries. There was not additional preprocessing performed on the text itself.

Phase 2: Tokenizer Training

For tokenizer training, I am experimenting with both BPE and SentencePiece Unigram tokenizers. The tokenizer was trained by interleaving text from all three languages with equal probability.

Statistics for BPE Tokenizer:

Here are the tokenizer evaluation results in a markdown table.

Language	Unknown Ratio	Chars/Token	Tokens/Word	Total Chars	Total Words	Total Tokens	Unknown Tokens
English	0.000000	4.20	1.42	30,503,959	5,144,303	7,284,362	0
Malayalam	0.000000	4.26	2.39	28,815,402	2,936,177	7,031,500	0
Sindhi	0.000000	3.73	1.37	7,790,177	1,640,539	2,240,778	0
Average	0.000000	4.06	1.73	67,109,538	9,721,019	16,556,640	0

Statistics for SentencePiece Unigram Tokenizer:

Language	Unknown Ratio	Chars/Token	Tokens/Word	Total Chars	Total Words	Total Tokens	Unknown Tokens
English	0.000000	4.22	1.41	30,503,959	5,144,303	7,261,193	0
Malayalam	0.000000	4.44	2.31	28,815,402	2,936,177	6,789,148	0
Sindhi	0.000000	3.78	1.35	7,790,177	1,640,539	2,220,958	0
Average	0.000000	4.15	1.69	67,109,538	9,721,019	16,271,299	0

Due to higher compression ratio, we choose SentencePiece BPE Tokenizer.

Plan for Future Phases

1. Phase 3: Model Pretraining

Since DeepseekV3 is an MoE model we need to convert it to dense format. Thankfully we can use DeepseekV3Config to achieve this. The configuration file configuration_deepseek.py provided in the source code gives us the necessary parameters to create a dense version of the model along with reduced parameters.

from transformers import DeepseekV3Config

# Configuration for a ~150M parameter dense SLM
config_150m_dense = DeepseekV3Config(
    # --- Core Dense Model Dimensions ---
    vocab_size=129280,                 # Keep the original vocabulary
    hidden_size=512,                   # Smaller embedding dimension
    num_hidden_layers=24,              # Number of transformer layers
    intermediate_size=2048,            # FFN intermediate size (4 * hidden_size)
    num_attention_heads=8,             # Attention heads
    num_key_value_heads=8,             # Use standard Multi-Head Attention

    # --- Disable MoE ---
    n_shared_experts=None,             # Set to None for a dense model
    n_routed_experts=None,             # Set to None for a dense model
    num_experts_per_tok=None,          # Set to None for a dense model
    
    # --- Important Optimization ---
    tie_word_embeddings=True,          # Share embedding and output layer weights

    # --- Other standard parameters ---
    hidden_act="silu",
    max_position_embeddings=4096,
    rms_norm_eps=1e-6,
    rope_theta=10000.0,
    bos_token_id=0,
    eos_token_id=1,
)

# We can now use this config object to initialize your model
# from transformers import DeepseekV3ForCausalLM
# model = DeepseekV3ForCausalLM(config_150m_dense)
# print(f"Model created with {model.num_parameters() / 1e6:.2f}M parameters.")

Now we can pretrain the model using the corpora collected.

2. Phase 4: Finetuning on Reasoning Tasks

Since there are no readily available data for Common Object Location, synthetic data using LLMs like Gemini and then performing Machine Translaton for L2 and L3

For Keyword Extraction, existing datasets will be used for L1, while synthetic data will be generated for L2 and L3 using Machine Translation.

Final Submission

Hugging Face Repo: JBalwaySUS/deepseekv3_150m_multilingual_eng-mal-snd

Phase 3: Model Pretraining

Approach

1. Model Configuration: We started by configuring a dense version of the DeepseekV3 model with approximately 150 million parameters. This involved setting key parameters such as hidden_size, num_hidden_layers, and intermediate_size to create a smaller yet effective model. To convert the MoE model to a dense format, we set n_shared_experts, n_routed_experts, and num_experts_per_tok to None and set first_k_dense = num_hidden_layers to convert all hidden layers to dense layer. Additionally, to reduce parameters, we enabled weight tying between the embedding and output layers by setting tie_word_embeddings=True. Using this configuration, we initialized the model and verified that it had approximately 149.2 million parameters.

2. Model Training: We trained the model using the corpus used to train the tokenizer (FineWeb-Edu and FineWeb-2), which included English, Malayalam, and Sindhi texts. The training was performed using the Hugging Face Trainer API. Since the Trainer API does not train based on tokens, a heuristic was used to estimate the number of epochs required to process a target number of tokens (3 billion). The heuritic is as follows:

   • No. of tokens processed in one step = batch_size * gradient_accumulation_steps * seq_length * num_gpus For this model with batch_size=32, gradient_accumulation_steps=1, seq_length=256, and num_gpus=4, we need 91552 steps to process 3 billion tokens.

   Evaluation was performed every 1000 steps using a held-out validation set.

   Final Loss and Perplexity Values:

   • Training - Loss: 3.3169, Perplexity: 27.59
   • Validation - Loss: 3.3890, Perplexity: 29.61

   10 Checkpoints were saved during training.

3. Model Evaluation: The model was evaluated manually by generating text prompts in all three languages. The model generated coherent text but lacked knowledge of real-world facts, which is expected given the limited training data and model size.
   Example generations:

• Prompt 1: The capital of France is
  Generated: the capital of the Netherlands, and it is also the capital of the Netherlands. The capital of the Netherlands is in the town of Kolbe. The capital is the town of the town, which is the capital of the Netherlands. The capital of

• Prompt 2: In machine learning, neural networks are
  Generated: the key to understanding the world and how it interacts with the environment. In machine learning, neural networks are the main areas of research for neural networks. As a result, there are three types of neural networks: - Machine Learning – This type of

Phase 4: Finetuning on Reasoning Tasks

Approach

1. Dataset Creation:

   • For Common Object Location, synthetic list of (object, location) pairs were generated using LLMs. These pairs were randomly picked to create a synthetic dataset of questions and answers in English. The dataset was then translated to Malayalam and Sindhi using Google Translate.

   • For Keyword Extraction, the xsum_keywords dataset was used for English. The dataset contains keywords for each summary in the XSum dataset generated using Language Models. A portion of these summaries and corresponding keywords were translated to Malayalam and Sindhi using Google Translate to create datasets for these languages.

2. Model Finetuning: The pretrained model from Phase 3 was finetuned using Supervised Fine-Tuning (SFT) on the datasets created for both tasks. The finetuning was performed using SFTTrainer from the trl library using LoRA adapters from unsloth library for memory efficiency. The model was finetuned for 3 epochs on the dataset

   Final Loss and Perplexity Values:

   • Training - Loss: 2.531, Perplexity: 12.57
   • Validation - Loss: 2.473, Perplexity: 11.86

3. Model Evaluation: The finetuned model was evaluated manually by generating answers to questions in all three languages for both tasks. The model was not able to perform well on both tasks, likely due to the limited size of the model and the synthetic nature of the datasets. However, it was able to generate relevant keywords and object locations in some cases.