---
license: llama2
---

# Toy LLaMA-39M

- This is a tiny LLaMA model pretrained on [Recag/Rp_C4_55](https://huggingface.co/datasets/Recag/Rp_C4_55), a small subset of C4 with `seq_len=512`.
  - Model architecture
      ```json
      {
        "hidden_size": 512,
        "intermediate_size": 2048,
        "max_position_embeddings": 2048,
        "num_attention_heads": 8,
        "num_hidden_layers": 2,
        "num_key_value_heads": 8
      }
      ```
  - Load model and tokenizer:
      ```python
      from transformers import AutoTokenizer, AutoModelForCausalLM
      model = AutoModelForCausalLM.from_pretrained("Cheng98/llama-39m")
      tokenizer = AutoTokenizer.from_pretrained("Cheng98/llama-39m")
      ```
  - Training script: [huggingface/transformers/examples/pytorch/language-modeling/run_clm.py](https://github.com/huggingface/transformers/blob/e9476832942a19cf99354776ef112babc83c139a/examples/pytorch/language-modeling/run_clm.py)
      ```python
      # "train" split is created from the last 95% samples of original "train" subset
      raw_datasets["validation"] = load_dataset("Recag/Rp_C4_55", split="train[5%:]")
      ```


- Evaluation (`seq_len=512`):
   
  | Dataset        | Eval loss | Perplexity | Accuracy | block_size |
  |----------------|-----------|------------|----------|------------|
  | Recag/Rp_C4_55 | 3.63      | 37.78      | 0.3561   | 512        |
  | Wikitext2      | 4.58      | 97.48      | 0.2719   | 512        |

  - Evaluation command (Wikitext2):
      ```bash
      # Evaluation command
      python run_clm.py --model_name_or_path Cheng98/llama-39m \
        --dataset_name wikitext \
        --dataset_config_name wikitext-2-raw-v1 \
        --block_size 512 \
        --do_eval \
        --output_dir ./results
      ```
  
  - Evaluation on Recag/Rp_C4_55 (`seq_len=512`):
      ```python
      # "validation" split is created from the first 5% samples of original "train" subset
      raw_datasets["validation"] = load_dataset("Recag/Rp_C4_55", split="train[:5%]")
      ```
      Results
      ```json
      {
        "eval_accuracy": 0.3561766818954313,
        "eval_loss": 3.6318140029907227,
        "eval_runtime": 190.8411,
        "eval_samples": 19413,
        "eval_samples_per_second": 101.723,
        "eval_steps_per_second": 1.593,
        "perplexity": 37.7812898658763
      }
      ```
      
  - Evaluation on Wikitext2 (`seq_len=512`):
      ```json
      {
        "eval_accuracy": 0.2718795201225219,
        "eval_loss": 4.579628944396973,
        "eval_runtime": 3.939,
        "eval_samples": 575,
        "eval_samples_per_second": 145.976,
        "eval_steps_per_second": 0.762,
        "perplexity": 97.47821765687856
      }
      ```