RichardErkhov/PerceiveIO_-_tinyllama_92M-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

tinyllama_92M - GGUF

• Model creator: https://huggingface.co/PerceiveIO/
• Original model: https://huggingface.co/PerceiveIO/tinyllama_92M/

Name	Quant method	Size
tinyllama_92M.Q2_K.gguf	Q2_K	0.03GB
tinyllama_92M.IQ3_XS.gguf	IQ3_XS	0.04GB
tinyllama_92M.IQ3_S.gguf	IQ3_S	0.04GB
tinyllama_92M.Q3_K_S.gguf	Q3_K_S	0.04GB
tinyllama_92M.IQ3_M.gguf	IQ3_M	0.04GB
tinyllama_92M.Q3_K.gguf	Q3_K	0.04GB
tinyllama_92M.Q3_K_M.gguf	Q3_K_M	0.04GB
tinyllama_92M.Q3_K_L.gguf	Q3_K_L	0.05GB
tinyllama_92M.IQ4_XS.gguf	IQ4_XS	0.05GB
tinyllama_92M.Q4_0.gguf	Q4_0	0.05GB
tinyllama_92M.IQ4_NL.gguf	IQ4_NL	0.05GB
tinyllama_92M.Q4_K_S.gguf	Q4_K_S	0.05GB
tinyllama_92M.Q4_K.gguf	Q4_K	0.05GB
tinyllama_92M.Q4_K_M.gguf	Q4_K_M	0.05GB
tinyllama_92M.Q4_1.gguf	Q4_1	0.05GB
tinyllama_92M.Q5_0.gguf	Q5_0	0.06GB
tinyllama_92M.Q5_K_S.gguf	Q5_K_S	0.06GB
tinyllama_92M.Q5_K.gguf	Q5_K	0.06GB
tinyllama_92M.Q5_K_M.gguf	Q5_K_M	0.06GB
tinyllama_92M.Q5_1.gguf	Q5_1	0.06GB
tinyllama_92M.Q6_K.gguf	Q6_K	0.07GB
tinyllama_92M.Q8_0.gguf	Q8_0	0.09GB

Original model description:

library_name: transformers tags: []

tinyllamas_92M

Model Details

  max_seq_len = 256
  vocab_size = 8192 
  dim = 768
  n_layers = 12
  n_heads = 12
  n_kv_heads = 12

Training Data

• https://huggingface.co/datasets/roneneldan/TinyStories
• Tokenized using: https://github.com/karpathy/llama2.c?tab=readme-ov-file#custom-tokenizers

Training Hyperparameters

  batch_size = 64  # if gradient_accumulation_steps > 1, this is the micro-batch size
  dropout = 0.0
  # adamw optimizer
  gradient_accumulation_steps = 8  # used to simulate larger batch sizes
  learning_rate = 1e-3  # max learning rate
  max_iters = 34000  # total number of training iterations
  weight_decay = 3e-4
  beta1 = 0.9
  beta2 = 0.95
  grad_clip = 1.0  # clip gradients at this value, or disable if == 0.0
  # learning rate decay settings
  decay_lr = True  # whether to decay the learning rate
  warmup_iters = 1000  # how many steps to warm up for

Results

4xV100 GPUs used.
Run summary:
  iter 34000
  loss/train 0.8704
  loss/val 0.9966
  tokens 983040000