Upload folder using huggingface_hub

README.md

@@ -0,0 +1,114 @@
+---
+license: apache-2.0
+base_model: hon9kon9ize/CantoneseLLMChat-v0.5
+tags:
+- llama-factory
+- full
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: open-lilm-v2
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# open-lilm-v2
+
+[Version 1](https://huggingface.co/0xtaipoian/open-lilm) can be found here.
+
+Warning: Due to the nature of the training data, this model is highly likely to return violent, racist  and discriminative content. DO NOT USE IN PRODUCTION ENVIRONMENT.
+
+
+Inspired by [another project](https://github.com/alphrc/lilm).
+This is a finetuned model based on [CantoneseLLMChat-v0.5](https://huggingface.co/hon9kon9ize/CantoneseLLMChat-v0.5) which everybody can use without the need for a Mac with 128GB RAM.
+
+Following the same principle, we filtered 1,916,944 post and reply pairs in LIHKG forum from the [LIHKG Dataset](https://huggingface.co/datasets/AlienKevin/LIHKG) and scrapped from the site for the latest posts.
+- Reply must be a direct reply to the original post by a user other than the author
+- The total number of reactions (positive or negative) must be larger than 20
+- The post and reply pair has to be shorter than 2048 words
+
+To avoid political complications, the dataset will not be made publicly available. 
+
+
+Compared to version 1, 
+- Training sample increased from 377,595 to 1,916,944, including the latest posts
+- Removed all URLs
+- Removed comments with only emojis
+
+
+
+
+## Intended uses & limitations
+
+Due to the nature of an online and anonymous forum, the training data and the model are full of rude, violent, racist and discriminative language. 
+This model is only intended for research or entertainment purposes.
+
+The comments on LIHKG also tend to be very short. Thus the model cannot generate anything more than a line.
+
+
+## How to use it?
+You can run it on [Colab](https://colab.research.google.com/drive/1veRH2GP3ZR3buYCG2_bFUKu0kS-hv1S2) or anywhere you want based on the code:
+```python
+
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, LlamaTokenizer, GenerationConfig, pipeline
+from peft import PeftModel, PeftMixedModel
+import torch
+
+
+model_name = "0xtaipoian/open-lilm-v2"
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16
+)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+  model_name,
+  torch_dtype=torch.bfloat16,
+  trust_remote_code=True,
+  quantization_config=bnb_config,
+)
+
+def chat(messages, temperature=0.9, max_new_tokens=200):
+    input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt').to('cuda:0')
+    output_ids = model.generate(input_ids, max_new_tokens=max_new_tokens, temperature=temperature, do_sample=True)
+
+    chatml = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+    print(chatml)
+
+    response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=False)
+
+    return response
+
+messages = [
+    # {"role": "system", "content": ""},
+     {"role": "user",
+
+             "content":
+"""
+密陽44人輪姦案」受害女隔20年現身：時間停在2004，不記得
+"""}]
+
+result = chat(messages, max_new_tokens=200, temperature=1)
+
+print(result)
+```
+
+### Training Procedures
+
+The model was trained for 11 hours on 8 NVIDIA H100 80GB HBM3 GPUs with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory).
+
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 22
+- seed: 42
+- gradient_accumulation_steps: 22
+- total_train_batch_size: 3872
+- num_epochs: 1.0
+
+

all_results.json

@@ -0,0 +1,12 @@
+{
+    "epoch": 0.9986738753442823,
+    "eval_loss": 2.5821733474731445,
+    "eval_runtime": 1288.7775,
+    "eval_samples_per_second": 148.742,
+    "eval_steps_per_second": 0.846,
+    "total_flos": 7.860958022007259e+18,
+    "train_loss": 2.698132219207421,
+    "train_runtime": 39941.2631,
+    "train_samples_per_second": 43.195,
+    "train_steps_per_second": 0.011
+}

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "/home/pj24001684/ku40000295/jc/models/CantonesellmChat-v0.5-sft",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 6,
+  "eos_token_id": 7,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 5000000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.3",
+  "use_cache": false,
+  "vocab_size": 64960
+}

config_old.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "hon9kon9ize/CantoneseLLM-v0.5",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 5000000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.1",
+  "use_cache": false,
+  "vocab_size": 64960
+}

eval_results.json

@@ -0,0 +1,7 @@
+{
+    "epoch": 0.9986738753442823,
+    "eval_loss": 2.5821733474731445,
+    "eval_runtime": 1288.7775,
+    "eval_samples_per_second": 148.742,
+    "eval_steps_per_second": 0.846
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 6,
+  "bot_token_id": 6,
+  "eos_token_id": 7,
+  "pad_token_id": 0,
+  "transformers_version": "4.43.0.dev0"
+}

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94197e548f61ab4a0216aab0db24191e90a92b4ea3aff17f15ef17a393767fda
+size 4856125168

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab0bd1f944a8954b41a71a1a2315897c8b6414250e358b34d350ad12bbdc3072
+size 4844657744

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6306110fffda4dbee4e72bfdfd10fdb9942d8b882d9c3c570d647fb95fffeb2
+size 4844657784

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f08472e79f98ebe54409fdde957269d65c2a65b1cdd63e956778dd066f6f83d
+size 4844657784

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:470c154000f1aee36369bac18a6dc366e85298abd30440cf0731e940e07bf5f5
+size 4885534456

model.safetensors.index.json

@@ -0,0 +1,298 @@
+{
+  "metadata": {
+    "total_size": 24275599360
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00005-of-00005.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.norm.weight": "model-00005-of-00005.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|Human|>",
+    "<|Assistant|>",
+    "<|System|>"
+  ],
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67909bb0045622af428982dee9b3f1033cf5c4bca5c9423a028d3748364ee14f
+size 1044277

tokenizer_config.json

@@ -0,0 +1,91 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|Human|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|Assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|System|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|Human|>",
+    "<|Assistant|>",
+    "<|System|>"
+  ],
+  "bos_token": "<|startoftext|>",
+  "chat_template": "{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ '<|im_start|><|System|>\n' + system_message + '<|im_end|>\n' }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|im_start|><|Human|>\n' + content + '<|im_end|>\n<|im_start|><|Assistant|>\n' }}{% elif message['role'] == 'assistant' %}{{ content + '<|im_end|>' + '\n' }}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "legacy": true,
+  "model_max_length": 4096,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "split_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 0.9986738753442823,
+    "total_flos": 7.860958022007259e+18,
+    "train_loss": 2.698132219207421,
+    "train_runtime": 39941.2631,
+    "train_samples_per_second": 43.195,
+    "train_steps_per_second": 0.011
+}

trainer_log.jsonl

@@ -0,0 +1,47 @@
+{"current_steps": 10, "total_steps": 445, "loss": 3.8476, "learning_rate": 2.222222222222222e-06, "epoch": 0.02244210955829848, "percentage": 2.25, "elapsed_time": "0:13:44", "remaining_time": "9:57:47", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 20, "total_steps": 445, "loss": 3.2218, "learning_rate": 4.444444444444444e-06, "epoch": 0.04488421911659696, "percentage": 4.49, "elapsed_time": "0:27:04", "remaining_time": "9:35:13", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 30, "total_steps": 445, "loss": 2.9756, "learning_rate": 6.666666666666667e-06, "epoch": 0.06732632867489544, "percentage": 6.74, "elapsed_time": "0:40:32", "remaining_time": "9:20:53", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 40, "total_steps": 445, "loss": 2.88, "learning_rate": 8.888888888888888e-06, "epoch": 0.08976843823319391, "percentage": 8.99, "elapsed_time": "0:54:05", "remaining_time": "9:07:43", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 50, "total_steps": 445, "loss": 2.8198, "learning_rate": 9.996145181203616e-06, "epoch": 0.11221054779149241, "percentage": 11.24, "elapsed_time": "1:08:29", "remaining_time": "9:01:03", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 60, "total_steps": 445, "loss": 2.8122, "learning_rate": 9.965342284774633e-06, "epoch": 0.13465265734979087, "percentage": 13.48, "elapsed_time": "1:22:54", "remaining_time": "8:51:57", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 70, "total_steps": 445, "loss": 2.7806, "learning_rate": 9.903926402016153e-06, "epoch": 0.15709476690808935, "percentage": 15.73, "elapsed_time": "1:36:58", "remaining_time": "8:39:31", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 80, "total_steps": 445, "loss": 2.7422, "learning_rate": 9.812276182268236e-06, "epoch": 0.17953687646638783, "percentage": 17.98, "elapsed_time": "1:50:49", "remaining_time": "8:25:40", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 90, "total_steps": 445, "loss": 2.736, "learning_rate": 9.690956679612422e-06, "epoch": 0.2019789860246863, "percentage": 20.22, "elapsed_time": "2:04:30", "remaining_time": "8:11:08", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 100, "total_steps": 445, "loss": 2.7361, "learning_rate": 9.540715869125407e-06, "epoch": 0.22442109558298481, "percentage": 22.47, "elapsed_time": "2:18:35", "remaining_time": "7:58:07", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 110, "total_steps": 445, "loss": 2.7135, "learning_rate": 9.362480035363987e-06, "epoch": 0.2468632051412833, "percentage": 24.72, "elapsed_time": "2:32:40", "remaining_time": "7:44:59", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 120, "total_steps": 445, "loss": 2.7064, "learning_rate": 9.157348061512728e-06, "epoch": 0.26930531469958174, "percentage": 26.97, "elapsed_time": "2:46:38", "remaining_time": "7:31:19", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 130, "total_steps": 445, "loss": 2.7018, "learning_rate": 8.926584654403725e-06, "epoch": 0.29174742425788025, "percentage": 29.21, "elapsed_time": "3:00:35", "remaining_time": "7:17:36", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 140, "total_steps": 445, "loss": 2.6961, "learning_rate": 8.671612547178428e-06, "epoch": 0.3141895338161787, "percentage": 31.46, "elapsed_time": "3:14:28", "remaining_time": "7:03:40", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 150, "total_steps": 445, "loss": 2.6968, "learning_rate": 8.39400372766471e-06, "epoch": 0.3366316433744772, "percentage": 33.71, "elapsed_time": "3:28:14", "remaining_time": "6:49:33", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 160, "total_steps": 445, "loss": 2.6879, "learning_rate": 8.095469746549172e-06, "epoch": 0.35907375293277566, "percentage": 35.96, "elapsed_time": "3:42:09", "remaining_time": "6:35:43", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 170, "total_steps": 445, "loss": 2.6686, "learning_rate": 7.777851165098012e-06, "epoch": 0.38151586249107416, "percentage": 38.2, "elapsed_time": "3:56:01", "remaining_time": "6:21:48", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 180, "total_steps": 445, "loss": 2.6497, "learning_rate": 7.443106207484776e-06, "epoch": 0.4039579720493726, "percentage": 40.45, "elapsed_time": "4:09:47", "remaining_time": "6:07:45", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 190, "total_steps": 445, "loss": 2.6413, "learning_rate": 7.093298687687141e-06, "epoch": 0.4264000816076711, "percentage": 42.7, "elapsed_time": "4:23:56", "remaining_time": "5:54:14", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 200, "total_steps": 445, "loss": 2.6472, "learning_rate": 6.730585285387465e-06, "epoch": 0.44884219116596963, "percentage": 44.94, "elapsed_time": "4:37:58", "remaining_time": "5:40:30", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 200, "total_steps": 445, "eval_loss": 2.642993450164795, "epoch": 0.44884219116596963, "percentage": 44.94, "elapsed_time": "4:59:33", "remaining_time": "6:06:57", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 210, "total_steps": 445, "loss": 2.6369, "learning_rate": 6.3572022493253715e-06, "epoch": 0.4712843007242681, "percentage": 47.19, "elapsed_time": "5:13:34", "remaining_time": "5:50:54", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 220, "total_steps": 445, "loss": 2.63, "learning_rate": 5.975451610080643e-06, "epoch": 0.4937264102825666, "percentage": 49.44, "elapsed_time": "5:27:23", "remaining_time": "5:34:49", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 230, "total_steps": 445, "loss": 2.6209, "learning_rate": 5.587686987289189e-06, "epoch": 0.5161685198408651, "percentage": 51.69, "elapsed_time": "5:41:21", "remaining_time": "5:19:06", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 240, "total_steps": 445, "loss": 2.6318, "learning_rate": 5.1962990787953436e-06, "epoch": 0.5386106293991635, "percentage": 53.93, "elapsed_time": "5:55:03", "remaining_time": "5:03:16", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 250, "total_steps": 445, "loss": 2.621, "learning_rate": 4.803700921204659e-06, "epoch": 0.561052738957462, "percentage": 56.18, "elapsed_time": "6:08:55", "remaining_time": "4:47:45", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 260, "total_steps": 445, "loss": 2.6161, "learning_rate": 4.4123130127108125e-06, "epoch": 0.5834948485157605, "percentage": 58.43, "elapsed_time": "6:22:39", "remaining_time": "4:32:16", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 270, "total_steps": 445, "loss": 2.6164, "learning_rate": 4.02454838991936e-06, "epoch": 0.605936958074059, "percentage": 60.67, "elapsed_time": "6:36:39", "remaining_time": "4:17:05", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 280, "total_steps": 445, "loss": 2.6091, "learning_rate": 3.6427977506746293e-06, "epoch": 0.6283790676323574, "percentage": 62.92, "elapsed_time": "6:50:40", "remaining_time": "4:02:00", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 290, "total_steps": 445, "loss": 2.5967, "learning_rate": 3.269414714612534e-06, "epoch": 0.6508211771906559, "percentage": 65.17, "elapsed_time": "7:04:34", "remaining_time": "3:46:55", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 300, "total_steps": 445, "loss": 2.6031, "learning_rate": 2.906701312312861e-06, "epoch": 0.6732632867489544, "percentage": 67.42, "elapsed_time": "7:18:40", "remaining_time": "3:32:01", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 310, "total_steps": 445, "loss": 2.6014, "learning_rate": 2.5568937925152272e-06, "epoch": 0.6957053963072529, "percentage": 69.66, "elapsed_time": "7:32:32", "remaining_time": "3:17:04", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 320, "total_steps": 445, "loss": 2.5978, "learning_rate": 2.2221488349019903e-06, "epoch": 0.7181475058655513, "percentage": 71.91, "elapsed_time": "7:46:30", "remaining_time": "3:02:13", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 330, "total_steps": 445, "loss": 2.5911, "learning_rate": 1.9045302534508298e-06, "epoch": 0.7405896154238498, "percentage": 74.16, "elapsed_time": "8:00:23", "remaining_time": "2:47:24", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 340, "total_steps": 445, "loss": 2.5913, "learning_rate": 1.6059962723352912e-06, "epoch": 0.7630317249821483, "percentage": 76.4, "elapsed_time": "8:14:27", "remaining_time": "2:32:42", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 350, "total_steps": 445, "loss": 2.5819, "learning_rate": 1.3283874528215735e-06, "epoch": 0.7854738345404468, "percentage": 78.65, "elapsed_time": "8:28:44", "remaining_time": "2:18:05", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 360, "total_steps": 445, "loss": 2.5833, "learning_rate": 1.0734153455962765e-06, "epoch": 0.8079159440987452, "percentage": 80.9, "elapsed_time": "8:42:44", "remaining_time": "2:03:25", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 370, "total_steps": 445, "loss": 2.5851, "learning_rate": 8.426519384872733e-07, "epoch": 0.8303580536570437, "percentage": 83.15, "elapsed_time": "8:56:50", "remaining_time": "1:48:49", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 380, "total_steps": 445, "loss": 2.5853, "learning_rate": 6.375199646360142e-07, "epoch": 0.8528001632153422, "percentage": 85.39, "elapsed_time": "9:10:53", "remaining_time": "1:34:13", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 390, "total_steps": 445, "loss": 2.5832, "learning_rate": 4.5928413087459325e-07, "epoch": 0.8752422727736408, "percentage": 87.64, "elapsed_time": "9:24:43", "remaining_time": "1:19:38", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 400, "total_steps": 445, "loss": 2.5779, "learning_rate": 3.0904332038757977e-07, "epoch": 0.8976843823319393, "percentage": 89.89, "elapsed_time": "9:38:32", "remaining_time": "1:05:05", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 400, "total_steps": 445, "eval_loss": 2.582942485809326, "epoch": 0.8976843823319393, "percentage": 89.89, "elapsed_time": "9:59:46", "remaining_time": "1:07:28", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 410, "total_steps": 445, "loss": 2.5833, "learning_rate": 1.8772381773176417e-07, "epoch": 0.9201264918902377, "percentage": 92.13, "elapsed_time": "10:13:48", "remaining_time": "0:52:23", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 420, "total_steps": 445, "loss": 2.597, "learning_rate": 9.607359798384785e-08, "epoch": 0.9425686014485362, "percentage": 94.38, "elapsed_time": "10:28:05", "remaining_time": "0:37:23", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 430, "total_steps": 445, "loss": 2.5725, "learning_rate": 3.465771522536854e-08, "epoch": 0.9650107110068347, "percentage": 96.63, "elapsed_time": "10:42:08", "remaining_time": "0:22:24", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 440, "total_steps": 445, "loss": 2.575, "learning_rate": 3.854818796385495e-09, "epoch": 0.9874528205651332, "percentage": 98.88, "elapsed_time": "10:55:58", "remaining_time": "0:07:27", "throughput": "0.00", "total_tokens": 0}
+{"current_steps": 445, "total_steps": 445, "epoch": 0.9986738753442823, "percentage": 100.0, "elapsed_time": "11:05:37", "remaining_time": "0:00:00", "throughput": "0.00", "total_tokens": 0}

trainer_state.json

@@ -0,0 +1,366 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9986738753442823,
+  "eval_steps": 200,
+  "global_step": 445,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.02244210955829848,
+      "grad_norm": 19.15482521057129,
+      "learning_rate": 2.222222222222222e-06,
+      "loss": 3.8476,
+      "step": 10
+    },
+    {
+      "epoch": 0.04488421911659696,
+      "grad_norm": 5.446422576904297,
+      "learning_rate": 4.444444444444444e-06,
+      "loss": 3.2218,
+      "step": 20
+    },
+    {
+      "epoch": 0.06732632867489544,
+      "grad_norm": 1.8523049354553223,
+      "learning_rate": 6.666666666666667e-06,
+      "loss": 2.9756,
+      "step": 30
+    },
+    {
+      "epoch": 0.08976843823319391,
+      "grad_norm": 2.139192581176758,
+      "learning_rate": 8.888888888888888e-06,
+      "loss": 2.88,
+      "step": 40
+    },
+    {
+      "epoch": 0.11221054779149241,
+      "grad_norm": 1.853474497795105,
+      "learning_rate": 9.996145181203616e-06,
+      "loss": 2.8198,
+      "step": 50
+    },
+    {
+      "epoch": 0.13465265734979087,
+      "grad_norm": 1.501637578010559,
+      "learning_rate": 9.965342284774633e-06,
+      "loss": 2.8122,
+      "step": 60
+    },
+    {
+      "epoch": 0.15709476690808935,
+      "grad_norm": 2.0072269439697266,
+      "learning_rate": 9.903926402016153e-06,
+      "loss": 2.7806,
+      "step": 70
+    },
+    {
+      "epoch": 0.17953687646638783,
+      "grad_norm": 1.7332258224487305,
+      "learning_rate": 9.812276182268236e-06,
+      "loss": 2.7422,
+      "step": 80
+    },
+    {
+      "epoch": 0.2019789860246863,
+      "grad_norm": 1.3256088495254517,
+      "learning_rate": 9.690956679612422e-06,
+      "loss": 2.736,
+      "step": 90
+    },
+    {
+      "epoch": 0.22442109558298481,
+      "grad_norm": 1.6238477230072021,
+      "learning_rate": 9.540715869125407e-06,
+      "loss": 2.7361,
+      "step": 100
+    },
+    {
+      "epoch": 0.2468632051412833,
+      "grad_norm": 1.326378583908081,
+      "learning_rate": 9.362480035363987e-06,
+      "loss": 2.7135,
+      "step": 110
+    },
+    {
+      "epoch": 0.26930531469958174,
+      "grad_norm": 1.3376497030258179,
+      "learning_rate": 9.157348061512728e-06,
+      "loss": 2.7064,
+      "step": 120
+    },
+    {
+      "epoch": 0.29174742425788025,
+      "grad_norm": 1.2815560102462769,
+      "learning_rate": 8.926584654403725e-06,
+      "loss": 2.7018,
+      "step": 130
+    },
+    {
+      "epoch": 0.3141895338161787,
+      "grad_norm": 1.5868873596191406,
+      "learning_rate": 8.671612547178428e-06,
+      "loss": 2.6961,
+      "step": 140
+    },
+    {
+      "epoch": 0.3366316433744772,
+      "grad_norm": 1.366570234298706,
+      "learning_rate": 8.39400372766471e-06,
+      "loss": 2.6968,
+      "step": 150
+    },
+    {
+      "epoch": 0.35907375293277566,
+      "grad_norm": 1.6603009700775146,
+      "learning_rate": 8.095469746549172e-06,
+      "loss": 2.6879,
+      "step": 160
+    },
+    {
+      "epoch": 0.38151586249107416,
+      "grad_norm": 1.4688373804092407,
+      "learning_rate": 7.777851165098012e-06,
+      "loss": 2.6686,
+      "step": 170
+    },
+    {
+      "epoch": 0.4039579720493726,
+      "grad_norm": 1.2386434078216553,
+      "learning_rate": 7.443106207484776e-06,
+      "loss": 2.6497,
+      "step": 180
+    },
+    {
+      "epoch": 0.4264000816076711,
+      "grad_norm": 1.3002716302871704,
+      "learning_rate": 7.093298687687141e-06,
+      "loss": 2.6413,
+      "step": 190
+    },
+    {
+      "epoch": 0.44884219116596963,
+      "grad_norm": 1.2603603601455688,
+      "learning_rate": 6.730585285387465e-06,
+      "loss": 2.6472,
+      "step": 200
+    },
+    {
+      "epoch": 0.44884219116596963,
+      "eval_loss": 2.642993450164795,
+      "eval_runtime": 1294.6794,
+      "eval_samples_per_second": 148.064,
+      "eval_steps_per_second": 0.842,
+      "step": 200
+    },
+    {
+      "epoch": 0.4712843007242681,
+      "grad_norm": 1.2680917978286743,
+      "learning_rate": 6.3572022493253715e-06,
+      "loss": 2.6369,
+      "step": 210
+    },
+    {
+      "epoch": 0.4937264102825666,
+      "grad_norm": 1.3160443305969238,
+      "learning_rate": 5.975451610080643e-06,
+      "loss": 2.63,
+      "step": 220
+    },
+    {
+      "epoch": 0.5161685198408651,
+      "grad_norm": 1.2467771768569946,
+      "learning_rate": 5.587686987289189e-06,
+      "loss": 2.6209,
+      "step": 230
+    },
+    {
+      "epoch": 0.5386106293991635,
+      "grad_norm": 1.208018183708191,
+      "learning_rate": 5.1962990787953436e-06,
+      "loss": 2.6318,
+      "step": 240
+    },
+    {
+      "epoch": 0.561052738957462,
+      "grad_norm": 1.2416397333145142,
+      "learning_rate": 4.803700921204659e-06,
+      "loss": 2.621,
+      "step": 250
+    },
+    {
+      "epoch": 0.5834948485157605,
+      "grad_norm": 1.1826361417770386,
+      "learning_rate": 4.4123130127108125e-06,
+      "loss": 2.6161,
+      "step": 260
+    },
+    {
+      "epoch": 0.605936958074059,
+      "grad_norm": 1.2550407648086548,
+      "learning_rate": 4.02454838991936e-06,
+      "loss": 2.6164,
+      "step": 270
+    },
+    {
+      "epoch": 0.6283790676323574,
+      "grad_norm": 1.2681384086608887,
+      "learning_rate": 3.6427977506746293e-06,
+      "loss": 2.6091,
+      "step": 280
+    },
+    {
+      "epoch": 0.6508211771906559,
+      "grad_norm": 1.2637056112289429,
+      "learning_rate": 3.269414714612534e-06,
+      "loss": 2.5967,
+      "step": 290
+    },
+    {
+      "epoch": 0.6732632867489544,
+      "grad_norm": 1.211774468421936,
+      "learning_rate": 2.906701312312861e-06,
+      "loss": 2.6031,
+      "step": 300
+    },
+    {
+      "epoch": 0.6957053963072529,
+      "grad_norm": 1.1411036252975464,
+      "learning_rate": 2.5568937925152272e-06,
+      "loss": 2.6014,
+      "step": 310
+    },
+    {
+      "epoch": 0.7181475058655513,
+      "grad_norm": 1.1422080993652344,
+      "learning_rate": 2.2221488349019903e-06,
+      "loss": 2.5978,
+      "step": 320
+    },
+    {
+      "epoch": 0.7405896154238498,
+      "grad_norm": 1.172059416770935,
+      "learning_rate": 1.9045302534508298e-06,
+      "loss": 2.5911,
+      "step": 330
+    },
+    {
+      "epoch": 0.7630317249821483,
+      "grad_norm": 1.1655080318450928,
+      "learning_rate": 1.6059962723352912e-06,
+      "loss": 2.5913,
+      "step": 340
+    },
+    {
+      "epoch": 0.7854738345404468,
+      "grad_norm": 1.1286932229995728,
+      "learning_rate": 1.3283874528215735e-06,
+      "loss": 2.5819,
+      "step": 350
+    },
+    {
+      "epoch": 0.8079159440987452,
+      "grad_norm": 1.1322216987609863,
+      "learning_rate": 1.0734153455962765e-06,
+      "loss": 2.5833,
+      "step": 360
+    },
+    {
+      "epoch": 0.8303580536570437,
+      "grad_norm": 1.1392606496810913,
+      "learning_rate": 8.426519384872733e-07,
+      "loss": 2.5851,
+      "step": 370
+    },
+    {
+      "epoch": 0.8528001632153422,
+      "grad_norm": 1.1811796426773071,
+      "learning_rate": 6.375199646360142e-07,
+      "loss": 2.5853,
+      "step": 380
+    },
+    {
+      "epoch": 0.8752422727736408,
+      "grad_norm": 1.1267277002334595,
+      "learning_rate": 4.5928413087459325e-07,
+      "loss": 2.5832,
+      "step": 390
+    },
+    {
+      "epoch": 0.8976843823319393,
+      "grad_norm": 1.116821527481079,
+      "learning_rate": 3.0904332038757977e-07,
+      "loss": 2.5779,
+      "step": 400
+    },
+    {
+      "epoch": 0.8976843823319393,
+      "eval_loss": 2.582942485809326,
+      "eval_runtime": 1274.0632,
+      "eval_samples_per_second": 150.46,
+      "eval_steps_per_second": 0.856,
+      "step": 400
+    },
+    {
+      "epoch": 0.9201264918902377,
+      "grad_norm": 1.1507278680801392,
+      "learning_rate": 1.8772381773176417e-07,
+      "loss": 2.5833,
+      "step": 410
+    },
+    {
+      "epoch": 0.9425686014485362,
+      "grad_norm": 1.0935174226760864,
+      "learning_rate": 9.607359798384785e-08,
+      "loss": 2.597,
+      "step": 420
+    },
+    {
+      "epoch": 0.9650107110068347,
+      "grad_norm": 1.1115341186523438,
+      "learning_rate": 3.465771522536854e-08,
+      "loss": 2.5725,
+      "step": 430
+    },
+    {
+      "epoch": 0.9874528205651332,
+      "grad_norm": 1.131402611732483,
+      "learning_rate": 3.854818796385495e-09,
+      "loss": 2.575,
+      "step": 440
+    },
+    {
+      "epoch": 0.9986738753442823,
+      "step": 445,
+      "total_flos": 7.860958022007259e+18,
+      "train_loss": 2.698132219207421,
+      "train_runtime": 39941.2631,
+      "train_samples_per_second": 43.195,
+      "train_steps_per_second": 0.011
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 445,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 7.860958022007259e+18,
+  "train_batch_size": 22,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:938afb7225b757e389466dbf73b64ea7c28ce1d3896a99335672249afcb6f74e
+size 5432