VTSNLP commited on
Commit
89be834
1 Parent(s): 8c1bb6e

VTSNLP/base_LLM_CAC_ND_400M_final

Browse files
README.md ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: wanderer2k1/base_LLM_433MB_chunk_and_concat_no_domain
3
+ tags:
4
+ - generated_from_trainer
5
+ model-index:
6
+ - name: base_LLM_CAC_ND_400M_final
7
+ results: []
8
+ ---
9
+
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
12
+
13
+ # base_LLM_CAC_ND_400M_final
14
+
15
+ This model is a fine-tuned version of [wanderer2k1/base_LLM_433MB_chunk_and_concat_no_domain](https://huggingface.co/wanderer2k1/base_LLM_433MB_chunk_and_concat_no_domain) on an unknown dataset.
16
+
17
+ ## Model description
18
+
19
+ More information needed
20
+
21
+ ## Intended uses & limitations
22
+
23
+ More information needed
24
+
25
+ ## Training and evaluation data
26
+
27
+ More information needed
28
+
29
+ ## Training procedure
30
+
31
+ ### Training hyperparameters
32
+
33
+ The following hyperparameters were used during training:
34
+ - learning_rate: 3.6e-05
35
+ - train_batch_size: 4
36
+ - eval_batch_size: 1
37
+ - seed: 42
38
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
39
+ - lr_scheduler_type: linear
40
+ - num_epochs: 1
41
+ - mixed_precision_training: Native AMP
42
+
43
+ ### Training results
44
+
45
+
46
+
47
+ ### Framework versions
48
+
49
+ - Transformers 4.40.1
50
+ - Pytorch 2.3.0+cu121
51
+ - Datasets 2.19.0
52
+ - Tokenizers 0.19.1
config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "wanderer2k1/base_LLM_433MB_chunk_and_concat_no_domain",
3
+ "abili": false,
4
+ "architectures": [
5
+ "MptForCausalLM"
6
+ ],
7
+ "attn_config": {
8
+ "model_type": ""
9
+ },
10
+ "d_model": 1024,
11
+ "emb_pdrop": 0.0,
12
+ "embedding_fraction": 1.0,
13
+ "expansion_ratio": 4,
14
+ "init_device": "cpu",
15
+ "initializer_range": 0.02,
16
+ "layer_norm_epsilon": 1e-05,
17
+ "learned_pos_emb": true,
18
+ "logit_scale": null,
19
+ "max_seq_len": 8192,
20
+ "model_type": "mpt",
21
+ "n_heads": 16,
22
+ "n_layers": 24,
23
+ "no_bias": true,
24
+ "norm_type": "low_precision_layernorm",
25
+ "pad_token_id": 128001,
26
+ "resid_pdrop": 0.0,
27
+ "rope": true,
28
+ "rope_theta": 100000,
29
+ "torch_dtype": "float32",
30
+ "transformers_version": "4.40.1",
31
+ "use_cache": false,
32
+ "verbose": 0,
33
+ "vocab_size": 128256
34
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "pad_token_id": 128009,
4
+ "transformers_version": "4.40.1",
5
+ "use_cache": false
6
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:58b327906b01610b768bd688fe2eb1131282433b6e926558ab27e983866d84c4
3
+ size 1733513312
runs/Oct21_02-56-56_kao-dgxa-e10-u17/events.out.tfevents.1729475849.kao-dgxa-e10-u17.1116475.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:08d470ae01aff756ae487955f21262f3bbe2c85eb1d823938d13b176eb3af7f0
3
+ size 122495
runs/Oct21_08-03-54_kao-dgxa-e10-u17/events.out.tfevents.1729494243.kao-dgxa-e10-u17.2649433.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33c75e22aaff89d1f351aeb894a622870772ae0a9e10c9e85118dc295cbd34b6
3
+ size 219390
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b72a7a3b6c0c99b310d9a7657e384f8270c6a7f53623e2c736167c279b48fdd
3
+ size 5112