End of training

README.md

@@ -0,0 +1,91 @@
+---
+license: mit
+base_model: gpt2
+tags:
+- generated_from_trainer
+model-index:
+- name: otu_gpt
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# otu_gpt
+
+This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 4.6030
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0002
+- train_batch_size: 256
+- eval_batch_size: 128
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- total_train_batch_size: 2048
+- total_eval_batch_size: 1024
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 30
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch | Step   | Validation Loss |
+|:-------------:|:-----:|:------:|:---------------:|
+| 5.413         | 1.0   | 6430   | 5.3753          |
+| 5.1384        | 2.0   | 12860  | 5.1224          |
+| 4.996         | 3.0   | 19290  | 4.9950          |
+| 4.9047        | 4.0   | 25720  | 4.9112          |
+| 4.8292        | 5.0   | 32150  | 4.8572          |
+| 4.7709        | 6.0   | 38580  | 4.8168          |
+| 4.7345        | 7.0   | 45010  | 4.7872          |
+| 4.6996        | 8.0   | 51440  | 4.7637          |
+| 4.6509        | 9.0   | 57870  | 4.7396          |
+| 4.6326        | 10.0  | 64300  | 4.7248          |
+| 4.6049        | 11.0  | 70730  | 4.7104          |
+| 4.5894        | 12.0  | 77160  | 4.6994          |
+| 4.5574        | 13.0  | 83590  | 4.6868          |
+| 4.5415        | 14.0  | 90020  | 4.6758          |
+| 4.5283        | 15.0  | 96450  | 4.6676          |
+| 4.4993        | 16.0  | 102880 | 4.6605          |
+| 4.486         | 17.0  | 109310 | 4.6532          |
+| 4.4675        | 18.0  | 115740 | 4.6467          |
+| 4.4588        | 19.0  | 122170 | 4.6410          |
+| 4.4402        | 20.0  | 128600 | 4.6347          |
+| 4.4182        | 21.0  | 135030 | 4.6292          |
+| 4.4031        | 22.0  | 141460 | 4.6262          |
+| 4.3857        | 23.0  | 147890 | 4.6200          |
+| 4.3726        | 24.0  | 154320 | 4.6150          |
+| 4.3575        | 25.0  | 160750 | 4.6130          |
+| 4.3369        | 26.0  | 167180 | 4.6102          |
+| 4.3106        | 27.0  | 173610 | 4.6064          |
+| 4.3068        | 28.0  | 180040 | 4.6044          |
+| 4.2803        | 29.0  | 186470 | 4.6026          |
+| 4.268         | 30.0  | 192900 | 4.6030          |
+
+
+### Framework versions
+
+- Transformers 4.39.3
+- Pytorch 2.2.2+cu121
+- Datasets 2.18.0
+- Tokenizers 0.15.2

config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 0,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 2,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 512,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.39.3",
+  "use_cache": true,
+  "vocab_size": 30000
+}

emissions.csv

@@ -0,0 +1,2 @@
+timestamp,project_name,run_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,on_cloud,pue
+2024-04-19T10:36:24,codecarbon,e0f1b38f-623d-4272-ad7f-42fab2e3fa40,62360.96308684349,3.5488854445795033,5.69087658193692e-05,82.5,0.0,282.9607129096985,1.4289430953671538,0.0,4.782612704686579,6.2115558000537385,Saudi Arabia,SAU,mecca region,,,Linux-5.14.0-162.23.1.el9_1.x86_64-x86_64-with-glibc2.34,3.9.13,2.2.3,24,Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz,8,8 x Tesla V100-SXM2-32GB,39.1165,22.2825,754.5619010925293,machine,N,1.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "transformers_version": "4.39.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4fa7bb0731432a2eeb943bd266c1b9170ab8e71c75044ae02cfd08e0c017916
+size 435544704

runs/Apr18_17-16-25_gpu212-02/events.out.tfevents.1713449794.gpu212-02.3270517.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9aeef4a3673d25b0411093835d971314650a775d1461927e5e828c13523c14a0
+size 115030

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,52 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d894311cb5ce24ea446aebb038953d746d7700b87545b4525c279842feda8e39
+size 4920