Upload 11 files

README.md

@@ -1,3 +1,108 @@
----
-license: apache-2.0
----
+# Pythia-14M Fine-Tuned for High-Quality English Sentence Generation
+
+This model is a fine-tuned version of the Pythia-14M language model, optimized for generating high-quality English sentences. It builds upon the base model [agentlans/pythia-14m-finewebedu-sentences](https://huggingface.co/agentlans/pythia-14m-finewebedu-sentences) and has been further trained on a curated dataset of well-formed English sentences [agentlans/high-quality-english-sentences](https://huggingface.co/datasets/agentlans/high-quality-english-sentences).
+
+## Model Description
+
+The model is based on the Pythia-14M architecture, which is a relatively compact language model. It has been fine-tuned specifically for generating (mostly) grammatically correct and coherent English sentences across a variety of topics and styles.
+
+## Intended Uses & Limitations
+
+This model is designed for:
+- Generating high-quality English sentences
+- Completing partial sentences
+- Assisting with writing tasks that require well-formed English
+
+Limitations:
+- Not suitable for tasks requiring deep domain knowledge
+- May struggle with very long-form text generation
+- Fails on non-English text
+- It's tiny so don't expect too much
+
+## Training Data
+
+The model was fine-tuned on a combination of datasets:
+- Web-scraped educational content (finewebedu)
+- High-quality web text (fineweb)
+- Filtered Common Crawl data (C4)
+
+For the composition and preprocessing of the training data, see [agentlans/high-quality-english-sentences](https://huggingface.co/datasets/agentlans/high-quality-english-sentences).
+
+## How To Use
+
+To generate 10 random sentences starting from an empty string on a CUDA device:
+
+```python
+from transformers import pipeline, set_seed
+
+generator = pipeline('text-generation', model='agentlans/pythia-14m-sentences', device='cuda')
+
+set_seed(1234)
+results = generator("", max_length=100, num_return_sequences=10, do_sample=True)
+
+for x in results:
+	print(x['generated_text'])
+```
+
+Output:
+```text
+The most common cause of the number of diseases is the common cause of death.
+And there are many people in the war.
+The average household income is 35.5 percent.
+He was the most influential theologians of the country in this world.
+On the other hand, the students will be able to learn the value of the current and the time.
+However, the effect of the study would be greater than that of a drug-related drug drug.
+To understand today, our nation's largest international commitment to the use of new technology and technology across the country.
+On Sunday, the UK was first held in the state of the Australian, where a foreign trade union was used since the first year.
+I've said that the program is most effective in education in the middle of the world.
+So a year, it is important to identify a community where a student has a disability.
+```
+
+To let the model continue the sentence:
+
+```python
+results = generator("The meaning of life is", max_length=100, num_return_sequences=10, do_sample=True)
+for x in results:
+	print(x['generated_text'])
+```
+
+Output:
+```text
+The meaning of life is one of the most extraordinary stories of the great world, and some of the most brilliant examples of the world of science.
+The meaning of life is to develop.
+The meaning of life is to the person, or to make it a personal impression of what is the case for the reader.
+The meaning of life is no longer the most important concept of the human language.
+The meaning of life is the form of a personal or personal character.
+The meaning of life is the world's real and our future.
+The meaning of life is the true one of the nation's largest historical experiences.
+The meaning of life is the basis of the Church's first, the church of the Holy Spirit, and a living faith.
+The meaning of life is that the law requires that the truth be lost.
+The meaning of life is the best reason for the poor and poor economy.
+```
+
+## Training Procedure
+
+The model was trained using the following hyperparameters:
+- Learning rate: 5e-05
+- Train batch size: 8 
+- Eval batch size: 8
+- Optimizer: Adam (betas=(0.9,0.999), epsilon=1e-08)
+- LR scheduler: Linear
+- Number of epochs: 3.0
+
+## Evaluation Results
+
+On the evaluation set, the model achieved:
+- Loss: 6.2540
+- Accuracy: 0.1776
+
+## Ethical Considerations
+
+As with any text generation model, users should be aware of potential biases in the training data that may be reflected in the model's outputs. The model should not be used to generate or propagate harmful content.
+
+## Technical Specifications
+
+- Library: Transformers 4.45.1
+- Framework: PyTorch 2.4.1+cu121
+- Datasets: 3.0.1
+- Tokenizers: 0.20.0

all_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 3.0,
+    "eval_accuracy": 0.1775901465932144,
+    "eval_loss": 6.25395393371582,
+    "eval_runtime": 26.5184,
+    "eval_samples": 4553,
+    "eval_samples_per_second": 171.692,
+    "eval_steps_per_second": 21.495,
+    "perplexity": 520.0650675835933,
+    "total_flos": 5764753863475200.0,
+    "train_loss": 6.457221655868903,
+    "train_runtime": 1165.3935,
+    "train_samples": 40997,
+    "train_samples_per_second": 105.536,
+    "train_steps_per_second": 13.193
+}

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "agentlans/pythia-14m-finewebedu-sentences",
+  "architectures": [
+    "GPTNeoXForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "eos_token_id": 0,
+  "hidden_act": "gelu",
+  "hidden_dropout": 0.0,
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 512,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neox",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 6,
+  "partial_rotary_factor": 0.25,
+  "rope_scaling": null,
+  "rope_theta": 10000,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 0.25,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.45.1",
+  "use_cache": true,
+  "use_parallel_residual": true,
+  "vocab_size": 50304
+}

eval_results.json

@@ -0,0 +1,10 @@
+{
+    "epoch": 3.0,
+    "eval_accuracy": 0.1775901465932144,
+    "eval_loss": 6.25395393371582,
+    "eval_runtime": 26.5184,
+    "eval_samples": 4553,
+    "eval_samples_per_second": 171.692,
+    "eval_steps_per_second": 21.495,
+    "perplexity": 520.0650675835933
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.45.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d57815e9690ab850851289477ac83e03a76c877aea0f87ed07a16c3f13da5507
+size 56279344

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,214 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 3.0,
+    "total_flos": 5764753863475200.0,
+    "train_loss": 6.457221655868903,
+    "train_runtime": 1165.3935,
+    "train_samples": 40997,
+    "train_samples_per_second": 105.536,
+    "train_steps_per_second": 13.193
+}

trainer_state.json

@@ -0,0 +1,252 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 15375,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0975609756097561,
+      "grad_norm": 19.442411422729492,
+      "learning_rate": 4.8373983739837406e-05,
+      "loss": 6.7559,
+      "step": 500
+    },
+    {
+      "epoch": 0.1951219512195122,
+      "grad_norm": 22.672739028930664,
+      "learning_rate": 4.6747967479674795e-05,
+      "loss": 6.6932,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2926829268292683,
+      "grad_norm": 21.795516967773438,
+      "learning_rate": 4.51219512195122e-05,
+      "loss": 6.6652,
+      "step": 1500
+    },
+    {
+      "epoch": 0.3902439024390244,
+      "grad_norm": 19.84381866455078,
+      "learning_rate": 4.3495934959349595e-05,
+      "loss": 6.6335,
+      "step": 2000
+    },
+    {
+      "epoch": 0.4878048780487805,
+      "grad_norm": 14.22912883758545,
+      "learning_rate": 4.186991869918699e-05,
+      "loss": 6.6248,
+      "step": 2500
+    },
+    {
+      "epoch": 0.5853658536585366,
+      "grad_norm": 14.391462326049805,
+      "learning_rate": 4.0243902439024395e-05,
+      "loss": 6.5929,
+      "step": 3000
+    },
+    {
+      "epoch": 0.6829268292682927,
+      "grad_norm": 19.81720733642578,
+      "learning_rate": 3.861788617886179e-05,
+      "loss": 6.5589,
+      "step": 3500
+    },
+    {
+      "epoch": 0.7804878048780488,
+      "grad_norm": 15.33761978149414,
+      "learning_rate": 3.699186991869919e-05,
+      "loss": 6.5327,
+      "step": 4000
+    },
+    {
+      "epoch": 0.8780487804878049,
+      "grad_norm": 14.190281867980957,
+      "learning_rate": 3.5365853658536584e-05,
+      "loss": 6.5175,
+      "step": 4500
+    },
+    {
+      "epoch": 0.975609756097561,
+      "grad_norm": 16.57828712463379,
+      "learning_rate": 3.373983739837399e-05,
+      "loss": 6.5137,
+      "step": 5000
+    },
+    {
+      "epoch": 1.0731707317073171,
+      "grad_norm": 16.75761604309082,
+      "learning_rate": 3.2113821138211384e-05,
+      "loss": 6.495,
+      "step": 5500
+    },
+    {
+      "epoch": 1.170731707317073,
+      "grad_norm": 18.840726852416992,
+      "learning_rate": 3.048780487804878e-05,
+      "loss": 6.4757,
+      "step": 6000
+    },
+    {
+      "epoch": 1.2682926829268293,
+      "grad_norm": 17.630483627319336,
+      "learning_rate": 2.886178861788618e-05,
+      "loss": 6.4633,
+      "step": 6500
+    },
+    {
+      "epoch": 1.3658536585365852,
+      "grad_norm": 16.721818923950195,
+      "learning_rate": 2.7235772357723577e-05,
+      "loss": 6.4462,
+      "step": 7000
+    },
+    {
+      "epoch": 1.4634146341463414,
+      "grad_norm": 14.650636672973633,
+      "learning_rate": 2.5609756097560977e-05,
+      "loss": 6.4404,
+      "step": 7500
+    },
+    {
+      "epoch": 1.5609756097560976,
+      "grad_norm": 13.825970649719238,
+      "learning_rate": 2.3983739837398377e-05,
+      "loss": 6.4326,
+      "step": 8000
+    },
+    {
+      "epoch": 1.6585365853658538,
+      "grad_norm": 11.85326862335205,
+      "learning_rate": 2.2357723577235773e-05,
+      "loss": 6.4239,
+      "step": 8500
+    },
+    {
+      "epoch": 1.7560975609756098,
+      "grad_norm": 13.92196273803711,
+      "learning_rate": 2.073170731707317e-05,
+      "loss": 6.4098,
+      "step": 9000
+    },
+    {
+      "epoch": 1.8536585365853657,
+      "grad_norm": 12.077308654785156,
+      "learning_rate": 1.9105691056910573e-05,
+      "loss": 6.3987,
+      "step": 9500
+    },
+    {
+      "epoch": 1.951219512195122,
+      "grad_norm": 12.406614303588867,
+      "learning_rate": 1.747967479674797e-05,
+      "loss": 6.3957,
+      "step": 10000
+    },
+    {
+      "epoch": 2.048780487804878,
+      "grad_norm": 14.001736640930176,
+      "learning_rate": 1.5853658536585366e-05,
+      "loss": 6.3752,
+      "step": 10500
+    },
+    {
+      "epoch": 2.1463414634146343,
+      "grad_norm": 12.691810607910156,
+      "learning_rate": 1.4227642276422764e-05,
+      "loss": 6.3566,
+      "step": 11000
+    },
+    {
+      "epoch": 2.2439024390243905,
+      "grad_norm": 10.062420845031738,
+      "learning_rate": 1.2601626016260162e-05,
+      "loss": 6.3492,
+      "step": 11500
+    },
+    {
+      "epoch": 2.341463414634146,
+      "grad_norm": 11.78906536102295,
+      "learning_rate": 1.0975609756097562e-05,
+      "loss": 6.3447,
+      "step": 12000
+    },
+    {
+      "epoch": 2.4390243902439024,
+      "grad_norm": 13.368131637573242,
+      "learning_rate": 9.34959349593496e-06,
+      "loss": 6.339,
+      "step": 12500
+    },
+    {
+      "epoch": 2.5365853658536586,
+      "grad_norm": 12.125652313232422,
+      "learning_rate": 7.723577235772358e-06,
+      "loss": 6.3305,
+      "step": 13000
+    },
+    {
+      "epoch": 2.6341463414634148,
+      "grad_norm": 13.748695373535156,
+      "learning_rate": 6.0975609756097564e-06,
+      "loss": 6.3205,
+      "step": 13500
+    },
+    {
+      "epoch": 2.7317073170731705,
+      "grad_norm": 13.787367820739746,
+      "learning_rate": 4.471544715447155e-06,
+      "loss": 6.3196,
+      "step": 14000
+    },
+    {
+      "epoch": 2.8292682926829267,
+      "grad_norm": 15.013029098510742,
+      "learning_rate": 2.8455284552845528e-06,
+      "loss": 6.3116,
+      "step": 14500
+    },
+    {
+      "epoch": 2.926829268292683,
+      "grad_norm": 15.244904518127441,
+      "learning_rate": 1.2195121951219514e-06,
+      "loss": 6.3107,
+      "step": 15000
+    },
+    {
+      "epoch": 3.0,
+      "step": 15375,
+      "total_flos": 5764753863475200.0,
+      "train_loss": 6.457221655868903,
+      "train_runtime": 1165.3935,
+      "train_samples_per_second": 105.536,
+      "train_steps_per_second": 13.193
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 15375,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5764753863475200.0,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}