Update README.md

README.md

@@ -1,3 +1,98 @@
----
-license: unknown
----
+I am not sure if anyone has ever demonstrated a 8-float embeddings, Two four-float attention heads, (approximately 9,526 parameters) one-megabyte GPT Model that produced coherent text (The Gettysburg Address) with correct punctuation, in response to a one-word prompt.  
+This result is reproducible.  I have accomplished this in practially all training runs with 'n_embd': 8, 'n_layer': 1, 'n_head': 2, 'n_inner': 128,   (with some undisclosed modifications of the model.py)
+
+This last training run was repeated to generate a pytorch parameter checkpoint on disk, for parameter-measurement purposes.
+model_checkpoint_epoch_30000_Nano_Gettysburg_GPT2_v1.5_loss_DisplayPerBatch.py_2024-11-26_00-48-36.pth
+Size_on_disk: 1.08 MB (1,138,688 bytes)
+Google Gemini Estimates that this nano LLM has about Parameters.
+
+Essential Hyperparameters (plus undisclosed modifications):
+'n_embd': 8, 'n_layer': 1, 'n_head': 2, 'n_inner': 128,
+##########################  OUTPUT PRINTED TO CMD CONSOLE IN WINDOWS 10 LAPTOP ##############################
+Epoch 29901/30000, Loss: 0.5203 Current learning rate: 0.000245
+...
+[Last] Batch Losses: 0.4611
+HyperParamters = {'vocab_size': [withheld], 'special_tokens': ['<[withheld]'], 'n_embd': 8, 'n_layer': 1, 'n_head': 2, 'n_inner': 128, 'max_sequence_len': [withehld], 'epochs': 30000, 'learning_rate': 0.001, 'batch_size': [withheld], 'dropout': 0.2}
+
+# --- Inference Examples --- at script line 507
+# Example 1: Recite the Gettysburg Address at script line 511
+Prompt: Four
+Total number of words and punctuation tokenized: 1
+
+Response:
+ four score and seven years ago our fathers brought forth , on this continent , a new nation , conceived in liberty , and dedicated to the proposition that all men are created equal . now we are engaged in a great civil war , testing whether that nation , or any nation so dedicated , or any nation so dedicated , can long endure . we have come to dedicate a new . we are met on a great battle - field of that sense , and so to dedicate a portion of that field , as a final resting - place for us that cause for us that those who here gave their lives , in final task remaining before us - that sense . that sense . it is altogether will us - that sense . but that sense . it is altogether for us - that that sense . it can never forget what they gave the world hatter bewildered badger baked blueberry strawberry biscuits popcorn fought of devotion - that that sense . it died that - that sense . the from that sense . the us - that sense . but that that - that that from . the from that cause for us to that - that that - that nation , for us to that that that that that that that - that - that that cause for us to that nation , and that - that - that that we of devotion - that that that that that that - that this ground . the us to that that that that this nation , to that that nation , to that that that that this war of the us to that ground . their lives , under god , under god , and that that nation , for - that that that that that we gave their lives , by the us the us to that this ground . apple score , by that this nation , under god , under god
+
+1562 Words in the Response
+
+#####################################################
+
+Google Gemini estimates the parameters within this model as follows
+
+Let's break down the parameter count for your Nano GPT-2 model based on its hyperparameters:
+
+1. Token Embeddings:
+
+[vocab size withheld]
+
+Embedding Dimension (n_embd): 8
+
+Parameters: 174 * 8 = 1392
+
+2. Positional Embeddings:
+
+Max Sequence Length: [withheld]
+
+Embedding Dimension (n_embd): 8
+
+Parameters: 512 * 8 = 4096
+
+3. Transformer Layers (n_layer = 1):
+
+For each layer, we have the following components:
+
+a) Self-Attention:
+
+Query (Q), Key (K), Value (V) Matrices: Each has dimensions (n_embd, n_embd / n_head) = (8, 8/2) = (8,4)
+so each Q, K and V matrix has (8x4 =32 parameters) and we have n_head * 3 matrices per layer
+so we have 2 * 3 = 6 matrices of size (8x4) or 6 * 32 = 192 parameters
+
+Output Projection: (n_embd, n_embd) = (8, 8) = 64 parameters
+
+Total Self-Attention Parameters per layer: 192 + 64 = 256 parameters
+
+b) Feed-Forward Network:
+
+Layer 1: (n_embd, n_inner) = (8, 128) = 1024 parameters
+
+Layer 2: (n_inner, n_embd) = (128, 8) = 1024 parameters
+
+Bias vectors = n_inner + n_embd = 128 + 8 = 136
+Total Feed-Forward Parameters per layer: 1024 + 1024+ 136 = 2184 parameters
+
+c) Layer Normalization:
+
+Two per layer - one in each section and we assume they have affine=True
+
+2 * n_embd (scale) + 2 * n_embd (bias) = 4 * n_embd = 4 * 8 = 32 parameters
+Total Layer Norm Parameters per layer: 32 parameters
+
+Total Parameters per Transformer Layer: 256 + 2184 + 32 = 2472 parameters
+
+4. Output Layer (Language Model Head):
+
+Final Linear Layer: (n_embd, vocab_size) = (8, [withheld]) = [withehld] parameters
+
+Bias vector: vocab_size = 174 parameters
+Total Output Layer Parameters: []= 1566 parameters
+
+5. Total Parameters in the model:
+
+Token Embeddings + Positional Embeddings + (n_layer * Layer Parameters) + Output Layer parameters
+
+1392 + 4096 + [] = 9526 parameters
+
+Therefore, your Nano GPT-2 model has approximately 9,526 parameters.
+
+---
+license: unknown
+---