Brahmavamsi/nlp

Custom Language Model for Text Generation

Model Overview

This project focuses on building a custom language model trained to generate coherent text sequences. The model was trained using a specific corpus and designed for tasks such as generating stories, dialogues, or other forms of text, based on a given prompt.

The model follows a basic neural network architecture and has been trained on a large dataset of text. It can generate sequences of reasonable length while maintaining syntactic and contextual coherence.

Training Process

The model was trained using a dataset of 72,000,000 tokens. The training process involved several iterations with the following key hyperparameter settings:

Hyperparameters

• Batch Size: 64
• Sequence Length (Block Size): 16
• Embedding Dimensions (n_embd): 32
• Hidden Layer Sizes:
  • Layer 1: 512 units
  • Layer 2: 256 units
  • Layer 3: 128 units
• Dropout: 0.3 (helps prevent overfitting)
• Learning Rate: 1e-4 (lower rate to ensure gradual, steady learning)
• Optimizer: AdamW (Adam with weight decay to improve generalization)
• Maximum Training Iterations: 1,000,000
• Evaluation Interval: Every 10,000 steps

Training was done over a million iterations, and model checkpoints were saved at regular intervals (every 50,000 steps) to allow resuming training from the last saved point.

Checkpoints and Model Progress

• Checkpoints: Saved at steps 50,000, 100,000, 150,000, etc.
• Training and Validation Losses: Logged during evaluation intervals to track model progress.

Loss Curves

Training and validation loss values consistently decreased over time, indicating effective learning and generalization by the model. Checkpoint data can be used to resume training and continue improving the model.

Text Generation

The model can be used to generate text given an initial prompt. Below is an example of how to load the model and use it to generate sequences:

# Save the model
torch.save({
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'vocab': chars,
    'stoi': stoi,
    'itos': itos,
    'hyperparams': {
        'n_embd': n_embd,
        'n_hidden': n_hidden,
        'dropout': dropout,
        'vocab_size': vocab_size,
        'block_size': block_size
    }
}, 'simple_rnn_model.pth')

print("Model saved successfully.")

# Function to load the model
def load_model(model_path):
    checkpoint = torch.load(model_path)
    model = SimpleRNNModel()
    model.load_state_dict(checkpoint['model_state_dict'])
    model.to(device)
    model.eval()
    return model, checkpoint['stoi'], checkpoint['itos'], checkpoint['hyperparams']

# Function to generate text
def generate_text(model, start_text, max_new_tokens):
    encode = lambda s: [model_stoi[c] for c in s]
    decode = lambda l: ''.join([model_itos[i] for i in l])

    context = torch.tensor(encode(start_text), dtype=torch.long, device=device).unsqueeze(0)
    generated = model.generate(context, max_new_tokens=max_new_tokens)
    return decode(generated[0].tolist())

# Load the saved model
loaded_model, model_stoi, model_itos, hyperparams = load_model('simple_rnn_model.pth')

# Generate text
start_text = "Once upon a time"
generated_text = generate_text(loaded_model, start_text, max_new_tokens=500)
print(generated_text)




example output
Once upon a time, there was a brave knight who ventured into the dark forest in search of a hidden treasure. Along the way, he met...