PharmaGPT-336M

A 336M parameter GPT language model trained entirely from scratch on 200K synthetic pharmaceutical documents across 6 manufacturing domains.

No pre-trained weights. No fine-tuning. Every component built from scratch: custom BPE tokenizer, full transformer architecture (RoPE + RMSNorm + SwiGLU), training loop, and evaluation pipeline.

Paper: ArXiv preprint (coming soon)
Blog: Medium article (coming soon)
Code: Included in this repository

---

Quick Start

import torch
from tokenizers import Tokenizer

# Download model files from this repo, then:
ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)

# Reconstruct model from saved config
from model import GPT  # model.py included in this repo
model = GPT(ckpt["model_config"])
model.load_state_dict(ckpt["model"])
model.eval()

# Load tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")

# Generate pharmaceutical text
prompt = "<|deviation|>\nDuring manufacturing of Batch B-NDL-2026"
ids = torch.tensor([tok.encode(prompt).ids])
output = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=50)
print(tok.decode(output[0].tolist()))

Generation with Different Domains

prompts = {
    "deviation": "<|deviation|>\nDuring routine inspection of the tablet coating line",
    "batch_record": "<|batch_record|>\nBATCH PRODUCTION RECORD\nProduct: Metformin HCl 500mg Tablets",
    "sop": "<|sop|>\nSOP-ENV-205 | Environmental Monitoring Program",
    "stability": "<|stability_study|>\nSTABILITY STUDY REPORT\nProduct: Adalimumab 40mg/0.8mL",
    "pharmacovigilance": "<|icsr|>\nA 72-year-old female patient with history of diabetes",
    "scientific": "<|scientific_paper|>\nObjective: To evaluate the impact of granulation",
}

for domain, prompt in prompts.items():
    ids = torch.tensor([tok.encode(prompt).ids])
    out = model.generate(ids, max_new_tokens=150, temperature=0.8, top_k=50)
    print(f"\n{'='*60}\n[{domain.upper()}]\n{'='*60}")
    print(tok.decode(out[0].tolist()))

---

Model Details

Property	Value
Parameters	336,380,928 (336M)
Architecture	Decoder-only Transformer (GPT)
Embedding Dimension	1024
Attention Heads	16
Layers	24
Context Length	512 tokens
Vocabulary	32,000 tokens (custom BPE)
Positional Encoding	Rotary (RoPE), base=10000
Normalization	RMSNorm (ε=1e-6)
Activation	SwiGLU (FFN hidden=2752)
Bias	None (all linear layers)
Weight Tying	Embedding ↔ LM Head
Dropout	0.1

Architecture Highlights

This model implements the same architectural innovations found in LLaMA/Mistral, all coded from scratch:

• RoPE (Rotary Positional Embeddings) — encodes relative position through rotation of Q/K vectors
• RMSNorm — faster, simpler alternative to LayerNorm (no mean subtraction)
• SwiGLU — gated feed-forward network with Swish activation
• No bias in any linear layer — modern simplification
• Weight tying — token embedding and output projection share parameters
• Pre-norm architecture — normalize before attention/FFN, not after

---

Training Details

Parameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.95, wd=0.1)
Learning Rate	2e-4 (peak), cosine decay
Warmup	500 steps (linear)
Batch Size	8 micro × 4 grad accum = 32 effective
Iterations	15,000
Precision	float16 mixed precision
Gradient Clipping	1.0 (global norm)
Gradient Checkpointing	Enabled
Hardware	NVIDIA T4 (16GB), Kaggle free tier
Training Time	~9 hours
Cost	$0 (free compute)

Training Results

Metric	Value
Final Training Loss	0.3748
Best Validation Loss	0.3748
Validation Perplexity	1.45
Tokens Processed	~245M

Note: Low perplexity reflects the structured/templated nature of synthetic training data. Real-world pharmaceutical text would yield higher perplexity.

---

Training Data: 6 Pharmaceutical Domains

The model was trained on 200K synthetic documents (~32M tokens) generated across six pharmaceutical manufacturing domains:

1. Manufacturing Deviation Reports (~33K samples)

Equipment failures, process excursions, out-of-specification results, root cause analysis (Ishikawa, 5-Why), CAPA documentation following ICH Q10.

2. Batch Production Records (~33K samples)

Raw material dispensing, process step documentation, in-process controls, critical process parameters (CPPs), yield calculations, lot disposition decisions.

3. Standard Operating Procedures (~33K samples)

Cleaning validation, environmental monitoring, aseptic processing, water system maintenance (WFI, PW), equipment qualification — in Q&A format.

4. Stability Studies (~33K samples)

ICH Q1A(R2) study designs, accelerated (40°C/75% RH) and long-term (25°C/60% RH) conditions, assay trending, degradation products, shelf-life determination.

5. Pharmacovigilance Case Reports (~33K samples)

Individual Case Safety Reports (ICSRs), adverse event narratives, MedDRA coding, WHO-UMC causality assessment (certain/probable/possible/unlikely).

6. Scientific Writing (~33K samples)

Formulation development, Design of Experiments (DoE), analytical method development/validation, dissolution studies, results and discussion sections.

---

Special Tokens

Token	Purpose	Example Use
<|deviation|>	Start of deviation report	Triggers investigation-style generation
<|batch_record|>	Start of batch record	Triggers manufacturing record format
<|sop|>	Start of SOP document	Triggers procedural/Q&A format
<|stability_study|>	Start of stability study	Triggers ICH-compliant study format
<|icsr|>	Start of pharmacovigilance case	Triggers adverse event narrative
<|scientific_paper|>	Start of scientific writing	Triggers academic/research style
<|end|>	End of document	Marks document boundary

---

Repository Contents

├── best_model.pt          # Full checkpoint (model weights + config + metadata)
├── config.json            # Architecture specification (JSON)
├── tokenizer/
│   └── tokenizer.json     # Trained BPE tokenizer (32K vocab)
├── model.py               # Complete model source code (GPT + all components)
├── tokenizer.py           # Tokenizer training/loading utilities
└── README.md              # This file

Loading Without model.py

If you want to inspect the architecture without running the custom code:

import torch, json

# Load config
with open("config.json") as f:
    config = json.load(f)
print(config)
# {'vocab_size': 32000, 'n_embd': 1024, 'n_head': 16, 'n_layer': 24, ...}

# Load checkpoint metadata
ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)
print(f"Keys: {ckpt.keys()}")
print(f"Val loss: {ckpt.get('best_val_loss')}")
print(f"Iteration: {ckpt.get('iter_num')}")

---

Intended Use

Primary Use Cases

• Educational: Understanding how modern GPT architectures work end-to-end
• Research baseline: Starting point for pharmaceutical NLP research
• Template generation: Generating draft pharmaceutical document structures
• Domain adaptation: Fine-tuning on real pharmaceutical data for production use

Out-of-Scope Uses

• Clinical decision-making: This model generates plausible but NOT factually verified content
• Regulatory submissions: Generated text requires expert review and verification
• Production deployment without validation: The model was trained on synthetic data only
• General-purpose chat: This is a domain-specific completion model, not a chatbot

---

Limitations and Risks

Limitation	Impact	Mitigation
Synthetic training data	May generate structurally correct but factually wrong content	Always verify with domain experts
336M parameters	Limited reasoning and knowledge capacity	Use as starting point, not final solution
English only	Cannot process multilingual pharmaceutical docs	Extend training data for other languages
No instruction tuning	Cannot follow complex instructions	Fine-tune with instruction data
Context length (512)	Cannot process long documents in one pass	Chunk documents or extend context

Ethical Considerations

• Generated pharmaceutical content should NEVER be used for actual drug manufacturing without expert review
• The model may reproduce biases present in the synthetic data templates
• Not intended as a replacement for qualified pharmaceutical professionals

---

Citation

If you use PharmaGPT in your research, please cite:

@misc{chaturvedi2026pharmagpt,
  title={PharmaGPT: A Domain-Specific Language Model for Pharmaceutical Manufacturing Intelligence Trained from Scratch on Synthetic Data},
  author={Chaturvedi, Parth},
  year={2026},
  howpublished={\url{https://huggingface.co/ParthChat1802/PharmaGPT-336M}},
}

---

Technical Notes for Reproducibility

Checkpoint Format

The best_model.pt file is a PyTorch checkpoint dictionary containing:

{
    "model": OrderedDict,        # model.state_dict()
    "model_config": GPTConfig,   # dataclass with architecture params
    "config": dict,              # training configuration
    "iter_num": int,             # iteration at save time
    "best_val_loss": float,      # best validation loss achieved
}

System Requirements

• Inference: Any machine with 2GB+ RAM and PyTorch installed
• Training (reproduce): NVIDIA GPU with 8GB+ VRAM, or Apple M-series with 16GB+ unified memory
• Dependencies: torch>=2.0, tokenizers>=0.13

Reproducing Training

git clone <source-repo>
cd gpt-from-scratch
pip install -r requirements.txt

# Generate data
python -m data.generators.master_generator

# Train tokenizer + model
python -m src.train_pharma

Or use the Kaggle notebook for GPU-accelerated training (see repository).

---

License

Apache 2.0 — Use freely for any purpose (commercial, research, educational). Attribution appreciated but not legally required beyond the license notice.