PatternSmith
PatternSmith is a decoder-only Transformer that converts natural-language descriptions into regular expressions.
Model details
- Architecture: GPT-style decoder-only Transformer
- Framework: PyTorch
- Parameters: 7,242,624
- Tokenizer: GPT-2 BPE (
tiktoken) - Vocabulary size: 50,257
- Context length: 128
- Transformer blocks: 4
- Attention heads: 4
- Embedding dimension: 128
- Feed-forward dimension: 512
- Training: Supervised next-token prediction
- Initialization: Random weights
Training dataset
NL-RX-Turk, cleaned and split into:
- Training: 6,245 examples
- Validation: 961 examples
- Test: 2,402 examples
Evaluation results
| Metric | Result |
|---|---|
| Best validation loss | 0.2044 |
| Validation perplexity | 1.2268 |
| Test exact match | 37.14% |
| Semantic equivalence | 55.54% |
| Python regex compilation | 96.96% |
Semantic equivalence was evaluated using regexbench BRICS FULLMATCH.
Intended use
Research and experimentation with natural-language-to-regex generation. This model is a portfolio project, not a production-ready regex generation system.
Limitations
- Exact-match accuracy is limited.
- Syntactically valid regex does not guarantee semantic correctness.
- Predictions should be validated against representative test cases.
- The model may fail on complex or ambiguous descriptions.
Checkpoint
The trained PyTorch checkpoint is available as best.pt in this repository.
Source code
GitHub: https://github.com/ramu1904/PatternSmith
The model architecture and inference implementation are available in the GitHub repository.