ankitbsingh/research-generation

Working DeiT implementation for Generation

Overview

This is an experimental DeiT codebase for Generation. It keeps the base setup intentionally manageable so architecture changes can be inspected before a full training run.

Expect rough edges — this is a working sketch, not a polished release.

---

Repository Status

One Python module contains the model and a runnable example. Generated configuration lives in config.json; the recipe in training_args.json. The checkpoint is an initialization artifact and should not be read as a trained model. Scores are deliberately absent.

---

Limitations

The checkpoint is untrained and unaudited. Treat the repository as an experimental starting point; anything learned from a future trained version must be documented separately.

---

Quick Check

python predict.py --help

Inspect the script's __main__ block for its generated smoke-test example. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.

---

Architecture

• Architecture: DeiT
• Scale: base
• Attention: sliding window
• Fusion: tucker
• Activation: gelu
• Normalization: scalenorm

---

Default Experiment Recipe

Defaults: novograd optimizer, exponential schedule. Treat these as starting points; a real run should sweep learning rate and batch size before drawing conclusions.

---

Files

• predict.py — primary artifact
• README.md — this documentation
• config.json — architecture configuration
• training_args.json — default experiment settings
• model.safetensors — initialization checkpoint
• examples/sample_usage.py — usage sketch

---

License

Distributed under the unknown license. External datasets keep their own terms — check them separately.