Karaageankita/deit-experiment

DeiT experiments: Multitask

Overview

A research-oriented DeiT prototype targeting Multitask. The included large setup documents defaults and file formats without presenting unverified performance numbers.

Training details

• Weights initialized with a kaiming normal scheme.
• Runs on a single consumer GPU.
• Full smoke test takes minutes, not hours.
• No distributed setup required.

Limitations

The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here.

Changelog

• 2026-10-17 — initial upload
• 2026-10-20 — config and docs sync

Default experiment recipe

Defaults: adam optimizer, cosine schedule. Treat these as starting points; a real run should sweep learning rate and batch size before drawing conclusions.

Quick check

python train.py --help

Inspect the script's __main__ block for its generated smoke-test example. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.

Repository status

• The Python file contains the model and runnable example or training entry point.
• config.json records the generated architecture settings.
• training_args.json records the default experiment recipe.
• model.safetensors is a valid initialization checkpoint for smoke tests; it is not presented as a trained benchmark checkpoint.
• No benchmark score is claimed in this repository.

Evaluation guidance

A useful first evaluation would use a task-specific held-out set, report the task metric across at least three seeds, and include a matched-capacity baseline. Keep training logs and environment versions with any published result.

Environment

• Python: >= 3.9
• torch: 2.x
• safetensors: any recent
• transformers: optional, for loaders

Files

• train.py — primary artifact
• README.md — this documentation
• config.json — architecture configuration
• training_args.json — default experiment settings
• model.safetensors — initialization checkpoint
• USE_POLICY.md — use and redistribution notes

License

Released under bsd-2-clause. Review the source-data terms separately when this repository is used with external datasets.