Update NeoLLM model and configuration sources
#1
by KitsuVp - opened
Synchronize the model and configuration sources with the current NeoLLM implementation.
- Implement the Copy-Paste head used by the saved configuration and checkpoint.
- Protect configured structural tokens during MEAP corruption and exclude MEAP mask positions from Copy-Paste sources.
- Provide optional token-specific lexical copy keys for patch training, disabled by default, and include the other already-supported optional model features in these source files.
Compatibility: the existing config.json enables Copy-Paste and its checkpoint contains the head weights, but the currently published modeling source does not use that head. Therefore this update changes inference to use the saved Copy-Paste head. The config.json and checkpoint weights are unchanged.
Validation: the public checkpoint loads with no missing or unexpected tensors; Transformers AutoConfig/AutoModelForCausalLM loading and a finite-logit GPU inference passed. Additional MEAP and Copy-Paste tests include K=1 and K>1 and a compiled Trainer smoke test. Model-quality effects require a controlled training evaluation.
KitsuVp changed pull request status to merged