feat(transformers): Add native Transformers config
What does this PR do?
→ Sets model_type from new to gte and adds rope_parameters, so the checkpoint loads directly with AutoModel / AutoConfig in Transformers 🤗 without trust_remote_code.
→ Transformers support is being added in https://github.com/huggingface/transformers/pull/48416
→ ⚠️ Please hold this until the Transformers PR above has landed 🤗
Compatibility
→ rope_scaling and rope_theta are replaced by rope_parameters. rope_scaling cannot be kept alongside it, because Transformers reads that key first and ntk is not one of its rope types.
→ auto_map is left in place, but on this revision a trust_remote_code=True load no longer finds rope_theta / rope_scaling and falls back to the remote defaults (rope_theta=10000.0, rope_scaling=None) instead of this checkpoint's values. That changes the embeddings, and the gap grows with sequence length, so anyone staying on an older Transformers version should pin a revision from before this PR.
Produced by?
→ rope_parameters is the standardised form of the existing rope_scaling / rope_theta pair. GTE applies NTK scaling statically at construction by rescaling the base and dividing the inverse frequencies by factor ** (2 / head_dim), which is exactly a linear scaling with base rope_theta * factor, so the inverse frequencies are unchanged.
→ Outputs are reproduced against the original implementation on this checkpoint (max relative difference 1.4e-06), and the slow integration tests in the Transformers PR run against this revision.