BigHuggyD/anthracite-org_magnum-v2-123b_exl2_3.0bpw_h6

This is the sixth in a series of models designed to replicate the prose quality of the Claude 3 models, specifically Sonnet and Opus. This model is fine-tuned on top of Mistral-Large-Instruct-2407.

Prompting

Model has been Instruct tuned with the Mistral formatting. A typical input would look like this:

<s>[INST] SYSTEM MESSAGE\nUSER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]

We also provide SillyTavern presets for Context and Instruct respectively.

The Mistral preset included in SillyTavern seems to be misconfigured by default, so we recommend using these as a replacement.

Credits

This model has been a team effort, and the credits goes to all members of Anthracite.

Training

The training was done for 1.5 epochs. We used 8x AMD Instinct™ MI300X Accelerators for the full-parameter fine-tuning of the model.

In addition to this, we noticed that Mistral Large models seemed much more sensitive to learning rate adjustments than other models:

We hypothesize this is primarily due to the particularly narrow and low variance weight distributions typical of Mistral derived models regardless of their scale.

In the end, due to the costs that would be involved in training another full 2 epochs run ($600) on an even lower rate, we settled on our third attempt: 2e-6 with an effective batch size of 64. We chose to publish the 1.5 epoch run after manually testing and comparing it.

Also, we notice a correlation between the significance of the 2nd epoch loss drop and the strength of the learning rate, implying 4e-6 leads to more catastrophic forgetting.

Safety

...