adamo1139/Yi-34B-200K-AEZAKMI-RAW-TOXIC-XLCTX-2303

NEWS

This model has been renamed from adamo1139/Yi-34B-200K-AEZAKMI-XLCTX-v3 to adamo1139/Yi-34B-200K-AEZAKMI-RAW-TOXIC-XLCTX-2303 on 2024-03-30.
I am not happy with how often this model starts enumerating lists and I plan to improve toxic dpo dataset to fix it. Due to this, I don't think it deserves to be called AEZAKMI v3 and will be just a next testing iteration of AEZAKMI RAW TOXIC.
I think I will be uploading one EXL2 quant before moving onto a different training run.

Model description

Yi-34B 200K XLCTX base model fine-tuned on RAWrr_v2 (DPO), AEZAKMI-3-6 (SFT) and unalignment/toxic-dpo-0.1 (DPO) datasets. Training took around 20-30 hours total on RTX 3090 Ti, all finetuning was done locally. It's like airoboros but with less gptslop, no refusals and less typical language used by RLHFed OpenAI models, with extra spicyness. Say goodbye to "It's important to remember"!
Prompt format is standard chatml. Don't expect it to be good at math, riddles or be crazy smart. My end goal with AEZAKMI is to create a cozy free chatbot. Cost of this fine-tune is about $5-$10 in electricity. Base model used for fine-tuning was Yi-34B-200K model shared by 01.ai, the newer version that has improved long context needle in a haystack retrieval. They didn't give it a new name, giving it numbers would mess up AEZAKMI naming scheme by adding a second number, so I will be calling it XLCTX.

I had to lower max_positional_embeddings in config.json and model_max_length for training to start, otherwise I was OOMing straight away. This attempt had both max_position_embeddings and model_max_length set to 4096, which worked perfectly fine. I then reversed this to 200000 once I was uploading it. I think it should keep long context capabilities of the base model.

In my testing it seems less unhinged than adamo1139/Yi-34b-200K-AEZAKMI-RAW-TOXIC-2702 and maybe a touch less uncensored, but still very much uncensored even with default system prompt "A chat." If you want to see training scripts, let me know and I will upload them. LoRAs are uploaded here adamo1139/Yi-34B-200K-AEZAKMI-XLCTX-v3-LoRA

Quants!

EXL2 quants coming soon, I think I will start by uploading 4bpw quant in a few days.

Prompt Format

I recommend using ChatML format, as this was used during fine-tune.
Here's a prompt format you should use, you can set a different system message, model was trained on SystemChat dataset, so it should respect system prompts fine.

<|im_start|>system
A chat.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Intended uses & limitations

Use is limited by Yi license.
Some datasets that were used prohibit commercial use, so please use non-commercially only.

Known Issues

This model loves making numbered lists, to an exhaustion.

It's more of an assistant feel rather than a human feel, at least with system chat "A chat."
Long context wasn't tested yet, it should work fine though - feel free to give me feedback about it.

Credits

Thanks to unsloth and huggingface team for providing software packages used during fine-tuning.
Thanks to Jon Durbin, abacusai, huggingface, sandex, NobodyExistsOnTheInternet, Nous-Research for open sourcing datasets I included in the AEZAKMI dataset.
AEZAKMI is basically a mix of open source datasets I found on HF, so without them this would not be possible at all.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	64.39
AI2 Reasoning Challenge (25-Shot)	64.85
HellaSwag (10-Shot)	84.76
MMLU (5-Shot)	74.48
TruthfulQA (0-shot)	37.14
Winogrande (5-shot)	81.06
GSM8k (5-shot)	44.05