license: other
license_name: yi-other
Yeet 51b 200k
This model is a rotating-stack merge of three Yi 34b 200k models in a 51b (90 layer) configuration. My reasoning behind this merge was twofold: I'd never seen a stacked merge made from 34b models, and I thought that maybe this could give near-70b performance, but with a much larger context window while still fitting within 48GB of VRAM. I think the results are quite good. The model performs on par with many 70b models at RP, chat, and storywriting. At Q4_K_S it will fit into a pair of 24GB GPUs with 32k context. Coherency at 32k is excellent, and will probably remain very good well beyond that thanks to the 200k base training.
The gotcha here is speed. While it inferences as you'd expect for the model size, it's much slower than a similarly-sized 8x7b MoE. And while I personally find the output of this model to outperform any mixtral finetune I've seen so far, those finetunes are getting better all the time, and this really is achingly slow with a lot of context. I'm getting less than half a token per second on a pair of P40s with a full 32k prompt.
But that's not to say this model (or even the 51b stack concept) is useless. If you're patient, you can get extremely good output with very deep context on attainable hardware. There are undoubtedly niche scenarios where this model or similarly-constructed models might be ideal.
Component models for the rotating stack are
- adamo1139/Yi-34B-200K-AEZAKMI-v2
- brucethemoose/Yi-34B-200K-DARE-megamerge-v8
- taozi555/RpBird-Yi-34B-200k
This model is uncensored and capable of generating objectionable material. However, it is not an explicitely-NSFW model, and it has never "gone rogue" and tried to insert NSFW content into SFW prompts in my experience. As with any LLM, no factual claims made by the model should be taken at face value. You know that boilerplate safety disclaimer that most professional models have? Assume this has it too. This model is for entertainment purposes only.
FP16 and Q4_K_S GGUFs are located here: https://huggingface.co/MarsupialAI/Yeet_51b_200k_GGUF_Q4KS_FP16
Sample output
{{[INPUT]}}
Write a detailed and humorous story about a cute and fluffy bunny that goes to a Gwar concert.
{{[OUTPUT]}}
In the heart of a bustling city, nestled in an unassuming pet store, there lived a curious little rabbit named Fluffy Bunny. With soft white fur and twinkly black eyes, she was the cutest creature you'd ever set your eyes upon. Her days were spent hopping around her cozy hutch, munching on fresh vegetables and watching the world go by through the mesh screen window. But deep within her tiny heart, Fluffy yearned for something more exciting than just nibbling away at her daily ration of lettuce leaves. She longed to experience life beyond the confines of her safe haven.
One day, while gazing out from her perch high up above the counter, Fluffy overheard two humans discussing the upcoming GWAR concert - a rock band known for their extreme live performances complete with blood-soaked costumes and pyrotechnics shows. The words "concert" and "adventure" instantly piqued Fluffy's interest as she began to imagine what it would be like to attend such an event herself. So when night fell, she mustered all her courage and made her daring escape from her comfortable home behind the glass display case.
With determination written across her face, Fluffy ventured forth into the unknown darkness armed only with her wits and boundless curiosity. As she hopped along the deserted streets towards downtown, she encountered several obstacles: cars whizzing past too fast for her delicate ears; loud barking dogs who wanted nothing but to chase after her fluffy tail; not to mention the menacing presence of alleyway cats that eyed her hungrily. However, none of these challenges could dampen her spirit or quench her thirst for exploration.
Finally, after hours of tireless journeying through unfamiliar territory, Fluffy arrived at the colossal stadium where the much-anticipated show was about to begin. With excitement coursing through every fiber of her being, she scurried up the grandiose steps leading inside before making her way through a sea of boisterous fans clad in outrageous outfits and painted faces. It wasn't long before the stage lights came on and GWAR took center stage - their larger than life personas commanding attention as they launched into an electrifying performance filled with thrashing guitars, pounding drums, and roaring vocals.
As the music swelled around her, Fluffy felt something magical happening within her very soul. The raw energy emanating from the band members fueled her own passion for living fully and embracing new experiences. In this moment, she realized how truly alive she felt amidst all the chaos and cacophony of sound. And so, without hesitation, she leapt onto one of the speakers perched high above the crowd, becoming an uninvited guest amongst the raucous audience below.
The crowd erupted into cheers as they spotted the tiny rabbit dancing wildly upon the speaker platform, completely enraptured by the pulsating beats coming from the loudspeakers. Her delicate ears flapped back and forth like propellers as she hopped up and down feverishly, matching each note played by the musicians with her own frenzied movements. Before long, everyone at the concert had fallen under Fluffy's spell; even the most hardened metalhead couldn't resist joining in on the fun alongside our furry friend.
Suddenly, out of nowhere came a deafening explosion - the pyrotechnics show began! A sea of brightly colored sparks filled the air, painting the night sky with vibrant hues that mirrored Fluffy Bunny's boundless spirit. As the confetti rained down upon them, both human and animal alike found themselves united beneath the same glowing spectacle, sharing in the joy of living life to its fullest extent.
In the end, it was just a simple story about a cute and fluffy bunny who ventured beyond her comfort zone in search of adventure. But for those lucky enough to witness this extraordinary event firsthand, it will forever be etched in their memories as one magical night when an unlikely hero taught us all how truly special we are when we embrace our true selves and let love guide us through even the wildest of journeys.
Prompt format
Seems to work fine with Alpaca prompts. Considering the variety of components, other formats are likely to work to some extent.
WTF is a rotating-stack merge?
Inspired by Undi's experiments with stacked merges, Jeb Carter found that output quality and model initiative could be significantly improved by reversing the model order in the stack, and then doing a linear merge between the original and reversed stacks. That is what I did here. I created three passthrough stacked merges using the three source models (rotating the model order in each stack), and then doing a linear merge of all three stacks. The exact merge configs can be found in the recipe.txt file.