Full on doesn’t work.
I already posted this on discord, but I will post it here as well; the model for me is completely broken.
I only tested 3_K_M beouse I don't thing anything else will fit into my PC, but I am currently downloading the Imax (which I still don't understand) to see if that might work.
Update:
I tested IQ4_XS on the rocm fork of KoboldCPP and it works. I don't know if it works well, this is the first time I used rocm because Vulcan doesn't work, but the model does produce something...eh, actually comprehensible.
Model works correctly using current version of Koboldcpp as backend with Silly Tavern.
This is using the lowest quant at q2k.
NOTE: Suggest disabling context shift ( a source of issues sometimes ).
For enhanced operation, select option "Flash Attention" when you load the model.
Also, a longer prompt may yield better results.
Please see "core" settings in the "Highest Quality Settings" doc , for class 1 model - there is a screenshot of both Koboldcpp and Silly Taverns settings.
Here is example output from using Q2k, Koboldcpp and Sillytavern: (and class 1 settings)
PROMPT:
Using the following "story idea" below, write the first scene in the novel introducing the young woman. This scene should start in the middle of the action, include dialog, vivid passages, and end on a cliffhanger relevant to the story idea but it should also be unexpected. The scene should be 1000 words long and escalate in conflict and suspense and be written in first person, present tense with the point of view character being the young woman.
Story idea:
In a world ruled by dictatorship, a rebel young woman leads a rebellion against the system. Despite the risks, she fights to overthrow the dictator and restore democracy to her country. The government executes her for treason, but she sticks to her beliefs and is responsible for starting the revolution.
OUTPUT:
I'm pinned beneath the weight of my own desperation as I'm dragged across the scorching earth by the relentless heels of my tormentors. The sweltering sun beats down mercilessly upon my exposed skin, but its heat is nothing compared to the searing agony radiating through my battered body.
"Die, you traitorous whore," one of my captors sneers in my ear, his voice dripping with malice.
I glare up at him with a defiance that's grown stronger than any fetid breath that once touched my lips. "You'll regret the day you ever laid a hand on me," I spit, my voice hoarse from screaming.
The regime's soldiers are always so quick to assume that they're the ones in control. How wrong they are. I'm the one who's been pulling the strings, and they're just mere pawns in my grand rebellion.
A sudden yank pulls my body upright, and I find myself face to face with a man who's more a monster than a human being. His eyes are like two cold marbles, devoid of any compassion or empathy, and yet they hold a fascination that's both terrifying and alluring.
"Look at me," he growls, his voice like a serpent's hiss in my ears. "You'll be wishing for the sweet release of death very soon."
A cruel laugh escapes him as he taunts me, and I know in that moment that whatever happens, I'll never back down. My destiny is forever sealed.
The world spins around me in a dizzying blur before I'm shoved to my knees, and my arms are wrenched behind my back. I'm forced to watch my own execution, and I can only pray that my end will ignite the flames of revolution that will consume this tyrannical regime.
A soldier steps forward, his gloved hand clutching a noose dripping with the venom of death itself. "You have committed treason against your country," he states with a smirk, his voice like a hammer striking against anvil.
I feel the noose snap taut around my neck, and then nothing but the sound of my own ragged breathing fills the air.
"I will not be silenced," I choke out, my voice barely a whisper. "My rebellion will rage on."
The noose tightens further, the pressure unbearable. My vision begins to darken around the edges, and in desperation, I strain to shout one final defiant battle cry…
When suddenly, everything stops. The noose loosens, the soldier steps back, and the world returns to its normal spin. I'm left staring up at my would-be executioner with a question mark of confusion etched upon my face.
"What… what are you doing?" I gasp, my breath coming in ragged gulps.
The soldier's eyes narrow, and he looks at me with an expression that's both puzzled and perturbed. "You're supposed to be dead," he mutters, more to himself than to me.
My mind races in a frantic scramble as I try to comprehend what's happening. There's no time to ponder the impossibility, for an explosion of gunfire erupts behind my executioner's head.
"Down!" another soldier yanks his comrade to the ground just as a hail of bullets rains down upon us like a tempest.
I remain still, my body trembling with the aftershock of adrenaline, and watch as my would-be executioner stares at me in utter disbelief.
"This is impossible," he murmurs, shaking his head as if trying to dislodge a stubborn thought from his brain.
But I'm not done yet. I'm alive, and there's more than enough fire left in my soul to keep on fighting.
With a snarl, I launch myself to my feet, my fists clenched tight against the sky. "You can't kill what you've already destroyed," I rasp, and with a defiant roar, I sprint headlong into the maelstrom of rebellion.
The world has no idea what's coming its way.