Pi3141

/

alpaca-lora-30B-ggml

like 133

No. I won't convert the old q4_0 because it's for people with alpaca.cpp (outdated). People with llama.cpp should just use q4_1 instead. In my testing, it was better than q4_0 so I'm not sure why you got the opposite result.

Fair enough! It's easy enough to do on the user end anyway.

That said, if it's not too much trouble, would you mind very much running the prompt in the comment I linked? I'm so curious why I was getting dramatically worse results with q4_1 vs q4_0, but maybe it's just something weird on my end (I had noticed in the ggml repo that the jury is still out on which approach is better). It was happening both with your model and with one I consolidated and quantized myself.

That said, if it's not too much trouble, would you mind very much running the prompt in the comment I linked? I'm so curious why I was getting dramatically worse results with q4_1 vs q4_0, but maybe it's just something weird on my end (I had noticed in the ggml repo that the jury is still out on which approach is better). It was happening both with your model and with one I consolidated and quantized myself.

I've just tested it myself. I've noticed that you did not follow the Alpaca prompt template:

Below is an instruction that describes a task. Write a response that appropriately completes the request. 

### Instruction: instruction goes here

### Response:

Here's the response i got from q4_0

Stoner by John Williams is a novel set in the American West. It follows the story of a man named Stoner, who is a loner living in the desert, with no connection to the modern world. He encounters various people and events during his journey, and through these encounters he is forced to confront his past and confront his inner demons. Ultimately, Stone must come to terms with his past and make peace with himself before he can move on.

Here's the response i got from q4_1

Stoner is a novel by John Williams that tells the story of a man named William Stoner, who struggles to find his place in the world. Stoner is a farm boy who goes to college to study agriculture, but later finds himself drawn to literature. He becomes a teacher, and his life is punctuated by moments of personal triumph and despair, as well as the larger events of the world around him. The novel is a poignant story of the search for meaning and connection in a life that feels small and unfulfilled.

Thank you! Really appreciate it. I have used the right prompt template when testing Alpaca, but I'd made that comment about LLaMA---sorry for the confusion.

At any rate, it does look like something on my end. You're getting almost the opposite results as me (your q4_0 inaccurate, q4_1 accurate).

Again, thanks for running it. I'm running the most recent llama.cpp on apple silicon, which feels like the best case scenario, but I guess there's always room for bugs

You're welcome 😁