|
--- |
|
language: |
|
- en |
|
tags: |
|
- testing |
|
- llm |
|
- rp |
|
- discussion |
|
--- |
|
|
|
# Why? What? TL;DR? |
|
|
|
Simply put, I'm making my methodology to evaluate RP models public. While none of this is very scientific, it is consistent. I'm focusing on things I'm *personally* looking for in a model, like its ability to obey a character card and a system prompt accurately. Still, I think most of my tests are universal enough that other people might be interested in the results, or might want to run those tests on their own. |
|
|
|
|
|
# Testing Environment |
|
|
|
- Frontend is staging version of Silly Tavern. |
|
- Backend is the latest version of KoboldCPP for Windows using CUDA 12. |
|
- Using **CuBLAS** but **not using QuantMatMul (mmq)**. |
|
- Fixed Seed for all tests: **123** |
|
- **7-10B Models:** |
|
- All models are loaded in Q8_0 (GGUF) |
|
- **Flash Attention** and **ContextShift** enabled. |
|
- All models are extended to **16K context length** (auto rope from KCPP) |
|
- Response size set to 1024 tokens max. |
|
- **11-15B Models:** |
|
- All models are loaded in Q4_KM or whatever is the highest/closest available (GGUF) |
|
- **Flash Attention** and **8Bit cache compression** are enabled. |
|
- All models are extended to **12K context length** (auto rope from KCPP) |
|
- Response size set to 512 tokens max. |
|
|
|
|
|
# System Prompt and Instruct Format |
|
|
|
- The exact system prompt and instruct format files can be found in the [file repository](https://huggingface.co/SerialKicked/ModelTestingBed/tree/main). |
|
- All models are tested in whichever instruct format they are supposed to be comfortable with (as long as it's ChatML or L3 Instruct) |
|
|
|
|
|
# Available Tests |
|
|
|
### DoggoEval |
|
|
|
The goal of this test, featuring a dog (Rex) and his owner (EsKa), is to determine if a model is good at obeying a system prompt and character card. The trick being that dogs can't talk, but LLM love to. |
|
|
|
- [Results and discussions are hosted in this thread](https://huggingface.co/SerialKicked/ModelTestingBed/discussions/1) ([old thread here](https://huggingface.co/LWDCLS/LLM-Discussions/discussions/13)) |
|
- [Files, cards and settings can be found here](https://huggingface.co/SerialKicked/ModelTestingBed/tree/main/DoggoEval) |
|
- TODO: Charts and screenshots |
|
|
|
### MinotaurEval |
|
|
|
TODO: The goal of this test is to check if a model is able of following a very specific prompting method and maintain situational awareness in the smallest labyrinth in the world. |
|
|
|
- Discussions will be hosted here. |
|
- Files and cards will be available soon (tm). |
|
|
|
### TimeEval |
|
|
|
TODO: The goal of this test is to see if the bot is able to behave in 16K context, recall and summarise "old" info accurately. |
|
|
|
- Discussions will be hosted here. |
|
- Files and cards will be available soon (tm). |
|
|
|
|
|
# Limitations |
|
|
|
I'm testing for things I'm interested in. I do not pretend any of this is very scientific or accurate: as much as I try to reduce the amount of variables, a small LLM is still a small LLM at the end of the day. The results for other seeds, or with the smallest of change, are bound to give very different results. |
|
|
|
I usually give the different models I'm testing a fair shake in a more casual settings. I regen tons of outputs with random seeds, and while there are (large) variations, it tends to even out to the results shown in testing. Otherwise I'll make a note of it. |
