Hugging Face's logo

SerialKicked
/
ModelTestingBed

English
testing
llm
rp
discussion
Model card Files Community
2
ModelTestingBed / README.md
SerialKicked's picture
SerialKicked
Update README.md
8615360 verified
preview code
|
raw history blame contribute delete
No virus
3.29 kB
metadata
language:
  - en
tags:
  - testing
  - llm
  - rp
  - discussion

Why? What? TL;DR?

Simply put, I'm making my methodology to evaluate RP models public. While none of this is very scientific, it is consistent. I'm focusing on things I'm personally looking for in a model, like its ability to obey a character card and a system prompt accurately. Still, I think most of my tests are universal enough that other people might be interested in the results, or might want to run those tests on their own.

Testing Environment

  • Frontend is staging version of Silly Tavern.
  • Backend is the latest version of KoboldCPP for Windows using CUDA 12.
  • Using CuBLAS but not using QuantMatMul (mmq).
  • Fixed Seed for all tests: 123
  • 7-10B Models:
    • All models are loaded in Q8_0 (GGUF)
    • Flash Attention and ContextShift enabled.
    • All models are extended to 16K context length (auto rope from KCPP)
    • Response size set to 1024 tokens max.
  • 11-15B Models:
    • All models are loaded in Q4_KM or whatever is the highest/closest available (GGUF)
    • Flash Attention and 8Bit cache compression are enabled.
    • All models are extended to 12K context length (auto rope from KCPP)
    • Response size set to 512 tokens max.

System Prompt and Instruct Format

  • The exact system prompt and instruct format files can be found in the file repository.
  • All models are tested in whichever instruct format they are supposed to be comfortable with (as long as it's ChatML or L3 Instruct)

Available Tests

DoggoEval

The goal of this test, featuring a dog (Rex) and his owner (EsKa), is to determine if a model is good at obeying a system prompt and character card. The trick being that dogs can't talk, but LLM love to.

MinotaurEval

TODO: The goal of this test is to check if a model is able of following a very specific prompting method and maintain situational awareness in the smallest labyrinth in the world.

  • Discussions will be hosted here.
  • Files and cards will be available soon (tm).

TimeEval

TODO: The goal of this test is to see if the bot is able to behave in 16K context, recall and summarise "old" info accurately.

  • Discussions will be hosted here.
  • Files and cards will be available soon (tm).

Limitations

I'm testing for things I'm interested in. I do not pretend any of this is very scientific or accurate: as much as I try to reduce the amount of variables, a small LLM is still a small LLM at the end of the day. The results for other seeds, or with the smallest of change, are bound to give very different results.

I usually give the different models I'm testing a fair shake in a more casual settings. I regen tons of outputs with random seeds, and while there are (large) variations, it tends to even out to the results shown in testing. Otherwise I'll make a note of it.