Context limit
Hello there!
I am currently testing your model and I was wondering if, with rope and a good GPU, can the model still be coherent at 65k context? How bad is the loss at that point?
Mistral-Small is generally believed to be coherent up to 32k tokens. I personally have not tested it beyond that.
Where would you recommend I can leave feedback for the model in regards to RP? I began testing the model with my own character cards and I was wondering if you are interested in feedback on how it behaves on different scenarios with original cards (not downloaded).
Leave your feedback where it's easier for you to do so
I will leave some feedback here for you.
Setup:
I will start off by saying that I use it for RP purposes, as mentioned above, using my own characters (No anime, no fandoms, no movie/videogames/tvshow/books/etc.). I prefer creating my own lore, worlds and build from there. Most of my scenarios are varying between psychological thrillers, slice of life and parts horror. My setup is RunPod, 48GB Vram or higher, and I am using a Q8 quant or higher if available. KoboldCPP on runpod and SillyTavern as front-end. Instructions are a mixture of Spaghetti Marinara's Mistral settings and some points from Behemoth's template suggested on their Discord channel (this is important for keeping most models in the moment and not go on a ramble with 5 actions and 3 questions before I can even reply). Presets are taken from your model's card, with the slight adjustment of lowering the XTC probability to 0.1 and temp was risen to 1. Context was kept at 32k per your instructions.
Character cards were kept in MD format, with simple to read descriptions and personality, as well as a backstory for the character. Example dialogue was given in an interview style (Ex: Interviewer: "Tell us about yourself?" {{char}}: "Well, let me tell you a bit about me. My name is ...."). Style of RP is novel style, third person, where I control my persona and some NPC's and the model controls {{char}}.
Positive Feedback:
I was generally impressed by how well the model was able to keep track of objects in the world and apply logic to certain actions. It was able to stay in character REALLY well and follow instructions to a tee, without the need to nudge it. I was particularly impressed when the character picked her phone up from the coffee table. She had a coffee in her hand and the coffee table was on her right hand side. The character described how she shifted the coffee cup from her right hand to her left, before picking up the phone and began texting someone. It might seem like a small detail, but it makes the action coherent and believable. Another instance was a buildup to a scene, where the character began building up tension between my persona and the character. Now, the tension buildup was done so smooth and naturally, that it almost felt like an actual book. It was testing the waters to see how I would react, asking specific questions to reveal my intentions (in the scene, my character is supposed to be a heretic, while {{char}} plays the queen who stood behind the church in order to avoid an inquiry into the death of her husband, the king. The queen suspected I might be a heretic and during a formal event, she began talking about faith and what it meant to her, all lies of course, ending up in asking about my faith. The conversation continued with more questions regarding the teachings of the priests and my opinion on the violent nature of the church.). It felt really impressive to have this scene buildup and tension.
The model also adopted well to other genres as well, playing a desperate character perfectly. It even began mentioning things the character despises doing, during a conversation on the topic of "means to an end", to further drive the feeling of loneliness and despair home.
Negative Feedback:
While the model impressed me so much with all the smarts and logic it applied, it seemed like up to 12k context was peak logic. Things began noticeably being...less smart. Not that it wouldn't react as it should, but that spark of imagination and logic seemed to dull slightly, making little mistakes which I probably wouldn't have noticed if it weren't so impressive before.
More so, the model seems to have some difficulties with characters that have subtle preferences (this might also be because 22B). As an example, I have a character which hates lewd words. This is further reinforced through example dialogue and word triggers in lorebook. However, an NPC (voiced by my persona) mentioned said lewd words, character never reacted to them in the way that her example dialogue would suggest. I checked the logs if the words triggered the entry in the lore book and it did so, but character never reacted to them. I tried multiple swipes, but it didn't react.
Another example, and this one is still baffling me, is the shift of tone in two separate characters, from two separate RP's. The desperate and neurotic character was portrayed splendidly, while the more happy and positive character's dialogue was very dry. I might need to tweak the example dialogue of the more positive one, but the same character was portrayed beautifully by a Nemo fine-tune, down to the shift in her demeanor and reluctance in certain situations.
Lastly, sometimes it feels like the model is holding back. While I do have the instruction set to "be in the moment", it sometimes feels like I have to drag it a bit to advance the plot. I know it is capable of doing this by itself, since in other instances it was able taking decisions on it's own, without my help. But there are many times when it feels like it stalls. (Example: My persona and the character are watching a movie, I talk about the movie, describing some other things in the scene, model agrees with me and describes their feeling and some parts of the scene. I say the movie is over and try to give it a way to progress further, model stays in the same scene and waits for me to make a decision on what happens next.). I have to start either making a decision, or heavily hint to the model to progress the scene further.
Thank you for your feedback!
While the model impressed me so much with all the smarts and logic it applied, it seemed like up to 12k context was peak logic.
Possibly a general Mistral-Small tunes problem
As an example, I have a character which hates lewd words. This is further reinforced through example dialogue and word triggers in lorebook. However, an NPC (voiced by my persona) mentioned said lewd words, character never reacted to them in the way that her example dialogue would suggest. I checked the logs if the words triggered the entry in the lore book and it did so, but character never reacted to them. I tried multiple swipes, but it didn't react.
Have you tried setting this entry to a low depth in the lorebook settings? Something like Position: @D👤or @D⚙️ and Depth:1 might help.
The desperate and neurotic character was portrayed splendidly, while the more happy and positive character's dialogue was very dry.
Kinda confuses me too. May or may not change in the next iteration.
Lastly, sometimes it feels like the model is holding back. While I do have the instruction set to "be in the moment", it sometimes feels like I have to drag it a bit to advance the plot. I know it is capable of doing this by itself, since in other instances it was able taking decisions on it's own, without my help. But there are many times when it feels like it stalls.
The next version will be based on Cydonia-v1.3. It's more adventure oriented, so it should be more eager to advance the plot. I'll probably finish it by the end of this week.
Possibly a general Mistral-Small tunes problem
Shame that it diminishes, but I understand that it is the restrictions of the current gen.
Have you tried setting this entry to a low depth in the lorebook settings? Something like Position: @D👤or @D⚙️ and Depth:1 might help.
I have not tried this, but I think you might be on to something. I have a similar thing for food preference which is mentioned in the char description, lorebook entry after char definition and sample dialogue on the same topic. The description says that instead of refusing, let's say pasta, the character will try to propose an alternative. During my testing, I noticed that the character, when served pasta, gently tried to take a bite and suggest that it might be more appropriate to have steak instead, based on the actual situation (again, very impressed with the smarts here as it didn't downright refused, but gently tried to find an alternative as it was instructed in the lorebook entry). I might need to set the lewd words as system or character level, as you suggested, since I think it might have issues differentiating what is a lewd word, based on it's training set.
The next version will be based on Cydonia-v1.3. It's more adventure oriented, so it should be more eager to advance the plot. I'll probably finish it by the end of this week.
I am looking forward to this. I have yet to try v1.3, but I have high hopes for it.