Repository: olliequ/KittyBot
Pull request #121: Reasoning meme rater
Author: login: walter-bd
Created: 2025-03-08T21:28:44Z | Merged: 2025-03-08T23:18:37Z

It includes a reasoning process for the meme rater and it explains why it gives a given rate.
To activate this feature just add a `REASONER_MEME=True` to the .env.
While kitty gives an explanation of why, this is only for debugging purposes and is include in the logs 
Edit: Also added short term memory to kitty
Edit2: Forgot to mention in the last commit, you can add this {user} to the prompt to add the user identity


Diff:
diff --git a/commons/agents.py b/commons/agents.py
index 590c2f5..4136775 100644
--- a/commons/agents.py
+++ b/commons/agents.py
@@ -3,9 +3,10 @@
 from pydantic_ai.messages import BinaryContent
 from pydantic_ai import Agent, RunContext
 import db
-from typing import Any, Final
+from typing import Any, Final, Optional
 import os
 import logging as log
+from collections import deque
 
 
 class KittyState(BaseModel):
@@ -19,6 +20,11 @@ class MemeState(BaseModel):
 
 class MemeAnswer(BaseModel):
     rate: int = Field(description="The rating of the meme")
+    explanation: Optional[str] = Field(description="The explanation of the rating")
+
+
+class MemeDescription(BaseModel):
+    description: str = Field(description="The textual description of the meme")
 
 
 class KittyAnswer(BaseModel):
@@ -43,7 +49,24 @@ class KittyAnswer(BaseModel):
 
 MEME_RATE_PROMPT: Final[str] = os.environ.get(
     "MEME_RATING_PROMPT",
-    "Rate this meme out of 10, with 10 being the funniest. Rate is solely on how funny a bunch of computer science nerds would find it. ONLY Return an integer.",
+    "Rate this meme out of 10, with 10 being the funniest. Rate it solely on how funny a bunch of computer science nerds would find it. ONLY Return an integer.",
+)
+
+EYE_RATE_PROMPT: Final[str] = os.environ.get(
+    "EYE_RATING_PROMPT",
+    "Describe this image, do not leave any  detail out, just describe it do not give any opinions on it unless is not a meme, then explain why. Ensure that the image is a meme and not debug or simple coding screenshots, that is code or debug output without any context. If you found this just said this is not a meme",
+)
+
+REASONER_MEME_PROMPT: Final[str] = os.environ.get(
+    "REASONER_MEME_PROMPT",
+    """
+    Rate the following meme on a scale from 1 to 10, where 10 is the funniest.
+    The rating should be based solely on how humorous a group of computer science enthusiasts—particularly those familiar with algorithms, programming languages, system design, debugging struggles, and internet culture—would find it.
+    Prioritize technical wit, inside jokes, and references to common CS experiences (e.g., recursion jokes, stack overflows, regex pain).
+    Do NOT factor in general audience appeal.
+    If the description said is not a meme, factors the explanation in your rate and give a 0. 
+    ONLY return an integer.
+    """,
 )
 
 
@@ -54,6 +77,7 @@ def __init__(
         self.model_settings = model_settings
         self.model = model
         self.prompt = prompt
+        self.messages = deque(maxlen=20)
         self.setup_agent()
 
     def setup_agent(self):
@@ -73,9 +97,13 @@ async def run(self, query: str, user: str = "ANON", prompt: str = None):
             self.prompt = prompt
         state = KittyState(query=query, user=user)
         try:
-            response = await self.agent.run(query, deps=state)
+            response = await self.agent.run(
+                query, deps=state, message_history=list(self.messages)
+            )
         except Exception as e:
             raise Exception(f"Error running agent: {e}")
+        for messages in response.new_messages():
+            self.messages.append(messages)
         return response.data.answer
 
 
@@ -112,6 +140,60 @@ async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None
         return response.data.rate
 
 
+class ReasonerMemeRater:
+    def __init__(
+        self,
+        eye_model_settings: ModelSettings,
+        eye_model: str,
+        reasoner_model: str,
+        eye_prompt: str,
+        reasoner_prompt: str,
+    ):
+        self.eye_model_settings = eye_model_settings
+        self.eye_model = eye_model
+        self.reasoner_model = reasoner_model
+        self.eye_prompt = eye_prompt
+        self.reasoner_prompt = reasoner_prompt
+        self.setup_agent()
+
+    def setup_agent(self):
+        self.eyes = Agent(
+            self.eye_model,
+            model_settings=self.eye_model_settings,
+            deps_type=MemeState,
+            result_type=MemeDescription,
+        )
+        self.reasoner = Agent(
+            self.reasoner_model, deps_type=MemeDescription, result_type=MemeAnswer
+        )
+
+        @self.eyes.system_prompt
+        def system_prompt_eye(state: RunContext[MemeState]):
+            return self.eye_prompt.format_map(state.deps.model_dump())
+
+        @self.reasoner.system_prompt
+        def system_prompt_reasoner(state):
+            return self.reasoner_prompt.format_map(state.deps.model_dump())
+
+    async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None):
+        if prompt:
+            self.prompt = prompt
+        state = MemeState(user=user)
+        image = BinaryContent(
+            data=image.content, media_type=image.headers["Content-Type"]
+        )
+        try:
+            eyes_response = await self.eyes.run([image], deps=state)
+            log.info(f"Eyes response: {eyes_response.data.description}")
+            response = await self.reasoner.run(
+                eyes_response.data.description, deps=eyes_response.data
+            )
+            log.info(response.data)
+        except Exception as e:
+            raise Exception(f"Error running agent: {e}")
+        return response.data.rate
+
+
 gemini_model_settings = GeminiModelSettings(
     **generation_config,  # general model settings can also be specified
     gemini_safety_settings=safety_settings,
@@ -119,6 +201,18 @@ async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None
 
 kitty_gemini_agent = KittyAgent(gemini_model_settings, "gemini-2.0-flash-lite")
 
+
 kitty_meme_agent = KittyMemeRater(
     gemini_model_settings, "gemini-2.0-flash-lite", MEME_RATE_PROMPT
 )
+
+if os.getenv("REASONER_MEME").lower() == "true":
+    kitty_reasoner_meme_rater = ReasonerMemeRater(
+        gemini_model_settings,
+        "gemini-2.0-flash-lite",
+        "gemini-2.0-flash-lite",
+        EYE_RATE_PROMPT,
+        REASONER_MEME_PROMPT,
+    )
+else:
+    kitty_reasoner_meme_rater = None
diff --git a/extensions/meme_rater.py b/extensions/meme_rater.py
index b0bbce6..207d615 100644
--- a/extensions/meme_rater.py
+++ b/extensions/meme_rater.py
@@ -6,7 +6,7 @@
 import lightbulb
 from PIL import Image
 import requests
-from commons.agents import kitty_meme_agent
+from commons.agents import kitty_meme_agent, kitty_reasoner_meme_rater
 from typing import Final
 from pydantic_ai import BinaryContent
 import asyncio
@@ -39,9 +39,15 @@ async def get_meme_rating(image_url: str, user: str):
     if not image.raw:
         logging.info("Not image")
         return ""
-    response = await kitty_meme_agent.run(image=image, user=user)
+    try:
+        if kitty_reasoner_meme_rater:
+            response = await kitty_reasoner_meme_rater.run(image=image, user=user)
+        else:
+            raise Exception("No reasoner meme rater")
+    except Exception as e:
+        logging.info(f"Error running reasoner meme rater: {e}")
+        response = await kitty_meme_agent.run(image=image, user=user)
     logging.info(f"Meme rating response: {response}")
-
     try:
         return min(max(0, int(response)), 10)
     except Exception as e:


Reviews:
user: login: flukiluke
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-08T23:11:33Z
user: login: walter-bd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T23:16:09Z
user: login: flukiluke
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-08T23:18:06Z
user: login: flukiluke
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-08T23:18:25Z

Review comments:
diff_hunk: @@ -112,13 +140,79 @@ async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None
         return response.data.rate
 
 
+class ReasonerMemeRater:
+    def __init__(
+        self,
+        eye_model_settings: ModelSettings,
+        eye_model: str,
+        reasoner_model: str,
+        eye_prompt: str,
+        reasoner_prompt: str,
+    ):
+        self.eye_model_settings = eye_model_settings
+        self.eye_model = eye_model
+        self.reasoner_model = reasoner_model
+        self.eye_prompt = eye_prompt
+        self.reasoner_prompt = reasoner_prompt
+        self.setup_agent()
+
+    def setup_agent(self):
+        self.eyes = Agent(
+            self.eye_model,
+            model_settings=self.eye_model_settings,
+            deps_type=MemeState,
+            result_type=MemeDescription,
+        )
+        self.reasoner = Agent(
+            self.reasoner_model, deps_type=MemeDescription, result_type=MemeAnswer
+        )
+
+        @self.eyes.system_prompt
+        def system_prompt_eye(state: RunContext[MemeState]):
+            return self.eye_prompt.format_map(state.deps.model_dump())
+
+        @self.reasoner.system_prompt
+        def system_prompt_reasoner(state):
+            return self.reasoner_prompt.format_map(state.deps.model_dump())
+
+    async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None):
+        if prompt:
+            self.prompt = prompt
+        state = MemeState(user=user)
+        image = BinaryContent(
+            data=image.content, media_type=image.headers["Content-Type"]
+        )
+        try:
+            eyes_response = await self.eyes.run([image], deps=state)
+            log.info(f"Eyes response: {eyes_response.data.description}")
+            response = await self.reasoner.run(
+                eyes_response.data.description, deps=eyes_response.data
+            )
+            log.info(response.data)
+        except Exception as e:
+            raise Exception(f"Error running agent: {e}")
+        return response.data.rate
+
+
 gemini_model_settings = GeminiModelSettings(
     **generation_config,  # general model settings can also be specified
     gemini_safety_settings=safety_settings,
 )
 
 kitty_gemini_agent = KittyAgent(gemini_model_settings, "gemini-2.0-flash-lite")
 
+
 kitty_meme_agent = KittyMemeRater(
     gemini_model_settings, "gemini-2.0-flash-lite", MEME_RATE_PROMPT
 )
+
+if os.getenv("REASONER_MEME").lower() == "true":
path: commons/agents.py
user: login: flukiluke
body: Don't bother with the option, just have it always enabled
created_at: 2025-03-08T23:05:44Z
author_association: COLLABORATOR
diff_hunk: @@ -112,13 +140,79 @@ async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None
         return response.data.rate
 
 
+class ReasonerMemeRater:
path: commons/agents.py
user: login: flukiluke
body: If we always use this agent can we get rid of the KittyMemeRater?
created_at: 2025-03-08T23:11:28Z
author_association: COLLABORATOR
diff_hunk: @@ -112,13 +140,79 @@ async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None
         return response.data.rate
 
 
+class ReasonerMemeRater:
path: commons/agents.py
user: login: walter-bd
body: It depends, do we want to use the standard direct one or the reasoner one, after testing we can get rid of one of them, the new one gives a description of the meme and then the reasoner gives a rating based on the description, the previous one just uses the image directly which is why I didn't put the reasoner one as a default. 
created_at: 2025-03-08T23:16:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,13 +140,79 @@ async def run(self, image: BinaryContent, user: str = "ANON", prompt: str = None
         return response.data.rate
 
 
+class ReasonerMemeRater:
path: commons/agents.py
user: login: flukiluke
body: I've been testing out the reasoner's ratings, it seems much more strict but reasonably fair. Maybe we'll leave the code as is for now but I'll keep it enabled in conf until someone complains.
created_at: 2025-03-08T23:18:06Z
author_association: COLLABORATOR

PR comments:
user: login: jimbxb
created_at: 2025-03-08T23:43:42Z
body: I think it would be good to post the reasoning as a reply to the meme that was posted. I'm thinking we can add a ? react, and when either the user or a prob mod+ click it, the bot replies to the original meme with the reasoning.
author_association: CONTRIBUTOR
user: login: adamnwerbik
created_at: 2025-03-08T23:55:56Z
body: Finally an idea I can get behind @jimbxb  😁
author_association: COLLABORATOR