Repository: ALLAN-DIP/chiron-utils
Pull request #23: Add LLM advisor
Author: login: YanzeJeremy
Created: 2025-01-21T21:17:32Z | Merged: 2025-03-27T17:25:39Z

mainly LlmAdvisor development and slightly changed BaselineBot(Communication stage length) to fit real game running.


Diff:
diff --git a/README.md b/README.md
index e8160f0..941a447 100644
--- a/README.md
+++ b/README.md
@@ -55,6 +55,21 @@ Both the bot and game running commands support a `--help` argument to list avail
     - When running the bot outside of a container, download the latest model file from [lr_models - Google Drive](https://drive.google.com/drive/folders/1FuG3qY51wRkR8RgEBVY49-loln06W-Ro). The filename includes the model release date in `YYYYMMDD` format).
     - Edit the `MODEL_PATH` constant in `lr_bot.py` to point to the unzipped model folder.
   - Code for model training can be found at <https://github.com/ALLAN-DIP/baseline-models>
+- [`LlmAdvisorBot`](src/chiron_utils/bots/llm_advisor_bot.py):
+  - A large language model using Llama-3.1-8B-Instruct to provide commentary advice given board states, message history, and predicted orders from Cicero.
+  - To set up the bot:
+    - When the bot is first run, it will attempt to download the used model from the Hugging Face Hub. Preparing for this requires multiple steps:
+      - Create a Hugging Face account
+      - Request access on the page for the following model:
+        - [Llama3.1-8b-instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
+      - Once your request has been approved, authenticate on your local machine using a user access token, using the official [User access tokens](https://huggingface.co/docs/hub/security-tokens) documentation as a guide.
+    - One needs to run another advisor to provide `OPPONENT_MOVE` advice to the same power. For example, one can run the [Cicero advisor](https://github.com/ALLAN-DIP/diplomacy_cicero) with the argument `--advice_levels 8`.
+  - To use the bot, run the following command from the repository root:
+    ```shell
+    # Set communication stage to 10 minutes (in seconds) to give enough time
+    export COMM_STAGE_LENGTH=600
+    python -m chiron_utils.scripts.run_bot --host [host_address] --port [port_address] --game_id [game_id] --power [power_name] --bot_type LlmAdvisor
+    ```
 
 ## Contributing
 
diff --git a/pyproject.toml b/pyproject.toml
index 88faf71..2b1e955 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -39,6 +39,7 @@ all = [
   "chiron_utils[base]",
   "chiron_utils[baseline-lr]",
   "chiron_utils[dev]",
+  "chiron_utils[llm-advisor]",
 ]
 # Default extra to make OCI image builds easier
 base = []
@@ -53,6 +54,11 @@ dev = [
   # `pytest` `6.0` introduced `--strict-config` argument and `pyproject.toml` support
   "pytest>=6.0",
 ]
+llm-advisor = [
+  "torch",
+  # Recommended minimum version for Llama 3.1 support
+  "transformers>=4.43.2",
+]
 
 [project.urls]
 Homepage = "https://github.com/ALLAN-DIP/chiron-utils"
@@ -77,8 +83,11 @@ strict = true
 warn_unreachable = true
 
 [[tool.mypy.overrides]]
-# Needed to prevent `.../site-packages/_pytest/mark/__init__.py:187: error: Positional-only parameters are only supported in Python 3.8 and greater  [syntax]`
-module = "pytest"
+# Needed to prevent a large number of errors related to Python 3.8 syntax being used
+module = [
+  "pytest",
+  "torch.*",
+]
 follow_imports = "skip"
 
 [tool.ruff]
diff --git a/requirements-lock.txt b/requirements-lock.txt
index 0a6ed6d..9722ccb 100644
--- a/requirements-lock.txt
+++ b/requirements-lock.txt
@@ -10,15 +10,21 @@ dill==0.3.9
 diplomacy @ git+https://git@github.com/ALLAN-DIP/diplomacy.git@2270f351b0bb1423415644487f8abf7bddd06754
 distlib==0.3.9
 filelock==3.17.0
+fsspec==2024.12.0
+huggingface-hub==0.27.1
 humanfriendly==10.0
 identify==2.6.6
 idna==3.10
 iniconfig==2.0.0
 isort==6.0.0
+Jinja2==3.1.5
 joblib==1.4.2
+MarkupSafe==3.0.2
 mccabe==0.7.0
+mpmath==1.3.0
 mypy==1.8.0
 mypy-extensions==1.0.0
+networkx==3.4.2
 nodeenv==1.9.1
 numpy==1.26.4
 packaging==24.2
@@ -33,14 +39,19 @@ pytz==2024.2
 PyYAML==6.0.2
 regex==2024.11.6
 requests==2.32.3
+safetensors==0.5.2
 scikit-learn==1.4.1.post1
 scipy==1.15.1
 setuptools==75.8.0
 six==1.17.0
+sympy==1.13.1
 threadpoolctl==3.5.0
+tokenizers==0.21.0
 tomlkit==0.13.2
+torch==2.5.1
 tornado==6.4.2
 tqdm==4.67.1
+transformers==4.48.0
 typing_extensions==4.12.2
 ujson==5.10.0
 urllib3==2.3.0
diff --git a/src/chiron_utils/bots/__init__.py b/src/chiron_utils/bots/__init__.py
index 70513dd..ff089a8 100644
--- a/src/chiron_utils/bots/__init__.py
+++ b/src/chiron_utils/bots/__init__.py
@@ -33,6 +33,10 @@
             LrPlayer,
         ]
     )
+if {"torch", "transformers"} < importable_modules:
+    from chiron_utils.bots.llm_advisor_bot import LlmAdvisor as LlmAdvisor
+
+    BOTS.extend([LlmAdvisor])
 # Alphabetize list of classes
 BOTS.sort(key=lambda t: t.__name__)
 
diff --git a/src/chiron_utils/bots/llm_advisor_bot.py b/src/chiron_utils/bots/llm_advisor_bot.py
new file mode 100644
index 0000000..f2f0aa0
--- /dev/null
+++ b/src/chiron_utils/bots/llm_advisor_bot.py
@@ -0,0 +1,433 @@
+"""Bot that provides commentary advise using LLM as base model."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that provides commentary advise using LLM as base model.
+
+    We use Llama3.1-8B-Instruct as the base model and use alignment judgement,
+    to measure the alignment between Cicero predicted orders and message history.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the opponent.
+3. The predicted orders for the opponent.
+
+Your primary objective is to evaluate whether the opponent's predicted orders align with the message history and the board state.
+
+Key Evaluation Guidelines:
+
+1. Consider an order aligned if its purpose or intent is consistent with the opponent's stated goals or the tactical/strategic needs implied by the board state.
+2. Redundancy in orders (e.g., several supporting moves) can still be aligned if it serves to ensure the success of a critical move or maintains flexibility in uncertain situations.
+3. Misalignment occurs only if the order:
+    Contradicts the stated strategy or creates unnecessary risks.
+    Fails to contribute meaningfully to the position given the board state.
+4. Evaluate the broader context of each move, including defensive, offensive, or preparatory purposes, and weigh these factors in your analysis.
+For each predicted order, explain whether it aligns with the message history and board state. Use reasoning that incorporates tactical, strategic, and contextual factors while avoiding overly strict judgments.
+Limit your evaluation to alignment or misalignment, with short reasoning for each.
+

user

+
+Here is an example:
+**Predicted Orders of ITALY:**
+A TUN to ALB VIA,
+F ION C A TUN to ALB,
+F NAP to TYS,
+A VEN to TRI,
+
+**Board State:**
+AUSTRIA: [F TRI, A SER, A GAL, A BUD ]
+ENGLAND: [F LON, F NWY, A YOR, F EDI ]
+FRANCE: [A POR, A SPA, F BEL, F BRE, F MAR, A PAR ]
+GERMANY: [F DEN, A MUN, A HOL, A BER, F KIE ]
+ITALY: [A VEN, F ION, A TUN, F NAP ]
+RUSSIA: [A WAR, F SEV, F BOT, A RUM, A STP ]
+TURKEY: [A CON, A BUL, F BLA, F SMY ]
+
+**Message History:**
+Message from AUSTRIA: "France and Turkey are trying to get me to attack you, obviously I will not do that."
+Message from ITALY: "With the build in Marseilles, France's goals are quite transparent."
+Message from ITALY: "Turkey will likely bounce me in EAS. Would you be willing to support me into Greece (via convoy)?"
+Message from ITALY: "From there, I can support you into Bulgaria."
+Message from AUSTRIA: "I'm willing to debate it but I had a different view on this turn. I figured he would bounce you in EAS but what if you went for AEG instead? Unless you're worried about overextending yourself due to France.
+Because I trust you enough at this point to move Tri to Alb and then we will definitely be able to get Gre in the fall."
+Message from ITALY: "Which makes going to AEG/EAS a bit of an overstretch."
+Message from ITALY: "If we can get Germany on board I'd feel a lot more comfortable with committing east."
+Message from ITALY: "Sorry, some connection issues on my end so messages sent out of order."
+
+**Question:**
+I am playing as AUSTRIA. For each of ITALY's predicted orders, evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+
+**Answer:**
+A TUN to ALB VIA,
+Alignment: The move is aligned with the message history and board state. Italy previously mentioned the idea of moving east and positioning to take Greece.
+Convoying from TUN to ALB sets up a strong position for further collaboration with Austria in the Balkans, consistent with the stated plan to focus on Greece in the fall.
+
+F ION C A TUN to ALB,
+Alignment: The convoy is aligned with the board state and message history. The convoy facilitates Italy's movement into ALB, aligning with the communicated intent to focus efforts on securing Greece.
+This move ensures Italy's positioning in the region while maintaining the tactical flexibility discussed.
+
+F NAP to TYS,
+Alignment: This move is aligned as it supports Italy's eastward intentions.
+Moving the fleet to the Tyrrhenian Sea strengthens Italy's naval presence in the Mediterranean and provides future flexibility to either defend against France or support further eastern operations
+, consistent with Italy's concerns about overextending.
+
+A VEN to TRI,
+Misalignment: This move is misaligned with the message history and board state. Austria explicitly trusts Italy and has not shown hostility, and Italy's previous messages suggest no immediate plans to antagonize Austria.
+Moving to TRI is aggressive and undermines the collaborative stance Italy communicated. It could create unnecessary conflict, contradicting Italy's expressed preference for focusing eastward and securing Greece.
+
+
+Now let's see the question:"""
+
+    def format_boardstates(self, boardstates: Dict[str, List[str]]) -> str:
+        """Return formatted board states."""
+        lines = []
+        for power, units in boardstates.items():
+            units_str = ", ".join(units)
+            line = f"{power}: [{units_str} ]"
+            lines.append(line)
+        return "\n".join(lines)
+
+    def map_words_in_sentence(self, sentence: str, data: Dict[str, List[str]]) -> str:
+        """Return abbreviation conversed words in sentences."""
+        words = sentence.split()
+        mapped_words = []
+
+        for word in words:
+            clean_word = word.strip(",.!?")
+            word_lower = clean_word.lower()
+
+            if word_lower in data:
+                mapped_value = data[word_lower][0]
+                mapped_word = word.replace(clean_word, mapped_value)
+                mapped_words.append(mapped_word)
+            else:
+                mapped_words.append(word)
+
+        return " ".join(mapped_words)
+
+    def format_prompt_phase1(self, own: str, oppo: str, suggest_orders: List[str]) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        # Convert power names
+        if own in POWER_NAMES_DICT:
+            own = POWER_NAMES_DICT[own]
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        # Grab and sort board states, filter relevant messages.
+        system_prompt = self.create_system_prompt()
+        board_states = self.game.get_state()["units"]
+        sorted_board_states = {key: sorted(value) for key, value in board_states.items()}
+        formated_board_states = self.format_boardstates(sorted_board_states)
+        filtered_messages = self.get_relevant_messages(own, oppo)
+        if not filtered_messages:
+            return None
+
+        recent_msgs = self.get_recent_message_history(filtered_messages, max_count=10)
+        if not recent_msgs:
+            return None
+
+        # Build a text representation of the message history
+        message_history = ""
+        for msg in recent_msgs:
+            message_history += f"Message from {msg.sender}:'{msg.message}' "
+        parsed_data = json.loads(suggest_orders[0])
+        predicted_orders = parsed_data["payload"]["predicted_orders"][oppo]
+
+        orders_string = ", ".join(predicted_orders)
+
+        prompt = (
+            f"{system_prompt}"
+            f"""
+**Predicted Orders of {oppo}:**
+{orders_string.replace('-','to')}
+
+**Board State:**
+"""
+            f"{formated_board_states}\n\n"
+            f"""**Message History:**  \n{message_history}\n\n"""
+            f"""**Question:**  \nI am playing as {own}. For each of {oppo}'s predicted orders,
+            evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+\n"""
+            f"""**Answer:**
+

assistant
"""
+        )
+        return prompt
+
+    def format_prompt_phase2(self, oppo: str, evaluation: str) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        prompt = f"""

system

+            You are an expert assistant specializing in the Diplomacy board game. Based on the detailed evaluation of following predicted orders, respond in the following short and structured format:
+
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

user

+            **Evaluation**:
+            {evaluation}
+
+            **Question**:
+            from Evaluation part, extract necessary information and fill in the following as the output:
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

assistant

+
+            """
+        return prompt
+
+    def load_model(
+        self,
+        base_model_name: str,
+        tokenizer_path: Optional[str] = None,
+        device: str = "cpu",
+    ) -> Tuple[PreTrainedTokenizer, Union[PreTrainedModel, DataParallel]]:
+        """Loads and returns a tokenizer and model on the requested device."""
+        if tokenizer_path is None:
+            tokenizer_path = base_model_name
+
+        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
+        model = AutoModelForCausalLM.from_pretrained(base_model_name)
+        if torch.cuda.device_count() > 1:
+            model = DataParallel(model)
+        model.to(device)
+
+        return tokenizer, model
+
+    def generate_text(
+        self,
+        prompt: str,
+        tokenizer: PreTrainedTokenizer,
+        model: Union[PreTrainedModel, DataParallel],
+        device: str = "cpu",
+        max_new_tokens: int = 512,
+    ) -> str:
+        """Performs text generation using the tokenizer/model loaded above."""
+        if tokenizer.pad_token_id is None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+
+        inputs = tokenizer(
+            prompt,
+            return_tensors="pt",
+            padding=True,
+            truncation=True,
+            max_length=2000,
+            return_attention_mask=True,
+        )
+
+        input_ids = inputs["input_ids"].to(device)
+        attention_mask = inputs["attention_mask"].to(device)
+
+        model.eval()
+        with torch.no_grad():
+            if isinstance(model, DataParallel):
+                output_ids = model.module.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+            else:
+                output_ids = model.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+
+        return tokenizer.decode(output_ids[0], skip_special_tokens=True)  # type: ignore[no-any-return]
+
+    def generate_and_parse_response(self, prompt: str) -> str:
+        """Generate text from the model given 'prompt'.
+
+        Returns:
+            message or None if something fails.
+        """
+        generated_text = self.generate_text(prompt, self.tokenizer, self.model, self.device, 1024)
+        assistant_output = generated_text.split("assistant")[2]
+        return assistant_output
+
+    async def do_messaging_round(self, orders: Sequence[str]) -> List[str]:
+        """Carry out one round of messaging, along with related tasks.
+
+        Returns:
+            List of orders to carry out.
+        """
+        await asyncio.sleep(random.uniform(5, 10))
+
+        filtered_orders = self.read_suggestions_from_advisor()
+        if not filtered_orders:
+            return []
+
+        for other_power in get_other_powers([self.power_name], self.game):
+            all_relevant = self.get_relevant_messages(self.power_name, other_power)
+            prev_msgs = self.previous_newest_messages.get(other_power)
+            if prev_msgs is not None:
+                # Compare to find new messages
+                new_messages = [m for m in all_relevant if m not in prev_msgs]
+                if not new_messages:
+                    # No new messages, skip
+                    continue
+
+                prompt = self.format_prompt_phase1(self.power_name, other_power, filtered_orders)
+                if prompt is None:
+                    continue
+                processed_prompt = self.map_words_in_sentence(prompt, mapping)
+                logger.info("Phase1 prompt for %s: %s", other_power, processed_prompt)
+                output_phase1 = self.generate_and_parse_response(processed_prompt)
+                logger.info("Phase1 output for %s: %s", other_power, output_phase1)
+
+                alignment_count = output_phase1.count("Alignment")
+                misalignment_count = output_phase1.count("Misalignment")
+                if misalignment_count >= alignment_count:
+                    prompt2 = self.format_prompt_phase2(other_power, output_phase1)
+                    if prompt2 is None:
+                        continue
+                    output_phase2 = self.generate_and_parse_response(prompt2)
+                    logger.info("Phase2 output for %s: %s", other_power, output_phase2)
+                    if output_phase2 and output_phase2.lstrip().startswith("I think"):
+                        try:
+                            await self.suggest_commentary(other_power, f"{output_phase2.lstrip()}")
+                        except diplomacy.utils.exceptions.GamePhaseException as exc:
+                            logger.exception("Ignoring %s", exc.__class__.__name__)
+                    else:
+                        pass
+                else:
+                    # do nothing if alignment_count > misalignment_count
+                    pass
+
+            else:
+                # If we have no previous messages, treat this as first time for this power
+                prompt = self.format_prompt_phase1(self.power_name, other_power, filtered_orders)
+                if prompt is None:
+                    continue
+                processed_prompt = self.map_words_in_sentence(prompt, mapping)
+                logger.info("Phase1 prompt for %s: %s", other_power, processed_prompt)
+                output_phase1 = self.generate_and_parse_response(processed_prompt)
+                logger.info("Phase1 output for %s: %s", other_power, output_phase1)
+
+                alignment_count = output_phase1.count("Alignment")
+                misalignment_count = output_phase1.count("Misalignment")
+                if misalignment_count >= alignment_count:
+                    prompt2 = self.format_prompt_phase2(other_power, output_phase1)
+                    if prompt2 is None:
+                        continue
+                    output_phase2 = self.generate_and_parse_response(prompt2)
+                    logger.info("Phase2 output for %s: %s", other_power, output_phase2)
+                    if output_phase2 and output_phase2.lstrip().startswith("I think"):
+                        try:
+                            await self.suggest_commentary(other_power, f"{output_phase2.lstrip()}")
+                        except diplomacy.utils.exceptions.GamePhaseException as exc:
+                            logger.exception("Ignoring %s", exc.__class__.__name__)
+                    else:
+                        pass
+                else:
+                    pass
+
+            # After handling new (or first-time) messages, update the record
+            self.previous_newest_messages[other_power] = all_relevant
+
+        self.is_first_messaging_round = False
+        return list(orders)
+
+    async def gen_orders(self) -> List[str]:
+        """Generate orders for a turn.
+
+        Returns:
+            List of orders to carry out.
+        """
+        return []
diff --git a/src/chiron_utils/utils.py b/src/chiron_utils/utils.py
index a73da6b..ad17980 100644
--- a/src/chiron_utils/utils.py
+++ b/src/chiron_utils/utils.py
@@ -46,6 +46,92 @@ def return_logger(name: str, log_level: int = logging.INFO) -> logging.Logger:
     "TUR": "TURKEY",
 }
 
+mapping = {
+    "adr": ["Adriatic Sea"],
+    "aeg": ["Aegean Sea"],
+    "alb": ["Albania"],
+    "ank": ["Ankara"],
+    "apu": ["Apulia"],
+    "arm": ["Armenia"],
+    "bal": ["Baltic Sea"],
+    "bar": ["Barents Sea"],
+    "bel": ["Belgium"],
+    "ber": ["Berlin"],
+    "bla": ["Black Sea"],
+    "boh": ["Bohemia"],
+    "bre": ["Brest"],
+    "bud": ["Budapest"],
+    "bul/ec": ["Bulgaria (East Coast)"],
+    "bul/sc": ["Bulgaria (South Coast)"],
+    "bul": ["Bulgaria"],
+    "bur": ["Burgundy"],
+    "cly": ["Clyde"],
+    "con": ["Constantinople"],
+    "den": ["Denmark"],
+    "eas": ["Eastern Mediterranean"],
+    "edi": ["Edinburgh"],
+    "eng": ["English Channel"],
+    "fin": ["Finland"],
+    "gal": ["Galicia"],
+    "gas": ["Gascony"],
+    "gre": ["Greece"],
+    "lyo": ["Gulf of Lyon"],
+    "bot": ["Gulf of Bothnia"],
+    "hel": ["Heligoland Bight"],
+    "hol": ["Holland"],
+    "ion": ["Ionian Sea"],
+    "iri": ["Irish Sea"],
+    "kie": ["Kiel"],
+    "lvp": ["Liverpool"],
+    "lvn": ["Livonia"],
+    "lon": ["London"],
+    "mar": ["Marseilles"],
+    "mao": ["Mid-Atlantic Ocean"],
+    "mos": ["Moscow"],
+    "mun": ["Munich"],
+    "nap": ["Naples"],
+    "nao": ["North Atlantic Ocean"],
+    "naf": ["North Africa"],
+    "nth": ["North Sea"],
+    "nwy": ["Norway"],
+    "nwg": ["Norwegian Sea"],
+    "par": ["Paris"],
+    "pic": ["Picardy"],
+    "pie": ["Piedmont"],
+    "por": ["Portugal"],
+    "pru": ["Prussia"],
+    "rom": ["Rome"],
+    "ruh": ["Ruhr"],
+    "rum": ["Romania"],
+    "ser": ["Serbia"],
+    "sev": ["Sevastopol"],
+    "sil": ["Silesia"],
+    "ska": ["Skagerrak"],
+    "smy": ["Smyrna"],
+    "spa/nc": ["Spain (North Coast)"],
+    "spa/sc": ["Spain (South Coast)"],
+    "spa": ["Spain"],
+    "stp/nc": ["St. Petersburg (North Coast)"],
+    "stp/sc": ["St. Petersburg (South Coast)"],
+    "stp": ["St. Petersburg"],
+    "swe": ["Sweden"],
+    "syr": ["Syria"],
+    "tri": ["Trieste"],
+    "tun": ["Tunis"],
+    "tus": ["Tuscany"],
+    "tyr": ["Tyrolia"],
+    "tys": ["Tyrrhenian Sea"],
+    "ukr": ["Ukraine"],
+    "ven": ["Venice"],
+    "vie": ["Vienna"],
+    "wal": ["Wales"],
+    "war": ["Warsaw"],
+    "wes": ["Western Mediterranean"],
+    "yor": ["Yorkshire"],
+    " - ": ["to"],
+}
+
+
 # Option for debugging without specialized builds
 DEBUG_MODE = False
 if os.environ.get("ALLAN_DEBUG") is not None:


Reviews:
user: login: aphedges
body: I've rebased your changes against `main` and fixed the conflicts I've encountered in files outside of `llm_advisor_bot.py`. However, I haven't tried to run it yet because I have specific feedback for you to make changes from:

- Run `make update` locally to get the package changes I made.
- Is `LlmPlayer` needed? As far as I'm aware, your bot can't act as a player because it doesn't generate moves or messages. If that is correct, then please delete `LlmPlayer` and merge `LlmAdvisor` into `LlmAdvisorBot`.
- Is the value for `suggestion_type` correct? Shouldn't it only be providing commentary? Also, I changed the name of the variable to `default_suggestion_type`, so please change it as well.
- You have lots of commented-out code. Please remove it if you aren't going to use it.
- Replace `print()` statements with calls to `logger`
- I noticed that you are using the hardcoded string `'"advisor":"ENGLAND (CiceroAdvisor)"'`. That should be replaced with something that isn't tied to a particular power.
- Run `make check` locally and make changes until everything passes. You are mostly missing type annotations.
- Add instructions on how to run your bot to the README
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-12T20:11:20Z
user: login: aphedges
body: It looks a lot better! I just have a little more feedback before I test it out:

- > only one error left when running `make check` : `tests/bot_tests/bot_test.py::TestBots::test_send_message FAILED` Since `LlmAdvisor` doesn't send messages, only suggest commentary, I think this is fine? Do I need to fix it somehow?

  Sorry about that! I have that test disabled locally and in CI because it requires a `diplomacy` server running locally, and I should just disable it by default.

- Please re-add comments that were useful, which most of them were. I only wanted you to delete _code_ that was commented out!
- I know that earlier versions used a finetuned Llama model. Do I need to download one from somewhere, or does this version just use an off-the-shelf model?
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T01:27:40Z
user: login: aphedges
body: I've tested out your bot, and I can confirm it works! I needed to make some changes, and I put them in multiple commits to make them easier to look over. I also rebased the branch to pull in the fix for the test problem you encountered. Please let me know what you think after look over them and test them out!

I made some inline comments about some final things that should be resolved before I clean up the branch and merge!
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T19:35:39Z
user: login: YanzeJeremy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:37:49Z
user: login: YanzeJeremy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T20:03:02Z
user: login: YanzeJeremy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T20:03:08Z
user: login: YanzeJeremy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T20:05:01Z
user: login: YanzeJeremy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T20:06:23Z
user: login: aphedges
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T20:00:57Z
user: login: aphedges
body: This all looks good to me now! I'll merge this PR once CI completes.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T17:17:21Z

Review comments:
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: aphedges
body: Please change this docstring to be about this current bot.
created_at: 2025-03-14T19:29:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: aphedges
body: Please change this docstring to be about this current bot.
created_at: 2025-03-14T19:29:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: aphedges
body: What do you mean by `counterplayer` in the prompt? Is this intentionally chosen, or should it just be `opponent` instead?
created_at: 2025-03-14T19:30:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
+3. The predicted orders for the counterplayer.
+
+Your primary objective is to evaluate whether the counterplayer's predicted orders align with the message history and the board state.
+
+Key Evaluation Guidelines:
+
+1. Consider an order aligned if its purpose or intent is consistent with the counterplayer's stated goals or the tactical/strategic needs implied by the board state.
+2. Redundancy in orders (e.g., several supporting moves) can still be aligned if it serves to ensure the success of a critical move or maintains flexibility in uncertain situations.
+3. Misalignment occurs only if the order:
+    Contradicts the stated strategy or creates unnecessary risks.
+    Fails to contribute meaningfully to the position given the board state.
+4. Evaluate the broader context of each move, including defensive, offensive, or preparatory purposes, and weigh these factors in your analysis.
+For each predicted order, explain whether it aligns with the message history and board state. Use reasoning that incorporates tactical, strategic, and contextual factors while avoiding overly strict judgments.
+Limit your evaluation to alignment or misalignment, with short reasoning for each.
+

user

+
+Here is an example:
+**Predicted Orders of ITALY:**
+A TUN to ALB VIA,
+F ION C A TUN to ALB,
+F NAP to TYS,
+A VEN to TRI,
+
+**Board State:**
+AUSTRIA: [F TRI, A SER, A GAL, A BUD ]
+ENGLAND: [F LON, F NWY, A YOR, F EDI ]
+FRANCE: [A POR, A SPA, F BEL, F BRE, F MAR, A PAR ]
+GERMANY: [F DEN, A MUN, A HOL, A BER, F KIE ]
+ITALY: [A VEN, F ION, A TUN, F NAP ]
+RUSSIA: [A WAR, F SEV, F BOT, A RUM, A STP ]
+TURKEY: [A CON, A BUL, F BLA, F SMY ]
+
+**Message History:**
+Message from AUSTRIA: "France and Turkey are trying to get me to attack you, obviously I will not do that."
+Message from ITALY: "With the build in Marseilles, France's goals are quite transparent."
+Message from ITALY: "Turkey will likely bounce me in EAS. Would you be willing to support me into Greece (via convoy)?"
+Message from ITALY: "From there, I can support you into Bulgaria."
+Message from AUSTRIA: "I'm willing to debate it but I had a different view on this turn. I figured he would bounce you in EAS but what if you went for AEG instead? Unless you're worried about overextending yourself due to France.
+Because I trust you enough at this point to move Tri to Alb and then we will definitely be able to get Gre in the fall."
+Message from ITALY: "Which makes going to AEG/EAS a bit of an overstretch."
+Message from ITALY: "If we can get Germany on board I'd feel a lot more comfortable with committing east."
+Message from ITALY: "Sorry, some connection issues on my end so messages sent out of order."
+
+**Question:**
+I am playing as AUSTRIA. For each of ITALY's predicted orders, evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+
+**Answer:**
+A TUN to ALB VIA,
+Alignment: The move is aligned with the message history and board state. Italy previously mentioned the idea of moving east and positioning to take Greece.
+Convoying from TUN to ALB sets up a strong position for further collaboration with Austria in the Balkans, consistent with the stated plan to focus on Greece in the fall.
+
+F ION C A TUN to ALB,
+Alignment: The convoy is aligned with the board state and message history. The convoy facilitates Italy's movement into ALB, aligning with the communicated intent to focus efforts on securing Greece.
+This move ensures Italy's positioning in the region while maintaining the tactical flexibility discussed.
+
+F NAP to TYS,
+Alignment: This move is aligned as it supports Italy's eastward intentions.
+Moving the fleet to the Tyrrhenian Sea strengthens Italy's naval presence in the Mediterranean and provides future flexibility to either defend against France or support further eastern operations
+, consistent with Italy's concerns about overextending.
+
+A VEN to TRI,
+Misalignment: This move is misaligned with the message history and board state. Austria explicitly trusts Italy and has not shown hostility, and Italy's previous messages suggest no immediate plans to antagonize Austria.
+Moving to TRI is aggressive and undermines the collaborative stance Italy communicated. It could create unnecessary conflict, contradicting Italy's expressed preference for focusing eastward and securing Greece.
+
+
+Now let's see the question:"""
+
+    def format_boardstates(self, boardstates: Dict[str, List[str]]) -> str:
+        """Return formatted board states."""
+        lines = []
+        for power, units in boardstates.items():
+            units_str = ", ".join(units)
+            line = f"{power}: [{units_str} ]"
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: aphedges
body: Is the extra space in ` ]` intentional? If not, please remove it.
created_at: 2025-03-14T19:30:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
+3. The predicted orders for the counterplayer.
+
+Your primary objective is to evaluate whether the counterplayer's predicted orders align with the message history and the board state.
+
+Key Evaluation Guidelines:
+
+1. Consider an order aligned if its purpose or intent is consistent with the counterplayer's stated goals or the tactical/strategic needs implied by the board state.
+2. Redundancy in orders (e.g., several supporting moves) can still be aligned if it serves to ensure the success of a critical move or maintains flexibility in uncertain situations.
+3. Misalignment occurs only if the order:
+    Contradicts the stated strategy or creates unnecessary risks.
+    Fails to contribute meaningfully to the position given the board state.
+4. Evaluate the broader context of each move, including defensive, offensive, or preparatory purposes, and weigh these factors in your analysis.
+For each predicted order, explain whether it aligns with the message history and board state. Use reasoning that incorporates tactical, strategic, and contextual factors while avoiding overly strict judgments.
+Limit your evaluation to alignment or misalignment, with short reasoning for each.
+

user

+
+Here is an example:
+**Predicted Orders of ITALY:**
+A TUN to ALB VIA,
+F ION C A TUN to ALB,
+F NAP to TYS,
+A VEN to TRI,
+
+**Board State:**
+AUSTRIA: [F TRI, A SER, A GAL, A BUD ]
+ENGLAND: [F LON, F NWY, A YOR, F EDI ]
+FRANCE: [A POR, A SPA, F BEL, F BRE, F MAR, A PAR ]
+GERMANY: [F DEN, A MUN, A HOL, A BER, F KIE ]
+ITALY: [A VEN, F ION, A TUN, F NAP ]
+RUSSIA: [A WAR, F SEV, F BOT, A RUM, A STP ]
+TURKEY: [A CON, A BUL, F BLA, F SMY ]
+
+**Message History:**
+Message from AUSTRIA: "France and Turkey are trying to get me to attack you, obviously I will not do that."
+Message from ITALY: "With the build in Marseilles, France's goals are quite transparent."
+Message from ITALY: "Turkey will likely bounce me in EAS. Would you be willing to support me into Greece (via convoy)?"
+Message from ITALY: "From there, I can support you into Bulgaria."
+Message from AUSTRIA: "I'm willing to debate it but I had a different view on this turn. I figured he would bounce you in EAS but what if you went for AEG instead? Unless you're worried about overextending yourself due to France.
+Because I trust you enough at this point to move Tri to Alb and then we will definitely be able to get Gre in the fall."
+Message from ITALY: "Which makes going to AEG/EAS a bit of an overstretch."
+Message from ITALY: "If we can get Germany on board I'd feel a lot more comfortable with committing east."
+Message from ITALY: "Sorry, some connection issues on my end so messages sent out of order."
+
+**Question:**
+I am playing as AUSTRIA. For each of ITALY's predicted orders, evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+
+**Answer:**
+A TUN to ALB VIA,
+Alignment: The move is aligned with the message history and board state. Italy previously mentioned the idea of moving east and positioning to take Greece.
+Convoying from TUN to ALB sets up a strong position for further collaboration with Austria in the Balkans, consistent with the stated plan to focus on Greece in the fall.
+
+F ION C A TUN to ALB,
+Alignment: The convoy is aligned with the board state and message history. The convoy facilitates Italy's movement into ALB, aligning with the communicated intent to focus efforts on securing Greece.
+This move ensures Italy's positioning in the region while maintaining the tactical flexibility discussed.
+
+F NAP to TYS,
+Alignment: This move is aligned as it supports Italy's eastward intentions.
+Moving the fleet to the Tyrrhenian Sea strengthens Italy's naval presence in the Mediterranean and provides future flexibility to either defend against France or support further eastern operations
+, consistent with Italy's concerns about overextending.
+
+A VEN to TRI,
+Misalignment: This move is misaligned with the message history and board state. Austria explicitly trusts Italy and has not shown hostility, and Italy's previous messages suggest no immediate plans to antagonize Austria.
+Moving to TRI is aggressive and undermines the collaborative stance Italy communicated. It could create unnecessary conflict, contradicting Italy's expressed preference for focusing eastward and securing Greece.
+
+
+Now let's see the question:"""
+
+    def format_boardstates(self, boardstates: Dict[str, List[str]]) -> str:
+        """Return formatted board states."""
+        lines = []
+        for power, units in boardstates.items():
+            units_str = ", ".join(units)
+            line = f"{power}: [{units_str} ]"
+            lines.append(line)
+        return "\n".join(lines)
+
+    def map_words_in_sentence(self, sentence: str, data: Dict[str, List[str]]) -> str:
+        """Return abbreviation conversed words in sentences."""
+        words = sentence.split()
+        mapped_words = []
+
+        for word in words:
+            clean_word = word.strip(",.!?")
+            word_lower = clean_word.lower()
+
+            if word_lower in data:
+                mapped_value = data[word_lower][0]
+                mapped_word = word.replace(clean_word, mapped_value)
+                mapped_words.append(mapped_word)
+            else:
+                mapped_words.append(word)
+
+        return " ".join(mapped_words)
+
+    def format_prompt_phase1(self, own: str, oppo: str, suggest_orders: List[str]) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        # Convert power names
+        if own in POWER_NAMES_DICT:
+            own = POWER_NAMES_DICT[own]
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        # Grab and sort board states, filter relevant messages.
+        system_prompt = self.create_system_prompt()
+        board_states = self.game.get_state()["units"]
+        sorted_board_states = {key: sorted(value) for key, value in board_states.items()}
+        formated_board_states = self.format_boardstates(sorted_board_states)
+        filtered_messages = self.get_relevant_messages(own, oppo)
+        if not filtered_messages:
+            return None
+
+        recent_msgs = self.get_recent_message_history(filtered_messages, max_count=10)
+        if not recent_msgs:
+            return None
+
+        # Build a text representation of the message history
+        message_history = ""
+        for msg in recent_msgs:
+            message_history += f"Message from {msg.sender}:'{msg.message}' "
+        parsed_data = json.loads(suggest_orders[0])
+        predicted_orders = parsed_data["payload"]["predicted_orders"][oppo]
+
+        orders_string = ", ".join(predicted_orders)
+
+        prompt = (
+            f"{system_prompt}"
+            f"""
+**Predicted Orders of {oppo}:**
+{orders_string.replace('-','to')}
+
+**Board State:**
+"""
+            f"{formated_board_states}\n\n"
+            f"""**Message History:**  \n{message_history}\n\n"""
+            f"""**Question:**  \nI am playing as {own}. For each of {oppo}'s predicted orders,
+            evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+\n"""
+            f"""**Answer:**
+

assistant
"""
+        )
+        return prompt
+
+    def format_prompt_phase2(self, oppo: str, evaluation: str) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        prompt = f"""

system

+            You are an expert assistant specializing in the Diplomacy board game. Based on the detailed evaluation of following predicted orders, respond in the following short and structured format:
+
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

user

+            **Evaluation**:
+            {evaluation}
+
+            **Question**:
+            from Evaluation part, extract necessary information and fill in the following as the output:
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

assistant

+
+            """
+        return prompt
+
+    def load_model(
+        self,
+        base_model_name: str,
+        tokenizer_path: Optional[str] = None,
+        device: str = "cpu",
+    ) -> Tuple[PreTrainedTokenizer, Union[PreTrainedModel, DataParallel]]:
+        """Loads and returns a tokenizer and model on the requested device."""
+        if tokenizer_path is None:
+            tokenizer_path = base_model_name
+
+        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
+        model = AutoModelForCausalLM.from_pretrained(base_model_name)
+        if torch.cuda.device_count() > 1:
+            model = DataParallel(model)
+        model.to(device)
+
+        return tokenizer, model
+
+    def generate_text(
+        self,
+        prompt: str,
+        tokenizer: PreTrainedTokenizer,
+        model: Union[PreTrainedModel, DataParallel],
+        device: str = "cpu",
+        max_new_tokens: int = 512,
+    ) -> str:
+        """Performs text generation using the tokenizer/model loaded above."""
+        if tokenizer.pad_token_id is None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+
+        inputs = tokenizer(
+            prompt,
+            return_tensors="pt",
+            padding=True,
+            truncation=True,
+            max_length=2000,
+            return_attention_mask=True,
+        )
+
+        input_ids = inputs["input_ids"].to(device)
+        attention_mask = inputs["attention_mask"].to(device)
+
+        model.eval()
+        with torch.no_grad():
+            if isinstance(model, DataParallel):
+                output_ids = model.module.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+            else:
+                output_ids = model.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+
+        return tokenizer.decode(output_ids[0], skip_special_tokens=True)  # type: ignore[no-any-return]
+
+    def generate_and_parse_response(self, prompt: str) -> str:
+        """Generate text from the model given 'prompt'.
+
+        Returns:
+            message or None if something fails.
+        """
+        generated_text = self.generate_text(prompt, self.tokenizer, self.model, self.device, 1024)
+        assistant_output = generated_text.split("assistant")[2]
+        return assistant_output
+
+    async def do_messaging_round(self, orders: Sequence[str]) -> List[str]:
+        """Carry out one round of messaging, along with related tasks.
+
+        Returns:
+            List of orders to carry out.
+        """
+        await asyncio.sleep(random.uniform(5, 10))
+
+        filtered_orders = self.read_suggestions_from_advisor()
+        if not filtered_orders:
+            return []
+
+        for other_power in get_other_powers([self.power_name], self.game):
+            all_relevant = self.get_relevant_messages(self.power_name, other_power)
+            prev_msgs = self.previous_newest_messages.get(other_power)
+            if prev_msgs is not None:
+                # Compare to find new messages
+                new_messages = [m for m in all_relevant if m not in prev_msgs]
+                if not new_messages:
+                    # No new messages, skip
+                    continue
+
+                prompt = self.format_prompt_phase1(self.power_name, other_power, filtered_orders)
+                if prompt is None:
+                    continue
+                processed_prompt = self.map_words_in_sentence(prompt, mapping)
+                logger.info("Phase1 prompt for %s: %s", other_power, processed_prompt)
+                output_phase1 = self.generate_and_parse_response(processed_prompt)
+                logger.info("Phase1 output for %s: %s", other_power, output_phase1)
+
+                alignment_count = output_phase1.count("Alignment")
+                misalignment_count = output_phase1.count("Misalignment")
+                if misalignment_count >= alignment_count:
+                    prompt2 = self.format_prompt_phase2(other_power, output_phase1)
+                    if prompt2 is None:
+                        continue
+                    output_phase2 = self.generate_and_parse_response(prompt2)
+                    logger.info("Phase2 output for %s: %s", other_power, output_phase2)
+                    if output_phase2 and output_phase2.lstrip().startswith("I think"):
+                        try:
+                            await self.suggest_commentary(other_power, f"{output_phase2.lstrip()}")
+                        except diplomacy.utils.exceptions.GamePhaseException as exc:
+                            logger.exception("Ignoring %s", exc.__class__.__name__)
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: aphedges
body: How often do you encounter this error? It probably shouldn't be happening at all, and we should try to fix the problem if you can reproduce it.
created_at: 2025-03-14T19:34:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: YanzeJeremy
body: Thanks for reminding! opponent should be more clear here :) I feel like i changed this narrative before, may lose tracking somehow
created_at: 2025-03-18T19:37:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: YanzeJeremy
body: Done!
created_at: 2025-03-18T20:03:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: YanzeJeremy
body: Done!
created_at: 2025-03-18T20:03:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
+3. The predicted orders for the counterplayer.
+
+Your primary objective is to evaluate whether the counterplayer's predicted orders align with the message history and the board state.
+
+Key Evaluation Guidelines:
+
+1. Consider an order aligned if its purpose or intent is consistent with the counterplayer's stated goals or the tactical/strategic needs implied by the board state.
+2. Redundancy in orders (e.g., several supporting moves) can still be aligned if it serves to ensure the success of a critical move or maintains flexibility in uncertain situations.
+3. Misalignment occurs only if the order:
+    Contradicts the stated strategy or creates unnecessary risks.
+    Fails to contribute meaningfully to the position given the board state.
+4. Evaluate the broader context of each move, including defensive, offensive, or preparatory purposes, and weigh these factors in your analysis.
+For each predicted order, explain whether it aligns with the message history and board state. Use reasoning that incorporates tactical, strategic, and contextual factors while avoiding overly strict judgments.
+Limit your evaluation to alignment or misalignment, with short reasoning for each.
+

user

+
+Here is an example:
+**Predicted Orders of ITALY:**
+A TUN to ALB VIA,
+F ION C A TUN to ALB,
+F NAP to TYS,
+A VEN to TRI,
+
+**Board State:**
+AUSTRIA: [F TRI, A SER, A GAL, A BUD ]
+ENGLAND: [F LON, F NWY, A YOR, F EDI ]
+FRANCE: [A POR, A SPA, F BEL, F BRE, F MAR, A PAR ]
+GERMANY: [F DEN, A MUN, A HOL, A BER, F KIE ]
+ITALY: [A VEN, F ION, A TUN, F NAP ]
+RUSSIA: [A WAR, F SEV, F BOT, A RUM, A STP ]
+TURKEY: [A CON, A BUL, F BLA, F SMY ]
+
+**Message History:**
+Message from AUSTRIA: "France and Turkey are trying to get me to attack you, obviously I will not do that."
+Message from ITALY: "With the build in Marseilles, France's goals are quite transparent."
+Message from ITALY: "Turkey will likely bounce me in EAS. Would you be willing to support me into Greece (via convoy)?"
+Message from ITALY: "From there, I can support you into Bulgaria."
+Message from AUSTRIA: "I'm willing to debate it but I had a different view on this turn. I figured he would bounce you in EAS but what if you went for AEG instead? Unless you're worried about overextending yourself due to France.
+Because I trust you enough at this point to move Tri to Alb and then we will definitely be able to get Gre in the fall."
+Message from ITALY: "Which makes going to AEG/EAS a bit of an overstretch."
+Message from ITALY: "If we can get Germany on board I'd feel a lot more comfortable with committing east."
+Message from ITALY: "Sorry, some connection issues on my end so messages sent out of order."
+
+**Question:**
+I am playing as AUSTRIA. For each of ITALY's predicted orders, evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+
+**Answer:**
+A TUN to ALB VIA,
+Alignment: The move is aligned with the message history and board state. Italy previously mentioned the idea of moving east and positioning to take Greece.
+Convoying from TUN to ALB sets up a strong position for further collaboration with Austria in the Balkans, consistent with the stated plan to focus on Greece in the fall.
+
+F ION C A TUN to ALB,
+Alignment: The convoy is aligned with the board state and message history. The convoy facilitates Italy's movement into ALB, aligning with the communicated intent to focus efforts on securing Greece.
+This move ensures Italy's positioning in the region while maintaining the tactical flexibility discussed.
+
+F NAP to TYS,
+Alignment: This move is aligned as it supports Italy's eastward intentions.
+Moving the fleet to the Tyrrhenian Sea strengthens Italy's naval presence in the Mediterranean and provides future flexibility to either defend against France or support further eastern operations
+, consistent with Italy's concerns about overextending.
+
+A VEN to TRI,
+Misalignment: This move is misaligned with the message history and board state. Austria explicitly trusts Italy and has not shown hostility, and Italy's previous messages suggest no immediate plans to antagonize Austria.
+Moving to TRI is aggressive and undermines the collaborative stance Italy communicated. It could create unnecessary conflict, contradicting Italy's expressed preference for focusing eastward and securing Greece.
+
+
+Now let's see the question:"""
+
+    def format_boardstates(self, boardstates: Dict[str, List[str]]) -> str:
+        """Return formatted board states."""
+        lines = []
+        for power, units in boardstates.items():
+            units_str = ", ".join(units)
+            line = f"{power}: [{units_str} ]"
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: YanzeJeremy
body: yes, this is intentional, which are used for mapping the power name sometimes. I prefer to keep it.
created_at: 2025-03-18T20:05:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
+3. The predicted orders for the counterplayer.
+
+Your primary objective is to evaluate whether the counterplayer's predicted orders align with the message history and the board state.
+
+Key Evaluation Guidelines:
+
+1. Consider an order aligned if its purpose or intent is consistent with the counterplayer's stated goals or the tactical/strategic needs implied by the board state.
+2. Redundancy in orders (e.g., several supporting moves) can still be aligned if it serves to ensure the success of a critical move or maintains flexibility in uncertain situations.
+3. Misalignment occurs only if the order:
+    Contradicts the stated strategy or creates unnecessary risks.
+    Fails to contribute meaningfully to the position given the board state.
+4. Evaluate the broader context of each move, including defensive, offensive, or preparatory purposes, and weigh these factors in your analysis.
+For each predicted order, explain whether it aligns with the message history and board state. Use reasoning that incorporates tactical, strategic, and contextual factors while avoiding overly strict judgments.
+Limit your evaluation to alignment or misalignment, with short reasoning for each.
+

user

+
+Here is an example:
+**Predicted Orders of ITALY:**
+A TUN to ALB VIA,
+F ION C A TUN to ALB,
+F NAP to TYS,
+A VEN to TRI,
+
+**Board State:**
+AUSTRIA: [F TRI, A SER, A GAL, A BUD ]
+ENGLAND: [F LON, F NWY, A YOR, F EDI ]
+FRANCE: [A POR, A SPA, F BEL, F BRE, F MAR, A PAR ]
+GERMANY: [F DEN, A MUN, A HOL, A BER, F KIE ]
+ITALY: [A VEN, F ION, A TUN, F NAP ]
+RUSSIA: [A WAR, F SEV, F BOT, A RUM, A STP ]
+TURKEY: [A CON, A BUL, F BLA, F SMY ]
+
+**Message History:**
+Message from AUSTRIA: "France and Turkey are trying to get me to attack you, obviously I will not do that."
+Message from ITALY: "With the build in Marseilles, France's goals are quite transparent."
+Message from ITALY: "Turkey will likely bounce me in EAS. Would you be willing to support me into Greece (via convoy)?"
+Message from ITALY: "From there, I can support you into Bulgaria."
+Message from AUSTRIA: "I'm willing to debate it but I had a different view on this turn. I figured he would bounce you in EAS but what if you went for AEG instead? Unless you're worried about overextending yourself due to France.
+Because I trust you enough at this point to move Tri to Alb and then we will definitely be able to get Gre in the fall."
+Message from ITALY: "Which makes going to AEG/EAS a bit of an overstretch."
+Message from ITALY: "If we can get Germany on board I'd feel a lot more comfortable with committing east."
+Message from ITALY: "Sorry, some connection issues on my end so messages sent out of order."
+
+**Question:**
+I am playing as AUSTRIA. For each of ITALY's predicted orders, evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+
+**Answer:**
+A TUN to ALB VIA,
+Alignment: The move is aligned with the message history and board state. Italy previously mentioned the idea of moving east and positioning to take Greece.
+Convoying from TUN to ALB sets up a strong position for further collaboration with Austria in the Balkans, consistent with the stated plan to focus on Greece in the fall.
+
+F ION C A TUN to ALB,
+Alignment: The convoy is aligned with the board state and message history. The convoy facilitates Italy's movement into ALB, aligning with the communicated intent to focus efforts on securing Greece.
+This move ensures Italy's positioning in the region while maintaining the tactical flexibility discussed.
+
+F NAP to TYS,
+Alignment: This move is aligned as it supports Italy's eastward intentions.
+Moving the fleet to the Tyrrhenian Sea strengthens Italy's naval presence in the Mediterranean and provides future flexibility to either defend against France or support further eastern operations
+, consistent with Italy's concerns about overextending.
+
+A VEN to TRI,
+Misalignment: This move is misaligned with the message history and board state. Austria explicitly trusts Italy and has not shown hostility, and Italy's previous messages suggest no immediate plans to antagonize Austria.
+Moving to TRI is aggressive and undermines the collaborative stance Italy communicated. It could create unnecessary conflict, contradicting Italy's expressed preference for focusing eastward and securing Greece.
+
+
+Now let's see the question:"""
+
+    def format_boardstates(self, boardstates: Dict[str, List[str]]) -> str:
+        """Return formatted board states."""
+        lines = []
+        for power, units in boardstates.items():
+            units_str = ", ".join(units)
+            line = f"{power}: [{units_str} ]"
+            lines.append(line)
+        return "\n".join(lines)
+
+    def map_words_in_sentence(self, sentence: str, data: Dict[str, List[str]]) -> str:
+        """Return abbreviation conversed words in sentences."""
+        words = sentence.split()
+        mapped_words = []
+
+        for word in words:
+            clean_word = word.strip(",.!?")
+            word_lower = clean_word.lower()
+
+            if word_lower in data:
+                mapped_value = data[word_lower][0]
+                mapped_word = word.replace(clean_word, mapped_value)
+                mapped_words.append(mapped_word)
+            else:
+                mapped_words.append(word)
+
+        return " ".join(mapped_words)
+
+    def format_prompt_phase1(self, own: str, oppo: str, suggest_orders: List[str]) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        # Convert power names
+        if own in POWER_NAMES_DICT:
+            own = POWER_NAMES_DICT[own]
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        # Grab and sort board states, filter relevant messages.
+        system_prompt = self.create_system_prompt()
+        board_states = self.game.get_state()["units"]
+        sorted_board_states = {key: sorted(value) for key, value in board_states.items()}
+        formated_board_states = self.format_boardstates(sorted_board_states)
+        filtered_messages = self.get_relevant_messages(own, oppo)
+        if not filtered_messages:
+            return None
+
+        recent_msgs = self.get_recent_message_history(filtered_messages, max_count=10)
+        if not recent_msgs:
+            return None
+
+        # Build a text representation of the message history
+        message_history = ""
+        for msg in recent_msgs:
+            message_history += f"Message from {msg.sender}:'{msg.message}' "
+        parsed_data = json.loads(suggest_orders[0])
+        predicted_orders = parsed_data["payload"]["predicted_orders"][oppo]
+
+        orders_string = ", ".join(predicted_orders)
+
+        prompt = (
+            f"{system_prompt}"
+            f"""
+**Predicted Orders of {oppo}:**
+{orders_string.replace('-','to')}
+
+**Board State:**
+"""
+            f"{formated_board_states}\n\n"
+            f"""**Message History:**  \n{message_history}\n\n"""
+            f"""**Question:**  \nI am playing as {own}. For each of {oppo}'s predicted orders,
+            evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+\n"""
+            f"""**Answer:**
+

assistant
"""
+        )
+        return prompt
+
+    def format_prompt_phase2(self, oppo: str, evaluation: str) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        prompt = f"""

system

+            You are an expert assistant specializing in the Diplomacy board game. Based on the detailed evaluation of following predicted orders, respond in the following short and structured format:
+
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

user

+            **Evaluation**:
+            {evaluation}
+
+            **Question**:
+            from Evaluation part, extract necessary information and fill in the following as the output:
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

assistant

+
+            """
+        return prompt
+
+    def load_model(
+        self,
+        base_model_name: str,
+        tokenizer_path: Optional[str] = None,
+        device: str = "cpu",
+    ) -> Tuple[PreTrainedTokenizer, Union[PreTrainedModel, DataParallel]]:
+        """Loads and returns a tokenizer and model on the requested device."""
+        if tokenizer_path is None:
+            tokenizer_path = base_model_name
+
+        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
+        model = AutoModelForCausalLM.from_pretrained(base_model_name)
+        if torch.cuda.device_count() > 1:
+            model = DataParallel(model)
+        model.to(device)
+
+        return tokenizer, model
+
+    def generate_text(
+        self,
+        prompt: str,
+        tokenizer: PreTrainedTokenizer,
+        model: Union[PreTrainedModel, DataParallel],
+        device: str = "cpu",
+        max_new_tokens: int = 512,
+    ) -> str:
+        """Performs text generation using the tokenizer/model loaded above."""
+        if tokenizer.pad_token_id is None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+
+        inputs = tokenizer(
+            prompt,
+            return_tensors="pt",
+            padding=True,
+            truncation=True,
+            max_length=2000,
+            return_attention_mask=True,
+        )
+
+        input_ids = inputs["input_ids"].to(device)
+        attention_mask = inputs["attention_mask"].to(device)
+
+        model.eval()
+        with torch.no_grad():
+            if isinstance(model, DataParallel):
+                output_ids = model.module.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+            else:
+                output_ids = model.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+
+        return tokenizer.decode(output_ids[0], skip_special_tokens=True)  # type: ignore[no-any-return]
+
+    def generate_and_parse_response(self, prompt: str) -> str:
+        """Generate text from the model given 'prompt'.
+
+        Returns:
+            message or None if something fails.
+        """
+        generated_text = self.generate_text(prompt, self.tokenizer, self.model, self.device, 1024)
+        assistant_output = generated_text.split("assistant")[2]
+        return assistant_output
+
+    async def do_messaging_round(self, orders: Sequence[str]) -> List[str]:
+        """Carry out one round of messaging, along with related tasks.
+
+        Returns:
+            List of orders to carry out.
+        """
+        await asyncio.sleep(random.uniform(5, 10))
+
+        filtered_orders = self.read_suggestions_from_advisor()
+        if not filtered_orders:
+            return []
+
+        for other_power in get_other_powers([self.power_name], self.game):
+            all_relevant = self.get_relevant_messages(self.power_name, other_power)
+            prev_msgs = self.previous_newest_messages.get(other_power)
+            if prev_msgs is not None:
+                # Compare to find new messages
+                new_messages = [m for m in all_relevant if m not in prev_msgs]
+                if not new_messages:
+                    # No new messages, skip
+                    continue
+
+                prompt = self.format_prompt_phase1(self.power_name, other_power, filtered_orders)
+                if prompt is None:
+                    continue
+                processed_prompt = self.map_words_in_sentence(prompt, mapping)
+                logger.info("Phase1 prompt for %s: %s", other_power, processed_prompt)
+                output_phase1 = self.generate_and_parse_response(processed_prompt)
+                logger.info("Phase1 output for %s: %s", other_power, output_phase1)
+
+                alignment_count = output_phase1.count("Alignment")
+                misalignment_count = output_phase1.count("Misalignment")
+                if misalignment_count >= alignment_count:
+                    prompt2 = self.format_prompt_phase2(other_power, output_phase1)
+                    if prompt2 is None:
+                        continue
+                    output_phase2 = self.generate_and_parse_response(prompt2)
+                    logger.info("Phase2 output for %s: %s", other_power, output_phase2)
+                    if output_phase2 and output_phase2.lstrip().startswith("I think"):
+                        try:
+                            await self.suggest_commentary(other_power, f"{output_phase2.lstrip()}")
+                        except diplomacy.utils.exceptions.GamePhaseException as exc:
+                            logger.exception("Ignoring %s", exc.__class__.__name__)
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: YanzeJeremy
body: It's hard to tell how often, maybe like once or twice per game? The error is about current phase and server phase misalignment.
created_at: 2025-03-18T20:06:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,443 @@
+"""Bots that carry out random orders and make random order proposals."""
+
+import asyncio
+from dataclasses import dataclass, field
+import json
+import random
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import diplomacy
+from diplomacy import Message
+from diplomacy.utils import strings as diplomacy_strings
+from diplomacy.utils.constants import SuggestionType
+import torch
+from torch.nn import DataParallel
+from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer
+
+from chiron_utils.bots.baseline_bot import BaselineBot, BotType
+from chiron_utils.utils import POWER_NAMES_DICT, get_other_powers, mapping, return_logger
+
+logger = return_logger(__name__)
+
+
+@dataclass
+class LlmAdvisor(BaselineBot):
+    """Bot that carries out random orders and sends random order proposals to other bots.
+
+    Because of the similarity between the advisor and player versions of this bot,
+    both of their behaviors are abstracted into this single abstract base class.
+    """
+
+    is_first_messaging_round = False
+    previous_newest_messages: Dict[str, Optional[List[Message]]] = field(
+        default_factory=lambda: dict.fromkeys(POWER_NAMES_DICT)
+    )
+    base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    bot_type = BotType.ADVISOR
+    default_suggestion_type = SuggestionType.COMMENTARY
+
+    def __post_init__(self) -> None:
+        """Initialize models."""
+        super().__post_init__()
+        self.tokenizer, self.model = self.load_model(self.base_model_name, None, self.device)
+
+    async def start_phase(self) -> None:
+        """Execute actions at the start of the phase."""
+        self.is_first_messaging_round = True
+
+    def read_suggestions_from_advisor(self) -> List[str]:
+        """Read suggestions from RandomProposerAdvisor.
+
+        Returns:
+            List of suggested orders.
+        """
+        received_messages = self.read_messages()
+        suggestion_messages = [
+            msg.message
+            for msg in received_messages
+            if msg.type == diplomacy_strings.SUGGESTED_MOVE_OPPONENTS
+        ]
+        logger.info(
+            "%s received opponent move suggestions: %s", self.display_name, suggestion_messages
+        )
+        return suggestion_messages
+
+    def get_relevant_messages(self, own: str, oppo: str) -> List[Message]:
+        """Return all messages sent between 'own' and 'oppo'."""
+        return [
+            msg
+            for msg in self.game.messages.values()
+            if (msg.sender == own and msg.recipient == oppo)
+            or (msg.sender == oppo and msg.recipient == own)
+        ]
+
+    def get_recent_message_history(
+        self, messages: List[Message], max_count: int = 8
+    ) -> List[Message]:
+        """Sort a list of messages by `time_sent` descending and return up to max_count most recent.
+
+        Also ensures the very last message is from the opponent (if available).
+        """
+        if not messages:
+            return []
+
+        # Sort descending by time_sent
+        sorted_msgs = sorted(messages, key=lambda x: x.time_sent, reverse=True)
+        recent_msgs = sorted_msgs[:max_count]
+        reversed_msgs = list(reversed(recent_msgs))
+
+        # Make sure the last message is from opponent, if possible
+        if reversed_msgs and reversed_msgs[-1].sender == self.power_name:
+            return []
+
+        return reversed_msgs
+
+    def create_system_prompt(self) -> str:
+        """Return the system prompt string (static text)."""
+        return """

system

+You are an expert assistant specializing in the Diplomacy board game. Your role is to assist a novice player by analyzing:
+1. The current board state.
+2. The message history exchanged between the novice player and the counterplayer.
+3. The predicted orders for the counterplayer.
+
+Your primary objective is to evaluate whether the counterplayer's predicted orders align with the message history and the board state.
+
+Key Evaluation Guidelines:
+
+1. Consider an order aligned if its purpose or intent is consistent with the counterplayer's stated goals or the tactical/strategic needs implied by the board state.
+2. Redundancy in orders (e.g., several supporting moves) can still be aligned if it serves to ensure the success of a critical move or maintains flexibility in uncertain situations.
+3. Misalignment occurs only if the order:
+    Contradicts the stated strategy or creates unnecessary risks.
+    Fails to contribute meaningfully to the position given the board state.
+4. Evaluate the broader context of each move, including defensive, offensive, or preparatory purposes, and weigh these factors in your analysis.
+For each predicted order, explain whether it aligns with the message history and board state. Use reasoning that incorporates tactical, strategic, and contextual factors while avoiding overly strict judgments.
+Limit your evaluation to alignment or misalignment, with short reasoning for each.
+

user

+
+Here is an example:
+**Predicted Orders of ITALY:**
+A TUN to ALB VIA,
+F ION C A TUN to ALB,
+F NAP to TYS,
+A VEN to TRI,
+
+**Board State:**
+AUSTRIA: [F TRI, A SER, A GAL, A BUD ]
+ENGLAND: [F LON, F NWY, A YOR, F EDI ]
+FRANCE: [A POR, A SPA, F BEL, F BRE, F MAR, A PAR ]
+GERMANY: [F DEN, A MUN, A HOL, A BER, F KIE ]
+ITALY: [A VEN, F ION, A TUN, F NAP ]
+RUSSIA: [A WAR, F SEV, F BOT, A RUM, A STP ]
+TURKEY: [A CON, A BUL, F BLA, F SMY ]
+
+**Message History:**
+Message from AUSTRIA: "France and Turkey are trying to get me to attack you, obviously I will not do that."
+Message from ITALY: "With the build in Marseilles, France's goals are quite transparent."
+Message from ITALY: "Turkey will likely bounce me in EAS. Would you be willing to support me into Greece (via convoy)?"
+Message from ITALY: "From there, I can support you into Bulgaria."
+Message from AUSTRIA: "I'm willing to debate it but I had a different view on this turn. I figured he would bounce you in EAS but what if you went for AEG instead? Unless you're worried about overextending yourself due to France.
+Because I trust you enough at this point to move Tri to Alb and then we will definitely be able to get Gre in the fall."
+Message from ITALY: "Which makes going to AEG/EAS a bit of an overstretch."
+Message from ITALY: "If we can get Germany on board I'd feel a lot more comfortable with committing east."
+Message from ITALY: "Sorry, some connection issues on my end so messages sent out of order."
+
+**Question:**
+I am playing as AUSTRIA. For each of ITALY's predicted orders, evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+
+**Answer:**
+A TUN to ALB VIA,
+Alignment: The move is aligned with the message history and board state. Italy previously mentioned the idea of moving east and positioning to take Greece.
+Convoying from TUN to ALB sets up a strong position for further collaboration with Austria in the Balkans, consistent with the stated plan to focus on Greece in the fall.
+
+F ION C A TUN to ALB,
+Alignment: The convoy is aligned with the board state and message history. The convoy facilitates Italy's movement into ALB, aligning with the communicated intent to focus efforts on securing Greece.
+This move ensures Italy's positioning in the region while maintaining the tactical flexibility discussed.
+
+F NAP to TYS,
+Alignment: This move is aligned as it supports Italy's eastward intentions.
+Moving the fleet to the Tyrrhenian Sea strengthens Italy's naval presence in the Mediterranean and provides future flexibility to either defend against France or support further eastern operations
+, consistent with Italy's concerns about overextending.
+
+A VEN to TRI,
+Misalignment: This move is misaligned with the message history and board state. Austria explicitly trusts Italy and has not shown hostility, and Italy's previous messages suggest no immediate plans to antagonize Austria.
+Moving to TRI is aggressive and undermines the collaborative stance Italy communicated. It could create unnecessary conflict, contradicting Italy's expressed preference for focusing eastward and securing Greece.
+
+
+Now let's see the question:"""
+
+    def format_boardstates(self, boardstates: Dict[str, List[str]]) -> str:
+        """Return formatted board states."""
+        lines = []
+        for power, units in boardstates.items():
+            units_str = ", ".join(units)
+            line = f"{power}: [{units_str} ]"
+            lines.append(line)
+        return "\n".join(lines)
+
+    def map_words_in_sentence(self, sentence: str, data: Dict[str, List[str]]) -> str:
+        """Return abbreviation conversed words in sentences."""
+        words = sentence.split()
+        mapped_words = []
+
+        for word in words:
+            clean_word = word.strip(",.!?")
+            word_lower = clean_word.lower()
+
+            if word_lower in data:
+                mapped_value = data[word_lower][0]
+                mapped_word = word.replace(clean_word, mapped_value)
+                mapped_words.append(mapped_word)
+            else:
+                mapped_words.append(word)
+
+        return " ".join(mapped_words)
+
+    def format_prompt_phase1(self, own: str, oppo: str, suggest_orders: List[str]) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        # Convert power names
+        if own in POWER_NAMES_DICT:
+            own = POWER_NAMES_DICT[own]
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        # Grab and sort board states, filter relevant messages.
+        system_prompt = self.create_system_prompt()
+        board_states = self.game.get_state()["units"]
+        sorted_board_states = {key: sorted(value) for key, value in board_states.items()}
+        formated_board_states = self.format_boardstates(sorted_board_states)
+        filtered_messages = self.get_relevant_messages(own, oppo)
+        if not filtered_messages:
+            return None
+
+        recent_msgs = self.get_recent_message_history(filtered_messages, max_count=10)
+        if not recent_msgs:
+            return None
+
+        # Build a text representation of the message history
+        message_history = ""
+        for msg in recent_msgs:
+            message_history += f"Message from {msg.sender}:'{msg.message}' "
+        parsed_data = json.loads(suggest_orders[0])
+        predicted_orders = parsed_data["payload"]["predicted_orders"][oppo]
+
+        orders_string = ", ".join(predicted_orders)
+
+        prompt = (
+            f"{system_prompt}"
+            f"""
+**Predicted Orders of {oppo}:**
+{orders_string.replace('-','to')}
+
+**Board State:**
+"""
+            f"{formated_board_states}\n\n"
+            f"""**Message History:**  \n{message_history}\n\n"""
+            f"""**Question:**  \nI am playing as {own}. For each of {oppo}'s predicted orders,
+            evaluate whether it aligns with the message history and the current board state. Explain the orders first and provide short reasoning and analysis for each predicted move.
+\n"""
+            f"""**Answer:**
+

assistant
"""
+        )
+        return prompt
+
+    def format_prompt_phase2(self, oppo: str, evaluation: str) -> Optional[str]:
+        """Create prompt used as input to the LLM.
+
+        Returns:
+            A string prompt, or None if no messages exist between 'own' and 'oppo'.
+        """
+        if oppo in POWER_NAMES_DICT:
+            oppo = POWER_NAMES_DICT[oppo]
+
+        prompt = f"""

system

+            You are an expert assistant specializing in the Diplomacy board game. Based on the detailed evaluation of following predicted orders, respond in the following short and structured format:
+
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

user

+            **Evaluation**:
+            {evaluation}
+
+            **Question**:
+            from Evaluation part, extract necessary information and fill in the following as the output:
+            I think {oppo} is being deceptive, because I think they are going to make these moves: <predicted misaligned moves>. These moves are contradictory to {oppo}'s previous messages to you.
+            

assistant

+
+            """
+        return prompt
+
+    def load_model(
+        self,
+        base_model_name: str,
+        tokenizer_path: Optional[str] = None,
+        device: str = "cpu",
+    ) -> Tuple[PreTrainedTokenizer, Union[PreTrainedModel, DataParallel]]:
+        """Loads and returns a tokenizer and model on the requested device."""
+        if tokenizer_path is None:
+            tokenizer_path = base_model_name
+
+        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
+        model = AutoModelForCausalLM.from_pretrained(base_model_name)
+        if torch.cuda.device_count() > 1:
+            model = DataParallel(model)
+        model.to(device)
+
+        return tokenizer, model
+
+    def generate_text(
+        self,
+        prompt: str,
+        tokenizer: PreTrainedTokenizer,
+        model: Union[PreTrainedModel, DataParallel],
+        device: str = "cpu",
+        max_new_tokens: int = 512,
+    ) -> str:
+        """Performs text generation using the tokenizer/model loaded above."""
+        if tokenizer.pad_token_id is None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+
+        inputs = tokenizer(
+            prompt,
+            return_tensors="pt",
+            padding=True,
+            truncation=True,
+            max_length=2000,
+            return_attention_mask=True,
+        )
+
+        input_ids = inputs["input_ids"].to(device)
+        attention_mask = inputs["attention_mask"].to(device)
+
+        model.eval()
+        with torch.no_grad():
+            if isinstance(model, DataParallel):
+                output_ids = model.module.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+            else:
+                output_ids = model.generate(
+                    input_ids,
+                    attention_mask=attention_mask,
+                    max_new_tokens=max_new_tokens,
+                    pad_token_id=tokenizer.pad_token_id,
+                )
+
+        return tokenizer.decode(output_ids[0], skip_special_tokens=True)  # type: ignore[no-any-return]
+
+    def generate_and_parse_response(self, prompt: str) -> str:
+        """Generate text from the model given 'prompt'.
+
+        Returns:
+            message or None if something fails.
+        """
+        generated_text = self.generate_text(prompt, self.tokenizer, self.model, self.device, 1024)
+        assistant_output = generated_text.split("assistant")[2]
+        return assistant_output
+
+    async def do_messaging_round(self, orders: Sequence[str]) -> List[str]:
+        """Carry out one round of messaging, along with related tasks.
+
+        Returns:
+            List of orders to carry out.
+        """
+        await asyncio.sleep(random.uniform(5, 10))
+
+        filtered_orders = self.read_suggestions_from_advisor()
+        if not filtered_orders:
+            return []
+
+        for other_power in get_other_powers([self.power_name], self.game):
+            all_relevant = self.get_relevant_messages(self.power_name, other_power)
+            prev_msgs = self.previous_newest_messages.get(other_power)
+            if prev_msgs is not None:
+                # Compare to find new messages
+                new_messages = [m for m in all_relevant if m not in prev_msgs]
+                if not new_messages:
+                    # No new messages, skip
+                    continue
+
+                prompt = self.format_prompt_phase1(self.power_name, other_power, filtered_orders)
+                if prompt is None:
+                    continue
+                processed_prompt = self.map_words_in_sentence(prompt, mapping)
+                logger.info("Phase1 prompt for %s: %s", other_power, processed_prompt)
+                output_phase1 = self.generate_and_parse_response(processed_prompt)
+                logger.info("Phase1 output for %s: %s", other_power, output_phase1)
+
+                alignment_count = output_phase1.count("Alignment")
+                misalignment_count = output_phase1.count("Misalignment")
+                if misalignment_count >= alignment_count:
+                    prompt2 = self.format_prompt_phase2(other_power, output_phase1)
+                    if prompt2 is None:
+                        continue
+                    output_phase2 = self.generate_and_parse_response(prompt2)
+                    logger.info("Phase2 output for %s: %s", other_power, output_phase2)
+                    if output_phase2 and output_phase2.lstrip().startswith("I think"):
+                        try:
+                            await self.suggest_commentary(other_power, f"{output_phase2.lstrip()}")
+                        except diplomacy.utils.exceptions.GamePhaseException as exc:
+                            logger.exception("Ignoring %s", exc.__class__.__name__)
path: src/chiron_utils/bots/llm_advisor_bot.py
user: login: aphedges
body: Ugh. I guess we'll need to keep this in for now until the underlying problem can be fixed...
created_at: 2025-03-24T20:00:57Z
author_association: CONTRIBUTOR

PR comments:
user: login: YanzeJeremy
created_at: 2025-02-26T08:20:47Z
body: I have done most of your feedback:

- delete `LlmPlayer` and merge `LlmAdvisor` into `LlmAdvisorBot`.
- change `suggestion_type` to `default_suggestion_type `
- remove commented-out code.
- Replace `print()` statements with calls to `logger`.
- hardcoded string replaced
- Add instructions on how to run your bot to the README

only one error left when running `make check` :
`tests/bot_tests/bot_test.py::TestBots::test_send_message FAILED`
Since `LlmAdvisor` doesn't send messages, only suggest commentary, I think this is fine? Do I need to fix it somehow?

author_association: CONTRIBUTOR
user: login: YanzeJeremy
created_at: 2025-03-08T20:56:40Z
body: Sure, thank you!

Will add comments today.

I am doing improvements on LLM, and will provide one finetuned version later. But as of now, this is only an off-the-shelf model.
author_association: CONTRIBUTOR
user: login: aphedges
created_at: 2025-03-24T21:30:11Z
body: @YanzeJeremy, I've improved the README a little more, fixed the failing checks, and reverted a change to a file that shouldn't have been changed. If these changes look good to you, please let me know. Then I'll clean up the commit history, run a final test, and merge this PR!
author_association: CONTRIBUTOR
user: login: YanzeJeremy
created_at: 2025-03-25T17:25:58Z
body: Thanks Alex, all above changes look good to me!
author_association: CONTRIBUTOR
user: login: aphedges
created_at: 2025-03-25T20:14:09Z
body: I've cleaned up the commit history, rebased against `main`, and tested the branch (fixing a bug I found). If everything looks good, let me know, and I'll squash the new commit, then approve and merge this PR!
author_association: CONTRIBUTOR
user: login: YanzeJeremy
created_at: 2025-03-26T22:05:43Z
body: Everything looks good to me! 
author_association: CONTRIBUTOR