Repository: jonbinney/deep_rabbit_hole
Pull request #170: Integrate elo ratings in wandb
Author: login: alejandromarcu
Created: 2025-04-24T00:08:39Z | Merged: 2025-04-28T22:43:56Z

When training an agent, send the elo rating and the win percentage to wandb, so we can have a single number (or two) of how it is performing.
The whole elo table is sent to wandb every interval, but the tables are clunky and I still haven't figured out how they work.
It also sends what I call the "relative elo" score, which is the score the agent got minus the score of the best opponent.  The rationale behind this is the following: we want to have information on how good the model is performing with just 1 number, but if we just see our agent's elo rating, it's hard to draw conclusions.  Is an elo rating of 1800 good? well, it depends, if a few other agents are above 1800 it's not that great, but if everyone is below 1800 it looks much better.
So, if the relative elo is positive, it means that the agent is in the first position.  A relative elo score of 200 would indicate for example that our agent has about 75% chances of beating the second one, and a score of 400 would indicate 91% chances.
When the score is negative, we know the agent is not the best.

I'm also logging the win percentage, that has the same shape than the relative elo, so we may end up just using the win percentage.  The main difference is that the elo takes into account how good the adversaries are, e.g. you get your elo increased more if you beat an adversary with higher elo, so winning against greedy is more significant than winning against random.

The win_perc has the advantage that if it reaches 100%, you know that you topped the metric, so you won't be trying to improve it without devising a harder metric first.  With elo score, you may think you still have room for improvement because there's no ceiling.

I'm currently running this training (copied from the command line from `dexp_B5W3_mv2:v15`, but added epsilon_min and reduced the number of episodes):

```
 /Users/amarcu/code/deep_rabbit_hole/.venv/bin/python /Users/amarcu/code/deep_rabbit_hole/deep_quoridor/src/train.py  -N 5 -W 3 -e 500 -i 43 -p dexp:rotate=true,split=true,turn=false,epsilon=1,epsilon_decay=0.9985,epsilon_min=0.1,final_reward_multiplier=20,training_mode=true,use_opponents_actions=true,update_target_every=100,gamma=0.95,target_as_source_for_opponent=true,use_negative_qvalue_function=true,batch_size=128,mask_targetq=true,nick=dexp_train greedy:p_random=0.3 greedy:p_random=0.1 simple -rp cucu -s
```

And this is what I got so far ([link to the run](https://wandb.ai/the-lazy-learning-lair/deep_quoridor/runs/cucu-20250423-164800)):
<img width="1039" alt="image" src="https://github.com/user-attachments/assets/a71c94f3-5e03-4a1c-a2dd-346a497428d3" />



Diff:
diff --git a/deep_quoridor/src/agents/core/agent.py b/deep_quoridor/src/agents/core/agent.py
index 1392340a..aef6b787 100644
--- a/deep_quoridor/src/agents/core/agent.py
+++ b/deep_quoridor/src/agents/core/agent.py
@@ -121,7 +121,7 @@ def create(friendly_name: str, **kwargs) -> Agent:
         return AgentRegistry.agents[friendly_name](**kwargs)
 
     @staticmethod
-    def create_from_encoded_name(encoded_name: str, **kwargs) -> Agent:
+    def create_from_encoded_name(encoded_name: str, remove_training_args=False, **kwargs) -> Agent:
         parts = encoded_name.split(":")
         agent_type = parts[0]
         if len(parts) == 1:
@@ -132,7 +132,10 @@ def create_from_encoded_name(encoded_name: str, **kwargs) -> Agent:
         if subargs_class is None:
             raise ValueError(f"The agent {agent_type} doesn't support subarguments, but '{parts[1]}' was passed")
 
-        subargs = parse_subargs(parts[1], subargs_class)
+        if remove_training_args:
+            subargs = parse_subargs(parts[1], subargs_class, ignore_fields=subargs_class.training_only_params())
+        else:
+            subargs = parse_subargs(parts[1], subargs_class)
 
         return AgentRegistry.agents[agent_type](params=subargs, **kwargs)
 
diff --git a/deep_quoridor/src/agents/core/trainable_agent.py b/deep_quoridor/src/agents/core/trainable_agent.py
index d4dfaba5..aface25d 100644
--- a/deep_quoridor/src/agents/core/trainable_agent.py
+++ b/deep_quoridor/src/agents/core/trainable_agent.py
@@ -58,6 +58,7 @@ class TrainableAgentParams(SubargsBase):
     # This is used to log the opponent's possible actions based on agent's qvalues
     inspect_opponent_possible_actions: bool = False
 
+    @classmethod
     def training_only_params(cls) -> set[str]:
         """Returns a set of parameter names that are only used during training."""
         # TODO: we should ideally have two set of params: one for training and one for inference
@@ -75,8 +76,6 @@ def training_only_params(cls) -> set[str]:
             "use_negative_qvalue_function",
             "wandb_dir",
             "model_dir",
-            "wandb_alias",
-            "model_filename",
             "mask_targetq",
             "softmax_exploration",
             "inspect_opponent_possible_actions",
@@ -144,6 +143,8 @@ def start_game(self, game, player_id):
         self.player_id = player_id
 
     def end_game(self, game):
+        if not self.training_mode:
+            return
         """Store episode results and reset tracking"""
         self.episodes_rewards.append(self.current_episode_reward)
         self.games_count += 1
diff --git a/deep_quoridor/src/agents/dexp.py b/deep_quoridor/src/agents/dexp.py
index c91db5eb..76bd0a2f 100644
--- a/deep_quoridor/src/agents/dexp.py
+++ b/deep_quoridor/src/agents/dexp.py
@@ -69,6 +69,7 @@ class DExpAgentParams(TrainableAgentParams):
     def __str__(self):
         return f"{int(self.rotate)}{int(self.turn)}{int(self.split)}{'1' if self.target_as_source_for_opponent else ''}"
 
+    @classmethod
     def training_only_params(cls) -> set[str]:
         """
         Returns a set of parameters that are used only during training.
diff --git a/deep_quoridor/src/arena.py b/deep_quoridor/src/arena.py
index 04f8e2f8..4e267fb7 100644
--- a/deep_quoridor/src/arena.py
+++ b/deep_quoridor/src/arena.py
@@ -97,7 +97,7 @@ def _play_game(self, agent1: Agent, agent2: Agent, game_id: str) -> GameResult:
         return result
 
     # Replace the existing _play_games method
-    def _play_games(self, players: list[str | Agent], times: int, mode: PlayMode):
+    def _play_games(self, players: list[str | Agent], times: int, mode: PlayMode) -> list[GameResult]:
         agents = []
         for p in players:
             if isinstance(p, Agent):
@@ -144,6 +144,7 @@ def _play_games(self, players: list[str | Agent], times: int, mode: PlayMode):
                     match_id += 1
 
         self.plugins.end_arena(self.game, results)
+        return results
 
     def _replay_games(self, arena_data: dict, game_ids_to_replay: list[str]):
         """Replays a series of games from previously recorded arena data.
diff --git a/deep_quoridor/src/metrics.py b/deep_quoridor/src/metrics.py
new file mode 100644
index 00000000..46ed7b74
--- /dev/null
+++ b/deep_quoridor/src/metrics.py
@@ -0,0 +1,88 @@
+from agents import AbstractTrainableAgent, Agent
+from agents.core.agent import AgentRegistry
+from arena import Arena, PlayMode
+from arena_utils import GameResult
+from utils.misc import compute_elo
+
+
+class Metrics:
+    """
+    Computes metrics for an agent by making it play against other agents.
+    """
+
+    def __init__(self, board_size: int, max_walls: int):
+        self.board_size = board_size
+        self.max_walls = max_walls
+        self.stored_elos = {}
+
+    def _win_perc(self, results: list[GameResult], agent_name: str):
+        played = 0
+        won = 0
+        for result in results:
+            if result.player1 == agent_name or result.player2 == agent_name:
+                played += 1
+
+            if result.winner == agent_name:
+                won += 1
+
+        return 100.0 * won / played if played > 1 else 0.0
+
+    def _compute_relative_elo(self, elo_table: dict[str, float], agent_name: str) -> int:
+        agent_rating = elo_table[agent_name]
+        best_opponent = 0
+        for name, elo in elo_table.items():
+            if name != agent_name:
+                best_opponent = max(best_opponent, elo)
+
+        return int(agent_rating - best_opponent)
+
+    def compute(self, agent_encoded_name: str) -> tuple[int, dict[str, float], int, float]:
+        """
+        Evaluates the performance of a given agent by running it against a set of predefined opponents and computing its Elo rating and win percentage.
+
+        Args:
+            agent_encoded_name (str): The encoded name of the agent to evaluate.  If there are training params, they will be ignored
+
+        Returns:
+            tuple[int, dict[str, float], int, float]: A tuple containing:
+                - VERSION (int): The version number of the scoring method.
+                - elo_table (dict[str, float]): A dictionary mapping agent names to their computed Elo ratings.
+                - relative_elo (int): The evaluated agent's Elo rating minus the elo for the best opponent.
+                - win_perc (float): The win percentage of the evaluated agent against the opponents.
+
+        Notes:
+            - The method disables training mode for trainable agents during evaluation and restores it afterward.
+            - Opponent Elo ratings are cached to avoid redundant computations.
+            - The evaluation is performed using a fixed set of baseline agents and a specified number of games.
+        """
+        # Bump if there's any change in the scoring
+        VERSION = 1
+        times = 50
+
+        players: list[str | Agent] = [
+            "greedy",
+            "greedy:p_random=0.1,nick=greedy-01",
+            "greedy:p_random=0.3,nick=greedy-03",
+            "greedy:p_random=0.5,nick=greedy-05",
+        ]
+
+        agent = AgentRegistry.create_from_encoded_name(
+            agent_encoded_name, board_size=self.board_size, max_walls=self.max_walls, remove_training_args=True
+        )
+
+        arena = Arena(self.board_size, self.max_walls)
+
+        # We store the elos of the opponents playing against each other so we don't have to play those matches
+        # every time
+        if not self.stored_elos:
+            results = arena._play_games(players, times, PlayMode.ALL_VS_ALL)
+            self.stored_elos = compute_elo(results)
+
+        results = arena._play_games([agent] + players, times, PlayMode.FIRST_VS_RANDOM)
+
+        elo_table = compute_elo(results, initial_elos=self.stored_elos.copy())
+        relative_elo = self._compute_relative_elo(elo_table, agent.name())
+
+        win_perc = self._win_perc(results, agent.name())
+
+        return VERSION, elo_table, relative_elo, win_perc
diff --git a/deep_quoridor/src/plugins/save_model.py b/deep_quoridor/src/plugins/save_model.py
index 84ee67bc..237e992c 100644
--- a/deep_quoridor/src/plugins/save_model.py
+++ b/deep_quoridor/src/plugins/save_model.py
@@ -1,3 +1,5 @@
+from typing import Callable, Optional
+
 from agents.core import AbstractTrainableAgent
 from arena_utils import ArenaPlugin
 from utils import resolve_path
@@ -11,6 +13,7 @@ def __init__(
         max_walls: int,
         save_final: bool = True,
         run_id: str = "",
+        after_save: Optional[Callable[[str], None]] = None,
     ):
         self.update_every = update_every
         self.episode_count = 0
@@ -18,6 +21,7 @@ def __init__(
         self.max_walls = max_walls
         self.save_final = save_final
         self.run_id = run_id
+        self.after_save = after_save
 
     def start_game(self, game, agent1, agent2):
         self.agents = [agent1, agent2]
@@ -39,3 +43,6 @@ def _save_models(self, suffix: str):
             save_file = resolve_path(agent.params.model_dir, agent.resolve_filename(suffix))
             agent.save_model(save_file)
             print(f"{agent_name} Model saved to {save_file}")
+
+            if self.after_save:
+                self.after_save(str(save_file))
diff --git a/deep_quoridor/src/plugins/wandb_train.py b/deep_quoridor/src/plugins/wandb_train.py
index 95cf967d..a595bfd7 100644
--- a/deep_quoridor/src/plugins/wandb_train.py
+++ b/deep_quoridor/src/plugins/wandb_train.py
@@ -3,20 +3,24 @@
 
 from agents.core.trainable_agent import AbstractTrainableAgent
 from arena_utils import ArenaPlugin
+from metrics import Metrics
 from utils import resolve_path
 
 import wandb
 
 
 class WandbTrainPlugin(ArenaPlugin):
-    def __init__(self, update_every: int, total_episodes: int, run_id: str = ""):
+    def __init__(self, update_every: int, total_episodes: int, agent_encoded_name: str, run_id: str = ""):
         self.update_every = update_every
         self.total_episodes = total_episodes
         self.episode_count = 0
         self.agent = None
         self.run_id = run_id
+        self.agent_encoded_name = agent_encoded_name
 
     def _initialize(self, game):
+        assert self.agent
+
         config = {
             "board_size": game.board_size,
             "max_walls": game.max_walls,
@@ -24,6 +28,7 @@ def _initialize(self, game):
             "player_args": self.agent.params,
         }
         config.update(self.agent.model_hyperparameters())
+        self.metrics = Metrics(game.board_size, game.max_walls)
 
         self.run = wandb.init(
             project="deep_quoridor",
@@ -48,6 +53,7 @@ def start_game(self, game, agent1, agent2):
         self._initialize(game)
 
     def end_game(self, game, result):
+        assert self.agent
         if self.episode_count % self.update_every == 0 or self.episode_count == (self.total_episodes - 1):
             avg_loss, avg_reward = self.agent.compute_loss_and_reward(self.update_every)
 
@@ -55,9 +61,12 @@ def end_game(self, game, result):
                 {"loss": avg_loss, "reward": avg_reward, "epsilon": self.agent.epsilon},
                 step=self.episode_count,
             )
+
         self.episode_count += 1
 
     def end_arena(self, game, results):
+        assert self.agent
+
         # Save the model in the wandb directory with the suffix "temp".  The file will be renamed
         # once we upload it to wandb and have the digest.
         save_file = resolve_path(self.agent.params.wandb_dir, self.agent.resolve_filename("temp"))
@@ -76,5 +85,22 @@ def end_arena(self, game, results):
         # the same path, just a different file name
         os.rename(save_file, wand_file)
         print(f"Model saved to {wand_file}")
-
+        self.compute_tournament_metrics(str(wand_file))
         wandb.finish()
+
+    def compute_tournament_metrics(self, model_filename: str):
+        _, elo_table, relative_elo, win_perc = self.metrics.compute(
+            self.agent_encoded_name + f",model_filename={model_filename}"
+        )
+        print(f"Tournament Metrics - Relative elo: {relative_elo}, win percentage: {win_perc}")
+        wandb_elo_table = wandb.Table(
+            columns=["Player", "elo"], data=[[player, elo] for player, elo in elo_table.items()]
+        )
+        self.run.log(
+            {
+                "elo": wandb_elo_table,
+                "relative_elo": relative_elo,
+                "win_perc": win_perc,
+            },
+            step=self.episode_count,
+        )
diff --git a/deep_quoridor/src/train.py b/deep_quoridor/src/train.py
index 14bcd094..f070583c 100644
--- a/deep_quoridor/src/train.py
+++ b/deep_quoridor/src/train.py
@@ -14,17 +14,23 @@ def train_dqn(
     episodes: int,
     board_size: int,
     max_walls: int,
-    save_frequency: int = 100,
+    save_frequency: int,
     step_rewards: bool = True,
     use_wandb: bool = True,
-    players: list | None = None,
+    players: list = [],
     renderers: list[ArenaPlugin] = [],
     run_id: str = "",
 ):
     plugins = []
+    total_episodes = episodes * (len(players) - 1)
 
+    after_save_method = None
     if use_wandb:
-        plugins.append(WandbTrainPlugin(update_every=10, total_episodes=episodes, run_id=run_id))
+        wandb_plugin = WandbTrainPlugin(
+            update_every=10, total_episodes=total_episodes, run_id=run_id, agent_encoded_name=players[0]
+        )
+        plugins.append(wandb_plugin)
+        after_save_method = wandb_plugin.compute_tournament_metrics
 
     plugins.append(
         SaveModelEveryNEpisodesPlugin(
@@ -33,13 +39,11 @@ def train_dqn(
             max_walls=max_walls,
             save_final=not use_wandb,
             run_id=run_id,
+            after_save=after_save_method,
         )
     )
 
-    print_plugin = TrainingStatusRenderer(
-        update_every=1,
-        total_episodes=episodes * (len(players) - 1),
-    )
+    print_plugin = TrainingStatusRenderer(update_every=1, total_episodes=total_episodes)
     arena = Arena(
         board_size=board_size,
         max_walls=max_walls,
diff --git a/deep_quoridor/src/utils/misc.py b/deep_quoridor/src/utils/misc.py
index 3490aec2..5633f0b9 100644
--- a/deep_quoridor/src/utils/misc.py
+++ b/deep_quoridor/src/utils/misc.py
@@ -118,8 +118,8 @@ def yargs(parser: argparse.ArgumentParser, default_path: str):
     return args_dict
 
 
-def compute_elo(results: list["GameResult"], k=32, initial_rating=1500) -> dict[str, float]:
-    ratings = {}
+def compute_elo(results: list["GameResult"], k=32, initial_rating=1500, initial_elos={}) -> dict[str, float]:
+    ratings = initial_elos
     random.shuffle(results)
 
     for result in results:
diff --git a/deep_quoridor/src/utils/subargs.py b/deep_quoridor/src/utils/subargs.py
index 49eba2aa..c381457e 100644
--- a/deep_quoridor/src/utils/subargs.py
+++ b/deep_quoridor/src/utils/subargs.py
@@ -23,7 +23,7 @@ class ParseSubargsError(ValueError):
     pass
 
 
-def parse_subargs(s: str, cls: Type[SubargsBase], separator=",", assign="="):
+def parse_subargs(s: str, cls: Type[SubargsBase], separator=",", assign="=", ignore_fields=set()):
     """Parses the string s and uses it to instantiate a class cls and return it."""
     if s == "":
         return cls()
@@ -35,6 +35,9 @@ def parse_subargs(s: str, cls: Type[SubargsBase], separator=",", assign="="):
             raise ParseSubargsError(f"The subargument '{part}' needs to have an assignment using '{assign}'")
 
         k, v = part.split(assign)
+        if k in ignore_fields:
+            continue
+
         if k not in class_fields:
             raise ParseSubargsError(
                 f"Field '{k}' not in class {cls.__name__}.  Available fields are: {', '.join(class_fields.keys())}"


Reviews:
user: login: muralx
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T12:54:13Z
user: login: adamantivm
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T13:40:22Z
user: login: alejandromarcu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-24T19:59:31Z
user: login: muralx
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-25T11:28:30Z
user: login: muralx
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-26T17:50:25Z
user: login: muralx
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-26T17:56:24Z
user: login: alejandromarcu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-26T21:44:45Z
user: login: muralx
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-27T19:26:15Z
user: login: muralx
body: Thanks Cucu!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-28T21:23:09Z

Review comments:
diff_hunk: @@ -24,7 +24,7 @@ def train_dqn(
     plugins = []
 
     if use_wandb:
-        plugins.append(WandbTrainPlugin(update_every=10, total_episodes=episodes, run_id=run_id))
+        plugins.append(WandbTrainPlugin(update_every=100, total_episodes=episodes, run_id=run_id))
path: deep_quoridor/src/train.py
user: login: muralx
body: I think we should upload the elo and results after running a tournament with the training complete, similar to the eval that Julian's training process is doing. Uploading the results from training (in relation to elo and % of winning is not really meaningful)
created_at: 2025-04-24T12:54:08Z
author_association: COLLABORATOR
diff_hunk: @@ -24,7 +24,7 @@ def train_dqn(
     plugins = []
 
     if use_wandb:
-        plugins.append(WandbTrainPlugin(update_every=10, total_episodes=episodes, run_id=run_id))
+        plugins.append(WandbTrainPlugin(update_every=100, total_episodes=episodes, run_id=run_id))
path: deep_quoridor/src/train.py
user: login: adamantivm
body: If I understand correctly, this PR is doing the same thing: running "play" against all players, but doing it many times instead of only one time at the end.

I think that's going to be pretty onerous. I think at the minimum we need to have the option of: 
 - Not running this
 - Running it just once at the end (default?)
 - Running it once every `update_every` (current impl) as an option
created_at: 2025-04-24T13:40:22Z
author_association: COLLABORATOR
diff_hunk: @@ -24,7 +24,7 @@ def train_dqn(
     plugins = []
 
     if use_wandb:
-        plugins.append(WandbTrainPlugin(update_every=10, total_episodes=episodes, run_id=run_id))
+        plugins.append(WandbTrainPlugin(update_every=100, total_episodes=episodes, run_id=run_id))
path: deep_quoridor/src/train.py
user: login: muralx
body: True, I glanced at the Wandb plugin and I thought it was just computing the metrics based on training information. I agree with your comment.

I think metrics should be run at the end of the training, and also we should allow to run metrics independent of the moment we are training. At this point, the ELO is associated to the artifact, for which we can upload information at any time. 

created_at: 2025-04-24T18:35:27Z
author_association: COLLABORATOR
diff_hunk: @@ -24,7 +24,7 @@ def train_dqn(
     plugins = []
 
     if use_wandb:
-        plugins.append(WandbTrainPlugin(update_every=10, total_episodes=episodes, run_id=run_id))
+        plugins.append(WandbTrainPlugin(update_every=100, total_episodes=episodes, run_id=run_id))
path: deep_quoridor/src/train.py
user: login: alejandromarcu
body: Yeah, this runs the "tournament" every N episodes and reports the results.  I think it's pretty useful because you can see how the training is progressing.  But I'll make a separate configuration for the frequency, so you can choose to run it it less often or not run it.   Doing it every 100 episodes in my case didn't significantly improve the time, since it was taking a bit over 1 minute to do 100 episodes and then 4-5 seconds to do the tournament. 
created_at: 2025-04-24T19:59:31Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,95 @@
+from agents import AbstractTrainableAgent, Agent
+from agents.core.agent import AgentRegistry
+from arena import Arena, PlayMode
+from arena_utils import GameResult
+from utils.misc import compute_elo
+
+
+class Metrics:
+    """
+    Computes metrics for an agent by making it play against other agents.
+    """
+
+    def __init__(self, board_size: int, max_walls: int):
+        self.board_size = board_size
+        self.max_walls = max_walls
+        self.stored_elos = {}
+
+    def _win_perc(self, results: list[GameResult], agent_name: str):
+        played = 0
+        won = 0
+        for result in results:
+            if result.player1 == agent_name or result.player2 == agent_name:
+                played += 1
+
+            if result.winner == agent_name:
+                won += 1
+
+        return 100.0 * won / played if played > 1 else 0.0
+
+    def _compute_relative_elo(self, elo_table: dict[str, float], agent_name: str) -> int:
+        agent_rating = elo_table[agent_name]
+        best_opponent = 0
+        for name, elo in elo_table.items():
+            if name != agent_name:
+                best_opponent = max(best_opponent, elo)
+
+        return int(agent_rating - best_opponent)
+
+    def compute(self, agent: str | Agent) -> tuple[int, dict[str, float], int, float]:
+        """
+        Evaluates the performance of a given agent by running it against a set of predefined opponents and computing its Elo rating and win percentage.
+
+        Args:
+            agent (str | Agent): The agent to evaluate, specified either as a string identifier or an Agent instance.
+
+        Returns:
+            tuple[int, dict[str, float], int, float]: A tuple containing:
+                - VERSION (int): The version number of the scoring method.
+                - elo_table (dict[str, float]): A dictionary mapping agent names to their computed Elo ratings.
+                - relative_elo (int): The evaluated agent's Elo rating minus the elo for the best opponent.
+                - win_perc (float): The win percentage of the evaluated agent against the opponents.
+
+        Notes:
+            - The method disables training mode for trainable agents during evaluation and restores it afterward.
+            - Opponent Elo ratings are cached to avoid redundant computations.
+            - The evaluation is performed using a fixed set of baseline agents and a specified number of games.
+        """
+        # Bump if there's any change in the scoring
+        VERSION = 1
+        times = 50
+
+        players: list[str | Agent] = [
+            "greedy",
+            "greedy:p_random=0.1,nick=greedy-01",
+            "greedy:p_random=0.3,nick=greedy-03",
+            "greedy:p_random=0.5,nick=greedy-05",
+        ]
+
+        if isinstance(agent, str):
+            agent = AgentRegistry.create_from_encoded_name(agent, board_size=self.board_size, max_walls=self.max_walls)
+
+        # Make sure the agent is not in training mode to do the evaluation
+        if isinstance(agent, AbstractTrainableAgent):
+            prev_training_mode = agent.training_mode
path: deep_quoridor/src/metrics.py
user: login: muralx
body: Mhh, this might not be enough. Some methods might not be checking this. In particular I think end_game is updating target network to match online network (it should not), but it will impact training (slightly).  (Just as an example). We should review methods are properly using this.
created_at: 2025-04-26T17:49:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,95 @@
+from agents import AbstractTrainableAgent, Agent
+from agents.core.agent import AgentRegistry
+from arena import Arena, PlayMode
+from arena_utils import GameResult
+from utils.misc import compute_elo
+
+
+class Metrics:
+    """
+    Computes metrics for an agent by making it play against other agents.
+    """
+
+    def __init__(self, board_size: int, max_walls: int):
+        self.board_size = board_size
+        self.max_walls = max_walls
+        self.stored_elos = {}
+
+    def _win_perc(self, results: list[GameResult], agent_name: str):
+        played = 0
+        won = 0
+        for result in results:
+            if result.player1 == agent_name or result.player2 == agent_name:
+                played += 1
+
+            if result.winner == agent_name:
+                won += 1
+
+        return 100.0 * won / played if played > 1 else 0.0
+
+    def _compute_relative_elo(self, elo_table: dict[str, float], agent_name: str) -> int:
+        agent_rating = elo_table[agent_name]
+        best_opponent = 0
+        for name, elo in elo_table.items():
+            if name != agent_name:
+                best_opponent = max(best_opponent, elo)
+
+        return int(agent_rating - best_opponent)
+
+    def compute(self, agent: str | Agent) -> tuple[int, dict[str, float], int, float]:
+        """
+        Evaluates the performance of a given agent by running it against a set of predefined opponents and computing its Elo rating and win percentage.
+
+        Args:
+            agent (str | Agent): The agent to evaluate, specified either as a string identifier or an Agent instance.
+
+        Returns:
+            tuple[int, dict[str, float], int, float]: A tuple containing:
+                - VERSION (int): The version number of the scoring method.
+                - elo_table (dict[str, float]): A dictionary mapping agent names to their computed Elo ratings.
+                - relative_elo (int): The evaluated agent's Elo rating minus the elo for the best opponent.
+                - win_perc (float): The win percentage of the evaluated agent against the opponents.
+
+        Notes:
+            - The method disables training mode for trainable agents during evaluation and restores it afterward.
+            - Opponent Elo ratings are cached to avoid redundant computations.
+            - The evaluation is performed using a fixed set of baseline agents and a specified number of games.
+        """
+        # Bump if there's any change in the scoring
+        VERSION = 1
+        times = 50
+
+        players: list[str | Agent] = [
+            "greedy",
+            "greedy:p_random=0.1,nick=greedy-01",
+            "greedy:p_random=0.3,nick=greedy-03",
+            "greedy:p_random=0.5,nick=greedy-05",
+        ]
+
+        if isinstance(agent, str):
+            agent = AgentRegistry.create_from_encoded_name(agent, board_size=self.board_size, max_walls=self.max_walls)
+
+        # Make sure the agent is not in training mode to do the evaluation
+        if isinstance(agent, AbstractTrainableAgent):
+            prev_training_mode = agent.training_mode
path: deep_quoridor/src/metrics.py
user: login: muralx
body: Also, epsilon should ideally be zero during play when not training (no exploration)
created_at: 2025-04-26T17:56:24Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,95 @@
+from agents import AbstractTrainableAgent, Agent
+from agents.core.agent import AgentRegistry
+from arena import Arena, PlayMode
+from arena_utils import GameResult
+from utils.misc import compute_elo
+
+
+class Metrics:
+    """
+    Computes metrics for an agent by making it play against other agents.
+    """
+
+    def __init__(self, board_size: int, max_walls: int):
+        self.board_size = board_size
+        self.max_walls = max_walls
+        self.stored_elos = {}
+
+    def _win_perc(self, results: list[GameResult], agent_name: str):
+        played = 0
+        won = 0
+        for result in results:
+            if result.player1 == agent_name or result.player2 == agent_name:
+                played += 1
+
+            if result.winner == agent_name:
+                won += 1
+
+        return 100.0 * won / played if played > 1 else 0.0
+
+    def _compute_relative_elo(self, elo_table: dict[str, float], agent_name: str) -> int:
+        agent_rating = elo_table[agent_name]
+        best_opponent = 0
+        for name, elo in elo_table.items():
+            if name != agent_name:
+                best_opponent = max(best_opponent, elo)
+
+        return int(agent_rating - best_opponent)
+
+    def compute(self, agent: str | Agent) -> tuple[int, dict[str, float], int, float]:
+        """
+        Evaluates the performance of a given agent by running it against a set of predefined opponents and computing its Elo rating and win percentage.
+
+        Args:
+            agent (str | Agent): The agent to evaluate, specified either as a string identifier or an Agent instance.
+
+        Returns:
+            tuple[int, dict[str, float], int, float]: A tuple containing:
+                - VERSION (int): The version number of the scoring method.
+                - elo_table (dict[str, float]): A dictionary mapping agent names to their computed Elo ratings.
+                - relative_elo (int): The evaluated agent's Elo rating minus the elo for the best opponent.
+                - win_perc (float): The win percentage of the evaluated agent against the opponents.
+
+        Notes:
+            - The method disables training mode for trainable agents during evaluation and restores it afterward.
+            - Opponent Elo ratings are cached to avoid redundant computations.
+            - The evaluation is performed using a fixed set of baseline agents and a specified number of games.
+        """
+        # Bump if there's any change in the scoring
+        VERSION = 1
+        times = 50
+
+        players: list[str | Agent] = [
+            "greedy",
+            "greedy:p_random=0.1,nick=greedy-01",
+            "greedy:p_random=0.3,nick=greedy-03",
+            "greedy:p_random=0.5,nick=greedy-05",
+        ]
+
+        if isinstance(agent, str):
+            agent = AgentRegistry.create_from_encoded_name(agent, board_size=self.board_size, max_walls=self.max_walls)
+
+        # Make sure the agent is not in training mode to do the evaluation
+        if isinstance(agent, AbstractTrainableAgent):
+            prev_training_mode = agent.training_mode
path: deep_quoridor/src/metrics.py
user: login: alejandromarcu
body: I already changed end_game in this PR to return early if it's not in training mode.
As I mentioned, one thing we could do to make sure we don't interfere would be to save the model to disk and evaluate a new instance of the agent loading the model from the disk, so we don't risk any interference. I think I could just remove the`training_only_params` from the parameters that were passed for training.
Another advantage is that if it's saved to disk, the evaluation could be run in parallel in another thread (but of course we incur in the cost of saving the model, which is probably not a big deal for now).
What do you think?

BTW, another thing I'm wondering if it may be worth doing in the future is to have separate classes for training and playing, probably extending the same class, so things are separate, and the playing agent is much easier than the training one anyway,
created_at: 2025-04-26T21:44:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,95 @@
+from agents import AbstractTrainableAgent, Agent
+from agents.core.agent import AgentRegistry
+from arena import Arena, PlayMode
+from arena_utils import GameResult
+from utils.misc import compute_elo
+
+
+class Metrics:
+    """
+    Computes metrics for an agent by making it play against other agents.
+    """
+
+    def __init__(self, board_size: int, max_walls: int):
+        self.board_size = board_size
+        self.max_walls = max_walls
+        self.stored_elos = {}
+
+    def _win_perc(self, results: list[GameResult], agent_name: str):
+        played = 0
+        won = 0
+        for result in results:
+            if result.player1 == agent_name or result.player2 == agent_name:
+                played += 1
+
+            if result.winner == agent_name:
+                won += 1
+
+        return 100.0 * won / played if played > 1 else 0.0
+
+    def _compute_relative_elo(self, elo_table: dict[str, float], agent_name: str) -> int:
+        agent_rating = elo_table[agent_name]
+        best_opponent = 0
+        for name, elo in elo_table.items():
+            if name != agent_name:
+                best_opponent = max(best_opponent, elo)
+
+        return int(agent_rating - best_opponent)
+
+    def compute(self, agent: str | Agent) -> tuple[int, dict[str, float], int, float]:
+        """
+        Evaluates the performance of a given agent by running it against a set of predefined opponents and computing its Elo rating and win percentage.
+
+        Args:
+            agent (str | Agent): The agent to evaluate, specified either as a string identifier or an Agent instance.
+
+        Returns:
+            tuple[int, dict[str, float], int, float]: A tuple containing:
+                - VERSION (int): The version number of the scoring method.
+                - elo_table (dict[str, float]): A dictionary mapping agent names to their computed Elo ratings.
+                - relative_elo (int): The evaluated agent's Elo rating minus the elo for the best opponent.
+                - win_perc (float): The win percentage of the evaluated agent against the opponents.
+
+        Notes:
+            - The method disables training mode for trainable agents during evaluation and restores it afterward.
+            - Opponent Elo ratings are cached to avoid redundant computations.
+            - The evaluation is performed using a fixed set of baseline agents and a specified number of games.
+        """
+        # Bump if there's any change in the scoring
+        VERSION = 1
+        times = 50
+
+        players: list[str | Agent] = [
+            "greedy",
+            "greedy:p_random=0.1,nick=greedy-01",
+            "greedy:p_random=0.3,nick=greedy-03",
+            "greedy:p_random=0.5,nick=greedy-05",
+        ]
+
+        if isinstance(agent, str):
+            agent = AgentRegistry.create_from_encoded_name(agent, board_size=self.board_size, max_walls=self.max_walls)
+
+        # Make sure the agent is not in training mode to do the evaluation
+        if isinstance(agent, AbstractTrainableAgent):
+            prev_training_mode = agent.training_mode
path: deep_quoridor/src/metrics.py
user: login: muralx
body: Since we are already saving snapshots of the model, you could use that one. That being said if you have already verified methods are handling the training_mode setting properly, just go ahead and submit. We can always improve things later.


created_at: 2025-04-27T19:26:15Z
author_association: COLLABORATOR

PR comments:
user: login: alejandromarcu
created_at: 2025-04-24T21:03:35Z
body: ## UPDATE 
cc: @adamantivm , @muralx 

With the last commit, we can pass a tournament frequency, by default I put 500 episodes, I think in general the overhead of this should be very small, but it can be set to a lower frequency from the command line.

e.g. run with command:
```
/Users/amarcu/code/deep_rabbit_hole/.venv/bin/python /Users/amarcu/code/deep_rabbit_hole/deep_quoridor/src/train.py  -N 5 -W 3 -e 600 -i 43 -p dexp:rotate=true,split=true,turn=false,epsilon=1,epsilon_decay=0.9985,epsilon_min=0.1,final_reward_multiplier=20,training_mode=true,use_opponents_actions=true,update_target_every=100,gamma=0.95,target_as_source_for_opponent=true,use_negative_qvalue_function=true,batch_size=128,mask_targetq=true,nick=dexp_train greedy:p_random=0.3 greedy:p_random=0.1 simple -rp cucu -s
```
Here's the [run]( https://wandb.ai/the-lazy-learning-lair/deep_quoridor/runs/cucu-20250424-134353?nw=nwuseramarcu
) in wandb

<img width="1584" alt="image" src="https://github.com/user-attachments/assets/5bf29b72-b181-44c0-860d-0f2e293e4ca4" />

author_association: COLLABORATOR
user: login: muralx
created_at: 2025-04-25T11:29:55Z
body: I think my comment was not submitted and it was pending. I've just sent it, but I will review the new version. I still think metrics generation for an agent that happens to be computed while playing, should be as decoupled as possible from the wand training plugin.
author_association: COLLABORATOR
user: login: alejandromarcu
created_at: 2025-04-25T18:47:55Z
body: > I think my comment was not submitted and it was pending. I've just sent it, but I will review the new version. I still think metrics generation for an agent that happens to be computed while playing, should be as decoupled as possible from the wand training plugin.

Is this because of risks of affecting the training? E.g. I already had an issue that the epsilon was discounting when playing to measure the ELO and I fixed it, but there may be other sides effects.  We could save the model and instantiate a completely new agent to remove the risk.

I think running the metrics at a certain frequency during training provides valuable information.  E.g. if you look at the graph I posted here, you can see that the win perc and elo was increasing during training, so this is a good signal, and if I was trying to make it better I'd increase the episodes in this case.  But in another case you may see that the ELO reaches a max and it's not improving, so this helps you know that more episodes won't help, probably something else needs to be changed to make it improve.



author_association: COLLABORATOR
user: login: alejandromarcu
created_at: 2025-04-28T20:59:40Z
body: # Update 2

With this new changes, the model is evaluated in a tournament after saving it.  The model is re-loaded from disk and a new instance of the agent is created, this way we eliminate the risk of interference between evaluation and training.
E.g. the previous version was evaluating the agent with the current value of the epsilon, which should be 0 during play.  Now it shows how it starts with better results, because we're actually using epsilon=0 during play.

<img width="1207" alt="image" src="https://github.com/user-attachments/assets/f9fd60e3-7b36-428e-9a43-0d64b8efe2c0" />

I tried to run the evaluation in a separate thread, but the wall time ended being a bit longer (not sure why), and it makes the code more messy and the output unclear, so I scratched the idea.  Each save adds about 5s in my computer with the current setup, and by default we save every 500 episodes.  So, it shouldn't have a lot of impact on run time, but we can always pass a parameter to run it less often.  E.g. if we were training for many thousands episodes, it would make sense anyway to save and evaluate less often
author_association: COLLABORATOR