Repository: google-deepmind/open_spiel (language: python)

Issue:
Exception when running alpha_zero.py on Othello
I attempted to train a AlphaZero model for the Othello game and followed the instruction. The running script is as below:
python3 alpha_zero.py --game othello --nn_model mlp  --actors 10 --path othello/

However, after running the program for 6 steps, the program throw an exception as below:

Traceback (most recent call last):
  File "alpha_zero.py", line 98, in <module>
    app.run(main)
  File "/home/zxc5262/.local/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/home/zxc5262/.local/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "alpha_zero.py", line 93, in main
    alpha_zero.alpha_zero(config)
  File "/home/zxc5262/.local/lib/python3.8/site-packages/open_spiel/python/algorithms/alpha_zero/alpha_zero.py", line 545, in alpha_zero
    learner(game=game, config=config, actors=actors,  # pylint: disable=missing-kwoa
  File "/home/zxc5262/.local/lib/python3.8/site-packages/open_spiel/python/algorithms/alpha_zero/alpha_zero.py", line 171, in _watcher
    return fn(config=config, logger=logger, **kwargs)
  File "/home/zxc5262/.local/lib/python3.8/site-packages/open_spiel/python/algorithms/alpha_zero/alpha_zero.py", line 425, in learner
    num_trajectories, num_states = collect_trajectories()
  File "/home/zxc5262/.local/lib/python3.8/site-packages/open_spiel/python/algorithms/alpha_zero/alpha_zero.py", line 372, in collect_trajectories
    game_lengths_hist.add(len(trajectory.states))
  File "/home/zxc5262/.local/lib/python3.8/site-packages/open_spiel/python/utils/stats.py", line 108, in add
    self._counts[bucket_id] += 1
IndexError: list index out of range

The problem remained to exist when I tried different actors like 40. 

It looks like the length of the trajectory collected by the actors is beyond the limit (8*8). But the grid for the othello game is actually 8 cols * 8 rows. Any suggestion on fixing this problem will be very helpful.




Discussion / hints:
@tewalds any idea what could be causing this?
I suspect this is a bug in othello's max game length, which is currently defined as kNumCells, but doesn't take pass actions into account.
I guess a non-crashing general way of dealing with this would be to change alpha_zero:372 to
`game_lengths_hist.add(min(len(trajectory.states), game.max_game_length()))`
so it's at least tolerant of games that underspecify their max game length. You could do that in utils/stats.py as well, but I think I'd prefer that not blindly tolerate invalid values without the higher level knowledge of whether that's fine or not.
If taking pass action into account, considering the worst case when one party keeps taking action while the other always takes pass, should the max game length of othello be 2*grid size?
Have you looked at that trajectory of 100? Seems unlikely to have 36 pass moves in one game. I suspect a (bigger) bug. Have you customized Othello on your end?
Can you add a try/catch and print out `state.serialize()` so we can inspect the game?
> Have you looked at that trajectory of 100? Seems unlikely to have 36 pass moves in one game. I suspect a (bigger) bug. Have you customized Othello on your end?

I didn't customize the othello environment. I just want to take an example when the trajectory length is greater than the current max length 64.. 100 just means a number bigger than 64. I saw some cases when 3 or 4 passes happened.
> Can you add a try/catch and print out `state.serialize()` so we can inspect the game?

Sure. Will do it now. It may take four or more hours to see the crash.
Oh I misunderstood, sorry. Yeah let's just fix the Othello max game length to something safer like 2*grid_size. Can you submit a PR?

> Sure. Will do it now. It may take four or more hours to see the crash.

Oh it's not necessary. I misunderstood. I thought you were claiming that you had already seen a trajectory with 100 moves. My bad, thanks anyway. 




> Oh I misunderstood, sorry. Yeah let's just fix the Othello max game length to something safer like 2*grid_size. Can you submit a PR?

I tried to modify the max game length in othello.h (https://github.com/deepmind/open_spiel/blob/master/open_spiel/games/othello.h), but it didn't work (still list index out of range). Am I missing something important here?


Did you recompile? (Could be that our initial guess was wrong, too :))
Oh, I realize that I used the simplest installation way, i.e., pip install. After modifying the code in the site-packages/open_spiel, the code worked. Thank you so much for your assistance!
I tried to train an alpha_zero model with mlp (10 actors) for the connect four game using two weeks and finally got the average reward of around 0.85. May I ask if any reference can be provided regarding alpha_zero models? If not, could you please provide any suggestions regarding the computing resources, i.e., how many actors are preferred? Any guidance would be very helpful.
> Oh, I realize that I used the simplest installation way, i.e., pip install. After modifying the code in the site-packages/open_spiel, the code worked. Thank you so much for your assistance!

Great! Can you submit a PR for the fix?

Fix patch:
diff --git a/open_spiel/games/othello.h b/open_spiel/games/othello.h
index fb80d11729..3f20c858e7 100644
--- a/open_spiel/games/othello.h
+++ b/open_spiel/games/othello.h
@@ -152,7 +152,9 @@ class OthelloGame : public Game {
   std::vector<int> ObservationTensorShape() const override {
     return {kCellStates, kNumRows, kNumCols};
   }
-  int MaxGameLength() const override { return kNumCells; }
+
+  // Conservative upper bound due to pass moves.
+  int MaxGameLength() const override { return 2*kNumCells; }
 };
 
 }  // namespace othello


Test patch:
diff --git a/open_spiel/integration_tests/playthroughs/othello.txt b/open_spiel/integration_tests/playthroughs/othello.txt
index 7be1513f3e..6811cdad2e 100644
--- a/open_spiel/integration_tests/playthroughs/othello.txt
+++ b/open_spiel/integration_tests/playthroughs/othello.txt
@@ -27,7 +27,7 @@ UtilitySum() = 0.0
 ObservationTensorShape() = [3, 8, 8]
 ObservationTensorLayout() = TensorLayout.CHW
 ObservationTensorSize() = 192
-MaxGameLength() = 64
+MaxGameLength() = 128
 ToString() = "othello()"
 
 # State 0