SlowGuess/NeurIPS_2023_OCR huggingface dataset:
assert state == [2, 3] and state != [4, 3]
reward = try_goal(state)
assert done
assert reward == 0 # completion
Here the initial action is optimal: right(). However, we once again see that the rollout policy is suboptimal because it favors the left() action over the optimal right() action.
assert state  $= = [2,3]$  and state  $! = [4,3]$    
state  $=$  right()   
assert not done   
assert reward  $= = 0$    
assert state  $= = [3,7]$  and state  $! = [4,7]$
state  $=$  left()   
assert not done   
assert reward  $\equiv = 0$    
assert state  $= = [2,4]$  and state  $! = [4,4]$    
state  $=$  left()   
assert not done   
assert reward  $= = 0$    
assert state  $= = [1,7]$  and state  $! = [4,7]$    
state  $=$  left()   
assert not done   
assert reward  $= = 0$    
assert state  $= = [0,0]$  and state  $! = [4,0]$    
state  $=$  left()   
assert not done   
assert reward  $= = 0$    
assert state  $= = [0,0]$  and state  $! = [4,0]$    
state  $=$  left()   
assert not done   
assert reward  $= = 0$    
assert state  $= = [0,0]$  and state  $! = [4,0]$    
state  $=$  left()
This causes the model to inaccurately produce a value estimate of 0 for the right() action.
Similar to the previous example, we study an example for maze early in training, when the rollout policy is still quite suboptimal. As a result, the rollout policy does not discover the reward, causing the rollouts to estimate equal value for each of the four actions.
The rollouts all start at state (1, 1).
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  left()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=0) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  up()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  down()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  up()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0, j=1) and state  $! =$  C(i=1, j=0)  
not done  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  up()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  left()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=0) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=1) and state  $! =$  C(i=1, j=0)  
state, reward  $=$  right()
assert reward == 0
assert state == C(i=2, j=2) and state != C(i=1, j=0)
state, reward = right()
assert not done
assert reward == 0
assert state == C(i=2, j=2) and state != C(i=1, j=0)
state, reward = right()
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward = up()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0, j=1) and state  $! =$  C(i=1, j=0)  
state, reward = down()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward = up()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0, j=1) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=0,  $\mathrm{j} = 2)$  and state  $! =$  C(i=1,  $\mathrm{j} = 0)$   
state, reward = right()
assert state  $= =$  C(i=1, j=1) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=1, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = down()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=2) and state  $! =$  C(i=1, j=0)  
state,reward = right()  
assert not done  
assert reward  $= = 0$   
assert state  $= =$  C(i=2, j=2) and state  $! =$  C(i=1, j=0)  
state, reward = right()