SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) Tabular policy parameterization
Figure 4: Cliff World – Linear/Tabular actor and Linear critic with estimated  $Q$ : Comparison of decision-aware, Adv-MSE, and MSE loss functions using a linear actor Fig. 4a and Fig. 4b coupled with a linear (with three different dimensions) critic in the Cliff World environment for direct and softmax policy representations with Monte-Carlo sampling. When employing a linear actor alongside an expressive critic ( $d = 80$ ), all algorithms have nearly identical performance. However, minimizing the MSE loss with a linear actor and a less expressive critic ( $d = 40, 60$ ) leads to a loss of monotonic policy improvement and converging towards a sub-optimal policy in both representations. Conversely, minimizing the decision-aware and Adv-MSE losses enables reaching the optimal policy. Notably, decision-aware demonstrates a faster rate of convergence when the critic has limited capacity (e.g.,  $d = 40$ ) in both policy representations. The disparity among algorithms becomes more apparent when using tabular parameterization. In this case, the decision-aware loss either achieves a faster convergence rate (in  $d = 80$  and  $d = 60$ ), or it alone reaches the optimal policy ( $d = 40$ ).
(b) Tabular policy parameterization
Figure 5: Frozen Lake - Linear/Tabular actor and Linear critic with exact  $Q$  computation: Comparison of decision-aware, Adv-MSE, and MSE loss functions using a linear actor Fig. 5a and Fig. 5b coupled with a linear (with four different dimensions) critic in the Frozen Lake environment for direct and softmax policy representations with known MDP. For  $d = 100$  (corresponding to an expressive critic) in both actor parameterizations and  $d = 60$  in linear paramterization, all algorithms have the same performance. In other scenarios, minimizing MSE loss functions leads to worse performance than decision-aware and Adv-MSE loss functions and for  $d = 40$  in linear parameterization MSE does not have monotonic improvement. Adv-MSE and decision-aware almost have a similar performance for all scenarios except  $d = 50$  with tabular actor where decision-aware reaches a better sub-optimal policy.
Frozen Lake - Linear/Tabular actor and Linear critic with estimated  $Q$ : For the Frozen Lake environment, when estimating the  $Q$  functions using Monte Carlo sampling (all other choices being the same as in Fig. 5), we found that the variance resulting from Monte Carlo sampling (even with  $\geq 1000$  samples) dominates the bias. As a result, the effect of the critic loss is minimal, and all algorithms result in similar performance.