Question: Does Table 1 represent a ‘per agent’ reward?
Evidence: As we can observe in Table, there is a significant difference in performance between MADDPG-Oracle and MADDPG (MADDPG-Oracle is 123% better on average): this confirms that developing ToM and recurrence is vital to perform successfully in SymmToM. MADDPG-Oracle is often not an upper bound: when c > a, the heuristic performs better (101% on average). This shows that even with perfect information, it can be difficult to learn the optimal policy using MADDPG. Moreover, models with a recurrence mechanism perform significantly better than MADDPG (61% better on average), also showing that remembering past information gives a notable advantage. As expected, having recurrent models tailored to our problem resulted in better performance than a general LSTM recurrence (RMADDPG). The performance of the best of the tailored models (MADDPG-CE, MADDPG-GE, MADDPG-EE) was 42% better on average than plain RMADDPG. LSTM was able to surpass the best of the tailored models only for a = 3, w = 12, c = 3a.
Increasing c generally decreases global rewards for learned agents (on average, rewards for c = 2a are 75.54% of those for c = a, and rewards for c = 3a are 75.66% of those for c = a).
Table: Average rewards per agent in trained models evaluated during 1000 episodes. 7 runs are averaged for each, using the best checkpoint: this compensates for collapses in performance seen in Fig. This suggests that probabilistic decisions are harder to learn, or impossible to successfully navigate when several events are equally likely. MADDPG-EE did not show improvements over the other learned agents, and in some cases decreased its performance more heavily than other learned agents (e.g. w = 6, c = 3a). MADDPG-EE uses an MLP in its definition of S [2] , which gives more flexibility but also implies a more complex function to learn. We leave to future work to explore other probabilistic agents, but the significant difference in performance between all of the learned models and the highest performing ones (MADDPG-Oracle and the heuristic) shows there is ample space for improvement in this task, and hence proves SymmToM to be a simple yet unsolved benchmark.
Increasing a results in a 10% reduction of performance on average for learned models. Nonetheless, the heuristic improved its rewards by an average of 46%, given the larger opportunities for rewards when including an additional listener. Overall, this implies that increasing a also makes the setup significantly more difficult. Finally, increasing w did not have a conclusive result: for a = 4 it consistently decreased performance in ∼17%, but for a = 3 we saw an improvement of 16% and 61% for c = a and c = 2a respectively.
In sum, modifying c and a provides an easy way of making a setting more difficult without introducing additional rules.
Table 1: Average rewards per agent in trained models evaluated during 1000 episodes. 7 runs are averaged for each, using the best checkpoint: this compensates for collapses in performance seen in Fig. 5 and Fig. 6. Values shown are individual rewards to normalize by the number of agents. Bold lettering represents the best result of a learned imperfect-information model for each setting. Standard deviations are detailed in A.5.
Answer: Yes! The original caption stated “Values shown are individual rewards”, but we have rephrased it to emphasize it represents a per agent reward.