Question: Quantitative evaluation of the two kinds of tests proposed?
Evidence: In this section, we compare the different algorithms explained in the previous section. The observation space will be constituted of a processed version of the last turn in the episode, to keep the input size controlled. More precisely, the observation space is composed of: the position of all agents, all recharge bases, the current direction each agent is moving towards to and what they communicated in the last turn, the presence of a wall in each of the immediate surroundings, and every agents' first-hand information. First-hand information is publicly available in our experiments to moderate the difficulty of the setup, but this constraint could also be removed. This simple setting is still partially observable, since the agents cannot hear interactions outside of their hearing range.
We use the reward as our main evaluation metric. This metric indirectly evaluates ToM capabilities, since information-seeking is at the core of SymmToM. We train through 60000 episodes, and with 7 random seeds to account for high variances in the rewards obtained. Our policies are parametrized by a two-layer ReLU MLP with 64 units per layer, as in the original MADDPG. MADDPG-EE's function f is also a two-layer ReLU MLP with 64 units per layer.
We test two board sizes (w = 6, and w = 12), two numbers of agents (a = 3 and a = 4), and three quantities of information pieces (c = a, c = 2a c = 3a). The length of each episode is set to 5w.
More detail about design decisions can be found in Section A.5.
In the past section we analyzed results using the metric of episode rewards. Although the rewards in SymmToM are designed to correlate with information seeking and knowledge state of the agent themselves and others, they do not show explicitly if agents are exhibiting theory of mind. To do so more directly, we develop two categories of possible analyses: scenarios specifically designed to test theory of mind, and post-hoc analyses of episodes.
A classic example of a scenario specifically designed to test ToM behavior is the Sally-Anne task. This false belief task, originally designed for children, aims to test if a passive observer can answer questions about the beliefs of another person, in situations where that belief may not match reality. If we were to use it for machine ToM, we could repeat the experiment and ask an agent to predict the position of an object varying the underlying conditions. This test is feasible because there is only one agent with freedom of action, which ensures that desired conditions are met every time. Testing becomes unfeasible when giving multiple agents freedom of action, as constraints planned in test design may be broken by a collective drift from the strategy thought by the designer. Testing becomes easier if we allow for controlling all agents but one, as shown in Fig.. Other tests besides the ones shown may be designed. In particular, in Fig. we show an example of probabilistic ToM where two communicative events are equally likely, but one could modify this scenario to have different probabilities and test the expected value of the turns until red successfully shares an information piece. One could also design retroactive deduction tests: for example, in Fig. if red communicates and receives no reward, it can deduce that green had Figure: Example of tests for zeroth, first, second order, and probabilistic ToM. We test red agents, immobilize gray agents, and control blue and green agents' movements. In Fig., red will go to the top right if it remembers to have heard the first piece, and to the left otherwise. In Fig., red will move to the right if and only if it assumes that the two agents on the left played optimally (red cannot hear what they communicated). In Fig., blue is controlled to ensure it will search the agent on the bottom left (its optimal play, in five moves). Red's optimal move is to meet blue, and hence must only move to the bottom left, even if the agent currently there will not provide any reward. In Fig., red will interact with green not knowing what it communicated with blue in the previous turn. It should be able to communicate the missing piece with an expected value of 1.5 turns. received that information from blue. If there had been another agent (let's say, a yellow agent) in range of blue when it spoke to green, the red agent could also update its knowledge about yellow. Results for the tests proposed in Figure are detailed in Appendix A.1.
Post-hoc analysis also has its challenges in multi-agent settings, even in the most direct cases. Thanks to our reward shaping, using recharge bases is always the optimal move when an agent has all the information available: an agent will have a reward of (n − 1)c for using the base, whereas it can only gain up to n − 1 + c − 1 per turn if it decides not to use it. Even in this case, small delays in using the base may occur, for example if the agent can gather additional rewards on its path to the base. More generally, having multiple agents makes a specific behaviors attributable to any of the several events happening at once, or a combination of them.
Even though it may be difficult to establish causality when observing single episodes, we developed metrics that comparatively show which models are using specific features of the environment better than others. Examples of metrics are unsuccessful recharge base usage count; number of times an agent shares an information piece everyone in its range already knows when having better alternatives; number of times an agent moves away from every agent when not having all the information pieces available; among others. See Appendix A.2 for detailed description and results. Reward can also be understood as a post-hoc metric with a more indirect intepretation.
Post-hoc analyses of single episodes can also be blurred by emergent communication. Because agents were trained together, they may develop special meaning assignment to particular physical movements or messages. Even though qualitatively this does not seem to be the case for the models presented in the paper, tests should also account for future developments. This also implies that one should not overinterpret small differences in the metrics described in the paragraph above.
In the past section we analyzed results using the metric of episode rewards. Although the rewards in SymmToM are designed to correlate with information seeking and knowledge state of the agent themselves and others, they do not show explicitly if agents are exhibiting theory of mind. To do so more directly, we develop two categories of possible analyses: scenarios specifically designed to test theory of mind, and post-hoc analyses of episodes.
A classic example of a scenario specifically designed to test ToM behavior is the Sally-Anne task. This false belief task, originally designed for children, aims to test if a passive observer can answer questions about the beliefs of another person, in situations where that belief may not match reality. If we were to use it for machine ToM, we could repeat the experiment and ask an agent to predict the position of an object varying the underlying conditions. This test is feasible because there is only one agent with freedom of action, which ensures that desired conditions are met every time. Testing becomes unfeasible when giving multiple agents freedom of action, as constraints planned in test design may be broken by a collective drift from the strategy thought by the designer. Testing becomes easier if we allow for controlling all agents but one, as shown in Fig.. Other tests besides the ones shown may be designed. In particular, in Fig. we show an example of probabilistic ToM where two communicative events are equally likely, but one could modify this scenario to have different probabilities and test the expected value of the turns until red successfully shares an information piece. One could also design retroactive deduction tests: for example, in Fig. if red communicates and receives no reward, it can deduce that green had Figure: Example of tests for zeroth, first, second order, and probabilistic ToM. We test red agents, immobilize gray agents, and control blue and green agents' movements. In Fig., red will go to the top right if it remembers to have heard the first piece, and to the left otherwise. In Fig., red will move to the right if and only if it assumes that the two agents on the left played optimally (red cannot hear what they communicated). In Fig., blue is controlled to ensure it will search the agent on the bottom left (its optimal play, in five moves). Red's optimal move is to meet blue, and hence must only move to the bottom left, even if the agent currently there will not provide any reward. In Fig., red will interact with green not knowing what it communicated with blue in the previous turn. It should be able to communicate the missing piece with an expected value of 1.5 turns. received that information from blue. If there had been another agent (let's say, a yellow agent) in range of blue when it spoke to green, the red agent could also update its knowledge about yellow. Results for the tests proposed in Figure are detailed in Appendix A.1.
Post-hoc analysis also has its challenges in multi-agent settings, even in the most direct cases. Thanks to our reward shaping, using recharge bases is always the optimal move when an agent has all the information available: an agent will have a reward of (n − 1)c for using the base, whereas it can only gain up to n − 1 + c − 1 per turn if it decides not to use it. Even in this case, small delays in using the base may occur, for example if the agent can gather additional rewards on its path to the base. More generally, having multiple agents makes a specific behaviors attributable to any of the several events happening at once, or a combination of them.
Even though it may be difficult to establish causality when observing single episodes, we developed metrics that comparatively show which models are using specific features of the environment better than others. Examples of metrics are unsuccessful recharge base usage count; number of times an agent shares an information piece everyone in its range already knows when having better alternatives; number of times an agent moves away from every agent when not having all the information pieces available; among others. See Appendix A.2 for detailed description and results. Reward can also be understood as a post-hoc metric with a more indirect intepretation.
Post-hoc analyses of single episodes can also be blurred by emergent communication. Because agents were trained together, they may develop special meaning assignment to particular physical movements or messages. Even though qualitatively this does not seem to be the case for the models presented in the paper, tests should also account for future developments. This also implies that one should not overinterpret small differences in the metrics described in the paragraph above.
In the past section we analyzed results using the metric of episode rewards. Although the rewards in SymmToM are designed to correlate with information seeking and knowledge state of the agent themselves and others, they do not show explicitly if agents are exhibiting theory of mind. To do so more directly, we develop two categories of possible analyses: scenarios specifically designed to test theory of mind, and post-hoc analyses of episodes.
A classic example of a scenario specifically designed to test ToM behavior is the Sally-Anne task. This false belief task, originally designed for children, aims to test if a passive observer can answer questions about the beliefs of another person, in situations where that belief may not match reality. If we were to use it for machine ToM, we could repeat the experiment and ask an agent to predict the position of an object varying the underlying conditions. This test is feasible because there is only one agent with freedom of action, which ensures that desired conditions are met every time. Testing becomes unfeasible when giving multiple agents freedom of action, as constraints planned in test design may be broken by a collective drift from the strategy thought by the designer. Testing becomes easier if we allow for controlling all agents but one, as shown in Fig.. Other tests besides the ones shown may be designed. In particular, in Fig. we show an example of probabilistic ToM where two communicative events are equally likely, but one could modify this scenario to have different probabilities and test the expected value of the turns until red successfully shares an information piece. One could also design retroactive deduction tests: for example, in Fig. if red communicates and receives no reward, it can deduce that green had Figure: Example of tests for zeroth, first, second order, and probabilistic ToM. We test red agents, immobilize gray agents, and control blue and green agents' movements. In Fig., red will go to the top right if it remembers to have heard the first piece, and to the left otherwise. In Fig., red will move to the right if and only if it assumes that the two agents on the left played optimally (red cannot hear what they communicated). In Fig., blue is controlled to ensure it will search the agent on the bottom left (its optimal play, in five moves). Red's optimal move is to meet blue, and hence must only move to the bottom left, even if the agent currently there will not provide any reward. In Fig., red will interact with green not knowing what it communicated with blue in the previous turn. It should be able to communicate the missing piece with an expected value of 1.5 turns. received that information from blue. If there had been another agent (let's say, a yellow agent) in range of blue when it spoke to green, the red agent could also update its knowledge about yellow. Results for the tests proposed in Figure are detailed in Appendix A.1.
Post-hoc analysis also has its challenges in multi-agent settings, even in the most direct cases. Thanks to our reward shaping, using recharge bases is always the optimal move when an agent has all the information available: an agent will have a reward of (n − 1)c for using the base, whereas it can only gain up to n − 1 + c − 1 per turn if it decides not to use it. Even in this case, small delays in using the base may occur, for example if the agent can gather additional rewards on its path to the base. More generally, having multiple agents makes a specific behaviors attributable to any of the several events happening at once, or a combination of them.
Even though it may be difficult to establish causality when observing single episodes, we developed metrics that comparatively show which models are using specific features of the environment better than others. Examples of metrics are unsuccessful recharge base usage count; number of times an agent shares an information piece everyone in its range already knows when having better alternatives; number of times an agent moves away from every agent when not having all the information pieces available; among others. See Appendix A.2 for detailed description and results. Reward can also be understood as a post-hoc metric with a more indirect intepretation.
Post-hoc analyses of single episodes can also be blurred by emergent communication. Because agents were trained together, they may develop special meaning assignment to particular physical movements or messages. Even though qualitatively this does not seem to be the case for the models presented in the paper, tests should also account for future developments. This also implies that one should not overinterpret small differences in the metrics described in the paragraph above.
We use the reward as our main evaluation metric. This metric indirectly evaluates ToM capabilities, since information-seeking is at the core of SymmToM. We train through 60000 episodes, and with 7 random seeds to account for high variances in the rewards obtained. Our policies are parametrized by a two-layer ReLU MLP with 64 units per layer, as in the original MADDPG (Lowe et al., 2017). MADDPG-EE’s function f is also a two-layer ReLU MLP with 64 units per layer.****To do so more directly, we develop two categories of possible analyses: scenarios specifically designed to test theory of mind, and post-hoc analyses of episodes.****Other tests besides the ones shown may be designed. In particular, in Fig. 3d we show an example of probabilistic ToM where two communicative events are equally likely, but one could modify this scenario to have different probabilities and test the expected value of the turns until red successfully shares an information piece.****Even though it may be difficult to establish causality when observing single episodes, we developed metrics that comparatively show which models are using specific features of the environment better than others. Examples of metrics are unsuccessful recharge base usage count; number of times an agent shares an information piece everyone in its range already knows when having better alternatives; number of times an agent moves away from every agent when not having all the information pieces available; among others. See Appendix A.2 for detailed description and results. Reward can also be understood as a post-hoc metric with a more indirect intepretation. Post-hoc analyses of single episodes can also be blurred by emergent communication. Because agents were trained together, they may develop special meaning assignment to particular physical movements or messages. Even though qualitatively this does not seem to be the case for the models presented in the paper, tests should also account for future developments. This also implies that one should not overinterpret small differences in the metrics described in the paragraph above.
Answer: We have added results of both types of analysis in the appendix, and will include more discussion of them for the camera ready version. We also expanded the number of runs throughout the paper (now we have 7 random seeds), with consistent results. We would like to emphasize that we believe that reward is the best metric to globally analyze performance. The discussion in Section 7.3 aimed to show that there are mainly two different ways to locally measure different levels/aspects of theory of mind in specific setups, that both could be useful to measure particular phenomena, but none generalize to a global metric as well as using the reward.