Question: What is the best way to think of the reward, versus the agent-wise utility?
Evidence: Here, we introduce our architectures for the action-value estimators. First, we construct the estimators using the utility functions z 1 (w agt ), . . . , z N (w agt ) with agent-wise risk level w agt . Our main contribution in designing the estimators is twofold: (i) using IQN for the true action-value estimator Z jt (w env ) with environment-wise risk level w env and (ii) using a monotonic mixing network for the transformed action-value estimators Z tran with agent-wise risk level w agt .
Agent-wise utility function z i . In a partially observable setting, agents can learn better policies by using their action-observation history τ i instead of their current observation. We represent each agent-wise utility function as a DRQN that receives action-observation history as input and outputs z i for each action and w agt . The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value.
True action-value estimator Z jt . The estimator Z jt aims to express distributions with additional representation power; note that the transformed action-value estimator has limited power due to the decentralization condition (equation 1). For the true action-value estimator, we employ a feed-forward network that takes the state s and set of utility functions z i for i ∈ N . Since we use z averaged over multiple w agt samples, the feed forward network is not conditioned on w agt .
To apply IQN for the true action-value estimator, we use an additional network φ that computes an embedding φ(w env ) for the sample point w env . We calculate the embedding of w env with cosine basis functions and utilize element-wise (Hadamard) product, as done in the IQN paper.
Table: Payoff matrix of the two-step game. N (µ, σ 2 ) denotes the normal distribution with a mean of µ and a variance of σ 2 . For the first step, agent-wise risk-seeking (cooperation) is crucial because the optimal action is (A, A) but taking action A is very risky because when the teammate chooses to be non-cooperative (i.e., by selecting action B or C), a catastrophic reward of -12 is provided. In the second step, handling environment-wise risks is highlighted since there are various combinations of mean and variance depending on joint actions, i.e., N (−1, 10) for the action (C, C) and N (−1, 0) for the action (B, B). Transformed action-value estimator Z tran . The architecture of the transformed action-value estimator is largely the same as the mixing network of. The transformed action-value estimator is expressed as follows:
where θ tran (s, w agt ) is a non-negative parameter obtained from state-dependent hypernetwork. A more detailed discussion is available in Appendix D.
Agent-wise utility function zi. In a partially observable setting, agents can learn better policies by using their action-observation history τi instead of their current observation. We represent each agent-wise utility function as a DRQN (Hausknecht & Stone, 2015) that receives action-observation history as input and outputs zi for each action and wagt. The utility functions do not estimate the action-value, but aim to accurately extract the optimal action from the joint action-value
Answer: They are different because the value of the agent-wise utility can be any value as long as the argmax action of the agent-wise utility and the joint action-value is the same. Conceptually, an agent-wise utility is similar to an actor in an actor-critic method. This is why we use the word “utility” rather than agent-wise “action-value”. In our revised draft, we further clarified this in Section 3.3.