SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifan Zang $^{1,2}$ , Jinmin He $^{1,2}$ , Kai Li $^{1,2*}$ , Haobo Fu $^{3}$ , Qiang Fu $^{3}$ , Junliang Xing $^{4}$ , Jian Cheng $^{1,2*}$
$^{1}$ Institute of Automation, Chinese Academy of Sciences
$^{3}$ Tencent AI Lab,  $^{4}$ Tsinghua University
{zangyifan2019,hejinmin2021,kai.li}@ia.ac.cn,
{haobofu,leonfu}@tencent.com,jlxing@tsinghua.edu.cn,jcheng@nlpr.ia.ac.cn
Grouping is ubiquitous in natural systems and is essential for promoting efficiency in team coordination. This paper proposes a novel formulation of Group-oriented Multi-Agent Reinforcement Learning (GoMARL), which learns automatic grouping without domain knowledge for efficient cooperation. In contrast to existing approaches that attempt to directly learn the complex relationship between the joint action-values and individual utilities, we empower subgroups as a bridge to model the connection between small sets of agents and encourage cooperation among them, thereby improving the learning efficiency of the whole team. In particular, we factorize the joint action-values as a combination of group-wise values, which guide agents to improve their policies in a fine-grained fashion. We present an automatic grouping mechanism to generate dynamic groups and group action-values. We further introduce a hierarchical control for policy learning that drives the agents in the same group to specialize in similar policies and possess diverse strategies for various groups. Experiments on the StarCraft II micromanagement tasks and Google Research Football scenarios verify our method's effectiveness. Extensive component studies show how grouping works and enhances performance.
Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents' actions through shared team rewards and has become a helpful tool for solving multi-agent decision-making problems [47, 9, 44]. Learning centralized policies to address this problem conditions on the full state, which is usually unavailable during execution due to partial observability or communication constraints [4, 3]. An alternative paradigm is to learn decentralized policies based on local observations [36, 31]. However, simultaneous exploration suffers from non-stationarity that may cause unstable learning [7, 50]. The centralized training with decentralized execution (CTDE) inherits the advantages of these two paradigms and learns decentralized policies in a centralized fashion [23, 15].
Value Function Factorization (VFF) under the CTDE paradigm is a popular approach to MARL, where a centralized value function is learned from global rewards and factorized into local values to train decentralized policies. In recent years, value factorization methods have been widely proposed. Existing methods usually adopt the flat VFF scheme [24], which directly estimates the joint action-value from local utilities. These methods achieve remarkable performances from various perspectives, such as enhancing the expressiveness of the mixing network [27, 33, 38, 37] and encouraging exploration [21, 19]. However, learning efficient cooperation by directly estimating the joint action-values from individual utilities is exceptionally difficult. Phan et al.[24] illustrate that the flat VFF
scheme leads to a performance bottleneck, where it gets challenging to provide sufficiently informative training signals for each agent. Although the state information provides complete knowledge, it is burdensome for agents to extract effective guidance that facilitates cooperative policy learning.
Research in natural systems [13, 43] and multi-agent systems [24] has validated grouping as a means to promote efficient collaboration. Dividing the team into smaller sets allows for fine-grained learning and opens up further opportunities to integrate informative group-wise learning signals. However, formulating a general division criterion without domain knowledge is still a matter of interest to the community. Most previous works are proposed for well-structured tasks and typically predefine specific responsibilities or forms of task decomposition [1, 18, 24]. They require apriori settings that are potentially unavailable in practice and discourage methods' transferring to diverse environments.
This paper proposes a novel formulation of Group-oriented MARL (GoMARL), which learns automatic grouping without domain knowledge for efficient cooperation. GoMARL holds a dual-hierarchical value factorization and learns dynamic groups with a "select-and-kick-out" scheme. Concretely, GoMARL continuously selects agents unsuitable for their current groups based on the learning weights of the decomposition from the group-wise value to local utilities and kicks them out to reorganize the group division. Furthermore, GoMARL transforms various informative training signals, including individual group-related information, group state, and global state, into network weights, which extracts effective guidance for policy improvement and enables flexible adaptation to the dynamic changes in the number of subgroups and the number of agents per group.
The advantages of GoMARL lie in two aspects. Firstly, group information offers richer knowledge for group-wise value factorization and provides more direct guidance for efficient learning. Our value factorization with a dual hierarchy is based on a more focused and compact input representation. Unlike existing methods that learn value factorization with no information or only with the global state which is complete but hard to extract efficient guidance, GoMARL proposes a fine-grained learning scheme to integrate information from the group perspective into the policy gradient, promoting intra-group coordination with the group state and facilitating inter-group cooperation with the global state. Secondly, GoMARL extracts individual group-related information to provide informative signals for policy diversity. The agents condition their behaviors on their individual group-related information embedded by a shared encoder, which is trained following specialization guidance, i.e., imposing similarity within a group and diversity among groups. In this way, GoMARL synergizes subgroups with policy styles, proposing a parameter-sharing mechanism for specialized policies.
We test our method on a challenging set of StarCraft II micromanagement tasks [29] and Google Research Football scenarios [16]. GoMARL achieves superior performance with greater efficiency compared with notable baseline methods. We also conduct detailed component analyses and ablation studies to give insights into how each module works and enhances learning efficiency.
Value function factorization under the CTDE [23] paradigm is a popular approach to MARL. Most existing methods learn flat value factorization by treating agents as independent factors and directly estimating the joint action-value from local utilities. The earlier work, VDN [35], learns a linear decomposition into a sum of local utilities used for greedy action selection. QMIX [27] learns a non-linear mixing network with the global state and enlarges the functions the mixing network can represent, but it still faces the monotonicity constraint. QTRAN [33] further improves the expressivity by proposing the Individual-Global-Max (IGM) principle between individual utilities and the global action-value. Subsequent works follow the IGM principle and further boost performance by encouraging exploration [21, 5], enhancing the expressiveness of the mixing network [38], preventing sub-optimal convergences [26, 37], and integrating functional modules [42, 46, 45, 51, 34, 49].
This paper explores automatic group division for multi-agent teams to realize group-wise learning. Early related works [30, 28, 20, 18] predefine specific responsibilities to each agent based on goal, visibility, capability, or by search. Some other efforts achieve implicit grouping by task allocation [32, 17, 10]. These methods only address tasks with a clear structure and require domain knowledge or apriori settings. Another class of approaches focuses on individuality [41, 19, 14] or role learning [39, 40]. Among them, ROMA [39] learns dynamic roles that depend on the context agents observe. RODE [40] decomposes the joint action spaces and integrates the action effects into the role policies to boost learning. A similar work, VAST [24], also studies the impact of subgroups
on value factorization but still requires a priori of group number. Our method, GoMARL, does not rely on domain knowledge and gradually adjusts the group division according to the learned factorization weights. Based on the dynamic group division, GoMARL models several group-related signals to learn specialized policies and promote efficient team cooperation.
Dec-POMDP. This paper focuses on cooperative tasks with  $n$  agents  $\mathcal{A} = \{a_1, \dots, a_n\}$  as a Dec-POMDP [22] defined by a tuple  $G = \langle S, U, P, r, Z, O, n, \gamma \rangle$ . The environment has a global state  $s \in S$ . Each agent  $a$  chooses an action  $u_t^a$  from its action space  $U_a$  at timestep  $t$  and forms a joint action  $\mathbf{u}_t \in (U_1 \times \ldots \times U_n) \equiv U^n$  that induces a transition according to the state transition distribution  $P(s_{t+1}|s_t, \mathbf{u}_t) : S \times U^n \times S \to [0,1]$ .  $r(s, \mathbf{u}) : S \times U^n \to \mathbb{R}$  is the reward function yielding a shared reward, and  $\gamma \in [0,1)$  is the discount factor. We consider partially observable scenarios in which agent  $a$  acquires its local observation  $z^a \in Z$  drawn from  $O(s_t, a) : S \times \mathcal{A} \to Z$  and has an action-observation history  $\tau^a \in T \equiv (U \times Z)^*$  on which it conditions a policy  $\pi^a(u^a|\tau^a) : T \times U \to [0,1]$ .
Value Function Factorization and the IGM Principle. We consider cooperative MARL with the CTDE paradigm, which has been a significant focus in recent efforts [2, 35, 27, 12, 21]. A majority of methods achieve CTDE through flat value function factorization [24], i.e., factoring action-value functions into combinations of per-agent utilities. The individual utility only depends on the local history of actions and observations, allowing agents to maximize their local utility functions independently under the Individual-Global-Max (IGM) Principle [33]:  $\arg \max_{\mathbf{u}} Q^{tot}(\tau, \mathbf{u}) = \{\arg \max_{u^1} Q^1(\tau^1, u^1), \dots, \arg \max_{u^n} Q^n(\tau^n, u^n)\}$ . Among these attempts, the representative deep MARL approach QMIX [27] improves the simple summation of individual utilities [35] by introducing a more expressive factorization:  $Q^{tot} = f(Q^1(\tau^1, u^1; \theta_Q), \dots, Q^n(\tau^n, u^n; \theta_Q); \theta_f)$ , where  $\theta_f$  denotes the parameters of the monotonic mixing function generated by a hypernetwork [6].
4 Group-oriented Multi-Agent Reinforcement Learning
Definition 1. (Individual and Group) Given a cooperative task with  $n$  agents  $\mathcal{A} = \{a_1, \dots, a_n\}$ , we have a set of groups  $\mathcal{G} = \{g_1, \dots, g_m\}$ ,  $1 \leq m \leq n$ . Each group  $g_j$  contains  $n_j$  ( $1 \leq n_j \leq n$ ) different agents,  $g_j = \{a_{j_1}, \dots, a_{j_{n_j}}\} \subseteq \mathcal{A}$ , where  $\bigcup_j g_j = \mathcal{A}$ ,  $g_j \cap g_k = \varnothing$  for  $j, k \in \{1, 2, \dots, m\}$  and  $j \neq k$ . The superscript describes the variable owner, e.g.,  $u^{j_i}$  is the action of the  $i$ -th agent  $a_{j_i}$  in group  $g_j$ . We denote joint quantities in bold and joint quantities over agents other than a given agent  $a$  with the superscript  $-a$ , e.g.,  $\mathbf{u}^{-j_i}$  is the joint action of agents in group  $g_j$  other than agent  $a_{j_i}$ .
GoMARL decomposes the global action-value  $Q^{tot}$  into group-wise values  $Q^g$  and trains agents by groups in a fine-grained manner. Figure 1 illustrates the overview of the learning framework. It consists of an automatic grouping module, specialized agent networks generating local utilities  $Q^i$ , and a mixing network among groups. In Section 4.1, we introduce the automatic grouping module. It progressively divides the team into dynamic groups as training proceeds. Based on the dynamic group division, we propose specialized agent networks that achieve similarity within each group and diversity among groups to generate local utilities  $Q^i$  in Section 4.2. Section 4.3 presents the overall training framework to detail the estimation of the group-wise action-values and the global action-values.
Figure 1: Overview of GoMARL.
4.1 Automatic Grouping Mechanism
The automatic grouping mechanism aims to learn a mapping relationship  $f_{g}:\mathcal{A}\mapsto \mathcal{G}$ . The key idea is to divide the team into dynamic groups in an end-to-end fashion by maximizing the expected global return  $Q_{\mathcal{G}}^{tot}(s_t,\mathbf{u}_t) = \mathbb{E}_{s_{t + 1:\infty},\mathbf{u}_{t + 1:\infty}}\left[\sum_{k = 0}^{\infty}\gamma^k r_{t + k}|s_t,\mathbf{u}_t;\mathcal{G}\right]$ . Value function factorization approaches represent the joint action-value as an aggregation of individual utilities, i.e., a weighted sum of  $Q^i$  and biases. We follow this setting and represent the group-wise value  $Q^g$  as an aggregation of the individual value  $Q^i$ . Intuitively, as illustrated in the right side of Figure 2, if the learned mixing
Figure 2: The left side illustrates the schematic diagram of the grouping mechanism. The group adjustment operator concatenates the weights  $\mathbf{w}_1^i$  of agents in the same group to form the group-wise weights  $\mathbf{w}_1^g$ . The right side gives an example to show how grouping  $\mathcal{G}$  changes during training.
weight of  $Q^i$  is small enough, then  $Q^i$  contributes a little to its group-wise value  $Q^g$ . In other words, when an agent  $a_{j_i}$  takes action  $u^{j_i}$  but does not affect its group-wise value  $Q^{g_j}$ , it indicates that agent  $a_{j_i}$  does not belong to group  $g_j$  anymore and needs group adjustment.
We factorize the group value  $Q^g$  into individual utilities  $Q^i$  with group-wise mixing weights  $\mathbf{w}_1$ , i.e.,  $Q^{g_j} = f(Q^{j_1}(\tau^{j_1},u^{j_1}),\dots ,Q^{j_{n_j}}(\tau^{j_{n_j}},u^{j_{n_j}});\mathbf{w}_1^{g_j})$ , where  $\mathbf{w}_1^{g_j}$  denotes group  $g_j$ 's mixing parameters generated by individual  $\mathbf{w}_1$  generators  $f_{\mathrm{w}_1}^i (\cdot ;\theta_{\mathrm{w}_1}^i):\tau^i\to \mathrm{w}_1^i$  that map each agent  $a_i$ 's history information  $\tau^i$  to a  $k$ -dimensional weight vector. A regularization on this  $\mathbf{w}_1$  generator  $f_{\mathrm{w}_1}^i$  drives the automatic grouping mechanism. Specifically, GoMARL "selects and kicks out" agents whose individual utilities hold small mixing weights and contribute a little to their group-wise  $Q$  values; thus, a sparsity regularization on  $\mathbf{w}_1$  is implemented to select those agents with small group-value mixing weights. The  $\mathbf{w}_1$  generators  $f_{\mathrm{w}_1}^i$  are trained by minimizing the following loss function:
$$
\mathcal {L} _ {g} \left(\theta_ {\mathrm {w} _ {1}}\right) = \mathbb {E} _ {\left(\mathbf {z}, \mathbf {u}, r, \mathbf {z} ^ {\prime}\right) \sim \mathcal {B}} \sum_ {i} \left(\left\| f _ {\mathrm {w} _ {1}} ^ {i} \left(\tau^ {i} \left(z ^ {i}, u ^ {i}\right); \theta_ {\mathrm {w} _ {1}} ^ {i}\right) \right\| _ {l _ {1}}\right), \tag {1}
$$
where  $\mathcal{B}$  is the replay buffer, and  $\| \cdot \|_{l_1}$  stands for the  $l_{1}$ -norm penalty.
Figure 2 illustrates the schematic diagram of the grouping mechanism. Initially, all the agents belong to the same group, and the grouping  $\mathcal{G}$  is adjusted as training proceeds. The grouping shifts every  $c$  timesteps, and each selected agent is assigned to the following group (a new one for agents in the last group) until it properly contributes to where it belongs. When adjusting groups, the  $\mathrm{w}_1$  of all agents are sent to a group adjustment operator  $\mathcal{O}_g$ . According to the adjusted grouping  $\mathcal{G}$ ,  $\mathcal{O}_g: \{\mathbf{w}_1^1, \dots, \mathbf{w}_1^n\} \xrightarrow{\mathcal{G}} \{\mathbf{w}_1^{g_1}, \dots, \mathbf{w}_1^{g_m}\}$  concatenates the  $\mathbf{w}_1^i$  of agents in the same group to form a set of group-wise  $\mathbf{w}_1^g$  to generate group action-value. This implementation flexibly adapts to the grouping dynamics since each  $\mathbf{w}_1^i$  is tied to agent  $a_i$  by the individual  $\mathbf{w}_1$  generator  $f_{\mathbf{w}_1}^i$ . No matter which group  $g_j$  agent  $a_i$  belongs to,  $Q^i$  can engage in the estimation of  $Q^{g_j}$  through  $\mathbf{w}_1^i \in \mathbf{w}_1^{g_j}$ .
In practice, the weights' shrinkage to zero is infeasible. It is also challenging to determine a fixed threshold for groups of various sizes in diverse environments. We empirically utilize seventy percent of each group's average weight to assess whether an agent fits its current group. Extensive experiments and component studies in Section 5 confirm the universality of this configuration.
4.2 Specialized Agent Network for Decentralized Execution
Group status describes the cooperation among agents within each group and is essential to local utility generation. Integrating group-wise information  $e$  into the decision-making process enables consideration of cooperative behaviors. In this section, we introduce GoMARL's agent network to generate local action-value functions with individual group-related information  $e$  embedded.
As shown in Figure 3, we construct a group-related info encoder  $f_{e}(\cdot;\theta_{e})$  to embed agents' hidden states. To achieve a group-related view, we train the encoder network as an extractor, where the extracted agent info  $e$  of agents from the same group should be similar. To avoid all agents'  $e^{i}$
collapsing to be alike, the regularizer also encourages diversity between agents from different groups. Formally, we minimize the following similarity-diversity objective to train the encoder  $f_{e}$ :
$$
\mathcal {L} _ {S D} \left(\theta_ {e}\right) = \mathbb {E} _ {\mathcal {B}} \left(\sum_ {i \neq j} I (i, j) \cdot \operatorname {c o s i n e} \left(f _ {e} \left(h ^ {i}; \theta_ {e}\right), f _ {e} \left(h ^ {j}; \theta_ {e}\right)\right)\right), \tag {2}
$$
where  $I(i,j) = \left\{ \begin{array}{ll} - 1, & a_i,a_j\in g_k.\\ 1, & a_i\in g_k,a_j\in g_l,k\neq l. \end{array} \right.$
The encoder trained by the SD-loss extracts agent info  $e^i$  that is recognizable to agents' groups. This group-related information is then fed into a decoder  $f_d(\cdot; \theta_d)$  to generate the parameters of the agent network's upper MLP. In this way, each agent  $a_i$  conditions its behaviors on  $e^i$  with group-related information embedded, promoting potential cooperation within each group during decentralized execution. The decoder hypernetwork  $f_d$  is trained by the TD-loss introduced in Section 4.3.
Figure 3: The specialized agent network.
The proposed actor network has two merits. Firstly, it enables diversified policies while sharing all the parameters. Policy decentralization with shared parameters is widely utilized to improve scalability and learning efficiency. However, agents tend to behave similarly when sharing parameters, preventing effective exploration and complex cooperative policies. It is also undesirable to entirely forgo shared parameters in pursuit of diversity since proper sharing accelerates learning. Our method hybridizes the efficiency of parameter-sharing and the policy diversity needed for complex collaboration. Secondly, the decoder hypernetwork  $f_{d}$  integrates the extracted agent info  $e$  into the policy gradients, providing informative group-related information to enrich local utilities and promote intra-group cooperation. Concretely, the partial derivative for updating the parameters  $\theta_{h}$  of the GRU and the bottom MLP is:
$$
\frac {\partial Q ^ {t o t}}{\partial \theta_ {h}} = \frac {\partial Q ^ {t o t}}{\partial Q ^ {a}} \frac {\partial Q ^ {a}}{\partial \theta_ {h}} = \frac {\partial Q ^ {t o t}}{\partial Q ^ {a}} \frac {\partial Q ^ {a}}{\partial v _ {h} ^ {a}} \frac {\partial v _ {h} ^ {a}}{\partial \theta_ {h}} = f _ {d} \left(e ^ {a}\right) \cdot \frac {\partial Q ^ {t o t}}{\partial Q ^ {a}} \frac {\partial v _ {h} ^ {a}}{\partial \theta_ {h}}, \tag {3}
$$
where  $v_{h}$  is the representation after the GRU illustrated in the left side of Figure 3. Eqn.(3) shows that  $e^{a}$  is deeply involved in the policy updating of agent  $a$ , providing richer group knowledge to generate local utilities and facilitating group-related guidance for cooperative decentralized execution.
4.3 Overall Learning Framework
We next introduce GoMARL's overall learning framework. It contains two mixing networks that generate the group-wise  $Q^g$  and the global  $Q^{tot}$ , respectively. As for the group mixing network, although  $\mathbf{w}_1^g$  enables a weighted mixing of local utilities of agents in group  $g$ , the naive mixture of  $\sum_{a\in g}\mathbf{w}_1^a Q^a +b$  lacks the guidance of group status to reflect the group action-value in a specific group-wise state. Therefore, we build the group mixing network with a two-layer structure that also embeds the group-wise state
Figure 4: Overall learning framework of GoMARL.
into the weights  $\mathbf{w}_2^g$  to generate  $Q^g$ . In particular, group  $g_j$ 's group-wise state  $s^{g_j}$  is a fusion of the agent info  $e^i$  for all  $a_i \in g_j$ . To cohesively summarize the group state based on the agent info of all agents in group  $g_j$ , we apply a pooling operation [25] over each dimension of the agent info  $e^i$  to generate the group-wise state  $s^{g_j}$  describing the current group status. The pooling operation also ensures adaptability to the dynamic group size (i.e., the number of agents per group). We build a group-wise  $\mathbf{w}_2$  generator  $f_{\mathrm{w}_2}(s^g)$  to map the fused group state into  $\mathbf{w}_2^g$ . Similar to the group-related info encoder  $f_e$ , the  $\mathbf{w}_2$  generator integrates the group state  $s^g$  into the policy gradient:  $\frac{\partial Q^g}{\partial Q^a} = \frac{\partial Q^g}{\partial v_Q^a} \frac{\partial v_Q^a}{\partial Q^a} = f_{\mathrm{w}_2}(s^g) \cdot \frac{\partial v_Q^a}{\partial Q^a}$ , providing group status information for  $Q^g$  generation that facilitates efficient intra-group cooperation.  $v_Q$  is the representation after  $\mathbf{w}_1^g$  marked in Figure 4. The
two-layer mixing structure of the group mixing network estimates the group action-value  $Q^g$  by  $\mathbf{w}_1$  which decides the group division and  $\mathbf{w}_2$  which carries group status information.
GoMARL estimates the global action-value  $Q^{tot}$  by mixing all the  $Q^g$  in a similar fashion. Concretely, the two layers of the total mixing network are generated by two hypernetworks, respectively taking group states  $s^g$  and the full state  $s$  as inputs. Likewise,  $s^g$  and  $s$  are deeply involved in the gradients, promoting inter-group cooperation. The architecture of the total mixing network is akin to the group mixing network and is omitted in Figure 4 for brevity. The TD-loss of the estimated  $Q^{tot}$  is:
$$
\mathcal {L} _ {T D} (\theta) = \mathbb {E} _ {\mathcal {B}} \left[ \left(r + \gamma \max  _ {\mathbf {u} ^ {\prime}} \bar {Q} ^ {t o t} \left(s ^ {\prime}, \mathbf {u} ^ {\prime}\right) - Q ^ {t o t} (s, \mathbf {u})\right) ^ {2} \right], \tag {4}
$$
where  $\bar{Q}^{tot}$  is a target network with periodic updates. The overall learning objective of GoMARL is:
$$
\mathcal {L} (\theta) = \mathcal {L} _ {T D} (\theta) + \lambda_ {g} \mathcal {L} _ {g} \left(\theta_ {\mathrm {w} _ {1}}\right) + \lambda_ {S D} \mathcal {L} _ {S D} \left(\theta_ {e}\right), \tag {5}
$$
where  $\theta = (\theta_h, \theta_e, \theta_d, \theta_{\mathrm{w}})$ .  $\theta_{\mathrm{w}}$  denotes the parameters of hypernetworks producing all the mixing weights and biases.  $\lambda_g$  and  $\lambda_{SD}$  are two scaling factors.