## 3 Definition and Motivation
### Limitation of Global Parameter-Sharing Methods with Action Semantic Conflict  
As mentioned in [28; 38], adopting parameter sharing among all agents is equivalent to adding a constraint \(\theta_{i}=\theta_{j},\forall i,j\in N\) to the joint policy space. Here, we point out that utilizing parameter sharing without properly handling the action semantic conflict can lead to a sub-optimal solution.  
**Proposition 1**.: _Consider a fully cooperative physically heterogeneous MARL problem under the global parameter-sharing constraint \(\theta_{i}=\theta_{j},\forall i,j\in N\). Let there be two groups of agents, marked as \(G_{0}\) and \(G_{1}\). Both groups have \(N\) agents. The actual action space of \(G_{0}\) is \(A_{0}=\{a_{0},\ldots,a_{m}\}\), while the actual action space of \(G_{1}\) is \(A_{1}=\{a_{0},\ldots,a_{m},\ldots,a_{n}\}\). The action dims of the two groups follow \(|a_{n}|>|a_{m}|\geq N\). The reward function is defined as:_  
\[r=\begin{cases}1&(\forall\;n_{id},\;a_{id}=n_{id},\;simultaneously)\\ 0&(\text{else})\end{cases}\quad. \tag{1}\]  
_Let \(J^{*}\) be the optimal joint reward, and \(J^{*}_{\theta}\) be the optimal joint reward under the parameter-sharing constraint. Then, we have:_  
\[\begin{split}&\frac{J^{*}}{J^{*}_{\theta}}=\frac{1}{(\frac{ \theta_{r}}{N})^{2N}}\;,\\ & 0<\frac{\theta_{r}}{N}<1\;.\end{split} \tag{2}\]  
Proof of Proposition 1.: Suppose the joint action policy distribution under the global parameter-sharing constraint is \(\{\theta_{0},\ldots,\theta_{m},\ldots,\theta_{n}\}\) for all \(2N\) agents in both groups, where \(\theta_{m}\) indicates the parameterized probability of action \(a_{m}\). It is obvious that \(J^{*}=1\), for we can let the joint policy be the deterministic policy that all agents with their ID \(i\leq N\) from both groups choose the same action \(a_{i}\). For \(J_{\theta}\), because it is required to simultaneously fulfill \(a_{id}=n_{id}\), the expected reward is:  
\[J_{\theta}=\prod_{m=0}^{N}\theta_{m}^{2}\;. \tag{3}\]  
To maximize \(J_{\theta}\), since \(\theta_{m}>0\) and \(N>0\), we can equivalently maximize \((\prod_{m=0}^{N}\theta_{m})^{1/N}\). According to the arithmetic-geometric means inequality, we have:  
\[(\prod_{m=0}^{N}\theta_{m})^{1/N}\leq\frac{1}{N}\sum_{m=0}^{N}\theta_{m}\;, \tag{4}\]where the equality holds if and only if \(\forall m,n\leq N,\theta_{m}=\theta_{n}\). Because \(|a_{n}|>|a_{m}|\geq N\), we have \(\sum_{m=0}^{N}\theta_{m}\leq 1\). Let \(\theta_{r}=\sum_{m=0}^{N}\theta_{m}\), and \(\forall m\in N,\theta_{m}=\theta_{r}/N\) and \(\forall m\
otin N,\theta_{m}=0\). Then, we have:  
\[J_{\theta}\leq J_{\theta}^{*}=\prod_{m=0}^{N}(\frac{\theta_{r}}{N})^{2}=( \frac{\theta_{r}}{N})^{2N}\, \tag{5}\]  
which is exactly the result.  
Through this example, we prove that parameter sharing can lead to an exponentially bad sub-optimal result as the number of agents increases. The action semantic conflict happens in the constrained probabilities \(\{\theta_{0},\ldots,\theta_{m}\}\). When agents from \(G_{0}\) intend to maximize the optimal probabilities \(\{\theta_{0},\ldots,\theta_{m}\}\), the agents from \(G_{1}\) drag down the shared probabilities and therefore prevent the MAS from reaching the global optimality. In addition, it can be inferred that increasing \(\theta_{r}\) helps to improve \(J_{\theta}^{*}\). However, according to the heterogeneity constraint and the parameter-sharing constraint, the maximum of \(\theta_{r}\) is \(\sum_{i=0}^{|a_{m}|}\theta_{i}\), which is still smaller than 1.