Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper studies a sub-class of Markov games that is linearly decomposable.
strength_and_weaknesses: There are many errors in even the basic definitions in this paper. It is very difficult to follow the paper and evaluate its contribution.
clarity,_quality,_novelty_and_reproducibility: Lots of claims, definitions, and notations in the paper are problematic.

> Definition 1

The reviewer is not convinced that this definition makes sense.

- Dou et al., 2022 gives the definition of decomposable Markov games. Why the authors cite it after Markov game?

- A subspace is better to be defined as a tuple, or the authors need to be clear that they want to define on a subspace of the action-oberservation space. Moreover, what does "a collection of" mean? 


>  Linear Value Factorization (LVF) recently gains growing attention, e.g., VDN (Sunehag et al., 2017) and QMIX (Rashid et al., 2018).

QMIX is not a linear value factorization method. Monotonic factorization function class is larger than the linear decomposition function class.

> The IGM principle is defined as the identity between the joint Q value function and the set of factorized local Q value functions

This is also misleading. What does identity mean? IGM only requires the maximum of joint and local Q values are the same..

> Note that we use bold symbols to denote the global and joint variables, e.g., $\mathbf{\mathcal{S}}$ and $\mathbf{\mathcal{u}}$.

Here the authors are defining Dec-POMDP, $\mathbf{\mathcal{S}}$ and $\mathbf{\mathcal{u}}$ here are not variables. They are state and action sets.

> Linear Value Factorization (LVF) ... and becomes the most popular value factorization method in recent years.

The reviewer is surprised by this claim. Linear decomposition is good for theoretical analysis, but, empirically, its performance is not good. The authors should try to follow the frontier of MARL.

> could be multiple convergences for the joint Q value function (Wan et al., 2021) and the policy would get trapped in sub-optimums.

Citation here is problematic.
summary_of_the_review: The direction may be promising, but please submit a mathematically sound (at least a complete) version of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies multi-player Markov games. The authors propose a concept of decomposable Markov game and factorized value function. For the indecomposable Markov games, the authors propose a new method to approximate the global value function by combining the decomposable components and some parameterized module $M$ to fix the gap between the global value function and the combination of decomposable parts.
strength_and_weaknesses: ### Strength:

- The proposed decomposable Markov Game is novel, and using a parameterized function $M$ to fill the gap between the indecomposable and decomposable Markov game is interesting

### Weakness:

- It's not clear to me how we select the subspace components $\hat {\mathcal U}$, despite its existence. If we don't know $\hat {\mathcal U}$, how should we define the parameterized function approximation $M$
- It seems that the $M$ is designed only to fix the gap between the decomposable MG and indecomposable MG? It would be better to highlight the intuition of this $M$.
clarity,_quality,_novelty_and_reproducibility: - This paper is well-written and easy to follow.
- Most of the parts is clear, except for the definition of $\hat {\mathcal U}$
- I appreciate the novelty of the decomposable Markov Game, but I'm not sure I understand the intuition about the function $M$ here. It would be more than helpful if the author could make it more clear.
- I did not find the implementation for reproducibility check
summary_of_the_review: This paper studies the decomposable Markov game and proposes a new method for approximating the indecomposable one. My major concern is mentioned above. Despite that, I appreciate its novelty and would recommend accept
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the relationship between linear value factorization and the decomposability of the Markov game. Then the paper studies the indecomposable Markov game setting and the proposed algorithm, Q Factorization with Representation Interference Suppression, QFRIS, to deal with the representation interference issue in the non-linear value factorization regime. Finally, the paper provides empirical studies on the proposed algorithm and shows that it works well on both the synthetic environment and the benchmark environments. 
strength_and_weaknesses: # Strength:
* The paper makes several nice technical contributions:
1. The paper first identifies the equivalence between linear function factorization and decomposability of the Markov game, thus motivating the study on indecomposable Markov game and non-linear function factorization.
2. The paper identifies the form of extended LVF and argues that one should use the parameterized module form of NVF in order for ensuring IGM and CRC conditions. 
3. The paper identifies the representation interference issues in NVF and proposed the new algorithm QFRIS to mitigate this issue. 
4. The experiments seem convincing in terms that the results match the theory in the synthetic environment and it achieves competitive performance in the standard benchmarks.
* The paper (main paper + appendix) seems relatively complete, and the technical results seem correct.

# Weakness
* I am having a hard time understanding what is the main message of the paper: section 3 seems a motivation for why we should focus on NVF and section 4 seems to identify how to design NVF that achieves both IGM and CRC. My understanding is that these previous sections serve as motivation for the main algorithm QFRIS, but the methodology section seems pretty short and I am also confused why the benchmark results are deferred to the appendix. Is the empirical success of QFRIS not an important message that the paper tries to deliver? In particular, I am confused about the functionality of several parts of the paper: 
1. Section 2.1: why do we introduce POMDP? Do the following sections specifically deal with partial observation? 
2. What is the motivation of introducing Lemma 1?
3. And what is the significance of the extended LVF?
* The definition of CRC condition seems vague to me and the proof for some form satisfying CRC seems quite handwavy, but I may miss its connection to some of the formal definitions and arguments. 
* The writing of the paper needs some refinement. There are several typos and undefined notations in the current version and some of them requires the readers to guess what the paper really wants to say at some point, for example:
  * In Eq. (1), should $s$ be bold because the above section mentions global variables are bold?
  * typo in the subscript $u^1$ in Eq. (2).
  * Again *u_{t+1}* is not bold under the expectation in Eq. (5).
  * What does it in by optimal trajectory under Eq. (6)? Does it only make sense in deterministic system?
  * The last sentence in the first paragraph in section 4 seems incomplete?
  * Is $u_{gre}$ defined the same way on $\mathcal{M}$ as it is defined on $Q$?
  * What does the subscript in the Q function, for example, $Q_{33}$ in the synthetic environment mean?
  * The figure reference numbers do not seem correct.
* I am not fully convinced by the following argument made under proposition 1: "The joint Q value function of LVF is capable to represent the true Q value function only if each decomposed Markov game of the MGD involves only a single agent." For example, how do you define the reward in the synthetic markov game when it is decomposed to one with only one agent?
* The main text has quite a few discontinuities in the arguments, not just the proofs. For example, it would be better to have at least some overview in section 4.2 on why introducing parametrized functions directly suffers poor convergence, instead of throwing everything into the appendix. 
clarity,_quality,_novelty_and_reproducibility: ## Clarity and Quality
As above, the current version of the paper suffers some clarity and writing issues. 
## Novelty
The paper makes nice theoretical observations and empirical contributions, novelty seems fair. 
## Reproducibility
Reproducibility is missing and no code is submitted. 
summary_of_the_review: I am mostly convinced by the technical merits of the paper, but the current version suffers quite a lot of organization and writing flaws. Thus I would recommend a borderline reject at the moment.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper studies value factorization in decentralized Markov games.

strength_and_weaknesses: The writing of this paper is extremely poor and there is a considerable amount of notation/technical errors.


clarity,_quality,_novelty_and_reproducibility: This paper studies value factorization in Markov games but cannot even provide a mathematically correct definition of decomposable Markov games. For example, in Definition 1, what do you mean by $S_i \times \hat{U}_i$ being a ''subspace'' of the joint state-action space $S_i \times U_i$? According to your later wording $S_i\times \hat{U}_i\subset S\times U$, you seems to mean ''subset'' instead of subspace. However, if you are using ''subset'' and ''subspace'' interchangeably, then the relation in the first bullet point in Definition 1 makes no sense at all. 

Getting extremely confused with Definition 1, I referred to the original definition of decomposable MG in Dou et al., 2022 (which is also cited in Definition 1 by this submission) and found it mathematically sound and also much clearer. I do not see how the submission invented this wrong definition but still attributed it to Dou et al., 2022. Also, the place where the submission cited (Dou et al., 2022) is wrong. It should be immediately after Decomposable Markov Game instead of Markov games.

There are tons of typos and vague claims in the submission. Just in Sec 2, I found:

1. $\mathcal{T}_i = (Z_i \times U_i)$:  a trajectory can contain more than one observation-action pair 
2. What do you mean the true Q value function? With respect to which policy? The optimal policy or any policy?
3. In Equation (1), you write $Q_i(\tau_i,u_i)$ but below you write $Q_i(u_i,\tau_i)$
4. what do you mean by "The IGM principle enables the coordination of local policies under the centralized trained joint Q value function"
5. what do you mean by "As a result, the optimal Bellman operator could be not a γ−constraint (Wang et al., 2020a) when faced with non-linear true Q value functions. In words, there could be multiple convergences for the joint Q value function (Wan et al., 2021) and the policy would get trapped in sub-optimums." Please explain what is γ−constraint otherwise what's the meaning of adding this sentence here? What do you mean by "multiple convergences"?


summary_of_the_review: This paper is far below the bar of acceptance and should be immediately rejected. Besides, I strongly urge the authors to carefully revise the draft before the next submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses:  
In this paper, the authors investigated the value function representation in decomposable and indecomposable Markov games. The authors also proposed a novel value function parametrization way and justified the new parametrization on several benchmarks. 


The major issues raised by the reviewers lies in several aspects:

- The correctness of the notations and definitions. In fact, as several reviewers pointed out, these errors make the paper difficult to be evaluated. 

- The clarity of the presentation. The current presentation does not provide enough emphasis on the major contribution, and thus hide the main message and make the reader confusing. 

The paper may contain significant contribution. However, the current presentation makes the paper difficult to follow and the contribution unclear. I suggest the authors to carefully consider the feedback from the reviewers to improve the paper before next submission. 
justification_for_why_not_higher_score: 
The variance of the scores for this paper is large, so I read the paper by myself. 

The paper in current version is not easy to follow, with misuse of notations and terminologies, making the significance of the paper difficult to be evaluated.
Decision: Reject