Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs

Abstract: Covering option discovery has been developed to improve the exploration of RL in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. Given that joint state space grows exponentially with the number of agents in multi-agent systems, existing researches still relying on single-agent option discovery either become prohibitive or fail to directly discover joint options that improve the connectivity of the joint state space. In this paper, we show how to directly compute multi-agent options with collaborative exploratory behaviors while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector using the Laplacian spectrum of individual agents' transition graphs. Further, considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method by estimating eigenfunctions through NN-based representation learning techniques. The evaluation on multi-agent tasks built with simulators like Mujoco, shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms the state-of-the-art. Codes are available at: https://github.itap.purdue.edu/Clan-labs/Scalable_MAOD_via_KP.

Reviews and author responses:

 I will raise my score to 6. I believe this is hard to parse from the paper itself (without your comment above). Please consider emphasizing the Mujoco results and the deep learning extension.

 ## About the approximation error of the estimated eigenvectors:
As mentioned in the third paragraph of Section 4.3, the NN-based estimation method of the eigenvectors is proposed in [1]. In Fig. 3 and 4 of [1], they have shown the accuracy of the estimations in both discrete (e.g., Grid Maze) and continuous (e.g., Mujoco Room) tasks by comparing with the groundtruth. In this paper, we use the method to get Laplacian spectrum estimations as mentioned in Section 4.3 and then adopt these estimations together with Theorem 4.1 to discover the multi-agent options.


## About the exploration in complex infinite-scale state-space environments:
Thanks for pointing this out. The exploration in complex environments with infinite-scale state space is carried out in an interactive fashion in multiple stages. To be specific, the agent explores the environment through a given policy (e.g., random walk) in the first stage. Then, in the next stage, the agent explores with its primitive actions and the options which are extracted from the samples collected in the first stage. Our results guarantee that these options can significantly improve collaborative exploration in the second stage. The process can continue with more efficient options being discovered in each later stage for improved joint exploration. We will add a discussion of this in the final version.

## About the difference in the frequency of changing options by different agents:
Each option has specific termination states which are determined by the Fiedler vector, so it could happen in our evaluation that a certain state is the termination state of one option but not the termination state of another option. Specifically, in a decentralized manner, if agents A and B choose different options and reach a state where A terminates but B doesn’t, A will switch to a new option but B will continue with the current option. Thus, the difference in the frequency of changing options by different agents due to the decentralized manner makes it harder for the agents to visit the joint termination state indicated by the joint Fidler vector.

## About using the NN-based DRL algorithm to compare with the SOTA MARL algorithm:
As mentioned in the last paragraph of Section 4.4, for the continuous control tasks, i.e., Mujoco Maze/Room, we use MAPPO to learn the high-level policy and SAC to learn the low-level policy, which are SOTA deep MARL and RL algorithms.

As for the evaluations on the Grid tasks, our aim is to justify the effectiveness of the tabular version of our algorithm, and the comparison between Fig. 5 and 11 has shown that tabular algorithms are much more efficient than NN-based algorithms for these tasks. Moreover, we have provided the comparisons among our algorithm with two baselines, i.e., agents without options and agents with single-agents options, in Fig. 5-8. Our algorithm and these two baselines adopt the same base tabular algorithm to learn the high-level and low-level policy, so the only difference in the algorithm aspect is that our method adopts the multi-agent options, of which the superiority is shown in Fig. 5-8.
In summary, the Grid Room/Maze tasks are for evaluating the tabular version of our algorithm, and Mujoco Room/Maze tasks are for evaluating the deep learning extension of our algorithm in this paper.

## About our algorithm used in the Mujoco environment:
As justified in the previous experiments, i.e., Fig. 5-8, the centralized manner to adopt the joint options performs much better. Hence, we also adopt the centralized manner in the Mujoco tasks, as mentioned in the last paragraph of Section 5.
The action space of the Mujoco tasks is continuous, so it’s difficult for the MARL baselines to make joint decisions. However, by introducing multi-agent options, as shown in Fig. 3, the high-level policy is trained to select among the options, which is a discrete space and makes joint decision-making possible. Thus, joint decision-making (i.e., centralized manner) is another advantage of using multi-agent options in MARL. Note that the baseline labeled with ‘Single-agent Options’ in Fig. 9 also adopts the centralized manner to select among the single-agent options, the comparison with which shows the necessity to use multi-agent options.

[1] Wang, Kaixin, et al. "Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing." International Conference on Machine Learning. PMLR, 2021.


 ## About the application of our algorithm on large-scale environment training:
We want to clarify that the deep learning version of our algorithm doesn’t need to construct or store state-transition graphs and thus can be applied to tasks with large-scale state and action spaces. As mentioned in the extension part of Section 4.3, we don’t need to calculate and store a state-transition graph to get the joint Fiedler vector for tasks with infinite-scale state space. Instead, we estimate the Laplacian spectrum using a DNN based on the transitions stored in the replay buffer, i.e., $\mathcal{T}=[(s, s’)]$ in Eq. 5 and 7, where $s$ denotes the current state and $s’$ denotes the next state. The replay buffer is a commonly-used technique in deep reinforcement learning, so it will not cause additional training difficulty. In our experiment, we use and store 100 trajectories in the replay buffer, each of which has a horizon of 500.

In Fig. 9, we have shown that our algorithm can scale to continuous state and action spaces and outperform SOTA MARL algorithms significantly. For tasks with higher dimensional state space, we only need to change the input of the neural networks that we train, and no changes are required for the algorithm part. Moreover, it can be observed from Fig. 9 that all the baselines have already failed to complete the Mujoco task, which is why we didn’t need to further evaluate on tasks with higher dimensional state space. The evaluations in Section 5 are to show that our algorithm can aid the joint exploration and boost the performance in tasks with sparse and delayed reward functions, so, as mentioned in the second paragraph of Section 5.1, we make the tasks (i.e., Grid and Mujoco tasks) challenging by setting the reward to be sparse and delayed.

## About the comparison with COMA, MAVEN, and Weighted QMIX:
For the tabular version of our algorithm, we use Grid Maze/Room as evaluation tasks. To show the difficulty of these grid tasks with quite sparse reward functions, we provide a comparison between our proposed tabular algorithm and deep SOTA MARL methods: COMA, MAVEN, and Weighted QMIX, on the 4-agent Grid Maze task in Fig. 5(c)(f) and Fig. 11 (Appendix E.1).

This comparison is fair because: (i) Given that the neural network training in the baselines requires a longer time, we set the training horizon as 5000 episodes which is five times that of the tabular methods, and the networks are trained for ten iterations in each episode. However, it can be observed from Fig. 11 that these deep learning baselines still fail to learn a policy for the task, demonstrating the need for multi-agent options in tasks with sparse reward functions and thus requiring thorough exploration. (ii) Moreover, we note that Fig. 5(c) shows the result of our algorithm (i.e., MAOD) using the decentralized manner, which is a fair comparison with the CTDE algorithms shown in Fig. 11. Still, our algorithm performs much better.

## About the related work mentioned by the reviewer: 
The work mentioned by the reviewer is only a 2-page extended abstract, which does not include a complete algorithm design for the tabular case nor consider the use of deep learning for Laplacian spectrum estimation. It only shows limited evaluation on a toy task, $n$-agent Grid Room. However, in our paper, we propose the complete algorithm including its theoretical foundations and propose a new design of our algorithm for the infinite-scale state spaces using DNN for the spectrum estimation. Moreover, we provide complete evaluations of our algorithm, including the evaluations on the centralized, decentralized, and group-based strategies to adopt the joint options, the evaluations on tasks with dynamic interactions among the agents, and the comparisons with SOTA MARL baselines on the infinite-scale Mujoco tasks.

## About the typo in Eq. 18:
Thanks for mentioning this. We will fix it in the final revision.



 I sincerely apologize for my carelessness. The review for your paper is now available.

 ## Regarding the experimental setup for the option discovery phase:

The options are built based on the Laplacian spectrum of the individual-state transition graphs using Theorem 4.1. Thus, we need to compute the state transition graph based on the transitions, i.e., $(s, a, s')$, where $s$ is the current state, $a$ is the action, and $s'$ is the next state. We use 10000 transitions for the discrete tasks and 50000 transitions for the continuous control tasks. The policy we use to collect these data is the random walk policy, i.e., sampling an action uniformly from its action space at each time step, and the data collection can be completed within 1 minute on a machine with one Geforce RTX 2060 GPU, for both the discrete and continuous tasks.

We learn 16 multi-agent options in the Grid tasks and 8 multi-agent options in the Mujoco tasks. Note that the options are discovered in an unsupervised manner, which is not based on any reward signal and not fine-tuned. Same as [1], the option discovery is only based on the property of the state space, e.g., the algebraic connectivity.

Thanks for your suggestion. We will add these necessary details in the final revision.

[1] Jinnai, Yuu, et al. "Finding options that minimize planning time." International Conference on Machine Learning. PMLR, 2019.
      
## Regarding the primitive option:

As mentioned in the first paragraph of Section 4.4, in typical RL settings, we need to train an agent to select among the primitive actions, aiming to maximize the accumulated reward. We define the agent that selects among only the primitive actions as the primitive option agent, e.g., the MAPPO agent in Fig. 9.

Once the primitive option is selected by the high-level policy, the primitive actions are sampled from the low-level policy of the primitive option, which is trained with MARL algorithms. For example, in the Mujoco tasks, we use MAPPO to train the low-level policy of the primitive option for 1000 episodes which is also the number of episodes for training the low-level policies of the multi-agent options (Fig. 3). Then, we fix these low-policies and only train the high-level policy to select among the primitive option and multi-agent options, of which the training curve is shown in Fig. 9. Note that to maintain fairness, we have pretrained the baselines, e.g., MAA2C, MADDPG, and MAPPO in Fig. 9, for the same number of episodes as well.

 ## On whether the proposed method can be used when the learning algorithm is non-tabular Q learning:

The proposed Multi-agent Option Discovery approach can be used when the learning algorithm is not tabular Q-learning. As mentioned in the last paragraph of Section 4.4, for the continuous control tasks, i.e., Mujoco Maze/Room, we use MAPPO to learn the high-level policy and SAC to learn the low-level policy, which are SOTA deep MARL and RL algorithms. The evaluation results on Mujoco Maze/Room are shown in Fig. 9, where our algorithm labeled with “Multi-agent Options” performs the best.

## On suggestions for further improvement:

Thank you for these suggestions. We provide a point-to-point response as follows: 

1. In the paper, we empirically demonstrate that multi-agent options acquired based on Theorem 4.1 can significantly improve learning performance in challenging tasks with both discrete and continuous states and actions. More rigorous theoretical study related to Theorem 4.1, e.g., the specific error bound, is an interesting direction for future research. We will discuss this in the conclusion section.

2. We will move the pseudo code of our algorithm from Appendix D to the main text and add a discussion of the limitations and future works in the final version when we can get an extra page.

3. As for the communication complexity, we have mentioned in the third paragraph of Section 4.4 that the agents need to share their views (i.e., observations) when co-exploring their joint observation space since the initiation and termination conditions of the multi-agent options are defined based on the joint observations. Thus, communication among the agents may be needed in cases where the joint observations are not directly available. The efficient communication strategy in MARL is a well-studied topic. In this paper, we mitigate communication complexity by grouping the agents randomly or based on sub-task assignments, such that only agents within the same sub-group need to share their views for option discovery. There are more opportunities to explore by combining efficient communications and option discovery. Unfortunately, we cannot solve everything in one paper :) We will discuss this in our conclusion section as a future research direction. Thanks for mentioning this in the review. 

4. As mentioned in the second paragraph of Section 4.3, for an MDP with $n$ agents and $m$ states for each agent, our solution significantly reduces the computation complexity from $\mathcal{O}(m^{3n})$ to $\mathcal{O}(nm^3)$. Thus, if $m$ is small and $n$ is large, the reduction in computation complexity will be more significant. Also, the communication complexity can be mitigated by grouping the agents when the number of agents (i.e., $n$) is large, as mentioned above.


 ## On extensions to MARL environments requiring non-tabular Q learning:

1. We have extended the Multi-agent Option Discovery algorithms to the infinite-scale state space case in Section 4.3. As shown in Eq. 4-7, we use a neural network (NN) to estimate the Laplacian spectrum based on which we can get the joint Fiedler vector and multi-agent options, rather than explicitly calculating the Laplacian matrix so that it can scale to infinite-scale state spaces. To adopt these discovered joint options in MARL tasks with continuous state and action spaces (i.e., requiring non-tabular Q-learning), we use MAPPO to train the high-level policy and SAC to train the low-level policy, as mentioned in the last paragraph of Section 4.4.

2. In Fig. 9, we have evaluated the deep learning extensions mentioned above on the Mujoco Room/Maze tasks, of which the state space and action space are continuous, as mentioned in the first paragraph of Section 5.1. It can be observed from Fig. 9 that multi-agent options can significantly improve the performance, compared with SOTA MARL baselines for continuous control tasks.

## On comparisons with MAPPO, MADDPG, and MAA2C: 

We note that the comparisons in Fig. 9 between our algorithm and the baselines: MAPPO, MADDPG, and MAA2C, are fair. 

1. We don’t use tabular approximation methods for our algorithm which is labeled with “Multi-agent Options” in Fig. 9. As mentioned above, we use an NN-based technique to discover the options, MAPPO to train the high-level policy, and SAC to train the low-level policy. Thus, the only difference is that the baselines (unlike our algorithm) do not have access to multi-agent options.

2. In Fig. 9, the options adopted are pre-trained with SAC for 10000 episodes, so we have pretrained other baselines: MAA2C, MADDPG, and MAPPO, for the same number of episodes to maintain fairness.

## On comparisons with COMA, MAVEN, and Weighted QMIX: 

For the tabular version of our algorithm, we use Grid Maze/Room as evaluation tasks. To show the difficulty of these grid tasks with quite sparse reward functions, we provide a comparison between our proposed tabular algorithm and deep SOTA MARL methods: COMA, MAVEN, Weighted QMIX, on the 4-agent Grid Maze task in Fig. 5 (c)(f) and Fig. 11 (Appendix E.1).

This comparison is also fair. Given that the neural network training in the baselines requires a longer time (as pointed out by the reviewer), we set the training horizon as 5000 episodes which is five times that of the tabular methods, and the networks are trained for ten iterations in each episode. However, it can be observed from Fig. 11 that these deep learning baselines still fail to learn a policy for the task, demonstrating the need for multi-agent options in tasks with sparse reward functions and thus requiring thorough exploration.

In summary, the Grid Room/Maze tasks are for evaluating the tabular version of our algorithm, and Mujoco Room/Maze tasks are for evaluating the deep learning extension of our algorithm in this paper.


 Thanks! But we believe the review is intended for another paper. Please take a look and we would appreciate if you could fix it.

 This work extends the Covering Option Discovery framework (for tabular Q-learning) to the multi agent setting. Their contributions lies in using the kronecker product of each agent's state transition graphs to get a joint graph who Felder vector (eignevector of second smallest eigen value) can be estimated. Then, like in covering option discovery where the states that have to be explored most are extracted from the Felder vector, they extract the joint states that have to be explored most. This is followed a projection of the important joint states into individual state spaces to obtain the individual states that have to be explored most. They perform experiments with a decentralized and centralized framework that learn these options (in a tabular framework and a deep learning extension) on grid room/grid maze/mujoco room/mujoco maze. Strengths
1. Interesting extension to the covering discovery framework for MARL. 
2. Helpful sensitivity analysis on sub-task grouping and random grouping for large n in the experiments.

Weaknesses
1. Missing analysis of the options learnt. It would be hlepful to understand what states were deemed most important to explore. Especially in the multi-agent setting.
2. Entension to MARL environments where tabular Q learning methods may fail. 1. Are the MAPPO, MADDPG, MAA2C baselines doubly set back: (1) because they don't have options and hence they require longer to learn and (2) use neural networks that would require longer to learn? Can tabular approximation methods be used in the baselines as well for a fair comparison? What about in COMA, MAVEN and QMIX? Alternatively would using an MLP in centralized and decentralizing Q learning slow down learning by a large factor? None

 This paper studies an empirical method to encourage the exploration of RL algorithm in multi-agent settings. The proposed method is inspired by the covering option discovery in single-agent scenarios. A naïve application of single-agent option discovery cannot work because the joint state space. To address this challenge, the authors proposed a novel method to compute multi-agent options based on a Kronecker-product approximation of the joint state space. The proposed method reduces the computing complexity significantly when each local state space is large. The authors also extend their method to tasks with infinite-scale state space, and provide four simulation results to show the effectiveness of their method. This paper makes a good contribution to the multi-agent reinforcement learning. It is interesting to see a method based on graph Laplacian can be used to encourage exploration and be implemented efficiently in a multi-agent setting. The paper is well-written and easy to follow with adequate intuitions and contexts.

I also have some suggestions for further improvement: In Theorem 4.1, it would be better to provide a more specific error bound instead of using a vague phrase “can approximate”. Further, the authors shall consider specify what assumptions are required in line 577. In Section 4.3, I recommend to move the pseudo code of the algorithm in Appendix D to the main body. In the second paragraph of Section 4.3, the authors emphasize the proposed method significantly reduces the computational complexity. What about the communication complexity? What will happen if $m$ is small and $n$ is large? (line 200)
 Besides the suggestions I gave in the previous part, I’m also curious about if the proposed multi-agent option recovery approach can be used when the learning algorithm is not tabular Q learning? Possible candidates include Q learning with function approximation and policy gradient methods. I encourage the authors to add a discussion about the limitations and future works at the end of this paper. I do not see any potential negative social impacts.

 This paper proposes a new multi-agent hierarchical reinforcement learning algorithm — Multi-agent Covering Option Discovery. Since the state space and action space of a multi-agent environment grow exponentially with the number of agents, this paper approximates the joint state transition graph as the Kronecker product of single-agent ones, which reduces the complexity of computing the joint state transition graph's Fiedler vector. Finally, the covering options method is applied to construct multi-agent options. In addition, the algorithm is extended to the continuous state space environment. The algorithm achieves state-of-the-art performance in some multi-agent tasks. Strengths：

1. The motivation is very clear. The paper gives an illustrative example to explain that the multi-agent option is more conducive to effective exploration compared to the single-agent option. And the Kronecker product is rationally applied to simplify the computation of the joint Fielder vector, and the robustness of the estimation error is experimentally analyzed.
2. The paper is well organized and clearly written.

Weaknesses：

1. The algorithm is difficult to apply to large-scale environment training. Whether or not the Kronecker product approximation is used, the computational effort required to construct the graph and the space required to store the graph is huge if the environment is complex. In this paper, although experiments are conducted on a Mujoco environment which is an infinite-scale state space, the state space is only 6 dimensions, still very small.
2. The experimental comparison with SOTA MARL algorithms (COMA, MAVEN,...) is not convincing. First, tabular and NN-based algorithms can not be compared based on the same training episodes. Second, these MARL algorithms are based on the CTDE (Centralized Training Distributed Execution) framework, so the agents' execution is independent, which is not a fair comparison with MAOD (Centralized manner).
3. The paper is an extended version of [1], only extending the algorithm to the continuous state space environment, and additionally replacing the experimental environment.
4. Typographical error. Eq.18 in Appendix should be $v_{11} = v_1^{G_1} \otimes v_1^{G_2}$ not $v_{11} = v_1^{G_1} \otimes v_2^{G_2}$.



[1] Chen, J., Chen, J., Lan, T. and Aggarwal, V., 2022, May. Multi-agent Covering Option Discovery through Kronecker Product of Factor Graphs. In *Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems* (pp. 1572-1574). https://www.ifaamas.org/Proceedings/aamas2022/pdfs/p1572.pdf 1. When the algorithm is applied to an infinite-scale state-space environment, the eigenvectors need to be represented as a neural network. Is it possible to provide an experimental analysis of this estimation error? For example, the same method can be used for experimental analysis in discrete tasks.
2. In a more complex infinite-scale state-space environment, does the random interaction still work? Cause agents may not be able to explore the environment effectively, or even only a small part of the environment, so the eigenvector representing may have a large error.
3. Does the difference in the frequency of changing options by different agents have an effect on the results? Does it happen that a certain state is the termination state of one option but not the termination state of another option? Specifically, in a decentralized manner, if agents A and B choose different options and reach a state where A terminates and B continues, B will switch options while A continues.
4. Can the comparison with the SOTA MARL algorithm all use the NN-based DRL algorithm? For example, DQN-based MAOD.
5. As stated in Weakness.2, is the MAOD in the Mujoco environment the centralized manner or the decentralized manner? The authors analyze the estimation error of the joint state transition graph and verify it experimentally accordingly. However, if the algorithm is applied to an environment with a more complex state space, the state transfer graph will be very large, leading to increased storage and computational complexity. So adding a section would be quite helpful.

 The paper studies the topic of covering option discovery in multi-agent reinforcement learning. The proposed approach extends in non-trivial way the single-agent approach that is based on greedily connecting distant state in the embedding space provided by the Fiedler vector of the MDP graph (the second smallest eigenvector of the MDP Laplacian). As the joint state space grows exponentially with the number of agents, performing option discovery directly on the joint space is computationally very intense. Instead, the authors propose to approximate the joint space by Kronecker product of single agents' state space, estimate the spectrum of each individual agent's Laplacian and then find the joint Fiedler vector on the whole space using their main theorem 4.1.
Moreover, the authors provide empirical investigation in 2d navigation mazes (discrete and continuous).  

  
## Strength:
+ The paper is well written and well motivated
+ I appreciated the illustrative example in Figure 1 and 2.
+ The proposed approach is technically sound and the authors show in theorem 4.1 how we can estimate the Laplacian spectrum of Kronecker product of factor graphs.
+ The authors provide first an algorithm in the tabular case and then extended it to large state space using the generalized graph drawing objective.
+ The authors investigate different variants of the algorithm (centralized, distributed, sub-task and random grouping) and provide a comparison with SOTA multi-agent algorithms with and without options. Overall, the proposed method provide a boost of performance consistently across tasks.

## Weakness:
-  The experimental setup for the option discovery phase is not explained at all. What is data used? how it was collected ? How many options are considered ? 
- The authors considered only 2-d navigation tasks. The scalability of the method in more challenging and practical environment remains a weak spot of the paper (The Mujoco experiments provide some insights but the domains have essentially  very small dimensionality)  - Could you please explain the experimental setup for the option discovery phase in each environment? how much data do you use ? under which policy you collect the data ? how many options are learned ? Is there any finetuning or pruning of options ?
- How the primitive option is defined in the experiments ? Once the primitive option is selected by the high-level policy, under which distribution the primitive actions are sampled ?  The authors adequately addressed the limitations and potential negative societal impact of their work

Meta-review: This paper proposes an extension of the covering discovery framework from single-agent to MARL. There was unanimous support for accepting this paper and agreement that it is an interesting and very useful contribution.

The responses to reviewers helped to clarify a few points. When preparing a revision, we strongly encourage you to address these points which came up during the review process:
* Consider putting more emphasis on the Mujoco experiments and DL extension in the main paper, to illustrate the generality of the results
* Discuss limitations and future work related to communication complexity of the proposed approach
* Clarify/justify details of how tabular and NN-based methods are compared in the grid world experiments
* Clarify the setup in the option discovery phase

Decision: Accept