Opponent portrait for multiagent reinforcement learning in competitive environment

International Journal of Intelligent SystemsVolume 36, Issue 12 p. 7461-7474 RESEARCH ARTICLE Opponent portrait for multiagent reinforcement learning in competitive environment Yuxi Ma, Yuxi Ma orcid.org/0000-0001-5639-5214 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorMeng Shen, Meng Shen orcid.org/0000-0002-1867-0972 School of Cyberspace Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorYuhang Zhao, Yuhang Zhao orcid.org/0000-0002-9598-5687 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorZhao Li, Zhao Li orcid.org/0000-0001-6918-0204 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorXiaoyao Tong, Xiaoyao Tong orcid.org/0000-0003-1698-8693 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorQuanxin Zhang, Corresponding Author Quanxin Zhang zhangqx@bit.edu.cn orcid.org/0000-0002-5094-7388 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China Correspondence Zhi Wang, College of Cyber Science, Nankai University, 300071 Tianjin, China. Email: zwang@nankai.edu.cn Quanxin Zhang, School of Computer Science and Technology, Beijing Institute of Technology, 100081 Beijing, China. Email: zhangqx@bit.edu.cnSearch for more papers by this authorZhi Wang, Corresponding Author Zhi Wang zwang@nankai.edu.cn orcid.org/0000-0002-3252-9254 College of Cyber Science, Nankai University, Tianjin, China Correspondence Zhi Wang, College of Cyber Science, Nankai University, 300071 Tianjin, China. Email: zwang@nankai.edu.cn Quanxin Zhang, School of Computer Science and Technology, Beijing Institute of Technology, 100081 Beijing, China. Email: zhangqx@bit.edu.cnSearch for more papers by this author Yuxi Ma, Yuxi Ma orcid.org/0000-0001-5639-5214 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorMeng Shen, Meng Shen orcid.org/0000-0002-1867-0972 School of Cyberspace Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorYuhang Zhao, Yuhang Zhao orcid.org/0000-0002-9598-5687 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorZhao Li, Zhao Li orcid.org/0000-0001-6918-0204 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorXiaoyao Tong, Xiaoyao Tong orcid.org/0000-0003-1698-8693 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, ChinaSearch for more papers by this authorQuanxin Zhang, Corresponding Author Quanxin Zhang zhangqx@bit.edu.cn orcid.org/0000-0002-5094-7388 School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China Correspondence Zhi Wang, College of Cyber Science, Nankai University, 300071 Tianjin, China. Email: zwang@nankai.edu.cn Quanxin Zhang, School of Computer Science and Technology, Beijing Institute of Technology, 100081 Beijing, China. Email: zhangqx@bit.edu.cnSearch for more papers by this authorZhi Wang, Corresponding Author Zhi Wang zwang@nankai.edu.cn orcid.org/0000-0002-3252-9254 College of Cyber Science, Nankai University, Tianjin, China Correspondence Zhi Wang, College of Cyber Science, Nankai University, 300071 Tianjin, China. Email: zwang@nankai.edu.cn Quanxin Zhang, School of Computer Science and Technology, Beijing Institute of Technology, 100081 Beijing, China. Email: zhangqx@bit.edu.cnSearch for more papers by this author First published: 15 August 2021 https://doi.org/10.1002/int.22594Citations: 1Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Existing investigations of opponent modeling and intention inferencing cannot make clear descriptions and practical explanations of the opponent's behaviors and intentions, which may inevitably limit the applicability of them. In this work, we propose a novel approach for opponent's policy explanation and intention inference based on the behavioral portrait of opponent. Specifically, we use the multiagent deep deterministic policy gradients (MADDPG) algorithm to train the agent and opponent in the competitive environment, and collect the behavioral data of opponent based on agent's observations. Then we perform pattern segmentation and extract the opponent's behavior events via Toeplitz inverse covariance-based clustering (TICC) algorithm; hence the opponent's behavior data can be encoded into a knowledge graph, named opponent's behavior knowledge graph (OKG). Based on this, we built a question-answer system (QA system) to query and match opponent historical information in OKG, so that the agent can obtain additional experience and gradually infer the intention of opponent with the episodes of iteration. We evaluate the proposed method on the competitive scenario in multiagent particle environment (MPE). Simulation results show that the agents are able to learn better policies with opponent portrait in competitive settings. Citing Literature Volume36, Issue12December 2021Pages 7461-7474 RelatedInformation