Q‐RTS: a real‐time swarm intelligence based on multi‐agent Q‐learning

Electronics LettersVolume 55, Issue 10 p. 589-591 Control engineeringFree Access Q-RTS: a real-time swarm intelligence based on multi-agent Q-learning M. Matta, Corresponding Author M. Matta matta@ing.uniroma2.it orcid.org/0000-0002-2415-1386 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorG.C. Cardarilli, G.C. Cardarilli Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorL. Di Nunzio, L. Di Nunzio orcid.org/0000-0002-4312-7939 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorR. Fazzolari, R. Fazzolari orcid.org/0000-0002-7383-2663 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorD. Giardino, D. Giardino Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorM. Re, M. Re Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorF. Silvestri, F. Silvestri Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorS. Spanò, S. Spanò orcid.org/0000-0002-8230-7211 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this author M. Matta, Corresponding Author M. Matta matta@ing.uniroma2.it orcid.org/0000-0002-2415-1386 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorG.C. Cardarilli, G.C. Cardarilli Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorL. Di Nunzio, L. Di Nunzio orcid.org/0000-0002-4312-7939 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorR. Fazzolari, R. Fazzolari orcid.org/0000-0002-7383-2663 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorD. Giardino, D. Giardino Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorM. Re, M. Re Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorF. Silvestri, F. Silvestri Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this authorS. Spanò, S. Spanò orcid.org/0000-0002-8230-7211 Department of Electronic Engineering, University of Rome ‘Tor Vergata’, Rome, ItalySearch for more papers by this author First published: 01 May 2019 https://doi.org/10.1049/el.2019.0244Citations: 12AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The authors introduce a novel approach for swarm reinforcement learning that extends the standard Q-learning to multi-agent systems. State-of-the-art methods implement a knowledge sharing mechanism between the agents that is triggered by the episodes succession. This causes an intrinsic limit in the convergence speed of the algorithms. They overcame this issue by developing a Q-learning real-time swarm algorithm (Q-RTS), which is iteration-based and suitable for real-time systems. Q-RTS was tested in different environments and compared to other related methods in the literature. They obtained positive results in terms of learning time and scalability, i.e. achieving a speed-up factor of at least 1.49 with respect to standard Q-learning. Moreover, Q-RTS shows enhanced learning performance as the environments complexity increases. Introduction Reinforcement learning (RL) is an artificial intelligence technique to train an agent to perform a task by interacting with the environment. By trial-and-error, the agent adapts its behaviour through a rewarding mechanism, called reinforcement, which is a measure of its task-solving performance [1]. In recent years, RL became increasingly popular in various fields, e.g. robotics [2], financial trading [3] and industrial applications [4]. In standard RL algorithms the environment is often modelled as a Markovian decision process. The agents gather knowledge from the environment through an exploration process: at time t, the agent in a state takes an action according to a policy and it observes the changes in the environment dynamics. The information derived from this reaction is the reward, which can be a stimulus or a deterrent. It is used to update a state-value function that is a record of the expected return of the cumulative reward over time. Through the state-value function , the agent learns the quality of its new state derived from doing when in following . If the agent successfully completes a task, it reaches a terminal state and concludes the current episode [1]. The next episode starts with an up-to-date state-value function which is exploited by the now expert agent. It explores the environment again, optimising its set of actions over time to reach the goal state and maximising the expected rewards. The use of an updated state-value function through the episodes is called knowledge exploitation. There are many RL algorithms and the most prominent example is Q-learning by Watkins [5]. RL algorithms for multi-agent systems are also present in the literature and they are often bio-inspired [6]. The agents are organised in ensembles, e.g. a swarm, with enhanced learning capabilities [7]. Coordination and knowledge sharing among the agents are key points in swarm RL. The literature proposes several methods for agent knowledge sharing that are based on the episodes succession [6, 8]. In the cited works, the proposed RL algorithms trigger the information exchange whenever all the agents reach a goal state. Therefore, the update of an agent state-value function is unaffected by the swarm dynamics until the current episode of the last agent is complete. This episodic triggering is critical for the convergence time of the algorithm and causes a limitation in sight of a real-time implementation. In this Letter, we propose a swarm learning solution that overcomes the explained swarm RL algorithms timing limitations by making the knowledge sharing iteration-based. We introduce the Q-learning real-time swarm algorithm (Q-RTS), a modified version of Q-learning by Watkins, that is suitable for real-time RL of intelligent swarms. Different experiments have been developed and we present the results as proof of concept. Single-agent Q-learning algorithm Q-learning is an algorithm to infer an optimal action-selection policy . The policy map shows the agent probability to take the action when in state . The agent goal is to maximise its total discounted cumulative future reward derived from the action taken in the current state and from the future steps. Its expected value is the state-value function , already introduced in the previous section. is the discounted cumulative future reward, where is the discount factor. In [5], it is proven that knowing the action-value matrix suffices to obtain . Each element of Q is a quality figure of the corresponding policy . At every iteration t, Q-learning computes the Q values using the update formula (1)where is the learning rate, is the discount factor and is the reward at time t. The literature provides different action-selection policies such as random, greedy, -greedy and Boltzmann [1]. The first one selects the action randomly disregarding the Q values; the second one selects the action with the highest Q value; according to the third policy, the chance of a random action is a value and the probability of a greedy choice is ; the fourth approach uses the Q values to weight the probabilities according to the Boltzmann distribution based on a temperature parameter . Iteration-based Q-learning real-time swarm algorithm: Q-RTS We propose Q-RTS to overcome the timing limitations imposed by the episode-based swarm knowledge sharing. This solution allows the pooling of knowledge from a global swarm matrix at each algorithm iteration. Using this novel approach, the interactions between agents take place in real-time avoiding the need to wait for the last agent to reach the goal state. The concept of Q-RTS algorithm is illustrated in Fig. 1. The aggregation function computes the global matrix merging the N-agents knowledge represented by their respective local matrix . Concurrently, the local combination functions (Comb) compute the update matrices which are linear combinations of the global matrix and . It is also possible to use non-linear combination functions. The global matrix is computed using the aggregation function in (2) on the set of all the local matrices (2)The update process is described by the system in (3). Q-RTS computes the new local matrices by applying Q-learning to instead of . The parameter , used in the combination function, is the independence factor and weights the local and global knowledge (3) Fig 1Open in figure viewerPowerPoint Block diagram of proposed Q-RTS algorithm The aggregation function is non-linear and, at each iteration, selects the with the highest absolute value between the lower and upper bounds taking into consideration all the matrices of . The purpose of such function is to emphasise the most valuable pieces of experience gathered by the agents during the iteration, both for positive and negative events. Experimental results In this section, we provide some experimental results to verify the faster convergence of Q-RTS to the same numerical values computed using standard Q-learning. As in [8], we benchmarked Q-RTS to obtain the shortest path to a goal cell in mazes of different sizes and in various conditions. The agents composing the swarm observe their own state through position coordinates (x,y) and they equip collision sensors to detect the walls. The action set A includes four position shifts: going up, right, down and left. In our experiments, the learning rate is , the discount factor , the action-selection policy is -greedy with and the independence factor is . All the are initialised as zero matrices. The agents gain when they reach the goal position, for each step taken and they are discouraged with if they hit a wall. Each agent experiments its own episode succession: when the goal is reached, the local episode is concluded and the agent position is reset randomly. All the results presented in this section are referred to the mean values of 40 experiments with the same set of hyperparameters, which include the number of agents and the maze size and structure. In the first set of experiments, we tested Q-RTS on mazes with random initial position and a fixed goal cell. Fig. 2 represents the optimal solution for a maze of size , obtained using both Q-learning and Q-RTS with a variable number of agents. As shown in Fig. 3, we observed a speedup in the convergence time based on the number of iterations to reach the optimal Q-values. Such values are the same obtained by Q-learning. The convergence speed increases as more agents are in the swarm. We tested Q-RTS also on mazes of sizes , , and . The speedup values with respect to Q-learning are shown in Table 1. We observed that the speedup values scale not only with the number of agents but also with the state-space dimension, i.e. the maze size. The speedup saturation in the cases and suggests that there is an optimal number of agents which depends on the complexity of the environment. This implies that Q-RTS obtains better results in applications with large state-spaces. Fig 2Open in figure viewerPowerPoint Optimal maze solution Fig 3Open in figure viewerPowerPoint Convergence of for a maze Table 1. Q-RTS speedup for different maze sizes and number of agents No. of agents Maze size 2 1.49 1.56 1.68 1.70 1.71 4 1.83 2.23 2.65 2.82 2.88 6 2.07 2.56 3.31 3.68 3.88 8 2.12 2.68 3.70 4.53 4.70 12 2.27 2.89 4.34 5.59 6.26 16 2.41 2.95 4.75 6.55 7.44 20 2.50 3.12 4.97 7.19 8.45 24 2.59 3.18 5.21 7.61 9.35 28 2.65 3.25 5.36 8.40 10.06 A second set of experiments was performed to compare our work to a similar state-of-the-art method for swarm intelligence, proposed in [8]. We implemented extended pheronomone-Q method (PHE2-Q) which is based on ant colony optimisation (ACO) [9]. Swarms of four agents deployed using Q-RTS and PHE2-Q were tested in a maze. The agents initial position and the goal cell were fixed. As described in [8], the additional parameters of PHE2-Q were set as follows: initial value of pheromone evaporation rate , number of episodes for which the individual Q-learning is performed between two successive information exchanges . Fig. 4 compares the two algorithms in terms of the number of actions required to reach the goal as a function of iterations. We observed that both methods achieve the same minimum number of actions (96 in the test case) but Q-RTS requires less iterations than PHE2-Q to solve the maze and to converge towards the optimal solution. In our specific test, the Q-RTS speedup with respect to PHE2-Q is . Fig 4Open in figure viewerPowerPoint Number of actions through iterations to reach the goal Conclusion In this Letter, we proposed a novel method for real-time Q-learning swarm intelligence (Q-RTS). It overcomes the timing limitation imposed by state-of-the-art algorithms by using an iteration-based update scheme instead of an episode-based one. In our work, we developed an extension to the Watkins model introducing the concept of a global knowledge Q-matrix (). The swarm agents share the knowledge by updating and simultaneously fetching pieces of information from . This mechanism is not episode-dependent and is regulated by the non-linear global aggregation function and the linear local combination function. We provided a mathematical representation of Q-RTS, and we validated our method using two set of experiments with different maze sizes. The first set of tests showed that the learning capabilities of Q-RTS are the same as standard Q-learning, whereas its convergence speed-up scales with the number of agents. Moreover, for the same number of agents, we measured that the timing performance increases as the environment size grows. This is a quality factor of the Q-RTS model and makes it suitable for complex environments in which the performance of traditional Q-learning is time consuming. The second test set was designed to compare Q-RTS to other state-of-the-art swarm RL methods. The results show that our iteration-based algorithm of knowledge sharing outperforms the related episode-based approach, obtaining a speedup factor of . References 1Sutton, R.S., Barto, A.G.: ‘ Reinforcement learning: an introduction’ ( MIT Press, Cambridge, MA, USA, 2018) 2Levine, S., Finn, C., Darrell, T. et. al.,: ‘End-to-end training of deep visuomotor policies’, J. Mach. Learn. Res., 2016, 17, (1), pp. 1334– 1373 3Deng, Y., Bao, F., Kong, Y. et. al.,: ‘Deep direct reinforcement learning for financial signal representation and trading’, Trans. Neural Netw. Learn. Syst., 2017, 28, (3), pp. 653– 664 (https://doi/org/10.1109/TNNLS.2016.2522401) 4Wei, C., Zhang, Z., Qiao, W. et. al.,: ‘Reinforcement-learning-Based intelligent maximum power point tracking control for wind energy conversion systems’, Trans. Ind. Electron., 2015, 62, (10), pp. 6360– 6370 (https://doi/org/10.1109/TIE.2015.2420792) 5Watkins, C.J., Dayan, P.: ‘Q-learning’, Mach. Learn., 1992, 8, (3–4), pp. 279– 292 (https://doi/org/10.1007/BF00992698) 6Brambilla, M., Ferrante, E., Birattari, M. et. al.,: ‘Swarm robotics: a review from the swarm engineering perspective’, Swarm Intell., 2013, 7, (1), pp. 1– 41 (https://doi/org/10.1007/s11721-012-0075-2) 7Navarro, I., Matía, F.: ‘An introduction to swarm robotics’, ISRN Robotics, 2013, 2013, pp. 1– 10 (https://doi/org/10.5402/2013/608164) 8Iima, H., Kuroe, Y., Matsuda, S.: ‘ Swarm reinforcement learning method based on ant colony optimization’. 2010 IEEE Int. Conf. on Systems, Man and Cybernetics, Istanbul, Turkey, October 2010, pp. 1726– 1733 9Dorigo, M., Stiitz1e, T.: ‘ Ant colony optimization’ ( MIT Press, Cambridge, MA, USA, 2004) Citing Literature Volume55, Issue10May 2019Pages 589-591 FiguresReferencesRelatedInformation