# Measuring Variables Effect to Statistically Model the Multi-Robot Patrolling Problem by Means of ANOVA

David Portugal, Rui P. Rocha

► To cite this version:

David Portugal, Rui P. Rocha. Measuring Variables Effect to Statistically Model the Multi-Robot Patrolling Problem by Means of ANOVA. 3rd Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), Feb 2012, Costa de Caparica, Portugal. pp.199-206, 10.1007/978-3-642-28255-3_22. hal-01365586

HAL Id: hal-01365586

https://inria.hal.science/hal-01365586

Submitted on 13 Sep 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Measuring Variables Effect to Statistically Model the Multi-Robot Patrolling Problem by Means of ANOVA

David Portugal¹ and Rui P. Rocha¹

¹ Institute of Systems and Robotics, Department of Electrical and Computer Engineering, University of Coimbra, 3030-290 Coimbra, Portugal {davidbsp, rprocha}@isr.uc.pt

**Abstract.** This paper focuses on analyzing extensive results generated from running diverse multi-robot patrolling algorithms with different configurations towards measuring the influence of the variables of the general problem. In order to do this, a statistical technique by the name of Analysis of Variance (ANOVA) is employed to compare the parameters and identify the ones which give raise to the total dispersion of the data set, at the same time accessing their contribution to the obtained results. It is shown that by applying such technique, it is possible to compute a data-related model for the Multi-Robot Patrolling Problem (MRPP).

**Keywords:** Multi-Robot Patrolling, ANOVA and Statistical Analysis.

## 1 Introduction

This work addresses cooperative multi-robot systems in area patrolling missions. In such tasks, agents must coordinately visit all regions of an area, at regular intervals, in order to protect or supervise them. Since it involves several variables, the multi-robot patrolling problem is highly complex and no optimal patrolling strategy has been proven to exist for all cases.

In a recent work [1], we implemented and compared five representative strategies to solve the problem using Stage/ROS simulator [2]. Aiming to promote transparent comparisons between all algorithms, it was necessary to test them in environments with different connectivity and teamsizes. Naturally, the combination of these factors produced a large set of results, which were only qualitatively discussed therein.

## 2 Contribution to Value Creation

Analyzing the state of the art in this field, the majority of works contain limitations like the absence of studies on scalability, performance and on other parameters of the problem, as well as many simplifications at the experimental work level. In this work, the main objective is to go beyond empirical analysis by conducting a quantitative analysis, identifying the contribution of each factor/parameter {Algorithm, Teamsize, Connectivity} to the dispersion of the results using ANOVA and building an
initiatory data-related model that explains the results obtained in the context of the
Multi-Robot Patrolling Problem (MRPP).

3 Related Work

During the last decade, several authors have proposed different algorithms for solving the MRPP, which mainly differ in terms of planning strategy, communication paradigm, coordination scheme, complexity, agent perception, decision-making and performance evaluation.

Pioneer work on this field was presented in [3] and [4], where very simple approaches with reactive and cognitive agents were proposed as well as a first evaluation criterion based on idleness. Following these works, more advanced approaches based on graph theory became also popular like [5] and [6], which use graph partitioning to assign efficient patrol routes to different robots. Based on a similar idea, in [7] a technique to reduce the patrolling graph to multiple spanning trees, assigning a spanning tour around each tree was recently presented. In contrast, many authors have proposed new algorithms for multi-robot patrolling based on other concepts, like task allocation [8], negotiation mechanisms [9], swarm intelligence [10], neural networks [11] or game theory [12]. For a more complete survey of multi-robot patrolling strategies, the interested reader is referred to [13].

On the other hand, ANOVA is a powerful statistical technique which enables the comparison between parameters of more than two populations [14]. From the analysis of the total dispersion present in a data set, it allows us to identify the source of the variations that led to that dispersion and evaluate the contribution of each factor, determining whether a significant relation exists between variables. The experiments presented in this work consider three factors (algorithm, teamsize and connectivity), thus the analysis of variance used to study their effects is called a three-way ANOVA.

ANOVA has been used as a tool in a wide range of scientific literature. For example, the relationships between group size and efficiency in robotic ant-like foraging tasks are studied in [15]. Different teams of robots were compared in 3 sets of experiments with different food distributions. To test the effect of group size, one-way ANOVAs were performed. Relationships between sets of experiments and group size were also inspected using two-way ANOVAs. Generally, it was concluded that group size has a significant effect on colony foraging efficiency. Results also show that in environments where robots are able to recruit other robots (call others to where the "food" is) the foraging efficiency significantly increases when compared to situations where robots cannot communicate.

4 Preliminaries

Every patrolling algorithm implicated in the results set, assume a topological environment, which is a common assumption in the MRPP [13], and the decision-making process is related to the computation of paths in the navigation graph so that robots effectively patrol a given environment, visiting all vertices of the graph.

Since they are governed by different patrolling rules, it is necessary to define an
evaluation metric to compare the performance of these algorithms. In this work the average graph idleness $Idl_G$ is used, defined as the average idle time of all vertices of the graph G during the multi-robot patrolling mission [1]. Results are therefore expressed by $Idl_G$ in seconds. The lower the $Idl_G$, the better the performance is.

The dataset considered is the result of the extensive simulation process described in [1]. Note that these simulations are fairly realistic, given that a community recognized robotic framework was used (ROS [17]), considering the robot's dynamics; and results are based on the actual time instead of simulation iterations, unlike previous works.

Five state-of-the art patrolling algorithms were implemented, namely: Conscientious Reactive (CR) [3], Heuristic Conscientious Reactive (HCR) [4], Heuristic Pathfinder Conscientious Cognitive (HPCC) [4], Cyclic Algorithm for Generic Graphs (CGG) [5] and Generalized MSP Algorithm (MSP) [5]. These algorithms were combined with six different teamsizes: 1, 2, 4, 6, 8 and 12; and three environments with different connectivity properties: low connectivity (A), medium connectivity (B) and high connectivity (C). All trials were repeated three times.

Tables 1, 2 and 3 present a summary of the performance results obtained in the experiments. In the next section three-way ANOVA is applied to quantitatively measure the influence of the parameters tested in the experiments.

# 5 Applying ANOVA to the Dataset

The purpose of this analysis is to access the influence of three important parameters in the results: patrolling algorithm or strategy, teamsize and environment connectivity.

Linear models are considered, assuming that the probability distribution of the response is normal, mutually independent and homoscedastic (i.e., the variance of the data inside the groups is equivalent). In the general case, the ANOVA fundamental test, which makes use of the F-statistics distribution usually with 95% of confidence bounds, verifies the significance of the factors by checking the group's variable effect ($\alpha_i$) [14]:

$$ 
\begin{aligned}
H_0: \alpha_1 = \alpha_2 = \dots = \alpha_i = 0 \\
H_1: \exists \alpha_i \neq 0,
\end{aligned} 
\qquad (1) $$

where $H_0$ is the null hypothesis and $H_1$ is the alternative hypothesis. If the probability of the null hypothesis is near 0, a main effect is present due to the associated factor, meaning that the result is statistically significant. Arbitrarily high n-way ANOVA divides the total variation, given by the deviation of all observations from the global mean, into variations given by different factors and residual variation (or error). Also, if the model has a non-additive effect, variations given by the interaction of factors are also considered. These variations are calculated through sums of squares. In this work, a model with first-order interaction effects between pairs of factors was adopted. This model explains 99.75% of the variation of the results.
<table><caption>**Table 1. Numerical Results for Map A ($Idl_G$ values in seconds)**</caption><thead><tr><th>Teamsize</th><th>CR</th><th>HCR</th><th>HPCC</th><th>CGG</th><th>MSP</th><th>x̄<sub>jA</sub></th></tr></thead><tbody><tr><td>1</td><td>1734.09</td><td>1962.42</td><td>1740.37</td><td>1717.36</td><td>1704.36</td><td>1771.72</td></tr><tr><td>2</td><td>843.93</td><td>1146.27</td><td>791.20</td><td>845.49</td><td>930.04</td><td>911.39</td></tr><tr><td>4</td><td>433.38</td><td>652.84</td><td>434.11</td><td>451.70</td><td>476.92</td><td>489.79</td></tr><tr><td>6</td><td>367.11</td><td>506.90</td><td>377.73</td><td>348.46</td><td>381.97</td><td>396.44</td></tr><tr><td>8</td><td>271.70</td><td>442.39</td><td>361.62</td><td>288.72</td><td>253.19</td><td>323.53</td></tr><tr><td>12</td><td>287.14</td><td>412.65</td><td>352.79</td><td>265.47</td><td>183.74</td><td>300.36</td></tr><tr><td>x̄<sub>1,A</sub></td><td>656.23</td><td>853.91</td><td>676.30</td><td>652.87</td><td>655.04</td><td>x̄<sub>λ</sub> = 698.87</td></tr></tbody></table>

<table><caption>**Table 2. Numerical Results for Map B ($Idl_G$ values in seconds)**</caption><thead><tr><th>Teamsize</th><th>CR</th><th>HCR</th><th>HPCC</th><th>CGG</th><th>MSP</th><th>x̄<sub>jB</sub></th></tr></thead><tbody><tr><td>1</td><td>1315.79</td><td>1283.59</td><td>1235.67</td><td>1347.30</td><td>1401.80</td><td>1316.83</td></tr><tr><td>2</td><td>675.44</td><td>654.61</td><td>670.44</td><td>675.64</td><td>749.42</td><td>685.11</td></tr><tr><td>4</td><td>363.46</td><td>373.45</td><td>298.77</td><td>335.45</td><td>375.15</td><td>349.25</td></tr><tr><td>6</td><td>238.57</td><td>273.60</td><td>254.96</td><td>234.18</td><td>248.92</td><td>250.05</td></tr><tr><td>8</td><td>198.90</td><td>217.38</td><td>225.44</td><td>172.39</td><td>185.28</td><td>199.88</td></tr><tr><td>12</td><td>172.4</td><td>255.62</td><td>212.3</td><td>143.94</td><td>-</td><td>x̄<sub>12B</sub></td></tr><tr><td>x̄<sub>i,B</sub></td><td>494.09</td><td>509.71</td><td>482.93</td><td>484.82</td><td>x̄<sub>MSP,B</sub></td><td>x̄<sub>B</sub></td></tr></tbody></table>

<table><caption>**Table 3. Numerical Results for Map C ($Idl_G$ values in seconds)**</caption><thead><tr><th>Teamsize</th><th>CR</th><th>HCR</th><th>HPCC</th><th>CGG</th><th>MSP</th><th>x̄<sub>jC</sub></th></tr></thead><tbody><tr><td>1</td><td>715.30</td><td>714.23</td><td>737.93</td><td>767.25</td><td>766.41</td><td>740.23</td></tr><tr><td>2</td><td>353.06</td><td>351.15</td><td>358.45</td><td>385.09</td><td>423.60</td><td>374.27</td></tr><tr><td>4</td><td>193.30</td><td>186.59</td><td>188.03</td><td>200.53</td><td>209.82</td><td>195.65</td></tr><tr><td>6</td><td>141.68</td><td>138.64</td><td>135.74</td><td>142.94</td><td>148.09</td><td>141.42</td></tr><tr><td>8</td><td>104.00</td><td>108.45</td><td>118.75</td><td>113.71</td><td>95.22</td><td>108.03</td></tr><tr><td>12</td><td>101.82</td><td>105.64</td><td>118.36</td><td>94.35</td><td>-</td><td>x̄<sub>12C</sub></td></tr><tr><td>x̄<sub>i,C</sub></td><td>268.19</td><td>267.45</td><td>276.21</td><td>283.98</td><td>x̄<sub>MSP,C</sub></td><td>x̄<sub>C</sub></td></tr></tbody></table>

The ANOVA table, presented in Figure 1, illustrates the model used. The only factor that has no relevant significance is the algorithm-teamsize interaction. In this case, the null hypothesis is accepted:

$$F - \text{test}_{\text{Alg*TS}} = \frac{\text{MS}_{\text{Alg*TS}}}{\text{MSE}} = \frac{1840.7}{1112} = 1.66 < F_{20,38}(\alpha=0.05) < 1.85 \quad (2)$$

In fact, this is expectable as seen on the $Idl_G$ values of each of the tables, which do not differ much when each of the associated columns are compared as a whole. The significance of the factor algorithm is also reduced; however, the null hypothesis cannot be accepted in this case. $Idl_G$ values vary deeply when teamsize changes which is the factor with more significance in the MRPP. After teamsize, the most significant factor is the connectivity of the environment as expected. Additionally, the interaction of these two factors is the most significant interaction. Figure 2 presents the approximately additive effect of interaction teamsize-algorithm and the non-additive effect of the interaction teamsize-connectivity is presented in Figure 3.
<figure><img src="image_2.png" /><figcaption>Fig. 1. ANOVA Table considering a model of three factors and first-order interactions</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. The factors algorithm and teamsize present an approximate additive effect, which is clear by the near parallel curves. In this chart, the environment used was map A</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3. The factors teamsize and connectivity present a clear non-additive effect. In this chart, the algorithm used was HCR</figcaption></figure>

A more complete model could eventually be obtained by considering second-order interaction of the three factors. However, this interaction is only responsible for the remaining 0.25% variation of the results and, therefore, it is ignored.

# 6 Extracting a Data-Related Model

As seen before, the ANOVA table in Figure 1 shows that the interaction between algorithm and teamsize has no relevant significance, which is an important property associated to the results obtained. Therefore we can build a simplified model which does not consider that interaction (as well as second-order interactions), accounting for 99.55% of the variation of the results. To describe such situations, the general expression of the model is [16]:

$$x_{ijk} = \mu_i + \alpha_j + \beta_j + \gamma_k + \delta_{jk} + \varepsilon_{ik} + E_{ijk}, \quad (3)$$

with the following restrictions:

$$\sum_{i} \alpha_{i} = 0; \quad \sum_{j} \beta_{j} = 0; \quad \sum_{k} \gamma_{k} = 0; \quad \sum_{j} \delta_{jk} = 0; \quad \sum_{i} \varepsilon_{ik} = 0, \quad (4)$$

and:

$$\begin{align*} i &= \{\text{CR, HCR, HPCC, CGG, MSP}\}, \\ j &= \{1, 2, 4, 6, 8, 12\}, \\ k &= \{\text{A, B, C}\}. \end{align*} \quad (5)$$
$\mu$ represents a global parameter, while the effects of algorithm, teamsize and connectivity are described by $\alpha_i$, $\beta_j$ and $\gamma_k$ respectively. $\delta_{jk}$ represents the effect of the teamsize-connectivity interaction and $\varepsilon_{ik}$ describes the effect of the algorithm-connectivity interaction. $E_{ijk}$ is the error and $E_{ijk} \sim N(0, \sigma^2)$. The parameters of the model can be estimated efficiently via the least squares methods. Thus, we obtain:

$$ 
\begin{aligned}
\hat{\mu} &= \overline{x_{...}} = \frac{1}{IJK} \sum_i \sum_j \sum_k x_{ijk} \\
\hat{\alpha}_i &= \overline{x_{i...}} - \hat{\mu} = \frac{1}{JK} \sum_j \sum_k x_{ijk} - \overline{x_{...}} \\
\hat{\beta}_j &= \overline{x_{..j}} - \hat{\mu} = \frac{1}{IK} \sum_i \sum_k x_{ijk} - \overline{x_{...}} \\
\hat{\gamma}_k &= \overline{x_{..k}} - \hat{\mu} = \frac{1}{IJ} \sum_i \sum_j x_{ijk} - \overline{x_{...}} \\
\hat{\delta}_{jk} &= \overline{x_{..jk}} - \overline{x_{..j}} - \overline{x_{..k}} + \overline{x_{...}} \\
\hat{\varepsilon}_{ik} &= \overline{x_{i..k}} - \overline{x_{i...}} - \overline{x_{..k}} + \overline{x_{...}}.
\end{aligned}
\qquad (6) $$

As seen in Tables II and III, the $Idl_G$ values for a team of 12 robots using the MSP algorithm are not available due to the nature of the algorithm, which is not able to partition those particular environments in 12 regions. In order to have a coherent model and calculate the average values ($\overline{x}_{MSP,B}$, $\overline{x}_{MSP,C}$, $\overline{x}_{12B}$ and $\overline{x}_{12C}$) and large averages ($\overline{x}_B$ and $\overline{x}_C$) left open in Tables 2 and 3, a curve fitting method by means of a general power function of the type $a \cdot x^b$ was used, as shown in Figures 4 and 5, to estimate the result for a team of 12 robots in those environments based on the data obtained with the same algorithm with lower teamsizes. The general power function model was applied because of the trend shown by the data as the teamsize grows when using the algorithm in the given environment, as well as its non-negative constraint, which completely suits the problem. The estimations obtained were:

$$ x_{MSP,12,B} = 133.45 \quad (\text{with } R^2: 0.9995) $$

$$ x_{MSP,12,C} = 77.26 \quad (\text{with } R^2: 0.9994). $$

Note that $R^2$ (or R-square) is a fraction between 0.0 and 1.0, which has no units and quantifies goodness of fit. Higher values indicate that the model fits the data better. Considering these values we obtain the following results for the values that were left open and present the results for the other large averages related to the algorithms and teamsize factors ($\overline{x}_A$ and $\overline{x}_B$) extracted from the three tables:

$$ 
\begin{aligned}
\overline{x}_{MSP,B} &= 515.67 & \overline{x}_{CR} &= 472.84 & \overline{x}_A &= 1276.26 \\
\overline{x}_{MSP,C} &= 286.73 & \overline{x}_{HCR} &= 543.69 & \overline{x}_B &= 656.92 \\
\overline{x}_{12B} &= 183.54 & \overline{x}_{HPCC} &= 478.48 & \overline{x}_A &= 344.90 \\
\overline{x}_{12C} &= 99.49 & \overline{x}_{COG} &= 473.89 & \overline{x}_B &= 262.63 \\
\overline{x}_A &= 497.44 & \overline{x}_{MSP} &= 485.81 & \overline{x}_A &= 210.48 \\
\overline{x}_B &= 276.51 & & & \overline{x}_{12} &= 194.46
\end{aligned}
$$
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Estimation of the $Idl_G$ value for a teamsize of 12 robots using MSP algorithm in map B. A General Power Model of type $ax^b$, with $a = 1408$ and $b = -0.9482$ was used to fit the curve to the 5 points previously obtained with the same algorithm and lower teamsize</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Estimation of the $Idl_G$ value for a teamsize of 12 robots using MSP algorithm in map C. A General Power Model of type $ax^b$, with $a = 772.2$ and $b = -0.9264$ was used to fit the curve to the 5 points previously obtained with the same algorithm and lower teamsize</figcaption></figure>

Having all data available, it is now possible to compute the parameters of the model using (6). Due to space limitations, the numerical results obtained are omitted. The model's parameters help us to understand the values of each $x_{ijk}$, clarifying the sources of variability and identifying the main factors that explain the results obtained.

# 6 Conclusions and Future Work

In this work, an extensive quantitative analysis of results obtained using multi-robot patrolling strategies with different teamsizes and environments was described. Analysis of Variance (ANOVA) was used to understand the significance of the different factors involved in the problem and a data-related model was calculated to explain the results.

The results presented were somehow unforeseen, given that the patrolling algorithms proved to be much less significant as a factor than teamsize and connectivity of the environment, which represents an important conclusion to this field. Therefore, future work should focus mostly on approaches that ensure scalability and are appropriate, or perhaps can adapt, to all kinds of environment.

**Acknowledgments.** This work was financially supported by a PhD grant (SFRH/BD/64426/2009) from the Portuguese Foundation for Science and Technology (FCT) and the Institute of Systems and Robotics (ISR-Coimbra) also under regular funding by FCT. The authors gratefully acknowledge António Portugal (Dept. Chemical Engineering, University of Coimbra), for his contribution and feedback.
References

1. D. Portugal and R. P. Rocha, "On the Performance and Scalability of Multi-Robot Patrolling Algorithms". In Proc. of the 2011 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR'2011), pp. 50-55, Kyoto, Japan, November 1-5, 2011.

2. ROS.org: Stage stack. Available at: http://www.ros.org/wiki/stage

3. A. Machado, G. Ramalho, J. Zucker and A. Drogoul, "Multi-Agent Patrolling: an Empirical Analysis of Alternative Architectures". Multi-Agent-Based Simulation, 3rd Int. Workshop, pp. 155-170, Italy, July, 2002.

4. A. Almeida, G. Ramalho, H. Sanana, P. Tedesco, T. Menezes, V. Corruble and Y. Chaveleyre, "Recent Advances on Multi-Agent Patrolling", In *Brazilian Symposium on Artificial Intelligence (SBIA 2004)*, pp. 474-483, Vol. 3171, São Luís, Brazil, 2004.

5. D. Portugal and R. P. Rocha, "MSP Algorithm: Multi-Robot Patrolling based on Territory Allocation using Balanced Graph Partitioning", In Proc. of the 25th *ACM Symposium on Applied Computing (SAC 2010)*, pp. 1271-1276, Sierre, Switzerland, March 22-26, 2010.

6. Y. Chevaleyre, "Theoretical Analysis of the Multi-agent Patrolling Problem", In Proc. of the Intelligent Agent Technology (IAT '04), IEEE/WIC/ACM Int. Conf., pp. 302-308, Beijing, China, Sep. 20-24, 2004.

7. P. Fazli, A. Davoodi, P. Pasquier, and A. K. Mackworth, "Complete and Robust Cooperative Robot Area Coverage with Limited Range". In Proc. of the IEEE/RSJ Int. Conference on Intelligent Robots and Systems (IROS 2010), Taipei, Taiwan, 2010.

8. F. Sempé and A. Drogoul, "Adaptive Patrol for a Group of Robots". In Proc. of the Int. Conf. on Intelligent Robots and Systems (IROS 2003), Las Vegas, Nevada, October, 2003.

9. T. Menezes, P. Tedesco and G. Ramalho, "Negotiator Agents for the Patrolling Task". In Proc. of the Int. Conf. IBERAMIA/SBIA, pp. 48-57, Ribeirão Preto, São Paulo, 2006.

10. H. Chu, A. Glad, O. Simonin, F. Sempé, A. Drogoul and F. Charpillet, "Swarm Approaches for the Patrolling Problem, Information Propagation vs. Pheromone Evaporation". In Proc. of the International Conference on Tools with Art. Intelligence, pp. 442-449, France, 2007.

11. Y. Guo, L. Parker, R. Madhavan, "Collaborative Robots for Infrastructure Security Applications". In *Studies in Computational Intelligence (SCI)*, vol. 50, pp. 185-200. Springer, Heidelberg, April 22, 2007.

12. N. Basilico, N. Gatti, T. Rossi, S. Ceppi and F. Amigoni, "Extending Algorithms for Mobile Robot Patrolling in the Presence of Adversaries to More Realistic Settings". In Proc. of WI- IAT 2009 IEEE/WIC/ACM Int. Joint Conference on Web Intelligence and Intelligent Agent Technology, IEEE Press, Vol. 02, pp. 557-564, Milan, Italy, September 15-18 2009.

13. D. Portugal and R. P. Rocha, "A Survey on Multi-Robot Patrolling Algorithms". In Proc. of the 2nd Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS'11), pp. 139-146, Costa da Caparica, Lisbon, Portugal, February 21-23, 2011.

14. H. Scheffé, *The Analysis of Variance*, John Wiley & Sons, New York, 1959.

15. M. Krieger, J. Billeter and L. Keller, "Ant-like task allocation and recruitment in cooperative robots". *Nature*, Macmillan Magazines Ltd, Vol. 406, pp. 992-995, 2000.

16. R.C Guimarães and J. A. Sarsfield Cabral, *Estatística*, McGraw-Hill, 1997.

17. M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler and A. Ng, "ROS: an open-source Robot Operating System". In Proceedings of the 2009 IEEE International Conference on Robotics and Automation (ICRA'2009), Workshop On Open Source Software, Kobe, Japan, May 12-17, 2009.