# Model-driven design space exploration for multi-robot systems in simulation

James Harbin, Simos Gerasimou, Nicholas Matragkas, Thanos Zolotas, Radu Calinescu, Misael Alpizar Santana

► To cite this version:

James Harbin, Simos Gerasimou, Nicholas Matragkas, Thanos Zolotas, Radu Calinescu, et al. Model-driven design space exploration for multi-robot systems in simulation. Software and Systems Modeling, 2022, 22, pp.1665-1668. 10.1007/s10270-022-01041-w. cea-04122525

HAL Id: cea-04122525

https://cea.hal.science/cea-04122525

Submitted on 1 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Model-driven design space exploration for multi-robot systems in simulation

James Harbin<fnref n="1" /> Simos Gerasimou<fnref n="1" /> Nicholas Matragkas<fnref n="2" /> Thanos Zolotas<fnref n="3" /> Radu Calinescu<fnref n="1" /> Misael Alpizar Santana<fnref n="1" />

Received: 16 April 2022 / Revised: 9 August 2022 / Accepted: 6 September 2022 / Published online: 8 October 2022
© The Author(s) 2022

## Abstract

Multi-robot systems are increasingly deployed to provide services and accomplish missions whose complexity or cost is too high for a single robot to achieve on its own. Although multi-robot systems offer increased reliability via redundancy and enable the execution of more challenging missions, engineering these systems is very complex. This complexity affects not only the architecture modelling of the robotic team but also the modelling and analysis of the collaborative intelligence enabling the team to complete its mission. Existing approaches for the development of multi-robot applications do not provide a systematic mechanism for capturing these aspects and assessing the robustness of multi-robot systems. We address this gap by introducing ATLAS, a novel model-driven approach supporting the systematic design space exploration and robustness analysis of multi-robot systems in simulation. The ATLAS domain-specific language enables modelling the architecture of the robotic team and its mission and facilitates the specification of the team’s intelligence. We evaluate ATLAS and demonstrate its effectiveness in three simulated case studies: a healthcare Turtlebot-based mission and two unmanned underwater vehicle missions developed using the Gazebo/ROS and MOOS-IvP robotic platforms, respectively.

**Keywords** MRS · Multi-robot systems · Model-driven engineering · MDE · Simulation · Design-space exploration

Communicated by Shiva Nejati and Daniel Varro.

B Simos Gerasimou
simos.gerasimou@york.ac.uk

James Harbin
james.harbin@york.ac.uk

Nicholas Matragkas
nikolaos.matragkas@cea.fr

Thanos Zolotas
a.zolotas@ljmu.ac.uk

Radu Calinescu
radu.calinescu@york.ac.uk

Misael Alpizar Santana
misael.alpizarsantana@york.ac.uk

## 1 Introduction

Multi-Robot Systems (MRS) are a class of systems in which distributed and interconnected robots are orchestrated to perform missions whose complexity and cost are too high for individual robots to accomplish on their own [1]. MRS play a strategic role in safety-critical and business-critical missions ranging from precision agriculture and fast delivery of medical samples to real-time road traffic monitoring and infrastructure inspection [2]. The intrinsic characteristics of these missions, i.e. distributed sensing and action, uncertain operating environment, and the need for endurance and robust behaviour, necessitate the use of MRS instead of single-robot solutions. MRS bring additional benefits including improved scalability and performance (missions can be performed more efficiently through parallelism if they are decomposable), mission enablement (use of collective intelligence to execute missions beyond the capabilities of individuals) and increased robustness and reliability through redundancy.

MRS can execute their missions through *collective intelligence (CI)* algorithms that encapsulate communication

<footnote n="1">¹ Department of Computer Science, University of York, York, UK</footnote>

<footnote n="2">² CEA, List, Université Paris-Saclay, 91120 Palaiseau, France</footnote>

<footnote n="3">³ School of Computer Science and Mathematics, Liverpool
John Moores University, Liverpool, UK</footnote>
policies within the closed-loop control of individual robots (e.g., MAPE-K [3]) and adaptation strategies for delegating responsibilities within the team. The use of CI enables capitalising on the unique benefits offered by MRS in these business and safety-critical application domains [4]. Consider, for instance, a team of UUVs, each equipped with sonar sensors, deployed to discover hazardous objects by scanning a large marine area. A CI instance may partition this area based on the UUV sensor capabilities (e.g., reliability, energy consumption) while also specifying how the team will respond and redistribute pending tasks when a team member's battery is exhausted, or when a team member fails or experiences difficulties (e.g., when a UUV enters an area where the water salinity or temperature is outside the operating envelope of its sensors).

Selecting a suitable CI is a very important problem that directly affects the MRS performance and resilience [1]. An effective CI would enable MRS to cope with uncertain environments, evolving mission objectives and unpredictable degradation of robotic components (e.g., sensor failure of multiple robots) [5]. Clearly, this is a non-trivial problem. Engineers have a multitude of options both in terms of adaptation strategies and communication policies that complicate the process of designing, implementing, debugging and assessing candidate CI algorithms. The large design space that comprises robotics teams of different sizes and individual robots with a wide range of performance and functionality characteristics (e.g., sensor width, energy capacity) only exacerbates the task of choosing the most suitable robotic team and desired CI instantiation.

Despite recent advances in the specification and analysis of MRS [6,7], existing approaches either focus on providing specialised robotic functionality (e.g., perception, control) [8–10] or software for specific robotic platforms (e.g., ROS [11] or MOOS-IvP [12]). This limits their applicability to MRS missions characterised by simplistic CI behaviour resulting in reduced MRS resilience or bespoke algorithms that intertwine the CI logic with low-level platform-specific code resulting in significantly higher than expected maintenance cost [13]. These important limitations of existing approaches increase significantly the effort to explore the tradeoff between candidate MRS designs [14] and adaptation strategies of different CI algorithms [15].

In this paper we introduce ATLAS, a model-driven, tool-supported framework for the systematic engineering of MRS that facilitates the exploration and tradeoff analysis of candidate MRS designs and CI algorithms. Driven by insights derived from recent robotics surveys [14,16], ATLAS is underpinned by the following principles. First, the ATLAS domain-specific language (DSL) enables the specification of (i) the MRS mission, including both functional and non-functional requirements; and (ii) the characteristics of individual robots comprising the MRS, including architecture,

internal behaviours, and capabilities (e.g., use of energy efficient or reliable sensors). Second, the ATLAS code generation engine consumes the MRS mission and system specifications, and produces the necessary infrastructure (i.e., ATLAS middleware, CI templates, target simulator logical interface) that enables the communication of the ATLAS components with the target robotic simulator (e.g., MOOS-IvP [12], ROS [11]). Finally, the low coupling between the ATLAS components supported by the CI templates improves system extensibility while also supporting tradeoff analysis between different CI algorithms.

The main contributions of this paper are:

* The ATLAS tool-supported framework for the systematic engineering of MRS enabling tradeoff analysis of different MRS architectures and CI algorithms from the early stages of the MRS development process;

* An extensive ATLAS evaluation of three MRS case studies built with ROS/Gazebo [11] and MOOS-IvP [12], widely used platforms for ground-based and UUV simulations, respectively.

* A prototype open-source ATLAS tool and case study repository, both available on our project webpage at https://www.github.com/jrharbin-york/ATLAS.

The design space exploration provided by ATLAS can be used to iteratively refine the system design as part of its systematic process. For example, ATLAS can show through tradeoff analysis that certain CI algorithms are not viable for the target robotic missions, or that some software components could lead to system configurations that do not satisfy the mission requirements. Therefore, ATLAS can support a cost-benefit analysis between CI choice and mission requirements. Although we do not attempt to trace the underlying cause of violations, visual inspection via the user interface provided by the simulator provides a mechanism for error tracing and performance analysis.

This paper extends significantly our previous work presented in [17] in the following ways: (1) the ATLAS framework is extended to fully support the ROS robotic platform and the Gazebo simulator, including considerable adaptation to the ATLAS DSL to facilitate the definition of MRS missions both for the MOOS-IvP and ROS/Gazebo robotic simulators; (2) new ATLAS components have been implemented enabling the automatic search and selection of robotic system configurations based on mission goals and model similarity leveraging robust model hashing [18]; (3) we adapt a ROS-based case study (developed by the project team as part of a related research project) that involves a team of mobile robots servicing patients within a healthcare facility and extended it with ATLAS -specific constructs and two complementary CI algorithms; (4) we evaluate the ATLAS framework on this new case study using three inter-
<figure><img src="image_2.png" /><figcaption>**Fig. 1** Healthcare case study example showing three robots in operation visiting rooms</figcaption></figure>

dependent research questions; and (5) we perform a more
thorough analysis of related work.

The paper is structured as follows. Section 2 presents the new case study from the healthcare that we use to illustrate ATLAS, which is detailed in Sect. 3. Sections 4 and 5 present our ATLAS implementation and evaluation, respectively. Finally, Sect. 6 discusses related work, and Sect. 7 summarises our results and suggests directions for future research.

## 2 Healthcare robotic team

We will illustrate our ATLAS approach using a team of multi-purpose mobile healthcare robots that collaborate to execute service tasks within a hospital facility. Healthcare robots are increasingly deployed in the clinical domain to carry out routine activities, supporting the often overwhelmed healthcare professionals (i.e., nurses, doctors and other medical staff) in their demanding and fast-paced working environment [19,20]. The robots can typically perform simple tasks including the delivery of medical materials to care units, transport of medical samples to the lab for analysis, and disinfection of patient rooms and operating suites [21]. For instance, using robots capable of disinfecting hospital environments by emitting concentrated Ultraviolet-C (UV-C) light is investigated within the context of the European Research Council funded project SESAME (https://www.sesame-project.org).

Figure 1 shows three Turtlebot 3 Burger robots<fnref n="1" /> operating within a simulated healthcare facility comprising a total of 20 rooms (with each room located between the vertical walls and either side of the long horizontal corridor). Each room may host a single patient or may be empty. When visiting an occupied room, a robot can check whether the patient needs assistance, measure their vital parameters or simply interact with the patient. To support localisation and navigation, each robot is equipped with laser distance and inertia measurement unit sensors (e.g., gyroscope, accelerometer), and a transceiver communicating with a centralised command-and-

<table><caption>**Table 1** Healthcare robots mission requirements</caption><thead><tr><td>ID</td><td>Description</td></tr></thead><tbody><tr><td>R1</td><td>The robots must service all their assigned rooms.</td></tr><tr><td>R2</td><td>Each robot must return to its starting location for recharging before its battery is depleted and become stranded in the hospital environment.</td></tr><tr><td>R3</td><td>The robots must fulfil requirements R1 and R2 in the shortest possible time and before T<sub>end</sub> (the maximum mission end time).</td></tr></tbody></table>

control computer centre which runs the CI and coordinates the activities of the robotic team. Figure 1a depicts the simulation environment in Gazebo, whereas Fig. 1b shows the rviz visualisation capturing also the localisation area and navigation path per robot (i.e., showing that two robots are moving towards entering the rooms at the bottom left, with the last robot moving to its home location at the left of the hospital corridor).

The CI employed by the command-and-control centre organises the robotic team by allocating the rooms that must be visited per robot so that the requirements in Table 1 are satisfied. To this end, an effective CI should partition the responsibilities among the robotic team so that all assignable rooms are serviced successfully (R1), enabling robots to return to their starting points when their assigned work is completed. Since the robots have limited energy capacity, the priority is to service as many rooms as possible before their batteries are exhausted (R2) in the shortest possible time, and before the maximum mission execution time $T_{end} \le 35s$ (R3). The energy consumption model uses information computed from the Turtlebot3 specification and comprises the following causes of energy loss: (i) energy required by robotic electronic components like processor and sensors per time unit during mission execution (11 J/s); (ii) energy consumed by vehicle motors for propelling the robot (40 J/m); and (iii) average energy per room to carry out the service task (18kJ/room). Once allocation is made, each robot visits their allocated rooms in sequence before returning to its starting location. Unallocated rooms will be serviced by healthcare professionals with higher overheads than those incurred when serviced by the robotic team.

<footnote n="1">¹ https://www.turtlebot.com.</footnote>
Engineering this robotic application entails considering the possible variability options that could influence the robots' behaviour, producing different system configurations. Accordingly, engineers may be interested in exploring whether the mission can be executed with a variable number of robots (e.g., two or three robots in this instance). While using all available robots can, in principle, help in completing the mission faster, it unavoidably increases the hardware maintenance costs and reduces the system's resilience, e.g., if two robots break down or deplete their battery at the same time they will compete for the repair and charging resources, respectively. Using fewer robots could also require a less demanding CI algorithm for controlling the room allocation and coordination between the robots. Furthermore, each robot can be equipped with a standard (71 kJ) or high capacity (220 kJ) battery, with the more powerful battery being significantly more expensive and requiring extra charging time. Thus, system configurations that can complete the mission with at least one robot using the standard battery are preferred. Finally, selecting the subset of occupied rooms that the robotic team can service best is another variability dimension with the number of possible room combinations given by $\sum_{i=1}^{n} \binom{n}{i} \cdot i!$, where or and i are the occupied rooms and the number of rooms to be allocated to robots, respectively. As we explain later (Sect. 3.2) and to preserve the tractability of simulation analysis, we arrange the rooms into three groups of six rooms each. Collectively, these variability options produce 60 possible mission configurations (i.e., designs) for the robotic team.

In addition to those robotic team configurations, the command-and-control centre could employ different CI algorithms to coordinate the robotic team. For this mission, robotic engineers are interested in evaluating the following standard and advanced CI algorithms.

- *Standard CI:* This CI assigns rooms to robots following a round-robin manner considering also the distance between the room and the robot. Thus, this CI is easy to implement and requires minimal system knowledge, i.e., knowing only the location of each robot and the rooms. Despite its simplicity, this CI ignores the battery capacity per robot and may underperform when the work distribution is too much for robots with a standard battery.

- *Energy-aware CI:* Room allocation to robots using this CI is proportional to their energy levels. Thus, a robot with double the battery capacity will be allocated twice as many rooms to service (although this may not be exact in cases with small numbers of rooms). The energy-aware CI also dynamically reallocates work from a robot with a battery close to depletion to other robots, proportional to their remaining capacity. This feature can improve performance, but involves additional implementation complexity, including extra development and debugging effort, together with communication overheads to reassign work dynamically during the mission.

<figure><img src="image_3.png" /><figcaption>Fig. 2 High-level ATLAS architecture</figcaption></figure>

# 3 ATLAS

## 3.1 Overview

The high-level ATLAS architecture is depicted in Fig. 2. At its core, lies the ATLAS DSL (Sect. 3.2) that enables the specification of the robotic's team structure and mission objectives, complemented by a model-driven code generation engine for the specialisation of a middleware (Sect. 3.4), a simulator-specific logical interface, and CI templates that facilitate tradeoff analysis of CI algorithms (Sect. 3.5).

The middleware facilitates the communication between MRS team members and between the collective intelligence algorithms. These algorithms implement the high-level command and control which is responsible for the completion and adequate performance of the mission. For instance, for our healthcare use case from Sect. 2, the CI is responsible for issuing the initial tasks to be performed by each robot or for reassigning tasks when a team member has a low battery level.

The low-level logic of robotic team members is implemented using MRS implementation-specific functionality. Low-level behaviours, including station keeping, inter-robot avoidance and path navigation, are mediated by state held in the target robotic platform (e.g., in publish-subscribe databases). Therefore, although Fig. 2 illustrates the high-level communication, the yellow MRS nodes on the right contain additional state and communication flows which are specific to the target robotic platform.

This key ATLAS feature of using a middleware to coordinate the high-level CI command and control exploits the modular architecture of modern robotic systems and leverages low-level support from the publish-subscribe communication protocol underpinning widely used robotic platforms including ROS [11] and MOOS-IvP [12].
Furthermore, the middleware enhances separation of concerns between the implementation of the robotic team and the devised CI algorithms which are responsible for steering the team to achieve its mission. This is another key feature of ATLAS that supports the investigation of multiple, and likely complementary, CI algorithms [15] (e.g., decentralised collective learning, leader-election algorithms) for carrying out the specified mission and comparing their non-functional attributes such as performance, reliability, and scalability.

The simulator-specific logical interface facilitates the communication between the middleware and the target robotic simulator, enabling not only to receive and transmit data but also to monitor the robotic team state during mission execution. Accordingly, this interface reduces the coupling between framework components and enhances extensibility. Furthermore, the interface provides a reusable extension point whose specialisation enables to connect and experiment with different robotic simulators with modest effort. These key ATLAS features contribute in alleviating some of the key challenges identified in recent robotics surveys for developing, debugging and maintaining robotic applications [14,16]. The following sections provide details of the main components underpinning ATLAS, delineating also the fully fledged ATLAS instances for both the ROS and MOOS-IvP robotic simulators, providing evidence for the ATLAS generality.

Figure 3 shows the high-level workflow of the ATLAS methodology. In Step 1, domain experts can use the robotic mission informed ATLAS DSL to produce models for their target MRS mission including requirements, mission goals, safety invariants, and the composition of the team. We note that, in principle, domain experts and system testers could employ other formalisms than the ATLAS DSL to represent these dimensions of their robotic mission and architecture. However, the following steps, including code generation and middleware interfacing, are fairly integrated with this specification. As expected, a corresponding set of model-to-model transformation activities must take place to produce the required artefacts. Since this activity is not within the scope of this work, we leave it for future research.

The invocation of the ATLAS code generation engine in Step 2 uses the full information in the supplied models to automatically generate (i) a middleware configuration that enables the communication of the various system components; (ii) CI templates that will be used by the middleware to orchestrate the robotic team; (iii) interface code that enables the middleware to communicate directly with the target robotic simulator and extract the required information from the simulators; and (iv) the necessary configuration files for the target robotic simulator. In our prototype implementation, these transformations are performed using the Epsilon Generation Language (EGL) [22].

<figure><img src="image_4.png" /><figcaption>Fig. 3 Stepwise MDE ATLAS methodology</figcaption></figure>

Next, in Step 3, engineers implement the logic and coordination of the robotic team. To achieve this, they populate the CI templates generated in Step 2 with suitable code that realises their selected CI algorithms. This separation of concerns between the implementation of the high-level behaviour (demonstrated by the CI algorithm) and the low-level functionality of a robot has two key benefits. First, it reduces the effort to analyse multiple CI algorithms without changing the underlying low-level system behaviour. Second, it enables reuse of already available low-level code developed for individual robotic team members that has been developed independently. Our current ATLAS implementation supports the generation of CI templates in Java. Further details about the CI implementation, alternative implementation strategies and choices for development language are presented in Sect. 3.5.

Once the necessary artefacts have been instantiated, the MRS simulation is automatically executed in Step 4. ATLAS records simulation results related to the MRS mission requirements in the form of logs comprising both messages exchanged between system components and events relevant to system goals and requirements that occurred during the simulation. The analysis of these results permits the computation of metrics that support the selection of the most effective MRS designs and the identification of the most suitable CI algorithm for the target mission.
<figure><img src="image_5.png" /><figcaption>**Fig. 4** Excerpt of the Mission-related part of the ATLAS metamodel</figcaption></figure>

## 3.2 DSL core concepts

The primary concepts of the ATLAS DSL consist of the *Mission* (Fig. 4) and the *VariationProgram* (Fig. 5). The *Mission* part includes the *Goals* that should be accomplished. Each goal is connected to the *Region* where it takes place and includes the specific *GoalAction* that should be executed for the fulfilment of the goal. Example actions include *Patrolling* an area, *Avoiding* an obstacle, etc. Although the DSL covers the vast majority of actions available in the current ROS and MOOS robotic environments, this is an extensibility point of the DSL. Accordingly, interested users can extend the *GoalAction* type to introduce new actions and the appropriate model-to-text (M2T) transformations to generate the corresponding robotic simulator specific implementations, which may be required for new use cases. For example, when testing a robotic monitoring scenario to discover potential collisions between robot arms and humans, the user could extend the DSL by implementing a new *GoalAction* subclass, e.g., *Arm-CollisionTracker*. The logic of this goal action would register potential collisions either directly supplied by the robotic simulator (if this information is available and ready to use) or 3D geometry information to be analysed post hoc after the simulation completes its execution.

Since goals may have their own constraints, (e.g., starting after or ending before a specific time), each goal is linked to a *GoalTemporalConstraint* defining the earliest starting and latest finish time of the goal. Each *GoalAction* is responsible for tracking the fulfilled/violated status of a particular high-level mission goal. As these are scenario-specific, the implementation of the action per goal is delegated to the user in Step 3 of the ATLAS methodology (cf. Fig. 3). For example, the *GoalAction* named *TrackDistances* will track the relative distances of robots to each other and check for possible intersection with environmental objects. This information is recorded by the middleware allowing the generation of related metrics for further analysis in Step 5.

Robotic simulators provide bespoke implementations of various behaviours such as navigating to specific coordinates (e.g., using the *MoveBaseGoal* and *WayPoint* behaviours in ROS and MOOS-IvP, respectively), and avoiding other robots and obstacles. These behaviours can be employed by low-level simulator code to facilitate the specification and execution of a robotic mission. This information is also important for the CI as it enables to dynamically alter the behaviour of a robot. In our healthcare robotic team, for instance, a conservative energy-aware CI can exploit this ROS construct to dynamically instruct the robot to return to its base for recharging when its battery level drops below a critical threshold, thus reducing the risk that the robot will get stuck within the healthcare facility and obstruct other robots or healthcare professionals.

In addition to the specification of high-level behaviours, the DSL supports the specification of *SimulatorVariables* that are relevant for the mission investigated and, therefore, need custom handling by the middleware and the CI algorithm. In ROS, for instance, a simulator variable is represented by the topic in a ROS computational graph instance that delivers this information to the middleware. More specifically, a *SimulatorVariable* is defined by its name, its type (e.g., for ROS, this is a string giving the ROS type name), a tag indicating the high-level nature of the variable (velocity, position, time, generic), and two boolean fields defining whether the variable is specific to a particular robot and if it should be propagated to the CI instance. The purpose of including these simulator variables in the DSL is to ensure the middleware subscribes to the variables and to enable the propagation of any updates to the CI. If the *propagate-ToCI* field is set to true, when an update occurs, the CI will be notified about the updated information and can respond appropriately to a particular scenario-specific MRS event, e.g., terminate the mission when notified about the successful traversal of a set of waypoints or raise an alarm when a robot is blocked within the healthcare facility because its battery has been exhausted. Multiple simulator variables can be linked within a behaviour and connected to high-level mission goals through the *behaviours* reference. In other words, the *Behaviour* class also enables specifying multiple simulator variables (topics in ROS) from the MRS simulator that capture the status of a robot.

Another important element of the ATLAS DSL is the set of *Robots* employed to satisfy a specific goal. Conforming to the hierarchical representation of many robotic systems [10], ATLAS enables the specification of the MRS architecture in a compositional manner (Fig. 5). A *Component* represents the top level element of this hierarchical representation. Components are divided into *SystemComponent* and *EnvironmentalComponent*. A *SystemComponent* can either be a *Robot* (e.g., the mobile robot in the healthcare robotic team from Sect. 2) or a *Computer* (e.g., the command-and-
<figure><img src="image_6.png" /><figcaption>**Fig. 5** Excerpt of the Component-related part of the ATLAS meta-model</figcaption></figure>

control centre that is responsible for the execution of the CI algorithm and the coordination of the robots). Robots and Computers comprise several *subcomponents* (e.g., *Sensors*, *Actuators*, *MotionSources*, etc.). Also, each component contains *ComponentProperties* of different datatypes that enable the specification of different characteristics of the associated component (e.g., nominal operating rate of a sensor or expected energy consumption). An *EnvironmentalComponent* permits modelling elements in the environment via the component concept. For example, there could be numerous objects in the environment, including rooms to be serviced in the healthcare case study, and target objects to be located in a marine detection case study (cf. Sect. 5). These environmental objects can be represented and grouped in clusters through the *EnvironmentalObjectGroup* class which is a subclass of the *EnvironmentalComponent*.

**Example 1** Figure 6 shows the model instance for the healthcare mission from Sect. 2. The model comprises the specifications of the three Turtlebots that are available to carry out the patient inspection tasks within the healthcare facility. The model includes also the goals that should be executed to assess the compliance with the case study requirements, i.e., track the distances travelled by the robots, check the completion tasks in the allocated rooms, and compute the total energy consumption by the robotic team. The two goals *checkRoomsCompleted* and *trackEnergyHealthcare* are custom implementations for the healthcare case study. Several simulator variables are also defined which allow ATLAS to monitor the completion status of rooms and the assigned work of robots. The */clock* topic defines the simulator operating time, while the */amcl_poses* topic defines the pose for each robot, with the *isRobotSpecific* field set to true indicating that a different subscription should be made for each of the three robots. Any changes to these variables will be monitored by the middleware and communicated to the CI. The configuration also includes the rooms as environmental

<figure><img src="image_7.png" /><figcaption>**Fig. 6** Model of the healthcare robotic mission from Sect. 2</figcaption></figure>

objects in the form of three groups (i.e., group 1: rooms 1–6; group 2: rooms 7–12, group 3: rooms 13–18).

## 3.3 Searching for optimal robot configurations

Another key feature of the ATLAS DSL is the support for analysing candidate system configurations through design space exploration. To achieve this, ATLAS leverages concepts from the domain of software product line engineering [23–25], enabling the specification of alternative candidate designs as a *VariationGroup* with a defined cardinality (see Fig. 5). The members of a variation group are components, at the same level of the system specification hierarchy, that can participate in the mission. Accordingly, variation groups can be formed by *SystemComponent* subclasses such as robots and their subcomponents. For example, if there are different types of *Sensor* or *Battery* that a robot can use for a mission, those subcomponents can form another variation group. Environmental features that are subclasses of *EnvironmentalComponents* can form a variation group too. The set of all variation groups forms a *VariationProgram*. Given such a program, ATLAS uses model-to-model (M2M) transformation to automatically generate informative mission models that conform to the mission metamodel in Fig. 4 and meet the *min/maxRequired* properties of each group. For example, consider the healthcare example from Sect. 2 and assume we want to assess if the mission can be fulfilled using a subset of the available robots. The robots are added in the *Mission* model and the *VariationGroup* referring
to these robots is created. The minRequired and maxRequired properties are set to two and three, respectively. ATLAS consumes the variation groups and through M2M transformation automatically generates the possible mission model configurations (designs). Since a VariationGroup directly references components from the model specification of the target robotic team, metamodel enhancements, e.g., adding other system components or environmental elements, can be handled in the same manner and require no adaptation to the logic underpinning the search for optimal robot configurations.

The large number of generated mission model configurations and the, typically, considerable time needed to execute a single configuration incur substantial computational overheads that, unavoidably, render prohibitive the exhaustive analysis of all possible configurations [26]. ATLAS aims at alleviating this issue and mitigating the impact of running simulations for different configurations through informed selection of the most promising configurations for analysis. To this end, ATLAS leverages recent MDE advances in model similarity estimation through robust hashing [18] and uses simulated annealing [27] to guide the selection of the most promising and informative model configurations. Robust hashing of MDE models [18] builds on techniques developed for the comparison of large datasets and enables the estimation of model similarity considering both the contents of model elements (e.g., attributes, operations) and their relative position with regards to other model elements. Given the set of models to compare, we use the robust hashing technique to produce a similarity matrix between these models. The similarity score between a pair of models ranges from 0 (completely dissimilar models) to 100 (identical models). For further details, we refer interested readers to [18].

The ATLAS search process of the configuration space is underpinned by the premise that candidate models which are *more similar* to a model that has been analysed (and yielded good results) are more likely to produce similar results than other candidate models which are *less similar* to the analysed model [28–30]. Accordingly, simulated annealing, as a probabilistic global search optimisation technique [27], offers a good tradeoff between searching the local area of the configuration space (until the optimal configuration is identified) and escaping local optimum configurations (by temporarily accepting configurations that yield worse results than the currently best configuration). This optimisation algorithm has shown promising results in supporting design space exploration in related research [31]. Nevertheless, we note that other heuristics and meta-heuristics such as evolutionary algorithms and swarm intelligence [32] could also be applied to solve this optimisation problem [33]. Investigating the applicability and tradeoffs between different global optimisation algorithms presents an interesting idea for future work.

**Algorithm 1** ATLAS intelligent search process

1: **function** SearchModels(*M*, *I*, *G*, *N*, *T*, *C*)
   *M*: candidate models set; *I*: max iterations;
   *G*: mission goals; *N*: neighbourhood factor;
   *T*: initial temperature; *C*: cooling rate
2:   *m*<sub>*r*</sub> ← SelectT(*M*)
3:   *V*<sub>*m*<sub>*r*</sub></sub> ← EvaluaTe(*m*<sub>*r*</sub>, *G*)
4:   *m̂* ← *m*<sub>*r*</sub>, *V*<sub>*m*̂</sub> ← *V*<sub>*m*<sub>*r*</sub></sub>
5:   *SM*<sub>*M*</sub> ← RobusTHashing(*M*)
6:   **for** (*i* = 1; *i* ≤ *I*; *i*++)<b> do</b>
7:     *M*<sup>*N*</sup><sub>*m*<sub>*r*</sub></sub> ← GeTSimilar(*SM*<sub>*M*</sub>, *m*<sub>*r*</sub>, *i*, *I*, *N*)
8:     *m*<sub>*i*</sub> ← SelectT(*M*<sup>*N*</sup><sub>*m*<sub>*i*</sub></sub>)
9:     *V*<sub>*m*<sub>*i*</sub></sub> ← EvaluaTe(*m*<sub>*i*</sub>, *G*)
10:    **if** IsBeTTer(*V*<sub>*m*<sub>*i*</sub></sub>, *V*<sub>*m*̂</sub>) **then**
11:     *m̂* ← *m*<sub>*i*</sub>
12:     *V*<sub>*m*̂</sub> ← *V*<sub>*m*<sub>*i*</sub></sub>
13:     **end if**
14:     *m*<sub>*r*</sub>, *V*<sub>*m*<sub>*r*</sub></sub> ← SimAnneal(*m*<sub>*i*</sub>, *V*<sub>*m*<sub>*i*</sub></sub>, *m*<sub>*r*</sub>, *V*<sub>*m*<sub>*r*</sub></sub>, *T*, *C*)
15: **end for**
16: **return** *m*̂, *V*<sub>*m*̂</sub>
17: **end function**
D [18]

Algorithm 1 shows the high-level ATLAS search process to intelligently find good configurations. The algorithm requires as inputs the set of candidate models *M*, the maximum number of iterations *I*, the mission goals *G*, the neighbourhood factor *N* and the simulated annealing specific parameters initial temperature *T* and cooling rate *C*. Initially, the SearchModels function randomly selects a reference candidate model *m*<sub>*r*</sub> (line 2) and evaluates its goodness based on the mission goals *G* (line 3), saving the derived model and evaluation information as the current best (line 4). Then, through the robust hashing approach described in [18], we construct a similarity matrix *SM*<sub>*M*</sub> of size *M*<sup>2</sup> for the model set *M*, with each model in columns ordered in decreasing similarity to its source model on each row. The loop in lines 6–15 is executed for *I* iterations during which more promising models are selected for evaluation. To this end, the GeTSimilar function (line 7) retrieves the most similar models to the reference model *m*<sub>*r*</sub> considering the current search status. Thus, the current iteration *i* combined with the maximum number of iterations *I* and the neighbourhood factor *N* control the trade-off between exploration and exploitation, enabling at the beginning of the search to consider all models as candidates, irrespective of their similarity to *m*<sub>*r*</sub> (favouring exploration), and progressively shrinking the neighbourhood to models that are more similar to *m*<sub>*r*</sub> (favouring exploitation). Based on the resulting set of similar models *M*<sup>*N*</sup><sub>*m*<sub>*r*</sub></sub>, a model is randomly selected and evaluated (lines 8–9), assigning it at the best model *m̂* if it yields better results than the current best (lines 10–13). Next, the algorithm executes the standard simulated annealing process where the reference model *m*<sub>*r*</sub> is updated if the currently evaluated model *m*<sub>*i*</sub> is better, i.e., the IsBeTTer(*V*<sub>*m*<sub>*i*</sub></sub>, *V*<sub>*m*<sub>*r*</sub></sub>) function holds, or accepts a seemingly worse solution to avoid poten-
<figure><img src="image_8.png" /><figcaption>**Fig. 7** Variation model for the healthcare mission from Sect. 2</figcaption></figure>

tial local optimum (line 14). The latter case is a key feature of simulated annealing and the probability of accepting a worse solution is controlled by the simulated annealing hyperparameters temperature *T* and cooling rate *C* [27]. Once the algorithm terminates, the best model found $\dot{m}$ and its result $V_m$ is returned (line 16). For the healthcare case study, for instance, the best model corresponds to a system design under which the robotic team satisfies requirements R1 (the robots service all their assigned rooms) and R2 (the robots return to their base to recharge), and optimises requirement R3 (the team completes its mission in the shortest possible time).

**Example 2** Figure 7 shows the variation model (irrelevant details are omitted for reasons of brevity) for the variation scenario of the healthcare case study from Sect. 2. The execution of the M2M using as input this variation model will generate 60 possible configurations that satisfy the constraints defined in the variation groups, i.e., each robot should use either the standard or high capacity battery and the mission should use at least two out of three available Turtlebots. In addition, two out of the three environmental object groups will always be chosen, giving at least 12 rooms to be serviced, but with three possible room selection choices, i.e., rooms (1–12), rooms (1–6, 13–18), or rooms (7–18).

## 3.4 ATLAS middleware

The middleware is a key component of ATLAS that enhances separation of concerns between the CI instances and the target robotic simulator. The CI receives information about the MRS status via the middleware, executes its logic, and relays back its decisions to the MRS via the middleware. Also, the middleware uses runtime monitoring to assess the status of the goals defined in the DSL. This internal state, including goal events, is stored in log files that allow the computation of relevant mission metrics and post hoc analysis. The middleware, CI components and the underlying MRS are all implemented as separate processes, which decouples them from each other, thus increasing debugging flexibility and facilitating their independent testing. Furthermore, the low coupling between the CI and the MRS simulator, mediated by the middleware, enables not only to experiment easily with several candidate CI algorithms but also reinforces maintainability and extensibility (e.g., the components can be computing platform and programming language independent).

Irrespective of the target robotic simulator, the ATLAS middleware comprises (i) a simulator-specific interface that enables ATLAS to connect to the simulator, and subscribe and publish messages/topics; (ii) a highly efficient message broker (e.g., ActiveMQ [34]) to facilitate fast inter-robot communication and interaction with the CI algorithm; and (iii) a CI mapping software module that converts the high-level CI commands into message/topic changes for the target simulator. ATLAS uses the mission model (e.g., Fig. 6) and automatically generates these components through a sequence of M2T transformations. The current ATLAS implementation fully supports the Gazebo/ROS and MOOS-IvP simulators. Clearly, for each target simulator the M2T transformation and required components are developed only once and can be reused thereafter. We discuss below the concrete instantiations for these two simulators. The full details of the M2T transformations are available on our project webpage.

**ATLAS middleware for MOOS-IvP:** The M2T transformation uses the mission model and configures appropriately the ATLAS middleware to enable monitoring the MRS status for the specified goals. Also, MOOS-specific configuration files are produced to represent the robot configurations, properties and behaviours necessary to run the simulation. Within MOOS-IvP, each robot is represented as a community comprising a set of C++ software modules that provides the functionality of robot components. Each community is served by an individual publish-subscribe database (MOOSDB) which contains key-value pairs. When robots communicate, these key-value pairs form a communication channel through which one robot will publish a message and subscribed robots will receive the update and act accordingly. Example messages that can be sent/received include sensor detection events, speed values and location information.

The communication between the ATLAS middleware and the simulator occurs through MOOSDBInterface, a generic MOOS software component that enables interfacing directly with robot communities. The simulator did not support this functionality. Thus, we developed this reusable software component which can now be instrumented through the middleware to publish/subscribe to messages within MOOSDB databases, e.g., receiving updated robot coordinates, activating the return home behaviour upon mission completion.

Messages received by the MOOS-IvP simulator are kept in the message broker (ActiveMQ) until they are automatically processed and translated into update nodes mapped to the
initial mission goals. These updates are transmitted to the CI algorithm enabling revision of its internal state and informing subsequent decision-making. The middleware also supports translation of generic requests to low-level MRS messages (e.g., return to home or go to this location commands).

**ATLAS middleware for Gazebo/ROS:** This simulator also realises the publish-subscribe architecture with a topic subscription graph being the primary communication mechanism between robots and their components. Accordingly, the M2T transformation produces the necessary ROS launch scripts, which set out the simulation component task graphs, and the necessary ROS configuration to launch the ROS processes and implement the simulation functionality.

The middleware communicates with the ROS simulator using *rosbridge* (http://wiki.ros.org/rosbridge suit), a stable and widely used ROS component that represents the logical interface module between ATLAS and Gazebo. Through rosbridge, ATLAS can subscribe to ROS topic updates, triggering notifications when these subscribed topics are updated. Dynamic topic subscriptions can be added/removed during simulation, thus enabling to reconfigure the interface during mission execution.

To connect to the ROS simulation, the Java-based middleware uses *jrosbridge* to initiate the topic subscriptions using the simulator variables defined in the DSL; see, for instance, the model elements `/amcl_pose` and `/roomCompleted` in Fig. 6. Updates on these variables update the middleware’s internal state and generate events that are forwarded to the CI activating CI-specific decisions. For the reverse communication pathway, i.e., from the CI to Gazebo, CI actuation commands (e.g., go home/robot stop) are translated into specific low-level ROS topic changes, instantiating the behaviour changes specified by these commands. For example, a generic action for “robot stop” (due to a depleted battery) is translated into a command that sets the robot’s current goal to its present point, and prevents it from moving further away from its current location.

## 3.5 Collective intelligence

Collective Intelligence (CI) is an important MRS mechanism that facilitates the encoding of the high-level logic to coordinate robot decisions and manage the behaviour of the overall MRS. In addition to supporting the analysis of different robotic team configurations using variation groups (cf. Sect. 3.3), ATLAS enables the automated evaluation of alternative CI algorithms. Users can inspect the mission metrics produced by each CI instance and select the best for hardening the MRS. This is another key feature of ATLAS that reduces further the coupling between the target robotic simulator and the high-level decision-making process.

To facilitate the development of CI algorithms, ATLAS performs an M2T transformation using the mission model to generate CI templates. These templates have empty placeholder methods that reflect the mission goals, behaviours and events mapped to robot components defined within the mission model (Fig. 6). Engineers are expected to specialise those templates with the appropriate logic to develop the target CI algorithm (cf. Step 3 in Fig. 3).

Although this CI generation facility is provided for convenient development, the decoupling of the CI into a separate process provides considerable flexibility. If system testers wish to exploit CI algorithms already implemented in other languages, it would be possible for users to use a custom CI instead of ATLAS-specific generated stubs. To use this external CI, the users would need to provide parsing and communications mechanisms by providing an API to their custom CI and to interface it to the middleware. Further discussion of this functionality is presented in Sect. 4.1.

The communication of fully fledged CI instances with the ATLAS middleware is underpinned by the *inversion of control* programming paradigm [35]. To achieve this, the middleware becomes aware of the methods within the CI template and the messages associated with each method during the M2T transformation. When the middleware receives an update to a subscribed message/topic from the robotic simulator, it maps the message to the appropriate CI method and proceeds with its invocation. The CI executes its logic and informs the middleware for its decision (e.g., instruct a robot to take over a failed peer) so that the latter can send the appropriate message updates to the simulator via the logical interface. Since the CI instance runs as an independent Java process, both the CI and the middleware run independently and operate in a non-blocking mode using JSON messages.

The current ATLAS version supports the delegation of CI control to a single MRS component. This component can be a centralised command-and-control station or a single robot that acts as the leader with full knowledge and total control over its peers. Supporting other CI variants like hierarchical or decentralised [4,36] is out of scope and is left for future work.

**Example 3** Listing 1 shows an example of the CI implementation for the energy-aware CI for the healthcare robotic team use case. The global variables *rbts*, *roomAssignments* and *roomLocations* hold information about the location of robots, the rooms assigned to each robot and the room locations, respectively. The *init()* method parses the defined model and loads information necessary for the CI; for example, the room locations and the initial energy per robot. Then, the invocation of the *assignRoomsByEnergy* method enables the allocation of pending rooms to the available robots proportionally to their energy capacity. Once room allocation is completed, the *sendRoomAssignments* method facilitates the
<caption>**Listing 1** Excerpt of energy-aware CI for the healthcare robotic team (Sect. 2)</caption>

1 //map maintaining the starting location of each robot
2 Map<Robot, Point> rbts;
3 //rooms assigned to each robot
4 private Map<Robot, List<Integer> roomAssignments;
5 //map maintaining the location of each room
6 Map<Integer, Point> roomLocations;
7
8 //initialise the CI
9 public void init() {
10     parseModel();
11     Set<Integer> pendingRooms = roomLocations keySet();
12     assignRoomsByEnergy(pendingRooms);
13     ...
14 }
15 //allocate rooms based on the energy level per robot
16 public void assignRoomsByEnergy(Set<-Integer> rooms) {
17     while (!rooms.isEmpty())
18         for (Robot rbt : rbts keySet())
19             if (!idleRbts contains(rbt)) {
20                 int rbtRooms = calculateRoomsForRbt(rbt);
21                 assignRoomsToRbt(rbt, rbtRooms, rbts.get(rbt));
22                 roomsaternallastrooms.get(rbt);
23             }
24     sendRoomAssignments();
25 }
26 //send new room assignments to robots
27 private void sendRoomAssignments() {
28     for (Robot rbt : roomAssignments keySet()) {
29         String roomStr = roomAssignments.get(rbt).stream()
30             .map (String::valueOf).collect (Collectors.joining (", "));
31         CILog.logCI("Sending to " + rbt + " rooms" + roomStr);
32         API.sendSimulatorVariable(rbt.getName(), "/rooms",
33             roomStr, true);
34 }
35 //update the energy level for the given robot
36 public void eUpdate(EnergyUpdate eUpd, Robot rbt) {
37     //if battery is below the threshold & robot is not idle
38     if (eUpd.getEnergyValue() < E_THRESHOLD) &
39         (!idleRbts contains(rbt)) {
39         API.returnHome(rbtOSENAME);
40         // Record that the robot is out of energy
41         idleRbts.put(rbt, eUpd.getEnergyValue());
42         // Reassign the work that the robot was doing
43         reassignRooms(rbt);
44     }
45 }
46 //reassign remaining rooms of robot with depleted battery
47 private synchronized void reassignRooms(Robot rbt) {
48     if (roomAssignments containsKey(rbt)) {
49         Set<-Integer> pendingRooms= new
50             HashMap<>(roomAssignments.get(rbt));
51         if (pendingRooms.size()>0)
52             assignRoomsByEnergy(pendingRooms);
53     }
54 }
55 //update local knowledge using information from simulator
56 public void simVarUpdate (String key, String val, Robot rbt) {
57     if (key.equals("/roomCompleted")) {
58         Integer room = Integer valueOf(val);
59         CILog.logCI("Room " + room + "done by " + rbt);
60         roomAssignments.get(rbt).remove (room);
61     } else if (...) {
62 }

transmission of the assigned rooms to the simulator by utilising the middleware-generic function *sendSimulatorVariable* which transmits a comma separated string of room numbers to the *rooms* simulator value of the target robot. The energy updates produced by the middleware's energy model are received by the CI and translated into calls to the *eUpdate* method, which compares the energy level to a fixed energy threshold for the robots. When a robot's energy is below this threshold, the CI considers the robot as being at risk of getting stuck into the healthcare facility. Thus, the CI instructs the robot to return to home (through using the corresponding middleware function), renders the robot idle, and reassigns the rooms not serviced yet to other robots (through the *reassignRooms* method). This method retrieves the outstanding rooms of the idle robot, and if there are any, it reassigns them to the other robots by leveraging the (*assignRoomsByEnergy*) method.

The *simVarUpdate* method provides a facility to update the local CI knowledge using information from the simulator. This method is invoked automatically by the middleware upon receiving an update to a variable specified in the mission model and to which the CI is subscribed to (e.g., Fig. 6). In this case, if the variable is an update to the ROS topic */roomCompleted*, it will automatically remove the room number from the given robot's outstanding work.

## 4 ATLAS prototype tool

The prototype ATLAS model-driven tool uses the Epsilon family of languages [22] to perform the MDE tasks. In particular, the DSL is implemented with EMF [37]. We use Epsilon's EMF Model Java API to check the variation groups and produce the candidate MRS designs (Sect. 3.3) and the Epsilon Generation Language to generate the ATLAS middleware (Sect. 3.4), the configuration files needed by the target simulator and the CI template (Sect. 3.5). We also use ActiveMQ [34] to link the MRS simulator, CI algorithm and the middleware. The open-source ATLAS source code and the full experimental results summarised next are available at https://www.github.com/jrharbin-york/ATLAS. Finally, a video showing the execution of ATLAS in a marine case study that involves a team of unmanned underwater vehicles (UUVs) (see Section 5.2) is available at https://tinyurl.com/ATLAS-ExampleVideo.

### 4.1 Code generation

We provide below further information about the code generation functionality provided in ATLAS.

*Simulator Interface Generation:* The code for the simulator interface is automatically generated using *EGL*, which consumes the model and generates a loader interface to con-
figure the middleware for the target use case. This loader contains instantiates objects from the model, representing the configuration of robots and sensors etc. to be loaded by the middleware. A *Mission* object is returned from the loader class, which the middleware uses internally to maintain the state and support further processing.

*Collective Intelligence Generation:* Templates are also generated for the collective intelligence algorithms using EGL, based upon the information provided in the model. ATLAS users should populate these templates with suitable code that delivers the expected functionality (e.g., see Listing 1). Currently, we assume that in the ATLAS architecture, the CI runs centrally upon a single computer which receives messages from the robotic team and coordinates the CI (i.e., the CI is not distributed). Users interested in exploiting CI algorithms already available in other programming languages (e.g., C++, Python) could achieve this by using their custom CI instead of the generated templates. However, parsing and communication mechanisms (e.g., a client-server architecture) must be provided to enable the interaction between their custom CI and the middleware.

<figure><img src="image_9.png" /><figcaption>**Fig. 8** UUV team deployed on a hazardous object detection mission, developed using the MOOS-IvP simulator [12]</figcaption></figure>

# 5 Experimental evaluation

## 5.1 Research questions

We performed extensive experiments to assess the benefits and performance of ATLAS and answer the following research questions.

**RQ1 (Configuration Analysis).** Can ATLAS help with finding the optimal configuration for a robotic team? We use this research question to analyse the extent to which ATLAS can support the specification and analysis of different variation options in an MRS and enable the selection of the optimal configuration.

**RQ2 (Collective Intelligence Analysis).** Can ATLAS support tradeoff analysis between different collective intelligence algorithms? We use this research question to analyse if ATLAS can assess the situations in which different CI algorithms perform well on specific metrics. Assessing these results will allow robotic developers to select optimal CIs for a given mission.

**RQ3 (Reproducibility).** How does the non-determinism of robot simulations affect the reproducibility of the ATLAS results? We analysed if ATLAS can enable the identification of non-reproducible behaviours of MRS configurations and the discovery of outliers, thus providing evidence that the MRS, CI or the overall system may produce sub-optimal behaviours.

## 5.2 Case studies

We followed the standard practice in empirical software engineering [38,39] and evaluated ATLAS using three distinct case studies: one from the domain of healthcare robotics using ROS/Gazebo [11] and two other from the domain of UUVs using the MOOS-IvP robotic simulator [12]. More specifically, the three case studies selected are: (1) the health-care robotic mission presented in Sect. 2; (2) an object detection UUV mission developed by the ATLAS team and described in Sect. 5.2.1; and (3) a variant of the Bo-Alpha mission [40] developed by the MOOS-IvP community and included within the standard package of the simulator, which we describe in Sect. 5.2.2.

### 5.2.1 UUV object detection mission

The first MOOS-IvP case study involves a UUV team deployed on an object detection mission within a large marine area containing both benign and hazardous objects. Each UUV is equipped with a sonar sensor that can detect environmental objects when they are in close proximity, localisation hardware and a radio transceiver to interface with a centralised command-and-control computer (shoreside) which runs the CI and coordinates the activities of the UUV team. Figure 8 shows the three UUVs executing lawnmower-style sweeps (horizontally back and forwards followed by vertical steps) over subdivided regions of the area, and three objects to be located (where green and red triangles indicate benign and malicious objects, respectively).
<table><caption>**Table 2** UUV requirements for the object detection mission</caption><thead><tr><td>ID</td><td>Description</td></tr></thead><tbody><tr><td>R1</td><td>All environmental objects within the area, both benign and malicious, should be detected</td></tr><tr><td>R2</td><td>When a UUV detects a malicious (benign) object, the detection must be verified by two (one) peer UUVs</td></tr><tr><td>R3</td><td>Subject to satisfying R1 and R2, the CI should coordinate the UUVs so that the mission execution time is minimised</td></tr></tbody></table>

In this case study, an environmental object represents a generic target for detection in the marine area, whose precise nature would depend on the case study. Here, we assume that environmental objects can be either “benign” or “malign” (hazardous), with the “malign” category requiring a more intensive response from the system and further investigation. The terminology used has been derived from similar use cases available on the MOOS-IvP simulator repository. In a mine detection mission, for instance, the “malign” object would represent an active mine, and the “benign” object would represent a coral reef or a sea creature. Similarly, in a search and rescue mission, the environmental objects could either include humans needing rescue or debris. Since in this paper we are investigating the CI response and not sensor accuracy, we assume that the status of the object is always correctly detected by the UUV sensors.

The shoreside uses a CI algorithm to coordinate the UUVs behaviour and fulfil the requirements shown in Table 2. Given the safety-critical nature of this mission, all objects should be detected (R1). Depending on the object’s type, one or two verifications by peer UUVs should be performed (R2), thus reducing the risk that an incorrect type has been assigned to the detected object. Since UUVs have limited battery capacity, it is important to partition the area effectively and complete the mission at the least possible time (R3) and before the maximum mission execution time given by $T_{max} \ge 400$ s.

When designing the UUV team, engineers want to investigate how effectively teams comprising three or four UUVs would accomplish the mission. Furthermore, each UUV can be equipped with a wide or a narrow sensor whose scanning area is 10 m or 20 m wide, respectively. Consequently, this results in 48 possible configurations (i.e., designs) for the UUV team.

In addition to the UUV team configurations, several CI algorithms could be used to support the execution of the object detection mission. We assume that robotic engineers are interested in assessing the following standard and advanced CI algorithms.

- Standard CI: This CI partitions the area between the UUVs equally, based on a lawnmower pattern with a constant vertical separation, independent of each UUV’s sonar sensor

range. When performing verifications of detected objects (R2), the CI selects the UUV that is the closest to the object. Verifications are performed for a constant fixed time of 600 s, scanning the area around the detection. When completing the verification, the UUV is commanded to resume its original sweep region from the beginning.

- Advanced CI: This CI partitions the area between the UUVs proportionally to the strength of their sonar sensors, i.e., a UUV equipped with a more capable sensor is assigned a larger area than a UUV equipped with a narrow sensor. The CI also monitors the status of the verification, returning the dispatched UUV back to its originally assigned area as soon as the final waypoint of the verification task is reached. When this happens, the CI instructs the UUV to resume its task from the point at which it was interrupted.

Evidently, the advanced CI is more efficient and aims at reducing the overall mission execution time but incurs significant communication costs due to the frequent communication between the dispatched UUV and the shoreside. On the contrary, the standard CI, albeit slower, is more meticulous and may help the UUV to discover previously missed objects.

### 5.2.2 Bo-Alpha mission

This case study is based on the standard Bo-Alpha<fnref n="2" /> case study supplied with the MOOS-IvP simulator. During this mission, UUVs monitor an area and make measurements (e.g., salinity, temperature) on the left and right sides of a topology, occasionally alternating sides. To avoid collisions, the UUVs must not come too close to each other and also avoid obstacle zones. The UUVs monitor their residual energy levels and communicate this information and positions to a control station running the CI. Figure 9 shows two robots crossing the central region containing five obstacles depicted as white octagons. For this UUV mission, there are 16 possible configurations: two robots with a “fast” or “slow” version each, travelling at 1.5 m/s and 3.0 m/s, respectively. Each UUV also supports a standard (2800 mAh) and a high capacity (5000 mAh) battery.

The mission is executed for $T_{max}=1200$ s and the CI coordinates the team’s behaviour to fulfil the requirements shown in Table 3. The metrics used to assess the satisfaction of the requirements are the residual energy of the UUVs after mission completion and the total UUVs distance from their return locations. A competent CI would steer the robots back to their base with sufficient residual energy. Engineers are interested in analysing the following CI instances.

- Standard CI: The UUVs alternate sides every 150 s. This CI recalls the robots to return home 150 s before $T_{max}$.

<footnote n="2">² https://oceanai.mit.edu/ivpman/pmwiki/pmwiki.php?n=IvPTools. UFIdObstacleSim.</footnote>
<figure><img src="image_10.png" /><figcaption>**Fig. 9** Bo-Alpha mission for UUV monitor and avoidance</figcaption></figure>

<table><caption>**Table 3** UUV requirements for the object detection mission</caption><thead><tr><td>ID</td><td>Description</td></tr></thead><tbody><tr><td>R1</td><td>Each vehicle must complete more than one sweep of the left and right areas, alternating periodically once both completed</td></tr><tr><td>R2</td><td>Vehicles must avoid entering into the obstacle regions</td></tr><tr><td>R3</td><td>Vehicles must return to their starting points before their battery is depleted</td></tr></tbody></table>

- *Energy-based CI:* This CI algorithm tracks the positions of the UUVs and alternates them when both have finished their assigned area. Also, this CI monitors the energy remaining on the UUV battery and sends the recall command to return to base when the critical energy threshold of 750mAh is satisfied.

## 5.3 Experimental setup

We performed a wide range of experiments for the healthcare-based use case described in Sect. 2 using the ROS simulator and the two UUV-based use cases (Sects. 5.2.1 and 5.2.2) using the MOOS-IvP simulator. The experimental setup for the healthcare use case involved running ATLAS and its supporting scripts for both CI variants (standard and energy-aware) and all 60 possible configurations described in the variation group example from Sect. 2. Similarly, we executed a corresponding number of experiments for the object detection and Bo-Alpha UUV missions. The experimental setup in these instances involved running ATLAS and MOOS-IvP in both missions for the two CI variants (standard and advanced; standard and energy-based, respectively) and all possible configurations (48 and 16, respectively).

All experiments were run on a *Ryzen 5 3600* machine with 16Gb of RAM. The MOOS-IvP experiments were executed in a Virtualbox environment, while the ROS configurations were executed natively in Ubuntu 18.04 with ROS Melodic.

## 5.4 Results and discussion

We assess ATLAS on the use cases for each robotic simulator independently. To this end, Sects. 5.4.1 and 5.4.2 focus on answering the research questions from Sect. 5.1 for the ROS- and MOOS-IvP-based case studies, respectively.

### 5.4.1 Healthcare case study

*RQ1 (Configuration Analysis).* Figure 10a shows the number of completed rooms for the possible MRS configurations of the healthcare case study with the robot using the standard CI. In this scenario, we set the threshold for requirement R1 to twelve, i.e., the robotic team must successfully service twelve rooms in order to fulfil requirement R1. As shown in the figure, twelve configurations fulfilled this requirement, servicing all assigned rooms, and returning the robots back to their base for recharging. All these configurations included the high-capacity battery option on all robots, with three configurations using three robots and the remaining nine featuring two robots.

The worst performing configuration serviced only four completed rooms during the evaluation. This configuration comprised one robot with a high capacity battery and another robot with a standard battery. Further investigation revealed that the behaviour occurred due to a rare event involving a failure either in the ROS simulation process or a failure of one of robot's path planning algorithm. This is an interesting behaviour of the simulation and we revisit this configuration in the analysis of RQ3 later in this section.

Figure 10b depicts the number of completed rooms versus the total cumulative energy of the robotic team at the end of simulation with the command-and-control centre using again the standard CI. Note that some points in the scatter plot are superimposed upon each other because the variance of the energy values is small. Looking at the figure, we can observe that some configurations violated requirement R1 (i.e., serviced less than twelve rooms) and also violated requirement R2 (as indicated by the almost zero residual energy at the end of the mission). Subsequent analysis indicated that these configurations involved robots equipped with standard capacity batteries, which were proven inadequate in enabling the robotic team to complete its mission using the standard CI. Configurations that completed the mission successfully with the standard CI and whose robots had a good amount of residual energy involved at least two robots with high capacity batteries, while a few successful configurations employed all three robots with high capacity batteries. Given this information, it is clear that the standard CI requires robots with relatively large energy resources in order to fulfil the mission requirements.

We complement the presentation for the configuration analysis by showing in Fig. 10c the collected results for
<figure><img src="image_11.png" /><figcaption>Fig. 10 Results for the healthcare mission over the 60 MRS configurations with the standard CI</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 11 Results for the healthcare case study over the 60 MRS configurations for both CI instances</figcaption></figure>

all mission requirements and all MRS configurations with the command-and-control centre using the standard CI. The three metrics shown are the number of completed rooms (R1), the total energy at the end of simulation (R2), and the mission completion time (R3). For configurations satisfying both requirements R1 and R2 the mission completion time ranges between 145 s and 275 s, while configurations that failed to complete the mission successfully (failing R1 and/or R2) received the fallback value of $T_{end}=335$ s. Overall, the results in this section demonstrated that ATLAS can automatically identify MRS configurations which fulfil the mission requirements, having identified twelve such configurations that service all rooms successfully and return the robots back to their base, within the required mission duration.

**RQ2 (Collective Intelligence Analysis).** To answer this research question, we analysed the differences in performance between the CI instances across all MRS configurations. To this end, Fig. 11a compares the number of completed rooms between the standard and energy-aware CI instances. The results show that 45 configurations successfully serviced all twelve rooms using the energy-aware CI, compared to only twelve configurations under the standard

CI. This result is not surprising due to the design advantages of the energy-aware CI which includes a more informed initial workload allocation considering each robot's available energy coupled with the ability to redistribute work from robot(s) whose battery has been exhausted to other robots with adequate residual energy.

We further assess the differences in fulfilling requirements R1 and R2 between both CIs by comparing the number of completed rooms and the total energy at mission end in Fig. 11b. As before (cf. Fig. 10b), some configuration results may be superimposed upon each other, as the variation between energy values for similar configuration is often small. In general, the total remaining energy when mission ends is lower for the energy-tracking CI than the standard CI in most configurations that successfully serviced eleven and twelve rooms. This is a useful insight and evidences the capability of the energy-tracking CI to redistribute workload at the expense of increased energy reduction for robots with a larger battery capacity. An interesting observation concerns the configurations that successfully serviced only six and nine rooms thus failing to satisfy requirement R1 under both CI instances. When using the standard CI, however, these
<figure><img src="image_13.png" /><figcaption>**Fig. 12** Repeatability analysis for both CIs for the healthcare case for two configurations</figcaption></figure>

configurations failed also to satisfy requirement R2 because they exhaust their batteries, whereas when using the energy-aware CI they were able to return to their base for recharging without getting stuck within the healthcare facility.

Figure 11c presents for both CI instances all requirements simultaneously, i.e., the completed rooms, the total energy at the end of the simulation, and the mission completion time. The results show that for the energy-aware CI, the configurations that fail requirement R1, servicing less than the expected twelve rooms, also violated requirement R3, receiving $T_{end}$ as their assigned mission completion timing. Considering requirement R3, for configurations that meet requirement R1 and service all twelve rooms, the energy-aware CI demonstrates its capability to complete the mission before $T_{end}$ but with an increased time compared to the standard CI, and with a lower cumulative residual energy level across the team members. This is expected, since the energy-aware CI either gives more work to robots with a larger battery or redistributes work from robots with exhausted batteries. Consequently, robots with a large battery capacity require more time to complete their work.

These results demonstrate that ATLAS can support the automatic investigation of performance differences between functionally equivalent CI algorithms. This process enables tradeoff and cost-benefit analysis of the optimal CI algorithm for a given set of mission requirements and possible MRS configurations. More specifically, we identified that 12 configurations can meet requirement R1 and complete all rooms successfully with the standard CI, but these require a high capacity battery on all robots. If the cost of these resources is acceptable, then it would be possible to deploy the robotic team with one of these configurations. However, if only standard capacity batteries are viable, then the energy-tracking CI needs to be used with one of the 45 viable configurations, incurring additional development work to add energy tracking to the CI. If neither of these are viable, then find-

ing a viable solution entails system engineers making further manual iterative redesigns of the system or mission, perhaps changing the location of robots’ starting points or altering the tasks performed by the robotic team.

**RQ3 (Reproducibility)** Non-determinism can be due to robot behaviour (e.g., path planning), components of the simulation engine, or the operating system [41]. Non-deterministic simulations can produce behaviour that is not representative of real systems. Regardless of the source of non-determinism, we can assess the behaviour of a system via repeated executions of a fixed system configuration. This provides a distribution of metrics over multiple runs, enabling the identification of outliers in which non-determinism may produce sub-optimal behaviour. Therefore, we investigated potential non-determinism in the simulations of the healthcare case study by selecting two configurations from the 60 possible MRS configurations and evaluated them for ten times using both CIs instances. First, we use one of the best configurations from Fig. 10a that completed the mission successfully with all twelve rooms serviced. For both the standard and energy-aware CI, all configuration runs successfully serviced all twelve rooms. Figure 12a presents the total residual energy on mission completion and the mission completion time for the ten runs. Across the repeated evaluations, there is a very low variation of the total residual energy (it may appear large due to the horizontal scale, but in fact the percentage difference is low). The completion time does show some variation, which we hypothesise is due to differences in the navigation path chosen by the robots as occasionally the robots have to alter their path to avoid each other or back up and choose an alternate path to prevent potential collisions with the walls.

Second, we used one of the worst configurations from Fig. 10a which completed only four rooms with the standard CI (cf. see RQ1 results for the healthcare case study). Figure 12b shows the completed rooms and total energy metrics across ten runs of this configuration. When the standard
CI is employed, this configuration successfully services nine rooms. In contrast, with the energy-aware CI, one run produced an output of ten completed rooms, and all the others achieved twelve. These results illustrate that there is some instability in the simulation for this particular configuration regarding robot selections, room choices, and the overall robotic behaviour, also demonstrating that the original value of four completed rooms with the standard CI being an outlier. We believe that the behaviour occurred because the path planning algorithm failed intermittently when one of the robots (with the high capacity battery) approached too close to a wall and got stuck. This shows that ATLAS can isolate potentially unstable or non-deterministic simulation configurations. However, the robotic engineer should exploit its domain knowledge to delve into the details of this behaviour and establish the actual root cause of the instability or non-determinism in the simulation.

### 5.4.2 MOOS case studies

**RQ1 (Configuration Analysis).** Figure 13a shows the number of missed detections for the 48 possible MRS configurations of the object detection mission using the standard (top) and advanced CI (bottom). Under the standard CI, ATLAS identified eight optimal configurations in which the MRS completed its mission successfully with zero missed detections. Other configurations produce a wide range of possible detection failures, up to the worst case of seven missed detections.

We analysed the produced MRS configurations to discover factors that can affect the system performance and reliability. Our analysis showed that the optimal MRS designs for a team of four UUVs always equipped the UUV gilda with a wide sensor, while for a three UUV team all UUVs should be equipped with a wide sensor. Engineers can factor in the cost of robots and sensors and decide whether a three- or four-robot team is preferred for this mission. Further analysis revealed a particularly sensitive MRS design when UUV gilda uses a narrow sensor. These designs tend to produce three missed detections, due to the UUV gilda missing the initial detection of the rightmost object in its assigned area and two subsequent verifications. Accordingly, such designs fail to meet the mission requirements and should not be preferred.

Figure 13b shows the relation between the number of missed detections and mission completion time for the object detection mission. Some MRS configurations using the standard CI exhausted the available time signifying that at least one UUV did not complete its task on time. We identified a general pattern indicating a tradeoff with a decline in sweep completion time at the cost of partial mission completion, i.e., the mission can complete earlier when fewer detections are performed. MRS configurations that detected no object

completed around 750–1600 s, where 750 s correspond to the shortest time needed to complete a sweep. Since no verifications were performed, this is expected. Subsequent analysis showed that most of these configurations comprise three UUVs all equipped with a narrow sensor. This configuration produces missed detections due to the width of the sweep patterns used in the standard CI, especially when objects are in the middle of the assigned UUV regions.

Considering the Bo-Alpha mission, we found one MRS configuration comprising two fast robots that completed two full sweeps using the standard CI (Fig. 14a). However, this configuration did not meet the other mission requirement as it failed to steer the UUV team back to its base with sufficient residual energy. Further analysis of the configurations that completed a single sweep yielded a configuration that reported some residual energy and a mean vehicle distance of less than 50 m. This configuration consists of a slow vehicle (henry) with a large battery, that enables the UUV to return to base with some residual energy, and a fast vehicle (gilda) with a smaller battery that can complete its sweep sooner.

These findings clearly demonstrate that ATLAS can support the analysis of different MRS configurations via its variation groups. The outcome of this analysis enables the selection of an optimal configuration for a given mission and requirements.

**RQ2 (Collective Intelligence Analysis):** To answer this research question, we compared the standard and advanced CI algorithms in both UUV case studies. For the object detection mission, Fig. 13a shows that the advanced CI produces a better worst-case scenario than the standard CI, with two missed detections at most. In fact, over 40 MRS designs succeed without missing an object failure. This behaviour is mainly caused by the CI adapting to the sensor range and using a smaller vertical sweep step size for narrow sensors. Also, many MRS configurations using the advanced CI completed the mission faster and had fewer missed detections than the corresponding configurations using the standard CI (Fig. 13b). This behaviour occurs because the advanced CI instructs the UUV, after completing its verification task, to resume its original task from the point at which it was interrupted.

The analysis of the Bo-Alpha mission showed no optimal MRS configurations with the standard CI, since every configuration fails to complete more than one sweep of the left and right areas (R1 violation) or fails to return to base (R3 violation). The low number of completed sweeps using the standard CI is due to the improperly short timing that alternates the UUVs between different areas before they have completed their sweeps (Fig. 14a). In contrast, the energy-based CI (with the low energy return threshold of 750 mAh) produces a larger number of completed sweeps with a modal value of six. Most of the energy-based MRS configurations
<figure><img src="image_14.png" /><figcaption>**Fig. 13** Results for the object detection UUV mission over the 48 MRS configurations for each CI instance</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**Fig. 14** Bo-Alpha mission results over the 16 MRS configurations for the standard CI (top) and energy-based CI (bottom)</figcaption></figure>

return the vehicles back to base (being at most 50 metres from the base).

The standard CI experiences many cases with very low residual energy (Fig. 14c). The overall better performance of the energy-based CI occurs because this CI uses a waypoint completion feedback to alternate the sweep sides between the vehicles, rather than a time limit as in the standard CI. The better performance is also due to using the energy feedback for sending the return command sufficiently early, rather than waiting until getting close to the simulation end time.

These results provide sufficient empirical evidence that ATLAS can support the comparison of different CI algorithms given a set of mission-specific metrics. Engineers can use these results to select optimal CIs for specific combinations of missions and robots. We note that the manual crafting and analysis of designs and CI algorithms for robotic teams was the status quo before ATLAS. Our experience with developing robotic missions using a non-ATLAS-based solution was the primary motivation for devising ATLAS. *RQ3 (Reproducibility).* As before, we assessed the impact of non-determinism in the reproducibility of simulation-based evaluation of MRS components by analysing the best MRS configuration and CI algorithm pair for each UUV case study over 30 independent runs.

In the object detection mission, we used a configuration that produced the lowest timing and zero missed detections in research question RQ2. Figure 15 shows the relationship between missed detections and completion time for both CIs

<figure><img src="image_16.png" /><figcaption>**Fig. 15** Results over 30 independent runs for the object detection mission using the best configuration and both CIs for the object detection UUV mission</figcaption></figure>

over 30 independent runs for this configuration. In all runs, there is a longer completion time under the standard CI than the advanced CI. This is not surprising since with the standard CI the UUV restarts its sweep pattern from the start after each verification. Moreover, the completion time for the advanced CI with no missed detections is generally clustered around the 1100–1300 s range, with some advanced CI executions producing a single missed detection and correspondingly a slightly shorter sweep time. This behaviour occurs because when there is a missed detection, at least one robot will not
<figure><img src="image_17.png" /><figcaption>**Fig. 16** Bo-Alpha mission results over 30 independent runs for the standard CI (top) and energy-based CI (bottom)</figcaption></figure>

be interrupted for verifications, thus allowing the robot to
complete its task slightly faster.

In the Bo-Alpha mission, we used a configuration that obtained the maximum number of sweeps in research question RQ2 with large batteries on both vehicles. The results over 30 independent runs show that there is little variability (Fig. 16). Generally, the energy tracking CI performs well and better than the standard CI. The final distance from the base in Fig. 16b shows that generally the metric is always constant, with a large mean distance with the standard CI and a small value with the energy tracking CI. Given that the robots are equipped with large batteries, the results show a considerable amount of final energy left upon the vehicle which is moving slower. Interestingly, there are a couple of outlier cases, which report a higher total final distance for the energy tracking CI. One possible explanation may be that given the variations in position, the faster vehicle was too far away from base when the return command was sent and therefore was unable to return home in time. Another explanation is that in rare cases, when returning home, robot collision avoidance strategies may incorrectly navigate the robots a considerable distance away from their intended home points. These results show the ability of ATLAS to identify potential instability of configurations in rare cases. However, it is up to the user to analyse the logs and determine precisely what happened in each case.

**5.5 Threats to validity**

**Construct validity threats** could arise by incorrect assumptions and simplifications during the specification of the experimental methodology and not using representative mission case studies. To mitigate this type of threat, we used three case studies, two developed by us and one provided by the MOOS-IvP community and is part of the standard version of the simulator. The criteria for selecting the case study from the MOOS-IvP mission library were the following: (i) the mission had to involve multiple robots; (ii) the team of robots had to coordinate via a collective intelligence mechanism to achieve the mission goals; and (iii) the mission had to be adequately documented.

**Internal validity threats** could produce incorrect analysis data and lead to incorrect insights. To mitigate this type of threat, the research questions are independent of each other, thus errors in one research question do not propagate to the other. Moreover, results were derived by automatically analysing the simulator-produced logs and reducing potential user bias. We also support reproducibility of our findings by making our analysis scripts available in the project's repository.

**External validity threats** could reduce the generalisation of ATLAS. We reduce this threat class by using established MDE practices and tools including EMF [37], Epsilon [22] and ActiveMQ [34]. The experimental evaluation involved three case studies (one provided by the MOOS-IvP community), while the use of two distinct robotic simulators reduces further this threat. Finally, the evaluation of ATLAS both in Gazebo/ROS and MOOS-IvP demonstrates the generality of our framework and its capability to support multiple robotic simulators. However, further experiments are needed to establish the applicability, feasibility and scalability of ATLAS in domains and applications with characteristics different from those used in our evaluation.

**6 Related work**

The work presented in this paper lies at the intersection of model-driven engineering, robotics, and design space exploration.

**6.1 Model-driven engineering for robotics**

Developing model-driven solutions for the robotics domain is an established area, which has produced several results over the years [42–44]. The majority of the proposed domain-specific modelling languages deals only with specific robot functions such as perception or control, while there are some model-driven toolchains like RobotML [8], BRICS [10], SmartSoft [7], and Robochart [9] which provide multiple modelling notations to be used together when developing a robotic system. For a detailed description of different
approaches to model-driven engineering of robots, the reader is referred to [16] and [45].

Despite the available literature on the application of MDE to robotics, the engineering of MRS is still inadequately investigated. Cattivera and Casalaro [46] conducted a systematic mapping study on the application of MDE to the engineering of mobile robots. Out of all the studies reviewed, the authors found that only 24% (i.e. 19 studies out of 80) deal with MRS compared to single robots. This is however an improvement from an earlier survey by the same authors in 2015 [47], in which only 19% of surveyed papers considered multiple robots.

The most common formalism used for modelling multi-robot behaviour is finite state machines and statecharts (e.g. [48–50]). Other approaches include Ciccozzi et al. [43], who propose the FLYAQ family of graphical domain-specific languages to model the structure and behaviour of multi-robot aerial systems, and Pinciroli and Beltrame [51] who propose a textual DSL for specifying the behaviour of robot swarms. Instead of developing a language for specifying the behaviour of multi-robot systems, Dragule et al. [52] extend FLYAQ with a specification language, which enables engineers to specify domain-specific constraints for robotic missions in a declarative manner. Finally, very few approaches propose solutions for modelling explicitly communication, task allocation, and coordination between robots with the exception of [53].

Notwithstanding their merits, the aforementioned languages and tools focus on the specification of the behaviour and structure of multi-robot systems. In contrast, ATLAS focuses on the design space exploration of robotic systems facilitating tradeoff analysis.

## 6.2 Design space exploration

Different techniques have been proposed in the literature for solving the ‘Design Space Exploration problem’, i.e., automatically exploring large numbers of system design alternatives and identifying designs that optimise properties of interest.

Learning-based techniques identify the optimal designs by predicting the quality of a design before the actual synthesis and thus sampling only a portion of interest from the entire design space. Mahapatra and Schafer [31] proposed a more efficient simulated annealing algorithm called Fast Simulated Annealer (FSA) which uses a decision tree algorithm to identify parameters that have a high impact on the final result. These parameters are maintained in the next iteration of the algorithm until the optimal solution is found. Similarly, Chen et al. [54] presented Co-Training Model Tree (COMT), which is a semi-supervised learning model for predicting the qualities of design configurations and does not depend on labelled design configurations. Finally, Liu and

Carloni [55] presented a learning-based method, which is based on the Random Forest learning model, transductive experimental design (i.e., a method for sampling the design space), and randomized selection, for finding an approximate Pareto front of designs.

Generic population-based meta-heuristic optimisation techniques, such as evolutionary algorithms [32], consider distinct designs as individuals in a population and a fitness function is used to qualify the solutions. Subsequent evolutions of the population are obtained by applying appropriate operators such as reproduction, selection, and gene mutation. The Non-dominated Sorting Genetic Algorithm (NSGA-II), which is a specific type of evolutionary algorithm for multi-objective optimisation, has been used for searching the space of possible solutions in the context of design space exploration [28,30,56]. In this work a number of good designs are grouped into candidate populations according to the optimization criteria. Similarly, Palesi and Givargis [57] use genetic algorithms to discover Pareto-optimal configurations of a system-on-a-chip architecture.

Design space exploration had been formulated as an optimization problem and explored using swarm intelligence algorithms like particle swarm optimisation (PSO) [58], bacterial foraging optimisation (BFO) [59], and ant colony optimisation [60]. Shathanaa and Ramasubramanian [61] used learning-based methods and achieved better performance in terms of the quality of the Pareto front created, whereas swarm intelligence-based methods outperform others in terms of execution speed.

Model-driven engineering has also been used in the context of design space exploration. Model-driven approaches specify the design problem as a set of models and model transformations and generate input for other tools that perform the space exploration. The Gaspard Framework [62] focuses on the design of massively parallel embedded systems. It uses automatic model refinement and model transformations to allow for design space exploration to evaluate performance characteristics through simulations. Neema et al. [63] propose the DESERT tool suite, which supports constrained-based design space exploration and model synthesis for embedded systems. Moreover, the OCTOPUS toolset [29] supports the design of software-intensive embedded systems and provides an intermediate representation for the specification of design problems. The tool integrates with analysis tools such as Uppaal [64] for the exploration of the design space. Finally, Hegedüs et al. [65] propose a model-driven framework for guided design space exploration, where the system states are graphs, operations are defined as graph transformation rules, while goals and constraints are defined as graph patterns. While most of the other model-driven approaches to design space exploration depend on external tools for searching the design space, Hegedüs et al. [65] they use model transformations as a way to perform the explo-
ration itself and they propose using information from analysis tools to guide this model transformation based exploration.

The majority of the aforementioned approaches focus on the exploration of optimal structural designs. In addition to these aspects, ATLAS focuses also on the exploration of optimal behaviours and particularly the exploration and evaluation of different algorithms for the collective intelligence of a robotic team.

## 6.3 Simulation-based analysis for robotics

Simulations are used extensively in the engineering of robotic systems. One of the main purposes of simulation for robotics is to provide a safe and fully controlled virtual testing and verification environment. Afzal et al. [26] propose a framework that can facilitate automated testing of robotic systems using software-in-the-loop (low-fidelity) simulations and anomaly detection. Similarly, Huck et al. [66] focus on testing industrial human-robot collaborative systems by using a human model and an optimization algorithm to generate high-risk human behaviour in simulation, thereby exposing potential hazards.

Simulation is also used to accelerate the engineering design cycle for robotic systems and reduce its costs. Serban et al. [67] propose Chrono, a multi-physics simulation package aimed at modelling, simulation, and visualisation of the mechanical parts of ground vehicles. Zhao et al. [68] introduce a simulation-based system for optimizing the physical structure and controllers of robots. The goal of the system is to take a set of user-specified primitive components and generate an optimal robot structure and controller for traversing a given terrain.

Lastly, simulations are used to generate at low cost large amounts of training data for the machine learning components of robots. Tobin et al. [69] trains models for object localisation on simulated images that transfer to real images by randomising rendering in the simulator. Similarly, Chebotar et al. [70] enable policy transfer to new real-world scenarios by training on a distribution of simulated scenarios. Finally, Andrychowicz et al. [71] teach a robotic arm using reinforcement learning dexterous in-hand manipulation policies that can perform vision-based object reorientation in a simulated environment.

Compared to the above approaches, ATLAS focuses on the exploration and evaluation of different algorithms for the collective intelligence of the robotic robot team. Moreover, our approach is simulator-agnostic, since its flexible, message-based architecture allows it to be easily extended to accommodate experimentation with different robotic platforms.

## 7 Conclusions and future work

This paper presents the ATLAS framework for design space exploration and tradeoff analysis of MRS architectures and CI algorithms. We described the framework's architecture, the core concepts of the ATLAS domain-specific language, and the ATLAS middleware. To this end, ATLAS is underpinned by: (i) a domain-specific modelling language, which enables the specification of MRS architectures, capabilities, and missions (including functional and non-functional requirements); and (ii) a code generation engine, which consumes models specified in the ATLAS language and produces the necessary infrastructure for experimenting on robotic simulators and platforms.

The evaluation of the framework is based on an MRS healthcare case study built with ROS/Gazebo [11] and two MRS case studies built with MOOS-IvP [12]. Our experimental evaluation indicated that ATLAS is capable of modelling MRS and their missions, and it enables the exploration and tradeoff analysis of different MRS configurations and CI algorithms.

In the future, we plan to evaluate the expressive power of the ATLAS DSL by applying it to more case studies, and extend its applicability to different robotic platforms [72]. Also, we would like to improve the variability modelling capabilities of ATLAS, enabling the analysis of more elaborate design alternatives [24] and investigate the incorporation of advanced intelligent techniques to search for optimal MRS configurations [56,73,74]. Finally, it would be useful to more formally apply a cost-benefit analysis of mission configurations. Using feedback from system users, ATLAS can be employed to analyse tradeoffs between CI algorithm development and requirement costs, and the resources required for implementing a mission.

**Acknowledgements** This research was supported by the European Union's Horizon 2020 projects SESAME (grant agreement No 1010172 58) and TAILOR (grant agreement No 952215), and by the Lloyds Register Foundation under the Assuring Autonomy International Programme grant SAFEMUV.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
References

1. Parker, L.E.: Reliability and Fault Tolerance in Collective Robot Systems (ch. 6). Pan Stanford Publishing, Singapore (2012)

2. Parker, L.E., Rus, D., Sukhatme, G.S.: Multiple mobile robot systems. In: Springer Handbook of Robotics, pp. 1335–1384. Springer (2016)

3. Kephart, J.O., Diaconescu, A., Giese, H., Robertsson, A., et al.: Self-adaptation in collective self-aware computing systems. In: Self-Aware Computing Systems, pp. 401–435. Springer (2017)

4. Weyns, D., Schmerl, B., Grassi, V., Malek, S., Mirandola, R., Prehofer, C., Wuttke, J., Andersson, J., Giese, H., Göschka, K.M.: On patterns for decentralized control in self-adaptive systems. In: Software Engineering for Self-Adaptive Systems II. Springer, pp. 76–107 (2013)

5. De Lemos, R., et al.: Software engineering for self-adaptive systems: a second research roadmap. In: Software Engineering for Self-Adaptive Systems II. Springer, pp. 1–32 (2013)

6. Bucchiarone, A., Cicchetti, A., De Sanctis, M.: Towards a domain specific language for engineering collective adaptive systems. In: 2nd International Workshops on Foundations and Applications of Self* Systems. IEEE, pp. 19–26 (2017)

7. Schlegel, C., Haßler, T., Lotz, A., Steck, A.: Robotic software systems: from code-driven to model-driven designs. In 2009 International Conference on Advanced Robotics. IEEE, pp. 1–8 (2009)

8. Dhouib, S., Kchir, S., Stinckwich, S., Ziadi, T., Ziane, M.: RobotML, a domain-specific language to design, simulate and deploy robotic applications. In: International Conference on Simulation, Modeling, and Programming for Autonomous Robots. Springer, pp. 149–160 (2012)

9. Miyazawa, A., Ribeiro, P., Li, W., Cavalcanti, A., Timmis, J., Woodcock, J.: RoboChart: modelling and verification of the functional behaviour of robotic applications. Softw. Syst. Model. 18(5), 3097–3149 (2019)

10. Bruyninckx, H., Klotzbücher, M., Hochgeschwender, N., Kraetzschmar, G., Gherardi, L., Brugali, D.: The BRICS component model: a model-based development paradigm for complex robotics software systems. In: 28th Annual ACM Symposium on Applied Computing, pp. 1758–1764 (2013)

11. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, 01 (2009)

12. Benjamin, M.R., Schmidt, H., Newman, P.M., Leonard, J.J.: Nested autonomy for unmanned marine vehicles with MOOS-IvP. J. Field Robot. 27(6), 834–875 (2010)

13. Gerasimou, S., Matragkas, N., Calinescu, R.: Towards systematic engineering of collaborative heterogeneous robotic systems. In: 2019 IEEE/ACM 2nd International Workshop on Robotics Software Engineering. IEEE 2019, pp. 25–28 (2019)

14. Parker, L.E.: Multiple mobile robot systems. In: Handbook of robotics. Springer, pp. 921–941 (2008)

15. Rossi, F., Bandyopadhyay, S., Wolf, M., Pavone, M.: Review of multi-agent algorithms for collective behavior: a structural taxonomy. IFAC-PapersOnLine 51(12), 112–117 (2018)

16. Nordmann, A., Hochgeschwender, N., Wigand, D., Wrede, S.: A survey on domain-specific modeling and languages in robotics. J. Softw. Eng. Robot. 7(1), 75–99 (2016)

17. Harbin, J., Gerasimou, S., Matragkas, N., Zolotas, A., Calinescu, R.: Model-driven simulation-based analysis for multi-robot systems. In: 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, pp. 331–341 (2021)

18. Martínez, S., Gérard, S., Cabot, J.: Efficient model similarity estimation with robust hashing. Softw. Syst. Model. 1–25 (2021)

19. Hawes, N., Burbridge, C., Jovan, F., Kunze, L., Lacerda, B., Mudrova, L., Young, J., Wyatt, J., Hebesberger, D., Kortner, T., et al.: The strands project: long-term autonomy in everyday environments. IEEE Robot. Autom. Mag. 24(3), 146–156 (2017)

20. Okamura, A.M., Mataric, M.J., Christensen, H.I.: Medical and health-care robotics. IEEE Robot. Autom. Mag. 17(3), 26–37 (2010)

21. Hebesberger, D., Körtner, T., Pripfl, J. Gisinger, C., Hanheide, M., et al.: What do staff in eldercare want a robot for? An assessment of potential tasks and user requirements for a long-term deployment. In: IROS Workshop on "Bridging user needs to deployed applications of service robots" (2015)

22. Kolovos, D., Rose, L., Paige, R., García-Domínguez, A.: The Epsilon book. Structure 178, 1–10 (2010)

23. Pohl, K., Böckle, G., van Der Linden, F.J.: Software Product Line Engineering: Foundations, Principles and Techniques. Springer, Berlin (2005)

24. Apel, S., Batory, D., Kästner, C., Saake, G.: Software product lines. In: Feature-Oriented Software Product Lines, pp. 3–15. Springer (2013)

25. Jézéquel, J.-M.: Model-driven engineering for software product lines. In: International Scholarly Research Notices, vol. 2012 (2012)

26. Afzal, A., Le Goues, C., Hilton, M., Timperley, C.S.: A study on challenges of testing robotic systems. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 96–107. IEEE (2020)

27. Kirkpatrick, S., Gelatt, C.D., Jr., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983)

28. Abdeen, H., Varró, D., Sahraoui, H., Nagy, A.S., Debreceni, C., Hegedüs, Á., Horváth, Á.: Multi-objective optimization in rule-based design space exploration. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 289–300 (2014)

29. Basten, T., Benthum, E. v., Geilen, M., Hendriks, M., Houben, F., Igna, G., Reckers, F., Smet, S. d., Somers, L., Teeselink, E., et al.: Model-driven design-space exploration for embedded systems: the octopus toolset. In: International Symposium On Leveraging Applications of Formal Methods, Verification and Validation, pp. 90–105. Springer (2010)

30. Gerasimou, S., Tamburrelli, G., Calinescu, R.: Search-based synthesis of probabilistic models for quality-of-service software engineering. In: 30th IEEE/ACM International Conference on Automated Software Engineering, pp. 319–330. IEEE (2015)

31. Mahapatra, A., Schafer, B.C.: Machine-learning based simulated annealer method for high level synthesis design space exploration. In: Proceedings of the 2014 Electronic System Level Synthesis Conference (ESLsyn), pp. 1–6. IEEE (2014)

32. Coello, C.A.C., Lamont, G.B., Van Veldhuizen, D.A., et al.: Evolutionary Algorithms for Solving Multi-objective Problems, vol. 5. Springer, Berlin (2007)

33. Hussain, K., Salleh, M.N., Cheng, S., Shi, Y.: Metaheuristic research: a comprehensive survey. Artif. Intell. Rev. 52(4), 2191–2233 (2019)

34. ActiveMQ, Activemq, https://activemq.apache.org/components/classic/

35. Martin, R.C.: Design principles and design patterns. Object Mentor 1(34), 597 (2000)

36. Calinescu, R., Gerasimou, S., Banks, A.: Self-adaptive software with decentralised control loops. FASE 15, 235–251 (2015)

37. Steinberg, D., Budinsky, F., Merks, E., Paternostro, M.: EMF: Eclipse Modeling Framework. Pearson Education, London (2008)

38. Runeson, P., Host, M., Rainer, A., Regnell, B.: Case Study Research in Software Engineering: Guidelines and Examples. Wiley, New York (2012)
39. Easterbrook, S., Singer, J., Storey, M.-A., Damian, D.: Selecting empirical methods for software engineering research. In: Guide to Advanced Empirical Software Engineering, pp. 285–311 (2008)

40. MOOS-IvP, Bo Alpha UUV Mission, https://oceanai.mit.edu/ivpman/pmwiki/pmwiki.php?n=IvPTools.UFIdObstacleSim

41. Afzal, A., Le Goues, C., Hilton, M., Timperley, C.S.: A study on challenges of testing robotic systems. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification. IEEE, pp. 96–107 (2020)

42. Schlegel, C., Steck, A., Brugali, D., Knoll, A.: Design abstraction and processes in robotics: from code-driven to model-driven engineering. In: International Conference on Simulation, Modeling, and Programming for Autonomous Robots, pp. 324–335. Springer (2010)

43. Ciccozzi, F., Di Ruscio, D., Malavolta, I., Pelliccione, P.: Adopting mde for specifying and executing civilian missions of mobile multi-robot systems. IEEE Access 4, 6451–6466 (2016)

44. Ciccozzi, F., Di Ruscio, D., Malavolta, I., Pelliccione, P., Tumova, J.: Engineering the software of robotic systems. In: 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), pp. 507–508. IEEE (2017)

45. de Araújo Silva, E., Valentin, E., Carvalho, J.R.H., da Silva Barreto, R.: A survey of model driven engineering in robotics. J. Comput. Lang, p. 101021 (2021)

46. Casalaro, Giuseppina Lucia, Cattivera, Giulio, Ciccozzi, Federico, Malavolta, Ivano, Wortmann, Andreas, Pelliccione, Patrizio: Model-driven engineering for mobile robotic systems: a systematic mapping study. Softw. Syst. Model. 21(1), 19–49 (2022). https://doi.org/10.1007/s10270-021-00908-8

47. Cattivera, G., Casalaro, G.: Model-Driven Engineering for Mobile Robot Systems: A Systematic Mapping Study. Malardalen University, Västerås (2015)

48. Skubch, H., Wagner, M., Reichle, R., Geih, K.: A modelling language for cooperative plans in highly dynamic domains. Mechatronics 21(2), 423–433 (2011)

49. Paraschos, A., Spanoudakis, N.I., Lagoudakis, M.G.: Model-driven behavior specification for robotic teams. In: AAMAS, pp. 171–178 (2012)

50. Gascuena, J.M., Navarro, E., Fernández-Caballero, A.: Model-driven engineering techniques for the development of multi-agent systems. Eng. Appl. Artif. Intell. 25(1), 159–173 (2012)

51. Pinciroli, C., Beltrame, G.: Buzz: an extensible programming language for heterogeneous swarm robotics. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3794–3800 (2016)

52. Dragule, S., Meyers, B., Pelliccione, P.: A generated property specification language for resilient multirobot missions. In: International Workshop on Software Engineering for Resilient Systems, pp. 45–61. Springer (2017)

53. Baer, P.A., Reichle, R., Geih, K.: The spica development framework-model-driven software development for autonomous mobile robots. In: Proceedings of the 10th International Conference on Intelligent Autonomous Systems (IAS-10-08), pp. 211–220 (2008)

54. Chen, T., Chen, Y., Guo, Q., Zhou, Z.-H., Li, L., Xu, Z.: Effective and efficient microprocessor design space exploration using unlabeled design configurations. ACM Trans. Intell. Syst. Technol. 5(1), 1–18 (2014)

55. Kim, R.G., Doppa, J.R., Pande, P.P.: Machine learning for design space exploration and optimization of manycore systems. In: 2018 IEEE/ACM International Conference on Computer-aided Design (ICCAD), pp. 1–6. IEEE (2018)

56. Gerasimou, S., Calinescu, R., Tamburrelli, G.: Synthesis of probabilistic models for quality-of-service software engineering. Autom. Softw. Eng. 25(4), 785–831 (2018)

57. Palesi, M., Givargis, T.: Multi-objective design space exploration using genetic algorithms. In: Proceedings of the Tenth International Symposium on Hardware/software Codesign, pp. 67–72 (2002)

58. Sengupta, A., Mishra, V.K.: Integrated particle swarm optimization (i-pso): an adaptive design space exploration framework for power-performance tradeoff in architectural synthesis. In: Fifteenth International Symposium on Quality Electronic Design, pp. 60–67. IEEE (2014)

59. Sengupta, A., Bhadauria, S.: Automated exploration of datapath in high level synthesis using temperature dependent bacterial foraging optimization algorithm. In: 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 1–5. IEEE (2014)

60. Schafer, B.C.: Probabilistic multiknob high-level synthesis design space exploration acceleration. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 35(3), 394–406 (2015)

61. Shathanaa, R., Ramasubramanian, N.: Design space exploration for architectural synthesis-a survey. In: Recent Findings in Intelligent Computing Techniques, pp. 519–527. Springer (2018)

62. Gamatié, A., Le Beux, S., Piel, É., Ben Atitallah, R., Etien, A., Marquet, P., Dekeyser, J.-L.: A model-driven design framework for massively parallel embedded systems. ACM Trans. Embedded Comput. Syst. 10(4), 1–36 (2011)

63. Neema, S., Sztipanovits, J., Karsai, G., Butts, K.: Constraint-based design-space exploration and model synthesis. In: International Workshop on Embedded Software, pp. 290–305. Springer (2003)

64. Larsen, K.G., Pettersson, P., Yi, W.: Uppaal in a nutshell. Int. J. Softw. Tools Technol. Transf. 1(1), 134–152 (1997)

65. Hegedüs, Á., Horváth, Á., Varró, D.: A model-driven framework for guided design space exploration. Autom. Softw. Eng. 22(3), 399–436 (2015)

66. Huck, T.P., Ledermann, C., Kröger, T.: Simulation-based testing for early safety-validation of robot systems. In: 2020 IEEE Symposium on Product Compliance Engineering-(SPCE Portland), pp. 1–6. IEEE (2020)

67. Serban, R., Taylor, M., Negrut, D., Tasora, A.: Chrono:: Vehicle: template-based ground vehicle modelling and simulation. Int. J. Veh. Perform. 5(1), 18–39 (2019)

68. Zhao, A., Xu, J., Konaković-Luković, M., Hughes, J., Spielberg, A., Rus, D., Matusik, W.: Robogrammar: graph grammar for terrain-optimized robot design. ACM Trans. Gr. 39(6), 1–16 (2020)

69. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30. IEEE (2017)

70. Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., Fox, D.: Closing the sim-to-real loop: adapting simulation randomization with real world experience. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 8973–8979. IEEE (2019)

71. Andrychowicz, O.M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al.: Learning dexterous in-hand manipulation. Int. J. Robot. Res. 39(1), 3–20 (2020)

72. Elkady, A., Sobh, T.: Robotics middleware: a comprehensive literature survey and attribute-based bibliography. J. Robot. 2012 (2012)

73. Henard, C., Papadakis, M., Harman, M., Le Traon, Y.: Combining multi-objective search and constraint solving for configuring large software product lines. In: 37th IEEE/ACM International Conference on Software Engineering, vol. 1, pp. 517–528. IEEE (2015)

74. Gerasimou, S., Camara Moreno, J., Calinescu, R., Alasmari, N., Alhwikem, F., Fang, X.: Evolutionary-guided synthesis of verified pareto-optimal MDP policies. In: 36th IEEE/ACM International Conference on Automated Software Engineering (2021)
**Publisher's Note** Springer Nature remains neutral with regard to juris-
dictional claims in published maps and institutional affiliations.

<figure><img src="image_18.png" /></figure>

**James Harbin** is a Research Associate at the University of York, UK. His research interests include cyber-physical and multi-robot systems, simulation development, and how model-driven engineering and evolutionary optimisation can improve the assessment of MRS system reliability, resilience and performance.

<figure><img src="image_19.png" /></figure>

**Simos Gerasimou** is an Assistant Professor at the Department of Computer Science at the University of York, UK. His research focuses on developing rigorous tool-supported approaches using model-based analysis, simulation and formal verification to support the engineering of trustworthy software for autonomous systems. His areas of expertise include safe AI, software engineering for AI and AI for software engineering, model-driven adaptive and autonomous systems, and

assurances for autonomous systems.

<figure><img src="image_20.png" /></figure>

**Nicholas Matragkas** is Research Engineer at CEA List, France. He is a member of the Embedded and Autonomous Systems Design Laboratory. Nicholas received his Ph.D. in Computer Science from the University of York in 2011. He has (co-)authored more than 60 scientific papers in international journals, conferences and workshops in the field of software engineering. His current research interests include model-driven engineering, software testing, and autonomous systems.

<figure><img src="image_21.png" /></figure>

domain, such as Rolls-Royce and Volkswagen.

<figure><img src="image_22.png" /></figure>

**Thanos Zolotas** is Assistant Professor in Software Engineering at the School of Computer Science and Mathematics at Liverpool John Moores University, UK. Athanasios received his EngD in Large-Scale Complex IT Systems from the University of York in 2017. His research interests are in model-driven engineering, big data analytics, safety-critical systems and requirements engineering, while he is collaborating with leading companies in the aerospace and automotive

**Radu Calinescu** is Professor of Computer Science at the University of York, UK. His main research interests are in formal methods for self-adaptive, autonomous, secure and resilient autonomous and AI systems, and in performance and reliability software engineering. He is an active promoter of formal methods at runtime as a way to improve the integrity and predictability of self-adaptive and autonomous systems and processes.

<figure><img src="image_23.png" /></figure>

**Misael Alpizar Santana** is a Ph.D. student at the University of York. His research area is in the field of machine learning, particularly deep neural networks applied to image classification and object detection and how risk factors can be considered during their training and verification.