# MBSE Testbed for Rapid, Cost-Effective Prototyping and Evaluation of System Modeling Approaches
## 4 Quantitative Analysis  
The MBSE testbed allows users to integrate and evaluate different machine-learning models with different parameters. We created an exemplar simulation environment. The exemplar simulation environment consists of a UAV in an indoor building setup searching for a predefined object. In the search mission, the UAV agent receives a negative reward for colliding with objects in the environment and a positive reward for touching the predefined goal object. Users can test multiple reinforcement learning algorithms on the UAV agent. We tested the integrated models with the UAV making observations in the environment and then taking corresponding actions. Five reinforcement learning algorithms were evaluated in the experiment: proximal policy optimization (PPO), soft actor-critic (SAC), PPO with generative adversarial imitation learning (GAIL), PPO with behavioral cloning (BC), and PPO combined with GAIL and BC [1; 2; 3; 4]. Model parameters such as cumulative reward, episode length, policy loss, and entropy were evaluated against the number of simulation runs for selected reinforcement learning algorithms. Figure 15 presents the learning cycle, while Figure 16 presents the testbed setup for quantitative analysis. As shown in Figure 16, the structural model in SysML is mapped to the 3D virtual environment. For this mapping, a Python XMI translation tool was built. This tool automatically populates the asset container in the simulation environment using the SysML models. Objects in the simulation environment are stored in an asset container. Figure 16 displays the SysML block definition diagram for the \"indoor search\" scenario context. This diagram presents various entities that are part of the scenario. The blocks in the diagram are arranged under the \"ScenarioEntities\" package. The structured packaging of SysML model entities facilitates the extraction of model elements. Figure 15: Learning cycle. The UAV, indoor environment, and goal object blocks are extracted from the scenario context SysML model. These block names are then matched with existing 3D objects in the repository. When a match is found, the object in the repository is duplicated and inserted into the 3D virtual environment's asset container. When a matching 3D object is not present in the repository, the translation program creates an empty object in the asset container for the user to further modify. The user can then employ the asset container to populate objects in the simulation. Observations and actions are defined for the simulation setup, and the rewards mechanism is created in the experiment. As noted earlier, five models were evaluated in the experiment: proximal policy optimization (PPO), soft actor-critic (SAC), PPO with generative adversarial imitation learning (GAIL), PPO with behavioral cloning (BC), and PPO combined with GAIL and BC [19; 20; 21; 22]. We used the Unity ML agents kit to extend the testbed capabilities [23]. Figure 17 presents simulation runs on the horizontal axis, and cumulative rewards gained by the agent for a given model on the vertical axis. For PPO and PPO with GAIL, the mean cumulative episode reward increases as the training progress. For the rest of the agents, reward decreases, indicating that the agent is not learning successfully for the given simulation environment and for the given simulation runs. Figure 16: Quantitative analysis setup using testbed. Figure 17: Simulation runs vs. cumulative reward. Figure 18a shows that the mean length of episodes goes down in the environment for successful agents as the training progresses. The \"is training\" Boolean in Figure 18b indicates whether the agent is updating its model or not. The different models exhibit different behaviors for a given simulation set up. The way the user sets up the training environment impacts the performance of the models differently. In Figure 19a,b, the policy loss parameter indicates how much the policy is changing for each agent.