## 4 Proposed Architecture
### Multi-Core and GPU Tightly Coupled Simulation
The process of copying data to the GPU can be ineffective if the copying process happens frequently or with too little data, causing the simulation-specific model evaluation and time synchronization to carefully consider memory management and copy strategies in CPU-GPU tightly-coupled architecture to increase utilization. Multiple CPUs, including multiple cores, execute the simulation kernel by assigning individual model evaluation in the GPU; this means that multiple access to GPU memory from CPU cores results in irregular bottleneck effects and overall performance degradation due to the CUDA context switching for the iterative evaluation of multiple DEVS-based simulation models. The integrated simulation of DEVS-based converted models of hybrid systems requires a simulator-level manipulation of memory interaction to increase the utilization of large-scale parallel execution in multiple GPU cores. Figure 3: Proposed CPU-GPU tightly coupled computation in DEVS-based simulation. In this paper, we propose the following two approaches for efficient data copy, distribution, and time synchronization in a large-scale simulation using asymmetric memory-interconnected architecture in heterogeneous CPU-GPU. First, we introduce the fine-grained simulation separation of event-driven models and continuous-time models to reduce the possibility of the drawback due to the data transfer overhead between CPU-GPU memory. Within maximum simulation time windows in which the two heterogeneous models are independently evaluated, event-driven simulation is performed in advance for all cell-based models in the fire-spreading territory; then, the simulation results are copied as a snapshot to GPU kernel, in which the continuous-time models of each fire-spreading cell are executed at the same time through multiple GPU cores. The experiment demonstrates that the proposed strategy is considered to be one of the best candidates with small data copy overhead between CPU and GPU memory. The simulation time-overhead reduction from this approach is because the small amount of simulation copy data is transferred in advance by interleaving the memory copy and simulation evaluation between two event-driven and continuous-time models. Second, the separated simulation kernels for a large number of simulation models are distributed into GPU cores, but GPU kernel-call overhead is inevitable, which is also one of the limiting factors of the overall performance of hybrid system simulation in CPU-GPU architecture. We introduce the simulation model description and the GPU utilization method based on CUDA MPS to reduce the time overhead due to the complexity of time synchronization between the large number of simulation models. To apply the proposed two approaches to extremely increase the utilization of CPU-GPU computing resources, we need to know the characteristics of the target model to accurately determine the maximum time window in which the event-driven simulation is evaluated in advance and in which the continuous-time models are simulated in parallel. Using this time value, we could optimally control the execution flow of the large-scale hybrid system simulation in terms of acceleration, as shown in Figure 4.  
The cellular model used as the target model in this study, and similar multiple-state connection models have large numbers of nodes, generally represented by differential equations. Figure 5 illustrates the concept of model specification that we used in our study. In the simulation, we used cellular models, which consist of many hybrid models, represented in Figure 5. The hybrid systems are combinations DES and CS. To accelerate them via partitioning based on their computation characteristics, we proposed a discrete-event simulation in multi-cores (CPUs) and continuous-time simulation in GPUs, as CS follows differential equations that require high-performance computing for accelerated simulation. In addition, we partitioned DES models into several sub-models for simulating in multi-cores.