# Development of Multimodal Physical and Virtual Traffic Reality Simulation System
## 1 Introduction
The simulators are synchronized through CARLA, a high-fidelity 3D driving simulation platform, which is connected to Vissim, a microscopic traffic simulation tool, ensuring that background traffic behaves realistically and adaptive control strategies can be implemented. Additionally, physiological sensors, including heart rate monitors, EDA sensors, and EEG devices, are fully integrated using Lab Streaming Layer (LSL) to achieve a unified and synchronized dataset, allowing for a holistic approach to studying road user interactions. This setup enables a wide range of research applications, from analyzing individual stress responses to evaluating traffic safety and multimodal transportation efficiency. To demonstrate the capabilities of the proposed system, we conducted three experimental case studies involving real-time interactions between human participants and simulated traffic environments. The first focused on cyclist-vehicle interactions, capturing how bicyclists dynamically adjusted their lateral position in response to overtaking vehicles, along with associated heart rate changes. The second examined pedestrian behavior in response to vehicles approaching at different speeds (30 mph vs. 50 mph), measuring both gap acceptance decisions and physiological stress using VR and wearable sensors. The third case study evaluated the system's ability to replicate real-world traffic dynamics under congested urban conditions. Twenty-five participants completed identical driving tasks in both field and simulation environments along two major corridors in Pittsburgh. By comparing segment-wise speed profiles, we validated the realism of the simulated traffic, ensuring that prior interaction studies were grounded in credible conditions. Together, these case studies highlight the system's ability to support multimodal research through high-fidelity simulation, human-in-the-loop design, and synchronized physiological monitoring. By uniting drivers, bicyclists, and pedestrians in a shared virtual space with naturalistic controls, the platform enables rich behavioral insights for transportation safety, design, and automation research. An overview of the system architecture is shown in Figure 1.  
The rest of the paper is organized as follows. Section 2 provides a comprehensive literature review on co-simulation frameworks and multimodal simulation studies, highlighting previous research efforts and existing gaps. Section 3 presents the components of the proposed co-simulation system, including the software and hardware elements necessary for the integration of driving, bicycle, and pedestrian simulators. Section 4 details the methodology used to integrate CARLA, PTV Vissim, and physiological sensors, addressing key challenges such as real-time synchronization, network design, and traffic signal coordination. Section 5 presents the experimental case studies conducted to evaluate the system's capabilities in modeling bicycle-vehicle and pedestrian-vehicle interactions. It also reports the results, analyzing both the behavioral and physiological responses of participants across different traffic scenarios. Finally, Section 6 presents conclusions and future directions for enhancing co-simulation environments.