# Presentation: 417810

## Parallel Discrete Event Simulation (PDES) at ORNL

- Kalyan S. Perumalla, Ph.D.
**Modeling & Simulation Group**
**Computational Sciences & Engineering**

## PDES: Selected application areas

**Network simulation**
**Internet protocols, security, ****P2P designs, ...**
**Traffic simulation**
**Emergency planning/response, environmental policy analysis, urban planning, ...**
**Social dynamics simulation**
**Operations planning, foreign policy, marketing, ...**
**Sensor simulations**
**Wide area monitoring, situational awareness, border surveillance, ...**
**Organization simulations**
**Command and control, business processes, ...**

**Emergencies**

**Current and future defense systems**

**Protection and awareness systems**

**Global and local events**

## High-performance PDES kernel requirements

**Global time synchronization**
**Total time-stamped ordering of events**
**Paramount for accuracy**
**Fast synchronization**
**Scalable, application-independent, time-advance mechanisms**
**Critical for real-time and as-fast-as-possible execution**
**Support for fine-grained events**
**Minimal overhead relative to event processing times**
**Application computation is typically only 5 μs to 50 μs per event**
**Conservative, optimistic, and mixed modes**
**Need support for the principal synchronization approaches**
**Useful to choose mode on per-entity basis at initialization**
**Desirable to vary mode dynamically during simulation**
**General-purpose API**
**Reusable across multiple applications**
**Accommodates multiple techniques**
**Lookahead, state saving, reverse computation, multicast, etc.**

## μsik—unique PDES “micro-kernel”

**Some recent results of ****fine-grained**** PDES benchmark (phold)**
**Among the largest/fastest scalability results in parallel discrete event simulation**

**LP = 1 thousand, MSG = 1 million**
**LP = 1 million, MSG = 100 million**
**LP = 1 million, MSG = 1 billion**

**Unique mixed-mode kernel**
**The only scalable mixed-mode kernel in the world**
**Supports conservative, optimistic, and mixed modes in a single kernel**

**Used in a variety of applications**
**DES-based vehicular traffic models**
**DES-based plasma physics models**
**DES-based neurological models**
**Largest Internet simulations**

## μsik scaled to more than 104 processors

**Some recent results of ****fine-grained**** PDES benchmark**
**On Blue Gene Watson (BGW) at IBM TJ Watson Research Center**
**Well-known PHOLD benchmark, with 1 million logical processes, 10 million pucks**
**The largest and fastest scalability results in PDES recorded to date**

## μsik micro-kernel internals

**Processable**

**P****p**

*User LPs*

*Kernel LPs*

*Micro-kernel*

**Future event list****Processed event list****Local virtual time**

**When update kernel Qs?**
** ****New LP added or deleted**
** ****LP executes an event**
** ****LP receives an event**

**LP = Logical process****KP = Kernel process****ECTS = Earliest committable time stamp****EPTS = Earliest processable time stamp****EETS = Earliest emittable time stamp****PEL = Processed event list****FEL = Future event list****LVT = Local virtual time**

**EPTS Q**

**Committable**

**P****c**

**ECTS Q**

**Emittable**

**P****e**

**EETS Q**

**LP**

**LP**

**LP**

**LP**

**LP**

**PEL→****t**

**LVT**

**FEL→****t**

**KP**

**KP**

**KP**

**KP**

## libSynk: μsik’s synchronization core

**OS/Hardware**

**libSynk**

**μsik**
**Process**

**μsik**

**μsik**
**Process**

**μsik**
**Process**

**TM**

**TM Red**

**TM Null**

**RM**

**RM Bar**

**FM Myr**

**FM TCP**

**FM**

**FM ShM**

**FM MPI**

**X**

**Y**

**Implies X uses Y**

**Network**

## μsik micro-kernel capabilities

**μsik is currently able to support the following:**
**Lookahead-based conservative and/or optimistic execution**
**Reverse computation-based optimistic execution**
**Checkpointing-based optimistic execution**
**Resilient optimistic execution (zero rollbacks)**
**Constrained, out-of-order execution**
**Preemptive event processing**
**Any combinations of the above**
**Automated, network-throttled flow control**
**User-level event retraction**
**Process-specific limits to optimism**
**Dynamic process addition/deletion**
**Shared and/or distributed memory execution**
**Process-oriented views**
**It accommodates addition of the following:**
**Synchronized multicast**
**Optimistic dynamic memory allocation**
**Automated load-balancing**

## SensorNet: Parallel simulation/ immersive test-bed

**Seamless integrated testbed to incorporate a variety of important simulations, stimulations, and live devices**
**Achieves unified capabilities and significant fidelity for test and evaluation of CB sensor device-based designs, concepts, and operations**

**TOSSIM**

**Environ-mental****Model Proxy**

**SCIPUFF**

**Weather Model**

**Plume Models**

**Comm. Models**

**Plume****Models**

**Sensor Net Simulator**

**Weather Simulator**

**Operations Simulator**

- SAF

**Live****Devices**

**CB Sensor Network Testbed Framework**

**Current**

**Envisioned**

## SensorNet: Simulation-based analysis for plume tracking

**Environmental phenomenon exhibits high variability.**
**Phenomenon drives the sensor network’s computation and communication.**
**Trace gathered at base station of sensed phenomenon reflects high variability.**
**Communication effects induce unpredictable gaps ****in series.**
**Accurate, integrated simulation of phenomenon and communication captures complex interdependencies.**

**-96.0**

**-91.2**

**-86.4**

**Longitude**

**40.6**

**42.6**

**44.6**

**46.6**

**48.5**

**50.5**

**Turbulent Winds**

**Surface Dosage**
**CHEM at T = 17.0 h**

**40.6**

**42.6**

**44.6**

**46.6**

**48.5**

**50.5**

**Light Winds**

**Surface Dosage**
**CHEM at T = 17.0 h**

**40.6**

**42.6**

**44.6**

**46.6**

**48.5**

**50.5**

**No Winds**

**Surface Dosage**
**CHEM at T = 17.0 h**

**Kg********s/cm****3**

**1.00E-02**

**1.00E-03**

**1.00E-04**

**1.00E-05**

**1.00E-06**

**1.00E-07**

**1.00E-08**

**Latitude**

## SCATTER: Ultra-scale PDES-based mobility simulations

**Scalable tool for transportation and ****energy/event/emergency research**
**Regional scale: multiple states**
**10****6****–10****7**** intersections**
**Current tool capabilities**
**At most 10****4**** intersections**
**Faster than real time is very useful**

***Higher***
***accuracy***

***Lower***
***accuracy***

***Fidelity***

***Speed***

***Faster***

***Slower***

**Desirable**

***Network flow***
***methods***

**OREMS**

**CORSIM**

**MITSIM**

**TRANSIMS**

**SCATTER**

**Good for emissions estimates, traffic analysis,  . . .**

**Good for rough estimates of evacuation delay,  . . .**

**Our approach: SCATTER**
**DES models**
**vs time-stepped**
**Parallel execution**
**vs sequential**
**Scalability to high-performance computing**
**10****2****–10****3**** CPUs**
**Important behaviors**
**kinetic + non-kinetic**

## SCATTER: Benchmark performance

**1**

**10**

**100**

**1000**

**10000**

**100000**

**1000000**

**μs/event**

**Number of vehicles**

**Nodes = 9**

**Nodes = 1089**

**Event processing speed**

**1**

**10**

**100**

**1000**

**10000**

**10000**

**100000**

**1000000**

**Real time/simulated time**

**Number of vehicles**

**Nodes = 9**

**Nodes = 1089**

**Speedup over real-time**

**0.0**

**0.5**

**1.0**

**1.5**

**2.0**

**2.5**

**3.0**

**3.5**

**1**

**2**

**3**

**4**

**Speedup over sequential run**

**Number of processors**

**Parallel Speedup**

***Significant speedup with parallel execution!***

***Significantly faster than real time with 1 million vehicles!***

## Contact

-    Perumalla_PDES_SC07

- Kalyan S. Perumalla
- Modeling & Simulation GroupComputational Sciences & Engineering
- (865) 241-1315
- perumallaks@ornl.gov