# Presentation: 553249

***Software pipeline in QuickPIC  ***
**Chengkun Huang**
**UCLA**

## Collaborators

***B. Feng,  T. Katsouleas ***
***J. H. Cooley, T. M. Antonsen  ***
***J. Vieira, L. O. Silva ***

***Collaborators***

## Challenge in PIC modeling

***Challenge in PIC modeling***

- Typical 3D high fidelity PWFA/LWFA simulation requirement

| PWFA | Feature | Grid size limit | Time step limit | Total time of simulation per GeV stage (node-hour)* |
| --- | --- | --- | --- | --- |
|  | Full EM PIC  | ~0.05c/p | t<     0.05p-1 | 1500 |
|  | Quasi-static PIC | ~0.05c/p | t<0.05-1
 = | 4 (20) |

| LWFA | Feature | Grid size limit | Time step limit | Total time of simulation per GeV stage (node-hour) |
| --- | --- | --- | --- | --- |
|  | Full EM PIC | ~0.05  | t< 0.05 0-1 | ~ 500000 |
|  | Ponderomotive
Guiding center PIC | ~0.05c/p
 | t< 0.05p-1
 | ~1500
 |
|  | Quasi-static PIC
 | ~0.05c/p
 | t < 0.05r
 | ~ 10
 |

**Notes:**
pwfa, 16x16x16 c/wp, 4 ppc, 2 micro sec/ particle/step

## QuickPIC

- New Features
- Pipelining
- Parallel scaling to 1000+ processors

- Massivelly Parallel, 3D Quasi-static particle-in-cell code
- Ponderomotive guiding center for laser driver
- 100-1000+ savings with high fidelity
- Field ionization and radiation reaction included
- Simplified version used for e-cloud modeling
- Developed by the UCLA+UMaryland+IST:

- Chengkun Huang: huangck@ee.ucla.edu
*http://exodus.physics.ucla.edu/*
*http://cfp.ist.utl.pt/golp/epp*

## Quasi-static Model

- There are two intrinsic time scales, one fast time scale associated with the plasma motion and one slow time scale associated with the betatron motion of an ultra-relativistic electron beam.
- Quasi-static approximation eliminates the need to follow fast plasma motion for the whole simulation.
- Ponderomotive Guiding Center approximation: High frequency laser oscillation can be averaged out, laser pulse will be repre-sented by its envelope.

- Quasi-static

- Ponderomotive Guiding Center

## Implementation

***Implementation***

- The driver evolution can be calculated in a 3D moving box, while the plasma response can be solved for slice by slice with the longitudinal index being a time-like variable.

## Exploiting more parallelism: Pipelining 

- Pipelining technique exploits parallelism in a sequential operation stream and can be adopted in various levels.
- Modern CPU designs include instruction level pipeline to improve performance by increasing the throughput.
- In scientific computation, software level pipeline is less common due to hidden parallelism in the algorithm.
- We have implemented a software level pipeline in QuickPIC.

**Moving Window**

|  | Instruction pipeline | Software pipeline |
| --- | --- | --- |
| Operand | Instruction stream | Plasma slice |
| Operation | IF, ID, EX, MEM, WB | Plasma/beam update |
| Stages | 5 ~ 31 | 1 ~(# of slices) |

## Pipelining: scaling QuickPIC 

- beam

**Initial plasma slab**

**Without pipelining: Beam is not advanced until entire plasma response is determined**

***Pipelining: scaling QuickPIC ***
***to 10,000+ processors***

## More details

- Step 1

- Step 2

- Step 4

- Step 3

**Plasma **
**update**

**Beam**
**update**

- Plasma slice

- Guard cell

- Particles leaving partition

- Computation in each block is also parallelized

- Time

- Stage

## Code Verification 

**Electron beam densities in the PWFA simulations at the 150****th**** time step**

**Horizontal and vertical spot sizes of the beam in basic quasistatic QuickPIC ecloud simulation **

**with pipelining **
**algorithm**** **

**original QuickPIC**

***Code Verification ***

- Feng et al, submitted to JCP

## Scaling to 1,000+ processors

- Near ideal speed up for 2D module in pipeline mode
- Data transfer between successive stages is inexpensive and independent of the number of stages.
- Data transfer overlaps with computation.
- Speedup drops when overhead  becomes comparable to computation time.

***Scaling to 1,000+ processors***

**Overall performance**
- 16 stages pipeline is 85% efficient compared with single (no) pipeline calculation
- 13.6 times throughput improvement with 16 stages pipeline.
- Relatively independent of # of pipeline

**Number of stages**

- 6416 CPUs

- 648 CPUs

## Performance in pipeline mode 

***Performance in pipeline mode****** ***

- Fixed problem size, strong scaling study, increase number of processors by increasing pipeline stages
- In each stage, the number of processors is chosen according to the transverse size of the problem.
- Benchmark shows that pipeline operation can be scaled to at least 1,000+ processors with substantial throughput improvement.

- Feng et al, submitted to JCP

## Enhancing pipeline operation

- Enhanced domain decomposition (in Z -> in Y, or 2D decomposition) for better job allocation. Also enables us to use more processors in the transverse direction and extremely small cell size which is required for simulating narrow trailing beam in PWFA-LC.
- For small pipeline stage, 3D beam update  should be load balanced.
-        For example, 16 pipeline stages,
- 2D: 1102.8 sec / 79.4 sec = 13.9 times faster
- 3D: 15.5 sec / 2.2 sec = 7.04

- y

- z

- x

***Enhancing pipeline operation***

## Pipelining for laser solver

**Ponderomotive guiding **
**center approximation:**
**      **** **Big 3D time step

**Plasma evolution: **
- Maxwell’s equations Lorentz Gauge
-        Quasi-Static Approximation

***Pipelining for laser solver***

## Summary

** ****QuickPIC enables scientific discovery in plasma-based acceleration by allowing 100-1000 times time-saving for simulations of state-of-art experiments and for exploring parameter space which are not easily accessible through conventional PIC code.**
** ****We are working to scale QuickPIC to the petascale platform using the software pipelining technique. Initial benchmark shows very promising performance enhancement.  **