# Presentation: 116059

## ORBITCalculation of Transverse Space Charge Forces  through Parallel Computing

- A.U. Luccio, N.L. D’Imperio

## Multiple-Instruction Multiple-Data(MIMD)Shared Memory Architecture

## Multiple-Instruction Multiple-Data(MIMD)Distributed Memory Architecture(Nodes are Shared Memory Architecture)

## Message Passing Interface(MPI)

- The Data is divided among the processes.  As the Programs are executed in parallel on the separate processes, communication between processes is through message passing.  A process is not to be confused with a processor.  A process is an instance of a program that is executing on a physical processor.

## Division of a Herd of Macro Particles to Sub-Herds

- In a Serial implementation of ORBIT the herd of Macro Particles is propagated through the accelerator lattice which consists of various magnets.  The lattice elements are represented by matrices in the simulation.  In the Parallel implementation, the herd is divided into sub-herds of equal size and each process propagates its respective sub-herd through the accelerator lattice.

## ORBITProgram FlowTransverse Space Charge

- Herd
- Creation

- Particle
- Binning

- Calculate
- Forces

- Propagate
- through
- Lattice
- Element

- The Sub-Herd of Particles is created by each process at the beginning of a turn. Each process bins its particles and calculates the forces created by its sub-herd.  The forces from each process are gathered by the root node which then sums the forces.  The sum of the forces are then scattered to the other processes.  Each process then propagates its sub-herd through a lattice element after which the sequence begins again.

- Particle
- Binning

- Particle
- Binning

- Calculate
- Forces

- Calculate
- Forces

- Propagate
- through
- Lattice
- Element

- Propagate
- through
- Lattice
- Element

- Herd
- Creation

- Herd
- Creation

## Binning the MacroParticles

- The macro particles are binned on a mesh.  Each
- macro particle is divided into 4 fractional parts
- which are then assigned to the 4 surrounding mesh points.

## Inverted L Matrix Scattered to Processes

- The inverted L matrix is read by the Root node.
- At the start of program execution the Root node
- distributes n/p rows of the matrix to each of the
- other nodes.  The inverted L is stored in a file and
- can therefore be distributed directly from disk thereby
- eliminating the necessity of storing the entire
- matrix on any single node.

## Calculating Potential on all processes

## Idealized Fields

- The shape of the
- macro-particle
- distribution on
- the grid.

- The potential of
- the combined
- charge distribution.

- The force
- representation
- on the grid

## MachineSpecifications

- Galaxy Cluster*

- Nodes (clc000):
- CPU specs:
- Nodes (clc001-clc018):
- CPU specs:
- Interconnection:
- File System:
- MPI:

- 2 CPU’s 256 Mbytes RAM
- 300 MHz Intel Pentium II (Klamath)
- 512 kBytes L2 cache
- 2 CPU’s 512 Mbytes RAM
- 500 MHz Intel Pentium III (Katmai)
- 512 kBytes L2 cache
- Cisco Catalyst 2900 switch
- 100 Mbit/sec Fast Ethernet
- NFS
- MPICH (MPI 1.2 compliant)

- *Note:  Galaxy consists of many more nodes than listed
-              here.  These were the nodes used for the analysis.

## ORBITTimings

- Grid		[33/65/33]     [33/33/33]     [65/129/65]
- _Routine__	__	__	__Elapsed Time [s]_
- LU Solver
- AllReduce charge	   0.0129	         0.0126	0.0443
- Potential calc.	   0.0020	         0.0020            0.0508
- Distribute potent.	   0.0127	         0.0132            0.0451
- Calc. Fx	                   6.0338E-5     6.0341E-5      2.4732E-4
- Calc. Fy	                   4.9372E-5     4.9338E-5      2.0733E-4
- Total cycle 	   0.0279           0.0278            0.1407
- FFT
- AllReduce charge	   0.0443	         0.0126            0.1594
- Calc. Forces	   0.0095	         0.0021            0.0964
- Total cycle	   0.0538           0.0147            0.2558
- Brute Force
- AllReduce charge      0.0127           0.0127             0.0469
- Calc. Forces	  0.1216	         0.1216             2.0832
- Total cycle                0.1343            0.1344             2.1300

## PerformanceScaling

- Performance results of ORBIT runs utilizing
- increasing numbers of processors.  The
- performance quotient is calculated by dividing
- the base runtime (1 calculating processor) by
- the test runtime.  The results show a linear
- speed up over 32 processors.

## Graphics

- Graphics have been added to ORBIT.  All processes
- currently write 4 components of data (x, y, z, dE)
- to a single file for each turn of the simulation.  The
- file can later be rendered for visualiztion using the
- package OpenDX.

## Future Directions

- Implement a different LU solver using ScaLapack.
- Implement an iterative solver.
- Add Increased Graphics capabilities.
- Model the effects of walls on the Space Charge