# Presentation: 134204

## Full Brillouin-Zone, Charge Self-consistent Quantum Transport Simulations Enabled by Parallelization of the Nanoelectronic Modeling Tool (NEMO 1-D) on a Beowulf Cluster

- Gerhard Klimeck
- Jet Propulsion Laboratory, California Institute of Technology
- gekco@jpl.nasa.gov
- (818) 354 2182
- http://hpc.jpl.nasa.gov/PEP/gekco

**Motivation:**
**	****	**Quantitative resonant tunneling diode simulation
- 	Need for parallelization
**Parallelization on a Beowulf cluster:**
**	**Beowulfs: Linux based PC’s coupled with commodity networking switch.
- 	Coarse, medium, fine and mixed grain parallel implementations
**Conclusions**

## NEMO: A User-friendly Quantum Device Design ToolFormalism, Physics, and Technology

**NEMO: A User-friendly Quantum Device Design Tool**Formalism, Physics, and Technology

**Approximately 250,000 lines of code**

## Resonant Tunneling Diode

**Conduction band diagrams ****for different voltages****and the resulting current flow.**

## Transport in Momentum and Energy Space

**2D integral**

**1D integral - Tsu-Esaki**

## Transport in Momentum and Energy Space

**2D integral**

**1D integral - Tsu-Esaki**

## Previously Used Approx.: 1D Charge & 2D Current

**Charge self-consistency results in a dramatic overshoot**
**Problem:**
**alignment of emitter bound and central resonance states **
**assumption of parabolic transverse subbands.**
**Partial Solution:**
**Perform self-consistency for N,V using the 1D integral.**
**Follow calculation with a single pass of of a 2D integral for the current.**

## Really Required: 2D integral throughout

**Really Required: ****2D integral throughout**

**Charge self-consistency results in a dramatic overshoot**
**Problem:**
**alignment of emitter bound and central rsonance states **
**assumption of parabolic transverse subbands.**
**Partial Solution:**
**Perform self-consistency for N,V using the 1D integral.**
**Follow calculation with a single pass of of a 2D integral for the current.**
**Needed:**
**Perform self-consistency for N, V, and I using the double integral in k and E.**
**Single CPU: 1-2 weeks to compute.**
**Need to parallelize the code.**

## Need for Parallelization / Look at Algorithm

**2D integral**

**Have three levels of possible parallelization:**
**Loop over bias points      ****	****	****coarse grain**
**Loop over momentum points****	****	****medium grain**
**Loop over energy points****	****	****fine grain**

## Parallelization Around Bias Points

**Master-Slave approach **
**master only distributes and gathers individual bias points**
**This I-V has 70 bias points:**
**Good scaling up to 15 CPUs, Strong steps at 36 and 24 CPUs / Load imbalance**
**Max speed-up: ****	****	****	****21 at 24 CPU****	****87% efficiency****	****	****	****	****	****30 at 36 CPU     ****	****83% efficiency        **
**Large CPU speed-up:****	****32 at 64 CPU****	****50% efficiency**

## Parallelization Around Momentum Integral

**J(k) is a smooth function for electrons**
**Need only a few k-points to resolve**
**Exception:**
**Hole transport J(k) is spiked in k**

## Parallelization Around Momentum IntegralHole Transport needs fine k-resolution

## Parallelization Around Momentum Integral

**J(k) is a smooth function for electrons**
**Need only a few k-points to resolve**
**Exception:**
**Hole transport J(k) is spiked in k**

## Parallelization Around Momentum Integral

**K-grid has 21 points:**
**Even bin distribution in one communication step.**
**Results:**
**Good scaling up to 8 CPUs, Strong steps at 11 and 21 CPUs**
**Load imbalance**
**Max speed-up:  ****	****	****10 at 11 CPU****	****91% efficiency****	****	****	****	****	****18 at 21 CPU    ****	****86% efficiency         **
**Large CPU Speed-up:****	****18 at 64 CPU****	****28% efficiency**

## Parallelization Around Energy Integral

**Adaptive mesh refinement starting from 50-200 nodes**
**Refinement ends on 2 nodes at a time for 1 resonance**
**Energy grid is at the lowest level -> communication overhead**
**Maximum speed-up:  ****	****7.5 at 10 CPU****	****75% efficiency****	****	****	****	****15 at  40 CPU****	****38% efficiency**
**Large CPU speed-up: ****	****13 at 64 CPU****	****20% efficiency**

## Parallelization On One Level

**Parallelization on 64 CPUs is unsatisfactory in all one level algorithms!**
**For this particular benchmark - typical for electron RTD computation**
**How about parallelization on multiple levels?**

## Parallelization Multiple Levels

**Parallelization on 64 CPUs is unsatisfactory in all one level algorithms!**
**How about parallelization on multiple levels?**
**4 possibilities:  I-K, I-E, K-E, I-K-E     all of them implemented**
**Try to maximize number of CPUs in the coarser grids**

## Parallelization Around Bias and Momentum

**Still see some load imbalance problems**
**Maximum speed-up:  ****	****40 at 43 CPU****	****93% efficiency****	****	****	****	****46 at 57 CPU****	****81% efficiency**
**Large CPU speed-up: ****	****45 at 64 CPU****	****70% efficiency**
**Compared to I parallelism only:**
**Large CPU speed-up:****	****32 at 64 CPU****	****50% efficiency**

## Parallelization Around Bias and Energy

**Much fewer load imbalance problems**
**Maximum speed-up:  ****	****43 at 58 CPU****	****74% efficiency**
**Large CPU speed-up: ****	****42 at 64 CPU      ****	****65% efficiency**
**Compared to I parallelism only:**
**Large CPU speed-up:****	****32 at 64 CPU****	****50% efficiency**

## Conclusions

**Efficient parallelization for realistic RTD simulations is non-trivial.**
**Parallelization on multiple levels:**
**Flexibility to tackle different problems**
**Enabled simulation of fully charge self-consistent simulations **
**Reduced compute time from 14 days to 6 hours.**
**Beowulf clusters are affordable and useful for computational electronics.**

## Acknowledgements

**NEMO was developed under a government contract to Texas Instruments and Raytheon from 1993-1997**
**Theory**
**Roger Lake, Chris Bowen, Gerhard Klimeck, Tim Boykin (UAH), Dejan Jovanovic **
**Graphical User Interface**
**Dan Blanks, Gerhard Klimeck**
**Programming Approach, Philosophy, and Prototypes**
**Bill Frensley (UTD), Gerhard Klimeck, Chris Bowen**
**Coding Help**
**Manhua Leng (UTD), Chenjing Fernando, Bill McMahon, Paul Sotirelis, Dejan Jovanovic, Mukund Swaminathan (UTA)**
**Experiments for verification**
**Ted Moise, Alan Seabaugh, Tom Broekaert, Berinder Brar, Yung-Chung Kao**
**NEMO 3-D is being developed (1998-...) at JPL by Chris Bowen, Fabiano Oyafuso, Tom Cwik and Gerhard Klimeck.**