Application of HPC-CFD for Industrial
Problems on the K Computer
and toward Fugaku
Makoto TSUBOKURATeam Leader of the Complex Phenomena Unified Simulation,
RIKEN Center for Computational ScienceProfessor, Kobe University
The 2nd R-CCS International Symposium
February 17th, 2020

RIKEN CENTER FOR COMPUTATIONAL SCIENCE

1

Backgrounds and Motivations

How HPC can change the industrial CFD?
 High-resolution turbulence simulation
Higher time resolution realizing the unsteady simulation, capturing the transientphenomena.

RANS

LES

Higher spatial resolution realizing the highly accurate simulation, independent onturbulence modeling.

 Coupled analysis realizing real-world simulation
Application of CFD to what conventional experiments are difficult to treat.

Coupling simulation of fluid motion with other physics such as structure
deformation/vibration, heat and mass transfer, or aero-acoustics.

 Big data analysis


Optimization, Machine/Deep learning based on AI technique.
2Backgrounds and Motivations

FrontFlow/red-HPC on the K computer

Unstructured Finite Volume Method.


Most popular and conventional data structure in industry.

Hybrid OpenMP/MPI for HPC.

Single node performance.



Thread parallelization by OpenMP.



9.5%/7.4% for hexa/tetra-hedral elements.

Parallel efficiency.
Domain decomposition by application “METIS”.

MPI among nodes.

96.5% parallelization efficiency (weak scaling).

Up to 10 billion unstructured meshes on 10,000nodes (80,000cores) on the K-computer .

Various moving boundary methods including ALE.

3

Backgrounds and Motivations

Success of the Unstructured CFD on the K ComputerPrecise prediction of aerodynamic forces comparable to wind-tunnel
measurements.
The key is the surface resolution of less than 1mm.
Coarse mesh(Conventional Supercomputer)

Fine mesh (K computer)

Real-World Aerodynamics Simulation.
Coupled with 6DoF vehicle motion anddriver’s reaction.

 Successful from the view point of academic research,

but…

4

Backgrounds and Motivations

Surface Clean-up for Mesh Generation
Industrial CAD is always very dirty…

Higher resolution requires very long time for CAD clean-up.

Resolution of less than 1mm surface is not realistic in industries.

・Red : gap
・Cyan: overlap

Original CAD data
Surface repair by 5mmresolution using conventional
wrapping technique (ANSA(R))

Surface repair by 1mm resolutionConventional wrapping techniquedoes not work…

5

Numerical Methods

CUBE: Building Cube Method for Unified Simulation
 Hierarchically structured Finite Volume MethodA solver for coupled phenomena:
fluid/structure/acoustics/chemical reaction…
Building Cube Method for the unified datastructure (Nakahashi et al., 2003)


Unstructured SAE 2014-01-0621

Easy tune for both single node and parallel performance

Immersed Boundary Method (Fadlun et al., 2002)(1) Dirty CAD treatment (Onishi et al., 2013)(2) Moving Boundary Method (Bale et al., 2016)(3) Unified Compressible/Incompressible analysis (Li)
(4) Unified Fluid/Structure analysis (Nishiguchi)
Structured(Cartesian) SAE 2014-01-0580

Domaindecomposition bydifferent size of cubes

Cube

Allocation of meshes
for each cube

Cell6

Numerical Methods

Performance on the K computer Single node performance.


16×16×16 cells per cube.

23.7% on the K computer (8 threads).

 Parallel efficiency.
othersgn_fill_corner_
cns_inflow_plug_syscall
flow_field.flow_field_fill_corner_field_
cns_sw_noslip_
field_intf.field_intf_q_e_
cns_solver.cns_solve._OMP_2_
vscf_ce2nd_tstep_rk3tvd_no_dt_
flux_roe_lm_

70

Effective parallelization ratio: 99.99954%.

60

75.324899% parallel efficiency (weak scaling).

40

50

30

Total nodes on the K computer

20

Expected to be 25 times faster on Fugaku
4×4×4

64

262,144

1

8×8×8

512

2,097,1528

16×16×16

4,096

16,777,216

64

32×32×32

32,768134,217,728

512

64×64×64

262,144 1,073,741,824

4,096

128x128x128

2,097,152 8,589,934,592

32,768100
1.E+09Total cell num. Node num.

1.E+081.E+071.E+06

GFLOPSCUBE num.

実行時間 [s]80

16 cubes per node.

90

1.E+05

1.E+04
1.E+03

1.E+02
1.E+01

1.E+001101001000

Number of nodes [-]10000

1000007

Numerical Methods

Full-scale vehicle aerodynamics simulation

 World-largest full-scale vehicle simulation (27 billion meshes).



Maximum of 27 billion meshes with 0.7 mm resolution within 1 hour from thedirty CAD data.

Condition
Finest grid size

0.763 [mm]Num. of cells

26,893,365,248

Grid generation time

About 20 min.

Pre-flow computational time (immersedboundary preparation)

About 30 min.

Delta t1.0×10-6 [s]Solution time

0.010 [s]Parallel num.

273,576 cores（37,197 nodes）

Flow computation time

Several days

8

Capacity Computing for Shape Optimization

Multi-Objective Shape Optimization
 4 objectives (0° and -3° yaws)

Drag and Lift at 0 yaw
 Delta Drag (difference bet. 0 and -3)
 Delta Lift (difference bet. 0 and -3)
 Smaller is better for all four variables 8 design parameters Multi-objective Genetic Algorithm

18 models for each generation

Aerodynamic characteristics（Objective function）
Text

Geometry
STL format

Shape parameters(Design variables)
Text

10

Capacity Computing for Shape Optimization

Results of multi-objective shape optimization

CDΔCDΔCDCL• Genetic Optimization until 12 th generations.
• Tradeoff between some objective functions
such as Cd and DCd.

CLΔCL

10

Real-World Simulation

Narrow band noise from a full-scale vehicle
 Acoustic feedback noise

generated at a small gap.
 Very peaky and uncomfortable…
 For the prediction, full coupling
simulation of flow and acoustics isneeded.
Grid size
Cube num.
Cell num.
Core num.

0.2 mm/1.6mm
471,059
1,929,457,664
13086x8 / 50 hrs
Cell

11

Real-World Simulation

Coupled Aerodynamics, Vehicle Motion and Driver’s Reaction Simulation•HPC-CFD for flow around a vehicle (Aerodynamics)•

Multi-Body vehicle motion analysis (6DoF body motion with suspension and steering)•

Autonomous Vehicle Motion by a Driver’s steering wheel, accelerator/braking actions•Lane changing motion at 100km/h

Toward FUGAKU

Coupling Data Science and HPC-CFD for Innovative CAE
Innovative Industrial CAE Solution by a Fusion of Data Science and HighPerformance Computing Simulation

Machine/Deep Learning, Data Assimilation, Multi-Objective Optimization…
Surrogate Model: Realizing real time evaluation of aerodynamic performance such asdrag and lift force.

Reduction Model: Realizing complicated real world simulation and reproducing flow fieldat lower cost.

Reduction Model by Proper-Orthogonal Decomposition

Neural Network

Surrogate Model
Real Time Performance Prediction

Physical Space

5Digital Twin

Cyber Space

432
10
空気抵抗

Co-operation design

操縦安定性

横風安定性

Multi-objective optimization
Virtual on-road test

Training AI for automatic driving

Toward FUGAKUShape Optimization based on a Surrogate Model Development of a Surrogate Model based on Neural Network.
Hundreds of HPC-CFD results as teaching data.

Prediction of Drag less than 5% error against the HPC simulation results.
0.28
0.26
0.24
0.22
0.2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

cal_Cd

est_CdCoupling the Surrogate Model with Multi-Objective Genetic Algorithm.
AI proposes high-performance vehicle shape.
Low emission car(Lowest CD)

CDHigh crosswind stability car(Lowest DCd)

ΔC14

Toward FUGAKUReduction Model of the Navier-Stokes Simulation
 Reduced order model by Proper-Orthogonal Decomposition.

Full flow simulation results are projected on
the reduced base functions.
𝑎0 𝑡 𝝋0 (𝒙)𝑎1 𝑡 𝝋1 (𝒙)𝑎2 (𝑡)𝝋2 (𝒙)𝑎3 (𝑡)𝝋3 (𝒙)𝑎5 (𝑡)𝝋5 (𝒙) The base functions are obtained by Neural Network(Murata).
Inner productScaling
Summationℱ𝑑𝑒𝑐1 𝑥 = 𝝋1 𝑥

ℱ𝑒𝑛𝑐 𝒙 =Full-NS simulation

𝝋1𝑇 𝒙
𝝋𝑇2 𝒙

Reduction by MD-CNN-AEℱ𝑑𝑒𝑐2 𝑥 = 𝝋2 𝑥

10,000 samples
2,000 epochs

2𝑷𝒙 =𝝋𝑘 𝝋𝑇𝑘 𝒙

𝑘=1

L2 Norm (Time averaged)15

Concluding Remarks
 CFD use in industries are conservative, which is just analternative to conventional experiments, so far…

 HPC expands the possibility of CFD by exceeding their accuracy

and applying to real-world problems, while data structure is thekey to massively utilize HPC environment.

 Hierarchically structured data realized very fast and real-world

aerodynamics simulation on the K computer.

 Coupling data science and HPC simulation will create next-generation Computer-Aided Engineering on FUGAKU.
Surrogate model for real time evaluation.

Reduction model for real world simulation.

16

Thanks to Industrial Partners
Consortium for next-generation Automotive CAE using HPC(Nov., 2017~)

Consortium for Combustion System CAE(April, 2018~)

Consortium for Construction CFD forWind-Resistance13