Report of JA large jobs
IFERC PL: N.Nakajima (NIFS)

1.
2.

3.
4.

Background• Experiences in the Lighthouse projects
Results of the large job session• Large job session• GT5D• MEGA• MIPS
Summary of large jobs
Measures up to coming March

12/11/22

Internal interim report11. BackgroundExperiences in the Lighthouse projects

Strong scaling for GT5D

Strong scaling for MEGA
• 34 M (=256x512x256) grids• 34 M parUcles for α and D beam, w/ FLR• Good scaling up to 8192 cores (512 nodes)

It was very hard to implement large jobs using a large fracUon of the system.
It is very important to smoothly implement large jobs for Japanese importantcodes; GT5D and MEGA proposed as the benchmark codes, as well as EUcodes; GENE and ORB5.
12/11/22

Internal interim report22. Results of large job session
Large job session :• To clarify problems on large jobs with almost full system and to improvesituaUons by using a most clean conﬁguraUon
 To contribute to improvement of system, hardware, and so`ware
conﬁguraUons
 To select adequate parameters in mpi‐library and so on• From 2012/07/25, under the support of CSC and HPC support teams• Target codes : GT5D, MEGA, and MIPS
Herea`er, the results of the large job session will be presented.
Note that the meanings of “slowness” and “scahering” of elapsed Umes are
fairly intuiUve, and their rough deﬁniUons are as follow;• “slowness” of the elapsed Umes; a deviaUon of the elapsed Umes from an
expected scaling low• “scahering” of the elapsed Umes; a wide scahering of the elapsed Umes
mainly depending on the Uming of job submission
12/11/22

Internal interim report32. Results of large job session ‐ GT5D ‐Period : 2012/9‐10 (tests only in this period are summarized)• Inevitable workarounds:
See Helios User Manual > FAQ > Jobs > 16, andSee Helios User Manual > FAQ > Jobs > 17.
• mpi‐library Bullxmpi: all jobs successfully done. (512, 1024, 2048, 4096 nodes)
 Intelmpi: someUmes hanged up at allreduce, failed for a long nodelist.
• CriUcal issues for both libraries
 slowness of elapsed Umes
 scahering of elapsed Umes
• Future measures To compare source code with running results for slowness
 To direct monitor execuUng jobs in the large job session by HPC teamfor scahering of elapsed Umes

12/11/22

Internal interim report42. Results of large job session ‐ MEGA ‐Period : 2012/8‐9 (tests only in this period are summarized)• Inevitable workarounds:
See Helios User Manual > FAQ > Jobs > 16, andSee Helios User Manual > FAQ > Jobs > 17.
[Jobs with 4096 nodes become possible under those workaround.]• mpi‐library Bullxmpi: all jobs successfully done. (1024, 2048, 4096 nodes)
 Intelmpi: someUmes failed with “unexpected disconnect compleUonevent”• CriUcal issues for both libraries
 slowness of elapsed Umes (independent of existence of I/O)
 scahering of elapsed Umes
• Future measures To compare the source code with running results for slowness
 To direct monitor execuUng jobs in the large job session by HPC teamfor scahering of elapsed Umes
12/11/22

Internal interim report52. Results of large job session ‐ MIPS ‐Period : 2012/5‐11• Inevitable workarounds:
See Helios User Manual > FAQ > Jobs > 16, andSee Helios User Manual > FAQ > Jobs > 17.
• mpi‐library Bullxmpi: Jobs with full cores could run a`er 2012/11.
slowness for 2048 nodes disappears a`er 2012/10.
 Intelmpi: Jobs with full cores could run a`er 2012/11. However,
someUmes failed with “SOCKOPT ERR ConnecUon refused”• Tuning of source Read from all process
read by rank0 + broadcast (10 Umes faster)• CriUcal issues for both libraries
 slowness of elapsed Umes
 scahering of elapsed Umes
• Future measures To compare the source code with running results for slowness
 To direct monitor execuUng jobs in the large job session by HPC team forscahering of elapsed Umes
12/11/22

Internal interim report63. Summary of Large jobs
Large job session :• To clarify problems on large jobs with almost full system and to improvesituaUons by using most clean conﬁguraUon• From 2012/07/25, under the support of CSC and HPC support teams• Target codes : GT5D, MEGA, and MIPS
Improvements have been obtained especially a`er 2012/10 ‐ 11:• Improvement of system [Workarounds in shell script (see FAQ)]
See Helios User Manual > FAQ > Jobs > 16, andSee Helios User Manual > FAQ > Jobs > 17.
• ConUnuous improvement of conﬁguraUons• Several patches of Luster file system
Very recent situations :• bullxmpi (OK), intelmpi (someUmes failed, intrinsic problems?)• CriUcal issues: slowness and scahering of elapse Umes for both libraries
12/11/22

Internal interim report74. Measures up to coming March
Report meeUng of CSC acUviUes will be held on 3/20 (just before the PC‐12 tobe held on 3/21‐22).
• Results of the LHP will be reported.
• SituaUon of large jobs and scaling of typical codes, GENE, ORB5, GT5D andMEGA will be reported. ImplementaUon for GT5D and MEGA (GENE andORB5) will be done by CSC and HPC (HLST) teams.
Expected near future improvements• ApplicaUon of memory alignment to bullxmpi will be default on 11/27.
• Some hardware will be replaced and so`ware of Lustre ﬁle system will beupdated in 2012/12 or 2013/1.
For source codes• MIPS has already been given to HPC team.
• GT5D and MEGA have been sent to CSC support team.
• GT5D has been sent to HPC team very recently, and MEGA will be sent toHPC team with some noUces very soon.
12/11/22

Internal interim report84. Measures up to coming March
Issues to be solved up to coming March• scahering of elapsed Umes (for gepng stable operaUon)(Eﬀect of memory alignment to bullxmpi on issues is not clear now.)• slowness of elapsed Umes (for improving scaling properUes)
Measures to solve issues reﬂecUng users opinions• to directly monitor job processing in the large job session by HPC team, andto compare running results with source codes,• to submit in the large job session a set of the same jobs in every month, tomonitor the change in the situaUon of the system conUnuously,• to exchange informaUon on large jobs among CSC and HPC teams, and HLST,• to make more close communicaUon with developing team of Bull in France,• to re‐consider the collaboraUon method with intel for improvement ofintelmpi library (how to treat the relaUon with BA intellectual properUes),• to use some budget for improvement, if possible and necessary.

12/11/22

Internal interim report

9