Chapter 20

System Operations andDevelopment Team
20.1

Members

Atsuya Uno (Team Head)
Hitoshi Murai (Research & Development Scientist)
Motoyoshi Kurokawa (Research & Development Scientist)
Keiji Yamamoto (Research & Development Scientist)
Fumio Inoue (Research & Development Scientist)
Yuichi Tsujita (Research & Development Scientist)
Mitsuo Iwamoto (Technical Staff)
Katsufumi Sugeta (Technical Staff)

20.2

Research Activities

K computer, a distributed-memory parallel computer comprising 82,944 computing nodes, has playeda central role in the High Performance Computing Infrastructure (HPCI) initiative granted by theMinistry of Education, Culture, Sports, Science and Technology. The HPCI has achieved the integrated operation of the K computer and other supercomputer centers in Japan and has enabled
seamless access from user machines to a cluster of supercomputers that includes the K computer.
Moreover, the HPCI has provided large-scale storage systems that are accessible from all over Japan.
The System Operations and Development Team (SODT) has conducted research and developmenton the advanced management and operations of the K computer. While analyzing operational
statistics collected during shared use, the SODT has improved the system configuration, including
aspects involving job scheduling, the file system, and user environments. As an example, achieving
higher system utilization is very difficult, because the K computer must simultaneously process
various sizes and types of jobs. The SODT has responded flexibly to user requests and analyzed
operational status, thereby realizing a high level of utilization of approximately 75% in the fiscalyear 2015 (FY2015). Moreover, the SODT has developed tools that improve the usability of the K
computer. The SODT has also helped users manage the K computer and utilize the K computerresources effectively by improving the system software. Note that this support was achieved together
with the software development team.
In FY2015, we primarily implemented improvements to operational issues. In particular, weaddressed the performance degradation issue of the file system and the long waiting times of somejobs. In addition, we implemented a performance monitoring system for the global file system (GFS)
and local file system (LFS). As for electric power use, we developed an automatic emergency jobstopping method and implemented a system to estimate future electric power needs.
165

Chapter 20. System Operations and Development Team
100%	  >80000	  90%	  50001-­‐80000	  80%	  20001-­‐50000	  70%	  10001-­‐20000	  60%	  5001-­‐10000	  50%	  2001-­‐5000	  40%	  1001-­‐2000	  501-­‐1000	  30%	  101-­‐500	  20%	  <101	  10%	  4/3-­‐4/5	  4/6-­‐4/12	  4/13-­‐4/19	  4/20-­‐4/26	  4/27-­‐5/3	  5/4-­‐5/10	  5/11-­‐5/17	  5/18-­‐5/24	  5/25-­‐5/31	  6/1-­‐6/7	  6/8-­‐6/14	  6/15-­‐6/21	  6/22-­‐6/28	  6/29-­‐7/5	  7/6-­‐7/12	  7/13-­‐7/19	  7/20-­‐7/26	  7/27-­‐8/2	  8/3-­‐8/9	  8/10-­‐8/16	  8/17-­‐8/23	  8/24-­‐8/30	  8/31-­‐9/6	  9/7-­‐9/13	  9/14-­‐9/20	  9/21-­‐9/27	  9/28-­‐10/4	  10/5-­‐10/8	  10/16-­‐10/18	  10/19-­‐10/25	  10/26-­‐11/1	  11/2-­‐11/8	  11/9-­‐11/15	  11/16-­‐11/22	  11/23-­‐11/29	  11/30-­‐12/6	  12/7-­‐12/13	  12/14-­‐12/20	  12/21-­‐12/27	  12/28-­‐1/3	  1/4-­‐1/10	  1/11-­‐1/17	  1/18-­‐1/24	  1/25-­‐1/31	  2/1-­‐2/7	  2/8-­‐2/14	  2/15-­‐2/21	  2/22-­‐2/28	  2/29-­‐3/6	  3/7-­‐3/13	  3/14-­‐3/20	  3/21-­‐3/27	  3/28-­‐3/31	  Compute	  node	  usage	  0%	  Figure 20.1: Resource usage in FY2015

20.3

Research Results and Achievements

Figure 20.1 shows resource usage details for FY2015. As shown in the figure, we achieved approximately 75% node usage, which is the same as FY2014. Each project had appropriate computing
resources for a year, and these resources were divided into the following two terms : (1) from April
to September and (2) from October to March. In FY2015, node usage during the first term was 71%,
whereas the usage during the second term was 80%. In contrast, node usage per term in FY2014
was 79% and 72%, respectively. Because many new projects were initiated in FY2015, utilization of
the first term (especially in April and May) decreased.

20.3.1Analysis and improvements of operational issues

Shortening job waiting times
Figure 20.2 shows the average waiting times of large jobs (i.e., 12289–36864 nodes) in both FY2014
and FY2015. To consume remaining computing resources before they expired, users tended tosubmit many jobs at the end of each term, the average waiting times in September and March weresubstantially longer than those of other months. In FY2014, job congestion occurred in August andSeptember; however, in the second term of FY2015, they occurred from December. In addition,
average waiting times for FY2015 were longer than those of FY2014. We therefore analyzed job
scheduling and found that the following two factors impacted long waiting times.
1. Influence of higher-priority jobs (prior/semiprior job)
From FY2014, we provide a priority use system by which users can run a job at a higher
priority than normal jobs. Figure 20.3 shows the number of higher-priority jobs per month. As
is evident in the figure, the number of priority jobs in FY2015 was greater than that of FY2014.
Normal priority jobs, especially, large-scale jobs, are impacted by these higher-priority jobs. InFebruary 2016, we therefore changed the region used by higher-priority jobs to a region that
may not inhibit the execution of normal jobs.
2. Influence of jobs specified in excess of node LFS quotas
Users specify LFS quotas when a job is submitted. There were many jobs specified with excess
node LFS quotas in FY2015. Figure 20.4 shows the number of occurrences of LFS space
166Chapter 20. System Operations and Development Team

Average	  wai*ng	  *me	  in	  FY2015	  (large	  :	  12289	  ～36864	  node)	

Average	  wai*ng	  *me	  in	  FY2014	  (large	  :	  12289	  ～36864	  node)	
Elapse	  limit	~24h	  768.0	  480.0	  384.0	  3/1～3/10	  2/1～2/10	  3/1～3/10	  2/1～2/10	  1/1～1/10	  12/1～12/10	  11/1～11/10	  10/1～10/10	  9/1～9/10	  7/1～7/10	  8/1～8/10	  0.0	  6/1～6/10	  96.0	  0.0	  5/1～5/10	  192.0	  96.0	  1/1～1/10	  288.0	  192.0	  12/1～12/10	  288.0	  ~24h	  576.0	  11/1～11/10	  384.0	  ~12h	  672.0	  9/1～9/10	  480.0	  ~4h	  10/1～10/10	  576.0	  7/1～7/10	  672.0	  8/1～8/10	  Average	  wai7ng	  7me	  (h)	864.0	  768.0	  ~1h	  6/1～6/10	  ~12h	  5/1～5/10	  ~4h	  '15	  4/1～ 4/10	  ~1h	  864.0	  '14	  4/1～ 4/10	  Average	  wai7ng	  7me	  (h)	Elapse	  limit	

Figure 20.2: Average waiting times in FY2014 and FY2015
140	  #	  of	  submi*ed	  jobs	

3,500	  3,000	  #	  of	  occurrences	  of	  LFS	  space	  shortage	

4,000	  FY2015	  FY2014	  2,500	  2,000	  1,500	  1,000	  500	  120	  100	  FY2015	  FY2014	  80	  60	  40	  20	  0	  0	  Apr	  May	  Jun	   Jul	   Aug	  Sep	  Oct	  Nov	  Dec	   Jan	   Feb	  Mar	  Apr	  May	  Jun	   Jul	   Aug	  Sep	  Oct	  Nov	  Dec	  Jan	   Feb	  Mar	  Figure 20.4: Number of occurrences of LFS
space shortages

Figure 20.3: Number of submission priority jobs

shortages in the 10,000 or more nodes of a job. Comparing Figure 20.2 and Figure 20.4, wenote a correlation between waiting times and LFS space shortages. We have therefore requested
a review of node LFS quotas in order not to occurrence of LFS space shortages. In addition,
we designed a system to monitor and detect LFS space shortages.

Response degradation problems in the GFS
Temporary reductions in the response rates of the GFS have occurred several times. The major
factor is the load concentration on the GFS caused by large-scale staging. Because this bandwidthis filled with access to very large files, access to other files was inhibited.
We analyzed job scheduling and found that the following two factors impacted response times. We
therefore performed the following three improvements. Through these improvements, load balancing
was achieved and response time improved.
• Automatic striping of stage-out file• Processing I/O thread allocation setting for access from the front-end node and staging system• Changing the default stripe count on the GFS

167

Chapter 20. System Operations and Development Team
Performance monitoring of the GFS and LFS
We started to collect performance data regarding the GFS and LFS to monitor load. We use thesedata to detect failures and analyze each job’s I/O performance. Every 10 min, our monitoring systemcollects and records the size of read and write operations. Similarly, our system collects the numberof metadata operations, such as open, close, getatter, etc. In the future, we will evaluate the job’sI/O performance using these gathered performance data.
Disclosure of information to the user
We constructed a webpage to present the following information on the K user portal.
• Resource use history, including job and storage information• Detailed performance information of each job• Login history• Performance information on jobs executed by pre/post processing nodes

20.3.2

Power consumption problem

The K computer’s power consumption exceeded the given limit several times during FY2013; this is
an important problem because it forces us to increase the contracted upper limit for power, thereby
increasing costs, which cannot be ignored. From FY2014, to prevent this problem, we performeda preliminary review that estimated the power consumption of each job, thereby enabling us tocontrol the K computer’s overall power consumption. Moreover, we investigated an emergency jobstopping method based on the estimated power consumption of each job in case power consumption
again exceeds the given limit. In FY2015, we primarily worked on improvements to our emergency
job-stopping method and algorithm to predict power consumption a few hours into the future.
Improvements to our emergency job-stopping method
In FY2014, the staff of the facility monitored and stopped jobs when power consumption exceeded
given limits. Because this method was manual, it took time to stop a job and was prone to humanerror. In FY2015, we built a management system for the K computer that can read power informationof the facilities. When this system detects excess power use, it automatically stops the current job.
Immediate response times are required when excess power is demanded, thus automated job stopping
led to further protection. In FY2014, we used a simple approach, i.e., we selected the largest job interms of the number of nodes as the job to be stopped.
In FY2015, we evaluated other selection methods that take into account the power consumption,
number of nodes and elapsed time of the jobs. Selecting jobs to stop is a combinatorial optimization
problem; thus, we used a genetic algorithm to solve the problem. In FY2016, we plan to operate
and evaluate this emergency job-stopping system.
Power consumption prediction
To prevent excess power consumption, the cogeneration system (CGS) is useful to temporarily increase power supply. Two CGS of 5 MW are operated alternately in AICS. If two CGSs are operated
simultaneously when excess power is expected, we can avoid the excess power. Because CGS takes
1–2 hours to initiate, we must expect excess power to be demanded in a few hours. We abandoned thefull-time operation of two CGSs, because fuel costs increasesd. We examined our prediction method
regarding power fluctuations after a few hours. According to job statistics, many users execute jobs
repeatedly with the same number of nodes. Moreover, 80% of users executed jobs with at most nine
patterns in terms of the number of nodes.
We therefore implemented a power forecast system that estimates the power of a scheduled job
using power history of already-executed jobs. Figure 20.5 presents an example of predicted powerusage at 10:30 AM, showing data from 12 hours before until 12 hours after 10:30 AM. In otherwords, the center of the figure represents now. The past shows the record of power, while the futureshows predicted power usage. The dashed lines in the figure indicate the upper and lower limitsof the predicted power plus statistical prediction errors. In FY2016, we plan to improve prediction
168

Chapter 20. System Operations and Development TeamFigure 20.5: Power Consumption Forecast at 10:30 AM45	  40	  35	  New	  issues	  Resolved	  issues	  30	  25	  20	  15	  10	  5	  0	  Apr	   May	  Jun	   Jul	   Aug	  Sep	   Oct	   Nov	  Dec	   Jan	   Feb	  Mar	  Figure 20.6: Number of issues addressed in FY2015
accuracy by using machine learning and evaluate our emergency job-stopping system. In addition,
we plan to study optimal operation methods of the CGS using our prediction results.

20.3.3User support

The K computer had approximately 170 groups and 1,900 users in FY2015. The total number of
HPCI users and AICS researchers were approximately 1,600 and 300, respectively. The number ofdaily active users was approximately 150.
We supported users through the K support desk, providing users with technical informationon the K computer, including information regarding its system environment, system tools, software
libraries. In addition, we performed user registrations, failures investigation and software installation,etc. Our consulting services were offered together with the software development team. Figure 20.6
present the number of issues addressed in FY2015, showing the number of new issues in FY2015 tobe approximately 230; the number of resolved issues was approximately 220. The number of newissues in FY2013 and FY2014 were approximately 230 and 170, respectively.

20.4

Schedule and Future PlanIn FY2015, we primarily performed improvements on operational issues. We analyzed the operational
status, identified a variety of issues, and made improvements in these areas. To implement theseimprovements, we constructed a log-gathering environment and a database. In the next fiscal year,
we are scheduled to publish the compiler, which supports the new standard. We continue to improvethe user environment and provide user support. Moreover, we continue to address the K computer’s
power consumption problem.

169

Chapter 20. System Operations and Development Team

20.5

Publications

Journal Articles[1]

Atsuya Uno et al. “Operation of the K computer Focusing on System Power Consumption (InJapanese)”. In: IPSJ ACS Vol.8 No.4 (2015).

Conference Papers[2]

Atsuya Uno et al. “Operation of the K computer Focusing on System Power Consumption (InJapanese)”. In: HPCS2015. 2015.

Posters and Presentations[3]

Fumiyoshi Shoji et al. “Long term failure analysis of 10 peta-scale supercomputer”. In: HPC
in Asia session at ISC2015. 2015.

[4]

Shinji Sumimoto et al. “Metadata Access Reduction of Large Scale Lustre Based File System”.
In: The Lustre User Group (LUG) conference. 2015.

[5]

Atsuya Uno. “Approaches to the power consumption problem on the K computer (In Japanese)”.
In: Annual Meeting on Advanced Computing System and Infrastructure (ACSI2016). 2016.

[6]

Keiji Yamamoto et al. “Analysis and Elimination of Client Evictions on a Large Scale Lustre
Based File System”. In: The Lustre User Group (LUG) conference. 2015.

170