JLUG2017

Issues and Directions for the Next
Generation Shared File System - 2- How SSD based storage should be used? Shinji Sumimoto, Ph.D.
Next Generation Technical Computing Unit

FUJITSU LIMITED

Copyright 2017 FUJITSU LIMITEDOutline of This Talk
Exascale Storage Design (JLUG2016)
SSD Characteristics for Local File System
How SSD Based Storage should be used?

1

Copyright 2017 FUJITSU LIMITEDExascale Storage Design
 From JLUG2016 Presentation2

Copyright 2017 FUJITSU LIMITEDFujitsu’s FEFS Development towards Exascale
Fujitsu will continue to develop Lustre based FEFS torealize the next generation exascale systems.
Needs continuous Lustre enhancements

FEFS already supports Exa-byte class file system size
However, several issues to realize real Exascale file system

Topics
Exascale File System Design
Exascale Storage Design

3

Copyright 2017 FUJITSU LIMITEDExascale File System Design
 K computer File System Design
 How should we realize High Speed and Redundancy together?
 How do we avoid I/O conflicts between Jobs?
 These are not realized in single file system.
•Therefore, we have introduced Integrated Layered File System.

 Exascale File System/Storage Design
 Another trade off targets: Power, Capacity, Footprint•Difficult to realize single 1EB and 10TB/s class file system
in limited power consumption and footprint.
Local File System Third Storage layer for Capacity is needed:Three Layered File System
Global File System•Local File System for Performance
Archive File System•Global File System for Easy to Use•Archive File System for Capacity
4Copyright 2017 FUJITSU LIMITEDThe Next Integrated Layered File System Architecture
for Post-peta scale System (Feasibility Study 2012-2013)
 Local File System o(10PB): Memory, SSD, HDD BasedApplication Specific, Existing FS, Object Based, etc..
 Global File System o(100PB): HDD BasedLustre Based, Ext[34], Object Based, Application Specific etc..
 Archive System o(1EB): HSM(Disk+Tape), Grid, Cloud Based HSM, Lustre, other file system
ComputeNodes
Nodes
ComputeCompute
Nodes
Compute Nodes

Thousands of Users
Login
ServerHigh Speed for Application
LustreBased
Based
LustreLustreBased
Ext[34]Based
Ext[34]Based
Ext[34]
Based
ObjectBased
Application
Specific
ObjectBased
ObjectBased
Application
Specific
Existing
FSSpecific
Application
Application
Specific
Object Based

SharedUsability

Lustre BasedTransparent Data Access

Job Scheduler
5OtherOrganization

OtherSystems

High Capacity & Redundancy& Interoperability
HSM, Other Shared FS,
Grid or Cloud Based/data

Copyright 2017 FUJITSU LIMITEDRequired Characteristics for the Next Integrated
Layered File System Application views:
 Local File System: Application Oriented File Accesses(Higher Meta&Data I/O)
 Global File System: Transparent File Access Archive System: In-direct Access or Transparent File Access(HSM) Transparent File Access to the Global File System
 Local File System Capacity is not enough as much as locating whole data ofGlobal File System
 File Cache on node memory and Local File System enables to accelerateapplication performance
MetaPerf.

DataBWs

Capacity

Scalability

Data Sharing
in a Job

Data Sharing
among Jobs

Local File System◎

◎×◎◎

×

Global File System

○○○○×

◎

Archive System××◎×××

This talk discusses about utilization of SSD for local file system
6

Copyright 2017 FUJITSU LIMITEDSSD Characteristics
for Local File System

7

Copyright 2017 FUJITSU LIMITEDDRAM, SSD, HDD Memory Device Comparison
 NAND Flash: Current SSD Devices

 PCM: Intel Optane

CIDR 2011January 9-12, 2011 Asilomar, California http://cidrdb.org/cidr2011/Papers/CIDR11_Paper3.pdf
Rethinking Database Algorithms for Phase Change Memory
Shimin Chen, Phillip B. Gibbons Intel Labs Pittsburghand Suman Nath Microsoft Research

Endurance of PCM is 10-1000 times better than NAND Flash
8Copyright 2017 FUJITSU LIMITEDEnterprise SSDs or Consumer SSDs
Enterprise Products

Consumer ProductsIntel
P3700

Intel
P3608

Intel750

Intel
600p

Samsung
950 proSamsung
960 ProSamsung
960 EVO

800GB

1.6TB

1.2TB

1TB

512GB

1TB

1TB

Read Perf. 2.8GB/s

5.0GB/s

2.4GB/s

1.8GB/s

2.5GB/s

3.5GB/s

3.2GB/sWrite Perf. 1.9GB/s

2.0GB/s

1.2GB/s

0.6GB/s

1.5GB/s

2.1GB/s

1.9GB/s

Warranty

5 years

5 years

5 years

5 years

5 years

5 years

3 years

MTBF

2.0M

1.0M

1.2M

1.6M

1.5M

1.5M

1.5M

AFR

0.44%

0.87%

0.73%

0.54%

0.58%

0.58%

0.58%

DWPD

8TB/Day

4.8TB/Day

70GB/Day

40GB/Day

210GB/Day

430GB/Day

360GB/Day

Capacity Same Level in Performance Differences in:

https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives.html
http://www.samsung.com/semiconductor/minisite/jp/ssd/consumer/overview.html

 DWPD(Data Writes per Day) MTBF and AFR

 Prices are increasing in proportion of their amount of flash cells
 Enterprise SSDs consist of flash cells as enough as their performance andendurance
9Copyright 2017 FUJITSU LIMITEDSpecification Difference in Intel P3700 Series
Enterprise Products
Intel
P3700
Capacity

400GB

800GB

1600GB

2000GB

Read Perf.

2.7GB/s

2.8GB/s

2.8GB/s

2.8GB/sWrite Perf.

1.1GB/s

1.9GB/s

1.9GB/s

1.9GB/s

Warranty

5 years

5 years

5 years5 years

MTBF

2.0M

2.0M

2.0M

2.0M

AFR

0.44%

0.44%

0.44%

0.4%

DWPD

4TB/Day

8TB/Day

24TB/Day

34TB/Day

https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/
data-center-ssds/dc-p3700-series.html

 DWPD increases in proportion of increasing its capacity

10Copyright 2017 FUJITSU LIMITEDHow about Intel Optane Products?
Enterprise Products

Enthusiast

Intel
P3700

Intel
P3608

Intel
P4600

Intel
P4500

Intel Optane
P4800XIntel Optane
900P

Capacity

800GB

1.6TB

1.6TB

1TB

375GB

480GB

Read Perf.

2.7GB/s5.0GB/s

3.3GB/s

3.3GB/s

2.4GB/s

2.5GB/sWrite Perf.

1.9GB/s

2.0GB/s

1.4GB/s

0.6GB/s

2.0GB/s

2.0GB/s

K IOPS(R/W)

460/90850/150

587/184394/32550/500

550/500

Latency(R/W) 20/20us

20/20us79/34us80/29us10/10 us10/10us

Warranty

5 years

5 years

5 years

5 years

5 years

5 years

MTBF

2.0M

1.0M

2.0M

2.0M

2.0M

1.6M

AFR

0.44%

0.87%

0.44%

0.44%

0.44%

0.54%

DWPD

8TB/Day

4.8TB/Day

4.7TB/Day

0.72TB/Day

11.2TB/Day

4.7TB/Day Intel Optane:https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds.html

 Write IOPs is 2.7 times higher than that of P4600, but 375GB capacity is too small to use
 DWPD 11.2TB/Day is not higher than expected, (3 times better than P3700/800G)
but actual number of cells should be investigated.
 Current cost is 30% higher than that of P3700 800GB (Amazon.com)
11

Copyright 2017 FUJITSU LIMITED3D Xpointhttps://www.intelsalestraining.com/infographics/memory/3DXPointc.pdf

12Copyright 2017 FUJITSU LIMITEDFrom the slides of Flash Memory Summit 2017https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170808_FR12_Choe.pdf

13

Copyright 2017 FUJITSU LIMITEDFrom the slides of Flash Memory Summit 2017https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170808_FR12_Choe.pdf

14

Copyright 2017 FUJITSU LIMITEDFrom the slides of Flash Memory Summit 2017https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170808_FR12_Choe.pdf

15

Copyright 2017 FUJITSU LIMITEDFrom the slides of Flash Memory Summit 2017https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170808_FR12_Choe.pdf

16

Copyright 2017 FUJITSU LIMITEDEvaluation of Intel Optane

From the slides of Flash Memory Summit 2017https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170808_FR12_Choe.pdf• vs. Intel P3700/800GB• Latency of XPoint is two times better• Endurance of XPoint is three times better
17

Copyright 2017 FUJITSU LIMITEDHow SSD Based Storage
should be used?

18

Copyright 2017 FUJITSU LIMITEDUtilizing SSD based storage
 Characteristics of SSD:
 Bandwidth Performance:• vs. HDD: - 10 times faster,• vs. DRAM(DIMM): - 10 times slower

 Latency:• vs. HDD: -1000 times faster• vs. DRAM(DIMM): -1000 times slower

 Capacity per cost (amazon.com price):• vs. HDD: 30- times higher• vs. DRAM(DIMM): -20 times lower

 Endurance:• Limited lifetime writes

 To utilize SSD characteristics:
 Reduction of HDD access Lifetime write control: Elimination of useless writes

 Whether useless or not depends on file I/O access pattern Needs to investigate file I/O usage on applications19

Copyright 2017 FUJITSU LIMITEDThree Scopes of File I/O Usages on Applications
 File Lifetime: Persistent Files: Input Files, Output Files Temporary Files: Input Files, Output Files Access Pattern: Distributed Files: for each process Single Shared File : partial access,concentrate access to same data
 Master-slave: Master does whole File I/O

Distributed Files
Process

Single SharedFile(1)

Single SharedFile(2)I/O MasterFile

 Data Sharing:Job Within a job
 Among multiple jobs(under designing)

JobProcess20FileFileWithin a Job

Among multipleProcesses

Copyright 2017 FUJITSU LIMITEDFile Lifetime for effective SSD use
Persistent files in a job are located on SSD as file cache
SSD based storage capacity is smaller than that of the global FS
Asynchronous data transfer is effective between the local andglobal FS

Temporary files in a job should be located on SSD toeliminate the global FS accessesBut, how persistent file cache on SSD should be used?
It depends on file access patterns

21

Copyright 2017 FUJITSU LIMITEDApplication’s Access Pattern and SSD Cache Effects Comparison of Effective Pattern for SSD based storage
Distributed
Files

Single Shared
Files (1)Single Shared
Files (2)Rereading Case：◎
Non Rereading ：×

I/O
Master

Processes

FileReading

Processes

FileWriting

File Read：
Effects

Rereading Case：◎
Non Rereading ：×

Rereading Case：◎
Non Rereading ：×

File Write：
Effects

Rewriting Case：◎
Non Rewriting ：○Rewriting Case：◎
Non Rewriting ：○22

Rereading Case：◎
Non Rereading ：×
Rewriting Case：◎
Non Rewriting ：○Copyright 2017 FUJITSU LIMITEDData Sharing in a Job on SSD
 Write-Read in a process and among processes are effective to use SSD
 For Persistent Files: File cache of global file system should be shared among processes
 For Temporary Files: Two types of temporary file systems are effective to use SSD
 Temporary Local System (in a process) Temporary Shared File System (among processes)
In a process
(1)

write

(2)

write

read

Process

Process(1) write(2) writeFile(1) write(1) read
Filewrite(2) read

File

Process

read

Among processes(2) write

Process(1) read(2) read

File23

Process

read

Process

Process(1) write(2) write(1) write

File

File

Process

Process(2) read

Process(1) read(2) write (1) readFileFile

Process

Process(2) read

Copyright 2017 FUJITSU LIMITEDData Sharing among multiple jobs on SSD
Write-Read among multiple jobs are effectiveIssues: Local File System Data Lifetime management•When file data will be removed from SSD?

 How to realize SSD capacity management•With relation with Job scheduler or not Performance and Availability

Needs to be designed how to share file on global filesystem and local file system

24

Copyright 2017 FUJITSU LIMITEDHow SSD based storage should be used?
 Life Time Persistent files in a job are located on SSD as file cache Temporary files in a job should be located on SSD to eliminate the global FSaccesses Application’s Access Pattern Non reusable file in file reading should not use SSD based storage

 Data Sharing in a Job
 Write-Read in a process and among processes are effective to use SSD
 For Persistent Files: File cache of global file system should be shared among processes
 For Temporary Files: Two types of temporary file systems are effective to use SSD• Temporary Local System (in a process)• Temporary Shared File System (among processes)

 Data Sharing among multiple jobs
 Write-Read among multiple jobs are effective to use SSD
 Needs to be designed how to share file cache on global and local file system

 SSD lifetime writes(DWPD) Issue
 SSD whose DWPD is higher than that of daily use will be a choice25

Copyright 2017 FUJITSU LIMITED26

Copyright 2017 FUJITSU LIMITED