Intel NVMe(P3600) on CentOS7利用メモ


NVMe初心者のメモです。

## 環境

* OS: CentOS7
* NVMe Intel P3600
* Kernel: 4.4 (自身でリビルド)

## 参考資料

[How to use and benchmark NVMe SSD, create a Software RAID and analyze performance – The answers are here](https://communities.intel.com/community/itpeernetwork/blog/2015/10/01/how-to-use-and-benchmark-nvme-ssd-create-a-software-raid-and-analyze-performance-the-answers-are-here)
のPDFが参考になる。

特に参考にしたPDF

[Hands-on Lab: How to Unleash Your Storage Performance by Using NVM ExpressTM Based PCI Express® Solid-State Drives](https://hubb.blob.core.windows.net/e5888822-986f-45f5-b1d7-08f96e618a7b-published/34337017-b157-4098-b35e-e576c12e0da5/SSDL001%20-%20SF15_SSDL001_100f.pdf?sv=2015-12-11&sr=c&sig=V6C54QhHPC5Z9E4OqmjHB4Y53bVlXQpfxr1nmQF8zk0%3D&se=2017-04-14T03%3A01%3A01Z&sp=rwd)


文中に記載されているリンク

[PCIe SSD PACC EP P3700 Intel® Solid-State Drive Data Center Tool](http://jp.fujitsu.com/platform/server/primergy/products/note/svsdvd/dvd/pdf/isdct-user-guide-jp.pdf) 富士通の日本語ドキュメント

[インテル® Solid-State Drive データセンター・ツールのユーザーおよびインストールガイド](http://www.intel.co.jp/content/www/jp/ja/support/solid-state-drives/ssd-software/000020016.html)


## データセンターツールの導入

[インテル® Solid-State Drive データセンター・ツール](https://downloadcenter.intel.com/ja/download/23931/Intel-Solid-State-Drive-Data-Center-Tool) からファイルをダウンロード

```
unzip ../DataCenterTool_3_0_0_Linux.zip 
Archive:  ../DataCenterTool_3_0_0_Linux.zip
  inflating: isdct-3.0.0.400-15.i386.rpm  
  inflating: isdct-3.0.0.400-15.x86_64.rpm  
  inflating: NSGSoftwareLicenseAgreement__EndUserNoModify__2016.txt
```

``isdct-3.0.0.400-15.x86_64.rpm``をインストール

```
rpm -ivh isdct-3.0.0.400-15.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:isdct-3.0.0.400-15               ################################# [100%]
```


```
rpm -ql isdct
/etc/isdct
/usr/bin/isdct
/usr/lib/isdct/FirmwareModules/generic_firmware_module.so
/usr/lib/isdct/libIntel.SSDFeatures.so.2.0.0
/usr/lib/isdct/libLSI.AdapterAPI.so.4.02
```


```
Help:
    help [-help|-h] [-output|-o (text|nvmxml|json)] [Name = (name)] [verb =
    (verb)]
IntelSSD:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -intelssd [(Index|SerialNumber)]
EnduranceAnalyzer:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) EnduranceAnalyzer = ('reset')
PhyConfig:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PhyConfig = (0|1|2|3)
PhysicalSectorSize:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PhysicalSectorSize = (512|4096)
PLITestTimeInterval:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PLITestTimeInterval = (0-6)
PowerGovernorMode:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PowerGovernorMode = (0|1|2)
ReadErrorRecoveryTime:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) ReadErrorRecoveryTimer = (value)
PhySpeed:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PhySpeed = (1.5|3|6)
SSCEnabled:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) SSCEnabled = ('true'|'false')
TemperatureLoggingInterval:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) TemperatureLoggingInterval = (time)
TempThreshold:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) TempThreshold = (value)
WriteErrorRecoveryTime:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteErrorRecoveryTimer = (value)
WriteCacheState:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteCacheState = (1|2|3)
MaximumLBA:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) MaximumLBA = (numGB|1-100%|LBA|'native')
FirmwareUpdate:
    load [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber)
SMART:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -smart [(id)] [-intelssd [(Index|SerialNumber)]]
Sensors:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -sensor [-intelssd [(Index|SerialNumber)]]
Performance:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -performance [-intelssd [(Index|SerialNumber)]]
NVMeLog:
    show [-help|-h] [-output|-o (text|nvmxml|json)] [-intelssd
    [(Index|SerialNumber)]] -nvmelog
    [('ErrorInfo'|'SmartHealthInfo'|'FirmwareSlotInfo'|'TemperatureStatistics')]
Delete:
    delete [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber)
SMBusAddress:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) SMBusAddress = (address)
LatencyTrackingEnabled:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) LatencyTrackingEnabled = (true|false)
DIPM:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) DIPMEnabled = ('true' | 'false')
WriteCache:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteCacheEnabled = ('true' | 'false')
WriteCacheReorderingState:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteCacheReorderingStateEnabled = ('true'|'false')
CreateNamespace:
    create [-help|-h] [-output|-o (text|nvmxml|json)] -namespace -intelssd
    (Index|SerialNumber) Size = (blocks) [LBAFormat = (0-NumLBAFormats)]
    [ProtectionInformation = (0|1)] [MultiPathIoCapabilities = (0|1)]
DeleteNamespace:
    delete [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -namespace
    (namespace id) -intelssd (Index|SerialNumber)
AttachNamespace:
    attach [-help|-h] [-output|-o (text|nvmxml|json)] -namespace (namespace id)
    -intelssd (Index|SerialNumber) [-nvmecontroller (controller ID integer)]
DetachNamespace:
    detach [-help|-h] [-output|-o (text|nvmxml|json)] -namespace (namespace id)
    -intelssd (Index|SerialNumber) [-nvmecontroller (controller ID integer)]
IdentifyDevice:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -identify [-namespace
    [(integer | 'attached' | 'allocated')]] [-nvmecontroller] [-intelssd
    [(Index|SerialNumber)]]
NVMeFormat:
    start [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) -nvmeformat [-namespace (namespace id)] [LBAFormat =
    (0-NumLBAFormats)] [SecureEraseSetting = (0|1|2)] [ProtectionInformation =
    (0|1)] [MetadataSettings = (0|1)]
LatencyStatistics:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -latencystatistics
    ('reads'|'writes') [-intelssd [(Index|SerialNumber)]]
HDATemperature:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -hdatemperature [-intelssd
    [(Index|SerialNumber)]]
Nlog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -nlog
AssertLog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -assertlog
EventLog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -eventlog
SATAGeneralPurposeLog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -gpl (log address) [PageNum = [(page
    #)]] [SectorCount = [(sectors)]]
PhyCounters:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -phycounters [-intelssd
    [(Index|SerialNumber)]]
DriveSelfTest:
    start [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) -selftest [('short'|'extended'|'conveyance')]
Standby:
    start [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) -standby
NVMeControllerList:
    show [-help|-h] [-output|-o (text|nvmxml|json)] [-namespace (namespace id)]
    -nvmecontroller [-intelssd [(Index|SerialNumber)]]
EnableLog:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -system EnableLog =
    ('true'|'false')
EnableLSIAdapter:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -system EnableLSIAdapter =
    ('true'|'false')
LogFile:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -system LogFile = (file name)
System:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -system
Version:
    version [-all|-a] [-display|-d (Property1,...)] [-help|-h] [-output|-o
    (text|nvmxml|json)]
```

```
isdct show -intelssd

- Intel SSD DC P3600 Series CVMD5395004X400AGN -

Bootloader : 8B1B012E
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
Firmware : 8DV10131
FirmwareUpdateAvailable : Firmware=8DV10171 Bootloader=8B1B0131
Index : 0
ModelNumber : INTEL SSDPEDME400G4
ProductFamily : Intel SSD DC P3600 Series
SerialNumber : XXXXXXXXXXX

- Intel SSD DC P3600 Series CVMD5395007L400AGN -

Bootloader : 8B1B012E
DevicePath : /dev/nvme1n1
DeviceStatus : Healthy
Firmware : 8DV10131
FirmwareUpdateAvailable : Firmware=8DV10171 Bootloader=8B1B0131
Index : 1
ModelNumber : INTEL SSDPEDME400G4
ProductFamily : Intel SSD DC P3600 Series
SerialNumber : XXXXXXXXXXX
```

なるほど


## RAID0

```
mdadm -C /dev/md/imsm0 /dev/nvme0n1 /dev/nvme1n1 -n 2 -e imsm -f 
mdadm: /dev/nvme0n1 appears to be part of a raid array:
       level=raid0 devices=0 ctime=Thu Jan  1 09:00:00 1970
Continue creating array? y
mdadm: container /dev/md/imsm0 prepared.
```

```
mdadm -C /dev/md0 /dev/md/imsm0 -n 2 -l 0 -c 128 -f 
mdadm: array /dev/md0 started.
```

```
parted /dev/md0 mklabel gpt
Warning: The existing disk label on /dev/md0 will be destroyed and all data on
this disk will be lost. Do you want to continue?
Yes/No? yes                                                               
Information: You may need to update /etc/fstab.
```


```
parted /dev/md0                                        

(parted) mkpart                                                           
Partition name?  []?                                                      
File system type?  [ext2]? xfs                                            
Start? 0%                                                                 
End? -1                                                                   
```


```
(parted) p                                                                
Model: Linux Software RAID Array (md)
Disk /dev/md0: 800GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  800GB  800GB

(parted) quit                                                             
Information: You may need to update /etc/fstab.
```



```
/sbin/mkfs.xfs -K /dev/md0p1 -f 
```


```
mount -o noatime,nodiratime,nobarrier /dev/md0p1 /mnt/nvme1
```

```
mdadm --detail --scan > /etc/mdadm.conf
```

mdadm.confの例

```
ARRAY /dev/md/imsm0 metadata=imsm UUID=a66e2dd4:e3167963:92df1c07:69c7f299
ARRAY /dev/md0 container=/dev/md/imsm0 member=0 UUID=0c3f3fbd:a1d6986b:b18d262b:f3ae58ff
```



## /proc/mdstatの値

```
cat /proc/mdstat 
Personalities : [raid0] 
md127 : inactive nvme0n1[1](S) nvme1n1[0](S)
      6306 blocks super external:imsm
       
md0 : active raid0 nvme1n1[1] nvme0n1[0]
      781416448 blocks super external:/md127/0 128k chunks
      
unused devices: <none>
```

## RAID0の手動停止

```
mdadm --stop /dev/md0
mdadm: stopped /dev/md0
```

```
mdadm --stop /dev/md/imsm0 
mdadm: stopped /dev/md/imsm0
```

```
cat /proc/mdstat 
Personalities : [raid0] 
unused devices: <none>
```

## RAID0の手動開始

```
mdadm --assemble --scan
mdadm: Container /dev/md/imsm0 has been assembled with 2 drives
mdadm: Started /dev/md0 with 2 devices
```

## madm.confを書き忘れた場合の手動アッセンブル

```
mdadm --assemble /dev/md/imsm0 /dev/nvme0n1 /dev/nvme1n1
```

```
mdadm --assemble /dev/md0 /dev/md/imsm0
```