Oracle Cloud(OCI) DBaaSのデフォルト・ストレージ構成

*2019/6/27 新規作成*
*2019/7/23 コマンド微修正,図の微修正,singleでも確認*

# はじめに
OCIのDBaaSについて、作成直後のストレージ構成を確認してみました。

## 前提
- 以下の環境で確認
    - OCI DBaaS(DB Systems)のVM.Standard2.2
    - DB ver 11.2.0.4 (RAC/Single両方で確認。以下コマンド結果はRAC版)
    - Oracle Linux 6.10
- 作成時に「Available Storage Size (GB)」は"256"を指定しました
- 2019/6/26作成のDB systemで確認しています
    - PaaSなので変更される可能性があります。


## 確認したこと
- Block Volumeのアタッチ数
- ファイルシステムの切られ方
- ASM Disk Groupの切られ方
- ASM Volumeの切られ方


# 確認結果

## 概要
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/109270/0fdc92e6-0abf-743f-5887-af47e58a8f36.png)




- Block Volumeは9つアタッチされている
    - それぞれudevのSYMLINKにて、デバイス名を固定している
- sdb ~ sbiはASMディスクとして使用される
    - sdb ~ sdeはASM Disk Groupの+DATA、およびASM VolumeのCOMMONSTOREとして利用
        - +DATAにはデータファイル、パラメータファイル、OCR、Voting Diskなどが格納
    - sdf ~ sdiはASM Disk Groupの+RECOとして利用
        - +RECOにはオンラインREDO、アーカイブREDOなどが格納
- ASM VolumeのCOMMONSTOREはacfsとして/opt/oracle/dcs/commonstoreへマウントされている
- sdjはext4として/u01へマウントされている
- Boot Volumeはsda1,sda2,sda3とパーティションが切られている
    - sda1は510MBで/boot/efiへマウント（Compute OL6.10の場合210MBでfat16）
    - sda2は1488MBで/bootへマウント（Compute OL6.10の場合/boot/efiと分けられていない）
    - sda3は42GBでLogVolRoot(35GB)とLogVolSwap(2GB)のLVが切られている（Compute OL6.10の場合LVは切られていない）
        - LogVolRootは/へマウント
        - LogVolSwapはSWAP
- ストレージを拡張すると、拡張されたブロックが追加アタッチされ、リバランスで移動させてから、元のブロックが削除される

## コマンド結果

### ブロックデバイスを確認

```
[opc@dbhost2 ~]$ sudo lsblk
NAME                              MAJ:MIN   RM  SIZE RO TYPE MOUNTPOINT
sda                                 8:0      0 46.6G  0 disk
|-sda1                              8:1      0  486M  0 part /boot/efi
|-sda2                              8:2      0  1.4G  0 part /boot
`-sda3                              8:3      0 39.1G  0 part
  |-VolGroupSys-LogVolRoot (dm-0) 249:0      0   35G  0 lvm  /
  `-VolGroupSys-LogVolSwap (dm-1) 249:1      0    2G  0 lvm  [SWAP]
sdb                                 8:16     0   64G  0 disk
`-sdb1                              8:17     0   64G  0 part
sdc                                 8:32     0   64G  0 disk
`-sdc1                              8:33     0   64G  0 part
sdd                                 8:48     0   64G  0 disk
`-sdd1                              8:49     0   64G  0 part
sde                                 8:64     0   64G  0 disk
`-sde1                              8:65     0   64G  0 part
sdf                                 8:80     0   64G  0 disk
`-sdf1                              8:81     0   64G  0 part
sdg                                 8:96     0   64G  0 disk
`-sdg1                              8:97     0   64G  0 part
sdh                                 8:112    0   64G  0 disk
`-sdh1                              8:113    0   64G  0 part
sdi                                 8:128    0   64G  0 disk
`-sdi1                              8:129    0   64G  0 part
sdj                                 8:144    0  200G  0 disk
`-sdj1                              8:145    0  200G  0 part
asm!commonstore-61                248:31233  0    5G  0 disk
```

### パーティション情報を確認

```
[opc@dbhost2 ~]$ sudo parted -l
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 3305472 blocks) or continue with
the current setting?
Fix/Ignore? Ignore
Model: ORACLE BlockVolume (scsi)
Disk /dev/sda: 50.0GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      2097kB  512MB   510MB   fat32        primary
 2      512MB   2000MB  1488MB  ext4         primary
 3      2000MB  44.0GB  42.0GB               primary


Error: /dev/sdb: unrecognised disk label

Error: /dev/sdc: unrecognised disk label

Error: /dev/sdd: unrecognised disk label

Error: /dev/sde: unrecognised disk label

Error: /dev/sdf: unrecognised disk label

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/VolGroupSys-LogVolSwap: 2147MB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  2147MB  2147MB  linux-swap(v1)


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/VolGroupSys-LogVolRoot: 37.6GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  37.6GB  37.6GB  ext4


Error: /dev/sdg: unrecognised disk label

Error: /dev/sdh: unrecognised disk label

Error: /dev/sdi: unrecognised disk label

Model: ORACLE BlockVolume (scsi)
Disk /dev/sdj: 215GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  215GB  215GB  ext4


Error: /dev/asm/commonstore-61: unrecognised disk label

Error: /dev/ofsctl: unrecognised disk label
```

### ブロックのFSTYPEを確認

```
[opc@dbhost2 ~]$ sudo lsblk -f
NAME                              FSTYPE      LABEL   UUID                                   MOUNTPOINT
sda
|-sda1                            vfat        bootefi CD3C-893F                              /boot/efi
|-sda2                            ext4        bootfs  58ab4698-de1d-4fac-916f-df800d826a65   /boot
`-sda3                            LVM2_member         g7qOjo-0121-Qf1j-jUwv-0dde-5NqK-duVwQ1
  |-VolGroupSys-LogVolRoot (dm-0) ext4                7a220ac5-5d95-48ed-b966-ef14fe36cc21   /
  `-VolGroupSys-LogVolSwap (dm-1) swap                f2eb68e5-ee82-404d-9a7c-613a8ff68d1b   [SWAP]
sdb                               oracleasm
`-sdb1
sdc                               oracleasm
`-sdc1
sdd                               oracleasm
`-sdd1
sde                               oracleasm
`-sde1
sdf                               oracleasm
`-sdf1
sdg                               oracleasm
`-sdg1
sdh                               oracleasm
`-sdh1
sdi                               oracleasm
`-sdi1
sdj                               ext4                7d75da54-1388-483c-95e8-d5961ba03a31
`-sdj1
asm!commonstore-61
```

-> sdb ~ sdiがasmであることを確認

### ASMを確認

- ASMディスク

```
SQL> select NAME,DISK_NUMBER,GROUP_NUMBER,PATH,MOUNT_STATUS,REDUNDANCY,OS_MB,TOTAL_MB,FAILGROUP from v$asm_disk order by  NAME,DISK_NUMBER;

NAME                 DISK_NUMBER GROUP_NUMBER PATH                           MOUNT_S REDUNDA      OS_MB   TOTAL_MB FAILGROUP
-------------------- ----------- ------------ ------------------------------ ------- ------- ---------- ---------- ------------------------------
DATA_0000                      0            1 /dev/DATADISK3                 CACHED  UNKNOWN      65536      65536 DATA_0000
DATA_0001                      1            1 /dev/DATADISK2                 CACHED  UNKNOWN      65536      65536 DATA_0001
DATA_0002                      2            1 /dev/DATADISK1                 CACHED  UNKNOWN      65536      65536 DATA_0002
DATA_0003                      3            1 /dev/DATADISK4                 CACHED  UNKNOWN      65536      65536 DATA_0003
RECODISK1                      0            2 /dev/RECODISK1                 CACHED  UNKNOWN      65536      65536 RECODISK1
RECODISK2                      1            2 /dev/RECODISK2                 CACHED  UNKNOWN      65536      65536 RECODISK2
RECODISK3                      2            2 /dev/RECODISK3                 CACHED  UNKNOWN      65536      65536 RECODISK3
RECODISK4                      3            2 /dev/RECODISK4                 CACHED  UNKNOWN      65536      65536 RECODISK4

8 rows selected.
```

-> PATHが見慣れない文字列

- ASMディスクグループ

```
[grid@dbhost1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304    262144   255220                0          255220              0             Y  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304    262144   257644                0          257644              0             N  RECO/
```

```
SQL> select NAME,GROUP_NUMBER,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;
NAME GROUP_NUMBER STATE TYPE TOTAL_MB
-------------------- ------------ ----------- --------------- ----------
DATA 1 MOUNTED EXTERN 262144
RECO 2 MOUNTED EXTERN 262144
```

- ASMボリューム

```
SQL> select VOLUME_NAME,VOLUME_NUMBER,GROUP_NUMBER,SIZE_MB,REDUNDANCY,VOLUME_DEVICE,MOUNTPATH from v$asm_volume;

VOLUME_NAME                    VOLUME_NUMBER GROUP_NUMBER    SIZE_MB REDUND VOLUME_DEVICE                  MOUNTPATH
------------------------------ ------------- ------------ ---------- ------ ------------------------------ ------------------------------
COMMONSTORE                                1            1       5120 UNPROT /dev/asm/commonstore-61        /opt/oracle/dcs/commonstore
```

- ASMファイル

```
SQL> select GROUP_NUMBER,FILE_NUMBER,TYPE from V$ASM_FILE;

GROUP_NUMBER FILE_NUMBER TYPE
------------ ----------- ------------------------------
           1         253 ASMPARAMETERFILE
           1         255 OCRFILE
           1         256 PASSWORD
           1         257 PASSWORD
           1         258 ASMVOL
           1         259 DATAFILE
           1         260 DATAFILE
           1         261 DATAFILE
           1         262 TEMPFILE
           1         263 DATAFILE
           1         264 PARAMETERFILE
           1         265 DATAFILE
           1         266 OCRBACKUP
           1         267 OCRBACKUP
           1         268 OCRBACKUP
           2         256 CONTROLFILE
           2         257 ONLINELOG
           2         258 ONLINELOG
           2         259 ONLINELOG
           2         260 ONLINELOG
           2         261 ARCHIVELOG
           2         262 ARCHIVELOG
           2         263 ARCHIVELOG
           2         264 ARCHIVELOG
           2         265 ARCHIVELOG
           2         266 ARCHIVELOG
           2         267 ARCHIVELOG

27 rows selected.
```

- OCRの場所

```
[grid@dbhost1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     491684
         Used space (kbytes)      :      84556
         Available space (kbytes) :     407128
         ID                       : 2018798373
         Device/File Name         :      +DATA <--★
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

```

- Voting Diskの場所

```
[grid@dbhost1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0785acad848a4ff0bf084ab0600111a9 (/dev/DATADISK3) [DATA]<--★
Located 1 voting disk(s).
```


### デバイスの命名を確認

- udevのルールをみてみる
 
```
[opc@dbhost2 ~]$ sudo ls -la /etc/udev/rules.d/
total 36
drwxr-xr-x 2 root root 4096 Jun 26 10:11 .
drwxr-xr-x 4 root root 4096 Apr 16 14:56 ..
-rw-r--r-- 1 root root 1574 Apr 16 16:03 40-redhat.rules
-rw-r--r-- 1 root root  190 Jun 26 10:11 55-usm.rules
-rw-r--r-- 1 root root  316 Jan 25  2018 60-raw.rules
-rw-r--r-- 1 root root 1431 Jun 26 09:49 70-names.rules
-rw-r--r-- 1 root root  589 Jun 26 09:16 70-persistent-net.rules
-rw-r--r-- 1 root root   83 Aug 17  2014 90-hal.rules
-rw-r--r-- 1 root root  308 Dec  2  2017 98-kexec.rules
```

- DISKと名前で、ルールがないか検索

```
[opc@dbhost2 ~]$ sudo find /etc/udev/rules.d/ -type f -name "*.rules" | xargs grep 'DISK' -n
/etc/udev/rules.d/70-names.rules:1:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360edc23284d54637b1b36221a19ce904", SYMLINK+="DATADISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:2:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a92e2a294d481497aeca6749839336", SYMLINK+="DATADISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:3:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360b476cdd2504f25bf1f6e4dd03480f8", SYMLINK+="DATADISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:4:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36056640a618a4f65993e5107a0086c8b", SYMLINK+="DATADISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:5:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:6:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:7:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
/etc/udev/rules.d/70-names.rules:8:KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
```

- ルールの中身を見る

```
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/55-usm.rules
#
# ADVM devices
KERNEL=="asm/*",      GROUP="asmadmin", MODE="0770"
KERNEL=="asm/.*",     GROUP="asmadmin", MODE="0770"
#
# ACFS devices
KERNEL=="ofsctl",     GROUP="asmadmin", MODE="0664"
```

```
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360edc23284d54637b1b36221a19ce904", SYMLINK+="DATADISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360a92e2a294d481497aeca6749839336", SYMLINK+="DATADISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360b476cdd2504f25bf1f6e4dd03480f8", SYMLINK+="DATADISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36056640a618a4f65993e5107a0086c8b", SYMLINK+="DATADISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360d566eec95b4c8f99aafa655558c6ad", SYMLINK+="localdisk",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
```

- 対応も確認

```
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sda | grep ID_SERIAL=
E: ID_SERIAL=36049ae4cf5c945378713d7eff5bd2b97													
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdb | grep ID_SERIAL=
E: ID_SERIAL=360edc23284d54637b1b36221a19ce904														--> DATADISK1
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdc | grep ID_SERIAL=
E: ID_SERIAL=360a92e2a294d481497aeca6749839336														--> DATADISK2
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdd | grep ID_SERIAL=
E: ID_SERIAL=360b476cdd2504f25bf1f6e4dd03480f8														--> DATADISK3
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sde | grep ID_SERIAL=
E: ID_SERIAL=36056640a618a4f65993e5107a0086c8b														--> DATADISK4
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdf | grep ID_SERIAL=
E: ID_SERIAL=36010ff9c8af14dbea0a5937fef871e28														--> RECODISK1
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdg | grep ID_SERIAL=
E: ID_SERIAL=36027594d89c741c1bd71096ca2b423e9														--> RECODISK2
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdh | grep ID_SERIAL=
E: ID_SERIAL=360c467190d3b42ddaef111492240cdc9														--> RECODISK3
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdi | grep ID_SERIAL=
E: ID_SERIAL=3606e77d57af6493ebee8c85ea10ac146														--> RECODISK4
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdj | grep ID_SERIAL=
E: ID_SERIAL=360d566eec95b4c8f99aafa655558c6ad														--> localdisk

```


```
 [opc@dbhost2 ~]$ sudo  ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root  9 Jun 26 11:40 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.10:3260-iscsi-iqn.2015-12.com.oracleiaas:d0a95eeb-9faf-4129-9dbd-d65dedf94f65-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.10:3260-iscsi-iqn.2015-12.com.oracleiaas:d0a95eeb-9faf-4129-9dbd-d65dedf94f65-lun-1-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Jun 26 11:40 ip-169.254.2.2:3260-iscsi-iqn.2015-12.com.oracleiaas:8ff76e53-8a0a-40ae-be86-3783f66fe750-lun-1 -> ../../sdj
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.2:3260-iscsi-iqn.2015-12.com.oracleiaas:8ff76e53-8a0a-40ae-be86-3783f66fe750-lun-1-part1 -> ../../sdj1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.3:3260-iscsi-iqn.2015-12.com.oracleiaas:05a4a7d3-e4cf-43f4-921d-83e31c1e364a-lun-1 -> ../../sdf
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.3:3260-iscsi-iqn.2015-12.com.oracleiaas:05a4a7d3-e4cf-43f4-921d-83e31c1e364a-lun-1-part1 -> ../../sdf1
lrwxrwxrwx 1 root root  9 Jun 26 12:23 ip-169.254.2.4:3260-iscsi-iqn.2015-12.com.oracleiaas:00467c13-215d-483d-8291-630a6ea256bd-lun-1 -> ../../sdd
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.4:3260-iscsi-iqn.2015-12.com.oracleiaas:00467c13-215d-483d-8291-630a6ea256bd-lun-1-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.5:3260-iscsi-iqn.2015-12.com.oracleiaas:637ed59e-b517-44c2-9f8d-828f8159ed57-lun-1 -> ../../sdh
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.5:3260-iscsi-iqn.2015-12.com.oracleiaas:637ed59e-b517-44c2-9f8d-828f8159ed57-lun-1-part1 -> ../../sdh1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.6:3260-iscsi-iqn.2015-12.com.oracleiaas:ae744a7f-23df-469f-aa5f-6bfdb9685f37-lun-1 -> ../../sdg
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.6:3260-iscsi-iqn.2015-12.com.oracleiaas:ae744a7f-23df-469f-aa5f-6bfdb9685f37-lun-1-part1 -> ../../sdg1
lrwxrwxrwx 1 root root  9 Jun 26 12:23 ip-169.254.2.7:3260-iscsi-iqn.2015-12.com.oracleiaas:c933f73c-87bc-44f5-ad32-657a77e19198-lun-1 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 26 09:49 ip-169.254.2.7:3260-iscsi-iqn.2015-12.com.oracleiaas:c933f73c-87bc-44f5-ad32-657a77e19198-lun-1-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.8:3260-iscsi-iqn.2015-12.com.oracleiaas:d169d00d-926e-4ba1-ae79-d25ad6b57d56-lun-1 -> ../../sde
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.8:3260-iscsi-iqn.2015-12.com.oracleiaas:d169d00d-926e-4ba1-ae79-d25ad6b57d56-lun-1-part1 -> ../../sde1
lrwxrwxrwx 1 root root  9 Jun 26 13:00 ip-169.254.2.9:3260-iscsi-iqn.2015-12.com.oracleiaas:c84336f3-21f7-4062-8d3b-044f4a2d7aa0-lun-1 -> ../../sdi
lrwxrwxrwx 1 root root 10 Jun 26 09:50 ip-169.254.2.9:3260-iscsi-iqn.2015-12.com.oracleiaas:c84336f3-21f7-4062-8d3b-044f4a2d7aa0-lun-1-part1 -> ../../sdi1
```

# (おまけ) Storage拡張
Available Storage Sizeを256GBから512GBへ変更

## 拡張後
- data用のブロックが大きくなっている（今回のスケールアップではRECOに変わりはない）

```
[opc@dbhost2 ~]$ sudo lsblk
NAME                              MAJ:MIN   RM  SIZE RO TYPE MOUNTPOINT
sda                                 8:0      0 46.6G  0 disk
|-sda1                              8:1      0  486M  0 part /boot/efi
|-sda2                              8:2      0  1.4G  0 part /boot
`-sda3                              8:3      0 39.1G  0 part
  |-VolGroupSys-LogVolRoot (dm-0) 249:0      0   35G  0 lvm  /
  `-VolGroupSys-LogVolSwap (dm-1) 249:1      0    2G  0 lvm  [SWAP]
sdf                                 8:80     0   64G  0 disk
`-sdf1                              8:81     0   64G  0 part
sdg                                 8:96     0   64G  0 disk
`-sdg1                              8:97     0   64G  0 part
sdh                                 8:112    0   64G  0 disk
`-sdh1                              8:113    0   64G  0 part
sdi                                 8:128    0   64G  0 disk
`-sdi1                              8:129    0   64G  0 part
sdj                                 8:144    0  200G  0 disk 
`-sdj1                              8:145    0  200G  0 part
sdk                                 8:160    0  128G  0 disk ★
`-sdk1                              8:161    0  128G  0 part
sdl                                 8:176    0  128G  0 disk ★
`-sdl1                              8:177    0  128G  0 part
sdm                                 8:192    0  128G  0 disk ★
`-sdm1                              8:193    0  128G  0 part
sdn                                 8:208    0  128G  0 disk ★
`-sdn1                              8:209    0  128G  0 part
asm!commonstore-61                248:31233  0    5G  0 disk
```

-> RECOに使ってたディスク以外のサイズが64GBずつ増加。
-> sdb ~ sdeはなくなり、sdk ~ sdnへ

- シリアルIDが変わってる

```
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdk | grep ID_SERIAL=
E: ID_SERIAL=360de2a6205ef4b4b9d2368812b28abee
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdl | grep ID_SERIAL=
E: ID_SERIAL=360c9ebbedfff4f3b84554d41440b238c
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdm | grep ID_SERIAL=
E: ID_SERIAL=3605f79bae3174834a01bbf0017fa1cce
[opc@dbhost2 ~]$ sudo udevadm info --query=all --name=/dev/sdn | grep ID_SERIAL=
E: ID_SERIAL=360da995ed1c844a3a24438dbc83f09f2
```

```
[opc@dbhost2 ~]$ sudo cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36010ff9c8af14dbea0a5937fef871e28", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36027594d89c741c1bd71096ca2b423e9", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c467190d3b42ddaef111492240cdc9", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e77d57af6493ebee8c85ea10ac146", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360d566eec95b4c8f99aafa655558c6ad", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360de2a6205ef4b4b9d2368812b28abee", SYMLINK+="DATADISK5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360c9ebbedfff4f3b84554d41440b238c", SYMLINK+="DATADISK6", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3605f79bae3174834a01bbf0017fa1cce", SYMLINK+="DATADISK7", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360da995ed1c844a3a24438dbc83f09f2", SYMLINK+="DATADISK8", OWNER="grid", GROUP="asmadmin", MODE="0660"
```

-> 再度ストレージ拡張を行うと、DATA9,10,11,12となった。


- ASM

```
SQL> select NAME,DISK_NUMBER,GROUP_NUMBER,PATH,MOUNT_STATUS,REDUNDANCY,OS_MB,TOTAL_MB,FAILGROUP from v$asm_disk order by  NAME,DISK_NUMBER;

NAME                           DISK_NUMBER GROUP_NUMBER PATH                           MOUNT_S REDUNDA      OS_MB   TOTAL_MB FAILGROUP
------------------------------ ----------- ------------ ------------------------------ ------- ------- ---------- ---------- ------------------------------
DATA_0004                                4            1 /dev/DATADISK5                 CACHED  UNKNOWN     131072     131072 DATA_0004
DATA_0005                                5            1 /dev/DATADISK6                 CACHED  UNKNOWN     131072     131072 DATA_0005
DATA_0006                                6            1 /dev/DATADISK7                 CACHED  UNKNOWN     131072     131072 DATA_0006
DATA_0007                                7            1 /dev/DATADISK8                 CACHED  UNKNOWN     131072     131072 DATA_0007
RECODISK1                                0            2 /dev/RECODISK1                 CACHED  UNKNOWN      65536      65536 RECODISK1
RECODISK2                                1            2 /dev/RECODISK2                 CACHED  UNKNOWN      65536      65536 RECODISK2
RECODISK3                                2            2 /dev/RECODISK3                 CACHED  UNKNOWN      65536      65536 RECODISK3
RECODISK4                                3            2 /dev/RECODISK4                 CACHED  UNKNOWN      65536      65536 RECODISK4

8 rows selected.
```

→ 再度ストレージ拡張を行うと、DATA_0000,0001,0002,0003へ戻った


```
SQL> select NAME,GROUP_NUMBER,STATE,TYPE,TOTAL_MB from v$asm_diskgroup;

NAME                           GROUP_NUMBER STATE       TYPE     TOTAL_MB
------------------------------ ------------ ----------- ------ ----------
DATA                                      1 MOUNTED     EXTERN     524288
RECO                                      2 MOUNTED     EXTERN     262144
```

```
SQL> select VOLUME_NAME,VOLUME_NUMBER,GROUP_NUMBER,SIZE_MB,REDUNDANCY,VOLUME_DEVICE,MOUNTPATH from v$asm_volume;

VOLUME_NAME     VOLUME_NUMBER GROUP_NUMBER    SIZE_MB REDUND VOLUME_DEVICE                  MOUNTPATH
--------------- ------------- ------------ ---------- ------ ------------------------------ ------------------------------
COMMONSTORE                 1            1       5120 UNPROT /dev/asm/commonstore-61        /opt/oracle/dcs/commonstore
```

-> ASM Volumeは変化無し

## 拡張中の動き
一時的にデバイスが4つ増え、ASMでリバランスした後、不要となったデバイスを削除する動きの模様。
（以下、再拡張時のログのため上とデバイス名等が合っていない）

- デバイス

デバイスが4つ増え、もともとデータが入っていたディスクが削除された。

`iostat 1`

```
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.42    0.00   11.93    1.78    0.00   74.87

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              21.00         0.00       192.00          0        192
dm-0             21.00         0.00       192.00          0        192
dm-1              0.00         0.00         0.00          0          0
sdf              80.00       640.00         0.00        640          0
sdg              80.00       640.00         0.00        640          0
sdh              71.00       568.00         0.00        568          0
sdi              81.00       672.00         0.00        672          0
sdj              87.00       640.00       144.00        640        144
asm!commonstore-61     0.00         0.00         0.00          0          0
sdk              84.00       642.00         9.00        642          9
sdl              80.00       640.00         0.00        640          0
sdm              80.00       640.00         0.00        640          0
sdn              80.00       640.00         0.00        640          0
sdo              83.00       664.00         0.00        664          0
sdp              83.00       664.00         0.00        664          0
sdq              83.00       664.00         0.00        664          0
sdr              83.00       664.00         0.00        664          0
```

- ASMディスク  
一時的に8つになった

`asmcmd iostat --io -G DATA 2`

```
Group_Name  Dsk_Name   Reads  Writes
DATA        DATA_0000  3.00   0.00
DATA        DATA_0001  3.00   0.00
DATA        DATA_0002  3.00   0.00
DATA        DATA_0003  3.00   0.00
DATA        DATA_0004  6.00   0.00
DATA        DATA_0005  3.00   0.00
DATA        DATA_0006  3.00   0.00
DATA        DATA_0007  3.00   4.00

Group_Name  Dsk_Name   Reads  Writes
DATA        DATA_0000  3.00   26.00
DATA        DATA_0001  2.00   0.00
DATA        DATA_0002  2.00   0.00
DATA        DATA_0003  2.00   0.00
DATA        DATA_0004  2.00   6.00
DATA        DATA_0005  2.00   0.00
DATA        DATA_0006  2.00   0.00
DATA        DATA_0007  2.00   2.00
```


- リバランス処理

REBAL RUNをとらえられず…ｗ

```
SQL> r
  1* select GROUP_NUMBER, OPERATION, STATE, POWER, ACTUAL,SOFAR, EST_WORK, EST_RATE, EST_MINUTES from V$ASM_OPERATION

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
           1 REBAL WAIT          1
           1 REBAL WAIT          1
           1 REBAL WAIT          1
```

以上