RHEL 8.6 ppc64le で MD RAID (RAID1) を構成

#### はじめに

RHEL on Power でディスク冗長化のため MD RAID を構成したログです。


(イメージ図)
![md-raid.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/517421/9fda49b2-2653-d05b-ac45-b89be3fabbdd.png)


---


#### 参照文書


https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/8/html/managing_storage_devices/converting-a-root-disk-to-raid1-after-installation_managing-raid


https://access.redhat.com/solutions/2390831

---

#### 環境


RHEL 8.6 (ppc64le, on Power S1022)
 - PowerVC でデプロイしてミラーリング用ディスクを接続　
 - testvmm というホスト名のサーバー


---

#### 実行ログ

##### 1 追加ディスク認識

PowerVC から RHEL VM をデプロイし、OS領域と同じ大きさのボリュームを作成して接続


1-1) デバイスのスキャン

```
[root@testvmm ~]# ls /sys/class/scsi_host/ | while read host ; do echo "- - -" > /sys/class/scsi_host/$host/scan ; done
```

接続確認

1-2) mpathb が認識されたことを確認

```
root@testvmm ~]# ls -l /dev/mapper
total 0
crw-------. 1 root root 10, 236 Jul 23 19:41 control
lrwxrwxrwx. 1 root root       7 Jul 23 19:41 mpatha -> ../dm-0
lrwxrwxrwx. 1 root root       7 Jul 23 19:41 mpatha1 -> ../dm-1
lrwxrwxrwx. 1 root root       7 Jul 23 19:41 mpatha2 -> ../dm-2
lrwxrwxrwx. 1 root root       7 Jul 23 19:41 mpatha3 -> ../dm-3
lrwxrwxrwx. 1 root root       7 Jul 23 20:03 mpathb -> ../dm-6
lrwxrwxrwx. 1 root root       7 Jul 23 19:41 rhel-root -> ../dm-5
lrwxrwxrwx. 1 root root       7 Jul 23 19:41 rhel-swap -> ../dm-4

[root@testvmm ~]#  multipath -ll
mpatha (360050763808106d7d800000000000128) dm-0 AIX,VDASD
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 0:0:1:0 sda 8:0  active ready running
  `- 1:0:1:0 sdb 8:16 active ready running
mpathb (360050763808106d7d800000000000129) dm-6 AIX,VDASD
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 0:0:3:0 sdc 8:32 active ready running
  `- 1:0:2:0 sdd 8:48 active ready running
```


---

##### 2 /dev/sda を /dev/mapper/mpatha で読み替えて手順を実行


2-1)パーティション情報を確認

```
[root@testvmm ~]# parted /dev/mapper/mpatha u s p
Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpatha: 104857600s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start     End         Size        Type     File system  Flags
 1      2048s     10239s      8192s       primary               boot, prep
 2      10240s    2107391s    2097152s    primary  xfs
 3      2107392s  104857599s  102750208s  primary               lvm
```

1 は IBM PowerSystems のRHELのため存在しています。(prep partition)


2-2) 追加ディスクでパーティション作成

```
[root@testvmm ~]# parted /dev/mapper/mpathb mklabel msdos
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb mkpart primary 2048s 10239s
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb mkpart primary 10240s 2107391s
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb mkpart primary 2107392s 104857599s
Information: You may need to update /etc/fstab.
```


2-3) パーティションに RAID フラクを立てる(mpatha, mpathb に対して実行)

```
[root@testvmm ~]# parted /dev/mapper/mpatha set 1 raid on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpatha set 2 raid on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpatha set 3 raid on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb set 1 raid on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb set 2 raid on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb set 3 raid on
Information: You may need to update /etc/fstab.
```

2-4) /dev/md0 を作成

```
[root@testvmm ~]# mdadm  --create /dev/md0 --level=1 --raid-disks=2 missing /dev/mapper/mpathb2 --metadata=1.0
mdadm: array /dev/md0 started.
```

2-5) /dev/md0 でファイルシステムを作成

```
[root@testvmm ~]# mkfs.xfs /dev/md0
meta-data=/dev/md0               isize=512    agcount=4, agsize=65532 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=262128, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1566, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@testvmm ~]#
```


2-6) /boot でデータ同期、マウント

```
[root@testvmm ~]# mkdir /mnt/md0
[root@testvmm ~]# mount /dev/md0 /mnt/md0
[root@testvmm ~]# rsync -a /boot/ /mnt/md0/
[root@testvmm ~]# sync
[root@testvmm ~]# umount /mnt/md0
[root@testvmm ~]# rmdir /mnt/md0
```


2-7) /dev/md0 に /bootをマウント

```
[root@testvmm ~]# umount /boot
[root@testvmm ~]# mount /dev/md0 /boot
```

2-8) mpatha を md0 に追加

```
[root@testvmm ~]# mdadm /dev/md0 -a /dev/mapper/mpatha2
mdadm: added /dev/mapper/mpatha2
```


2-9) RAID 化状態を確認

```
[root@testvmm ~]# mdadm -D /dev/md0
/dev/md0:
           Version : 1.0
     Creation Time : Tue Jul 25 00:20:29 2023
        Raid Level : raid1
        Array Size : 1048512 (1023.94 MiB 1073.68 MB)
     Used Dev Size : 1048512 (1023.94 MiB 1073.68 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jul 25 00:21:23 2023
             State : clean, degraded, recovering
    Active Devices : 1
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

    Rebuild Status : 95% complete     ##<= 95% complete が表示

              Name : testvmm:0  (local to host testvmm)
              UUID : 5dfd2d76:9a030e84:1fe2e429:a5348eec
            Events : 29

    Number   Major   Minor   RaidDevice State
       2     253        2        0      spare rebuilding   /dev/dm-2
       1     253        8        1      active sync   /dev/dm-8
[root@testvmm ~]#
```

数分待ち、Rebuild Status が complete になりました。


2-10) /etc/fstab を編集


/dev/md0 のUUID を確認

```
[root@testvmm ~]# blkid | grep md0
/dev/md0: UUID="41435ad6-deae-470a-a6dd-5597e6260a7f" BLOCK_SIZE="512" TYPE="xfs"

UUID=4693aeba-49a9-4864-8edf-44dcacf5a3ad /boot                   xfs     defaults        0 0
UUID=41435ad6-deae-470a-a6dd-5597e6260a7f /boot                   xfs     defaults        0 0

```


```
[root@testvmm ~]# vi /etc/fstab

-> /boot の UUID を変更


[root@testvmm ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed May  3 00:39:24 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
#UUID=9fc56ab1-d18f-4311-9c08-a1c1c17417f8 /boot                   xfs     defaults        0 0
/dev/mapper/rhel-swap   none                    swap    defaults        0 0
UUID=41435ad6-deae-470a-a6dd-5597e6260a7f /boot                   xfs     defaults        0 0
```



2-11) /dev/md1 の作成

```
[root@testvmm ~]# mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/mapper/mpathb3 --metadata=1.0
mdadm: array /dev/md1 started.
```

確認

```
[root@testvmm ~]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  rhel   1   2   0 wz--n- 48.99g    0
```

VG "rhel" の PV は 1 です。


2-12) /dev/md1 に rhel を追加

```
[root@testvmm ~]# vgextend rhel /dev/md1
  Physical volume "/dev/md1" successfully created.
  Volume group "rhel" successfully extended
```

確認


```
[root@testvmm ~]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  rhel   2   2   0 wz--n- 97.98g 48.99g
```


VG "rhel" の PV が2 になりました。


2-13) mpatha3 の 物理エクステントを新しいアレイに移動

```
[root@testvmm ~]# pvmove /dev/mapper/mpatha3 /dev/md1
  /dev/mapper/mpatha3: Moved: 0.01%
  /dev/mapper/mpatha3: Moved: 8.25%
  /dev/mapper/mpatha3: Moved: 22.94%
  /dev/mapper/mpatha3: Moved: 36.53%
  /dev/mapper/mpatha3: Moved: 47.19%
  /dev/mapper/mpatha3: Moved: 58.71%
  /dev/mapper/mpatha3: Moved: 76.09%
  /dev/mapper/mpatha3: Moved: 92.44%
  /dev/mapper/mpatha3: Moved: 100.00%
```



2-14) mpatha3 を VG "rhel" 、PV から削除

```
[root@testvmm ~]# vgreduce rhel /dev/mapper/mpatha3
  Removed "/dev/mapper/mpatha3" from volume group "rhel"

[root@testvmm ~]# pvremove /dev/mapper/mpatha3
  Labels on physical volume "/dev/mapper/mpatha3" successfully wiped.
[root@testvmm ~]#
```


2-15) mpatha3 を /dev/md1 に追加

```
[root@testvmm ~]# mdadm /dev/md1 -a /dev/mapper/mpatha3
mdadm: added /dev/mapper/mpatha3
```


2-16) RAID 化状態を確認

```
[root@testvmm ~]# mdadm -D /dev/md1
/dev/md1:
           Version : 1.0
     Creation Time : Tue Jul 25 00:22:36 2023
        Raid Level : raid1
        Array Size : 51374976 (48.99 GiB 52.61 GB)
     Used Dev Size : 51374976 (48.99 GiB 52.61 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jul 25 00:25:38 2023
             State : clean, degraded, recovering
    Active Devices : 1
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

    Rebuild Status : 2% complete   ##< RAID化進行中

              Name : testvmm:1  (local to host testvmm)
              UUID : 6660f874:01c32e81:783d5695:ef10722f
            Events : 22

    Number   Major   Minor   RaidDevice State
       2     253        3        0      spare rebuilding   /dev/dm-3
       1     253        9        1      active sync   /dev/dm-9
[root@testvmm ~]#
```

　数分後、完了しました。


2-17) /etc/mdadm.conf に定義追加

```
[root@testvmm ~]#  mdadm --examine --scan > /etc/mdadm.conf
[root@testvmm ~]# cat /etc/mdadm.conf
ARRAY /dev/md/0  metadata=1.0 UUID=5dfd2d76:9a030e84:1fe2e429:a5348eec name=testvmm:0
ARRAY /dev/md/1  metadata=1.0 UUID=6660f874:01c32e81:783d5695:ef10722f name=testvmm:1
```

2-18) /etc/default/grub の更新 


```
[root@testvmm ~]# vi /etc/default/grub

[root@testvmm ~]#  cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="ofconsole"
GRUB_CMDLINE_LINUX="rd.auto=1 rd.lvm.lv=rhel/root crashkernel=auto  rd.lvm.lv=rhel/swap"  ##<=修正した行
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_TERMINFO="terminfo -g 80x24 console"
GRUB_DISABLE_OS_PROBER=true
```


2-19) grub2.cfg を更新


```
[root@testvmm ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Generating boot entries from BLS files...
done
[root@testvmm ~]#
```

2-20) /boot/grub2/device.map を更新

```
[root@testvmm ~]# cat /boot/grub2/device.map
# this device map was generated by anaconda
(hd0)      /dev/mapper/mpatha
[root@testvmm ~]# vi /boot/grub2/device.map

[root@testvmm ~]# cat /boot/grub2/device.map
# this device map was generated by anaconda
(hd0)      /dev/mapper/mpatha
(hd1)      /dev/mapper/mpathb   ##<= 追加
```

2-21) grubを再導入 

```
[root@testvmm ~]# grub2-install /dev/mapper/mpatha
Installing for powerpc-ieee1275 platform.
grub2-install: error: the chosen partition is not a PReP partition.
[root@testvmm ~]#  grub2-install /dev/mapper/mpathb
Installing for powerpc-ieee1275 platform.
grub2-install: error: the chosen partition is not a PReP partition.
[root@testvmm ~]#
```

PReP パーティションはエラーが発生することは既知の事象。


---

2-22) PReP パーティションの修正

https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/8/html/managing_storage_devices/converting-a-root-disk-to-raid1-after-installation_managing-raid

```
root@testvmm ~]# dd if=/dev/mapper/mpatha1 of=/dev/mapper/mpathb1
8192+0 records in
8192+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0284917 s, 147 MB/s
[root@testvmm ~]#


[root@testvmm ~]# parted /dev/mapper/mpatha set 1 prep on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpatha set 1 boot on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb set 1 prep on
Information: You may need to update /etc/fstab.

[root@testvmm ~]# parted /dev/mapper/mpathb set 1 boot on
Information: You may need to update /etc/fstab.
```

---

2-23) mdデバイス確認

```
[root@testvmm ~]# mdadm -D /dev/md0
/dev/md0:
           Version : 1.0
     Creation Time : Tue Jul 25 00:20:29 2023
        Raid Level : raid1
        Array Size : 1048512 (1023.94 MiB 1073.68 MB)
     Used Dev Size : 1048512 (1023.94 MiB 1073.68 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jul 25 00:30:50 2023
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : testvmm:0  (local to host testvmm)
              UUID : 5dfd2d76:9a030e84:1fe2e429:a5348eec
            Events : 31

    Number   Major   Minor   RaidDevice State
       2     253        2        0      active sync   /dev/dm-2
       1     253        8        1      active sync   /dev/dm-8
[root@testvmm ~]#
```

-> /dev/dm-2, /dev/dm-8 がメンバーであることを確認


```
[root@testvmm ~]# mdadm -D /dev/md1
/dev/md1:
           Version : 1.0
     Creation Time : Tue Jul 25 00:22:36 2023
        Raid Level : raid1
        Array Size : 51374976 (48.99 GiB 52.61 GB)
     Used Dev Size : 51374976 (48.99 GiB 52.61 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jul 25 00:30:51 2023
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : testvmm:1  (local to host testvmm)
              UUID : 6660f874:01c32e81:783d5695:ef10722f
            Events : 71

    Number   Major   Minor   RaidDevice State
       2     253        3        0      active sync   /dev/dm-3
       1     253        9        1      active sync   /dev/dm-9
[root@testvmm ~]#
```

-> /dev/dm-3, /dev/dm-9 がメンバーであることを確認


---

2-24) initramfs イメージを mdadmconf で再度ビルド

intramfs のバックアップ

```
[root@testvmm ~]# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
```

```

[root@testvmm ~]# dracut -f --mdadmconf
[root@testvmm ~]# echo $?
0
[root@testvmm ~]# date
Tue Jul 25 00:32:01 EDT 2023

[root@testvmm ~]# ls -ltr /boot
total 383072
-rw-------. 1 root root   3620730 Apr 15  2022 System.map-4.18.0-372.9.1.el8.ppc64le
-rw-r--r--. 1 root root    150984 Apr 15  2022 config-4.18.0-372.9.1.el8.ppc64le
-rwxr-xr-x. 1 root root  35634133 Apr 15  2022 vmlinuz-4.18.0-372.9.1.el8.ppc64le
drwxr-xr-x. 3 root root        17 May  2 20:40 efi
drwxr-xr-x. 3 root root        21 May  2 20:43 loader
lrwxrwxrwx. 1 root root        50 May  2 20:44 symvers-4.18.0-372.9.1.el8.ppc64le.gz -> /lib/modules/4.18.0-372.9.1.el8.ppc64le/symvers.gz
-rwxr-xr-x. 1 root root  35634133 May  2 20:44 vmlinuz-0-rescue-7d4848a6a4584ea7adae49b33f3b7450
-rw-------. 1 root root 107181634 May  2 20:45 initramfs-0-rescue-7d4848a6a4584ea7adae49b33f3b7450.img
-rw-------. 1 root root  34939392 May  2 20:49 initramfs-4.18.0-372.9.1.el8.ppc64lekdump.img
drwx------. 4 root root       104 Jul 25 00:29 grub2
-rw-------. 1 root root  58110011 Jul 25 00:30 initramfs-4.18.0-372.9.1.el8.ppc64le.img.07-25-003018.bak
-rw-------. 1 root root  58777371 Jul 25 00:31 initramfs-4.18.0-372.9.1.el8.ppc64le.img
[root@testvmm ~]#
```

=< 無事再作成されています。

2-25) VM 停止、LPAR起動

```
[root@testvmm ~]# shutdown -h now
```


HMC 仮想端末からRHEL のブートシークエンスを眺めました。(ログ省略)

```
Red Hat Enterprise Linux 8.6 (Ootpa)
Kernel 4.18.0-372.9.1.el8.ppc64le on an ppc64le

Activate the web console with: systemctl enable --now cockpit.socket

testvmm login:

```


ログイン後確認：

```
[root@testvmm ~]# multipath -ll
mpatha (360050763808106d7d800000000000128) dm-0 AIX,VDASD
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 0:0:1:0 sda 8:0  active ready running
  `- 1:0:1:0 sdc 8:32 active ready running
mpathb (360050763808106d7d800000000000129) dm-1 AIX,VDASD
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 0:0:3:0 sdb 8:16 active ready running
  `- 1:0:2:0 sdd 8:48 active ready running
```

```
[root@testvmm ~]# mdadm -D /dev/md0
/dev/md0:
           Version : 1.0
     Creation Time : Tue Jul 25 00:20:29 2023
        Raid Level : raid1
        Array Size : 1048512 (1023.94 MiB 1073.68 MB)
     Used Dev Size : 1048512 (1023.94 MiB 1073.68 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jul 25 00:34:22 2023
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : testvmm:0  (local to host testvmm)
              UUID : 5dfd2d76:9a030e84:1fe2e429:a5348eec
            Events : 31

    Number   Major   Minor   RaidDevice State
       2     253        3        0      active sync   /dev/dm-3
       1     253        6        1      active sync   /dev/dm-6
```


-> md0のメンバーの/dev/dm-X の番号が変わっています。


```
[root@testvmm ~]# mdadm -D /dev/md1
/dev/md1:
           Version : 1.0
     Creation Time : Tue Jul 25 00:22:36 2023
        Raid Level : raid1
        Array Size : 51374976 (48.99 GiB 52.61 GB)
     Used Dev Size : 51374976 (48.99 GiB 52.61 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Jul 25 00:35:23 2023
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : testvmm:1  (local to host testvmm)
              UUID : 6660f874:01c32e81:783d5695:ef10722f
            Events : 73

    Number   Major   Minor   RaidDevice State
       2     253        4        0      active sync   /dev/dm-4
       1     253        7        1      active sync   /dev/dm-7
```

-> md0のメンバーの/dev/dm-X の番号が変わっています。



```
[root@testvmm ~]# ls -l /dev/mapper
total 0
crw-------. 1 root root 10, 236 Jul 25 00:34 control
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpatha -> ../dm-0
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpatha1 -> ../dm-2
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpatha2 -> ../dm-3
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpatha3 -> ../dm-4
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpathb -> ../dm-1
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpathb1 -> ../dm-5
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpathb2 -> ../dm-6
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 mpathb3 -> ../dm-7
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 rhel-root -> ../dm-8
lrwxrwxrwx. 1 root root       7 Jul 25 00:34 rhel-swap -> ../dm-9
```

認識順番が入れ替わっていました。(LVデバイス(rhel-root、rhel-swap が後ろ)



----

#### おわりに


これでできたはず..、


詳細説明は省略していますが、冒頭にリンクしたRHELの knowledgebase を参照ください。



bootlist の設定は以下で確認しました。

https://qiita.com/c_u/items/db684b21bbffd7f4a677



以上です。