Proxmox にディスクを追加 (btrfs)

既存の 1TB に 4TB の SSD を追加

# 既存のディスクを確認
## 使用量を表示

```
root@hair:~# btrfs filesystem usage /
Overall:
    Device size:                 952.87GiB
    Device allocated:            225.02GiB
    Device unallocated:          727.85GiB
    Device missing:                  0.00B
    Device slack:                  3.50KiB
    Used:                        208.12GiB
    Free (estimated):            743.12GiB      (min: 743.12GiB)
    Free (statfs, df):           743.12GiB
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              257.08MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:223.01GiB, Used:207.73GiB (93.15%)
   /dev/nvme0n1p3        223.01GiB

Metadata,single: Size:2.01GiB, Used:403.19MiB (19.61%)
   /dev/nvme0n1p3          2.01GiB

System,single: Size:4.00MiB, Used:48.00KiB (1.17%)
   /dev/nvme0n1p3          4.00MiB

Unallocated:
   /dev/nvme0n1p3        727.85GiB

root@hair:~# btrfs filesystem df /
Data, single: total=223.01GiB, used=207.73GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=2.01GiB, used=403.22MiB
GlobalReserve, single: total=257.08MiB, used=0.00B
```

## ディスクの一覧を表示
```
root@hair:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 953.9G  0 disk
├─nvme0n1p1 259:1    0  1007K  0 part
├─nvme0n1p2 259:2    0     1G  0 part /boot/efi
└─nvme0n1p3 259:3    0 952.9G  0 part /
nvme1n1     259:4    0   3.7T  0 disk
├─nvme1n1p1 259:5    0    16M  0 part
└─nvme1n1p2 259:6    0   3.7T  0 part
root@hair:~#
root@hair:~#
root@hair:~# fdisk -l
Disk /dev/nvme0n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: KINGSTON OM8PGP41024N-A0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F7D607A5-5933-41DF-B419-1EDF56F85A7A

Device           Start        End    Sectors   Size Type
/dev/nvme0n1p1      34       2047       2014  1007K BIOS boot
/dev/nvme0n1p2    2048    2099199    2097152     1G EFI System
/dev/nvme0n1p3 2099200 2000409230 1998310031 952.9G Linux filesystem


Disk /dev/nvme1n1: 3.73 TiB, 4096805658624 bytes, 8001573552 sectors
Disk model: SPCC M.2 PCIe SSD
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 994A3D8A-9029-4317-A21F-0D17ED84B21B

Device         Start        End    Sectors  Size Type
/dev/nvme1n1p1    34      32767      32734   16M Microsoft reserved
/dev/nvme1n1p2 32768 8001570815 8001538048  3.7T Linux filesystem
root@hair:~#
```
今使っているのは /dev/nvme0n1p3 (nvme0n1)
追加するのは /dev/nvme1
あら？最初は Type が NTFS って表示されてたのに色々とやっているうちに Linux filesystem ってなってますねぇ。まぁ、NTFS だったということで。

やってなかったけども、ここで `btrfs scrub start /` と `btrfs scrub status /` でディスクを壊れていないことを確認した方が良いかもしれない。

# 既存のパーティションを削除する
Windows で使っていたので /dev/nvme1n1 のパーティションを削除する
```
root@hair:~# gdisk /dev/nvme1n1
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): ?
b       back up GPT data to a file
c       change a partition's name
d       delete a partition
i       show detailed information on a partition
l       list known partition types
n       add a new partition
o       create a new empty GUID partition table (GPT)
p       print the partition table
q       quit without saving changes
r       recovery and transformation options (experts only)
s       sort partitions
t       change a partition's type code
v       verify disk
w       write table to disk and exit
x       extra functionality (experts only)
?       print this menu

Command (? for help): i
Partition number (1-2): 1
Partition GUID code: E3C9E316-0B5C-4DB8-817D-F92DF00215AE (Microsoft reserved)
Partition unique GUID: FAC8A871-BABA-4E17-AD0E-7B1DFCD7A34D
First sector: 34 (at 17.0 KiB)
Last sector: 32767 (at 16.0 MiB)
Partition size: 32734 sectors (16.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'Microsoft reserved partition'

Command (? for help): d
Partition number (1-2): 1		(パーティションの 1つめを削除)

Command (? for help): d
Using 2		(パーティションの 2つめを削除)

Command (? for help): 2
b       back up GPT data to a file
c       change a partition's name
d       delete a partition
i       show detailed information on a partition
l       list known partition types
n       add a new partition
o       create a new empty GUID partition table (GPT)
p       print the partition table
q       quit without saving changes
r       recovery and transformation options (experts only)
s       sort partitions
t       change a partition's type code
v       verify disk
w       write table to disk and exit
x       extra functionality (experts only)
?       print this menu

Command (? for help): w		(書き込み)

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/nvme1n1.
The operation has completed successfully.
```

パーティションが削除されたことを確認
```
root@hair:~# fdisk -l
...snip

Disk /dev/nvme1n1: 3.73 TiB, 4096805658624 bytes, 8001573552 sectors
Disk model: SPCC M.2 PCIe SSD
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 994A3D8A-9029-4317-A21F-0D17ED84B21B
root@hair:~#
```

# パーティションを作成
予備領域に 1割程度残しておく
```
root@hair:~# gdisk /dev/nvme1n1
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y

Command (? for help): p
Disk /dev/nvme1n1: 8001573552 sectors, 3.7 TiB
Model: SPCC M.2 PCIe SSD
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 6D322235-AF6F-475E-878E-9D586E15FF5F
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 8001573518
Partitions will be aligned on 2048-sector boundaries
Total free space is 8001573485 sectors (3.7 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-8001573518, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-8001573518, default = 8001572863) or {+-}size{KMGTP}: -370G	(予備領域に 1割程度残しておく)
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/nvme1n1: 8001573552 sectors, 3.7 TiB
Model: SPCC M.2 PCIe SSD
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 6D322235-AF6F-475E-878E-9D586E15FF5F
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 8001573518
Partitions will be aligned on 2048-sector boundaries
Total free space is 775948254 sectors (370.0 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      7225627278   3.4 TiB     8300  Linux filesystem

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/nvme1n1.
The operation has completed successfully.
root@hair:~#
```

## 作成されたパーティションの確認
3.4TB の領域を確保された
```
root@hair:~# fdisk -l /dev/nvme1n1
Disk /dev/nvme1n1: 3.73 TiB, 4096805658624 bytes, 8001573552 sectors
Disk model: SPCC M.2 PCIe SSD
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6D322235-AF6F-475E-878E-9D586E15FF5F

Device         Start        End    Sectors  Size Type
/dev/nvme1n1p1  2048 7225627278 7225625231  3.4T Linux filesystem
```

## btrfs でフォーマット
```
root@hair:~# mkfs.btrfs -L mylabel -n 64k /dev/nvme1n1p1
btrfs-progs v6.2
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM /dev/nvme1n1p1 (3.36TiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              mylabel
UUID:               66cf8320-0e40-481c-94d8-92a17ac46b26
Node size:          65536
Sector size:        4096
Filesystem size:    3.36TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       yes
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     3.36TiB  /dev/nvme1n1p1

```

# ディスクを追加
`/` パーティションにディスクを追加
```
root@hair:~# btrfs device add -f /dev/nvme1n1p1 /
Performing full device TRIM /dev/nvme1n1p1 (3.36TiB) ...
root@hair:~#
root@hair:~#
root@hair:~# btrfs filesystem show
Label: none  uuid: f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
        Total devices 2 FS bytes used 207.67GiB
        devid    1 size 952.87GiB used 226.02GiB path /dev/nvme0n1p3
        devid    2 size 3.36TiB used 0.00B path /dev/nvme1n1p1
```

# 使用量を確認
ディスクの使用量を確認
```
root@hair:~# btrfs filesystem usage /
Overall:
    Device size:                   4.29TiB
    Device allocated:            226.02GiB
    Device unallocated:            4.07TiB
    Device missing:                  0.00B
    Device slack:                  7.00KiB
    Used:                        207.67GiB
    Free (estimated):              4.09TiB      (min: 4.09TiB)
    Free (statfs, df):             4.09TiB
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              260.78MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:224.01GiB, Used:207.27GiB (92.53%)
   /dev/nvme0n1p3        224.01GiB

Metadata,single: Size:2.01GiB, Used:410.36MiB (19.96%)
   /dev/nvme0n1p3          2.01GiB

System,single: Size:4.00MiB, Used:48.00KiB (1.17%)
   /dev/nvme0n1p3          4.00MiB

Unallocated:
   /dev/nvme0n1p3        726.85GiB
   /dev/nvme1n1p1          3.36TiB
root@hair:~#
```


# ディスクのチェック
`btrfs scrub start /` で開始
`btrfs scrub status /` で実行状況を表示
```
root@hair:~# btrfs scrub start /
scrub started on /, fsid f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8 (pid=735633)
root@hair:~#
root@hair:~#
root@hair:~# btrfs scrub status /
UUID:             f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
Scrub started:    Mon Jan 20 05:49:53 2025
Status:           running
Duration:         0:00:10
Time left:        0:01:14
ETA:              Mon Jan 20 05:51:17 2025
Total to scrub:   207.67GiB
Bytes scrubbed:   24.70GiB  (11.90%)
Rate:             2.47GiB/s
Error summary:    no errors found
root@hair:~#
root@hair:~# btrfs scrub status /
UUID:             f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
Scrub started:    Mon Jan 20 05:49:53 2025
Status:           running
Duration:         0:01:10
Time left:        0:00:06
ETA:              Mon Jan 20 05:51:10 2025
Total to scrub:   207.67GiB
Bytes scrubbed:   189.80GiB  (91.39%)
Rate:             2.71GiB/s
Error summary:    csum=10
  Corrected:      0
  Uncorrectable:  10		← エラーを検出
  Unverified:     0
root@hair:~# ERROR: there are uncorrectable errors

root@hair:~# btrfs scrub status /
UUID:             f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
Scrub started:    Mon Jan 20 05:49:53 2025
Status:           finished
Duration:         0:01:16
Total to scrub:   207.67GiB
Rate:             2.73GiB/s
Error summary:    csum=10
  Corrected:      0
  Uncorrectable:  10
  Unverified:     0
root@hair:~#
```

# btrfs check  でディスクをチェックしてみる
エラーが出たので `btrfs check` を実行してみる
ディスクをマウントしているので readonly で --force を使う

```
root@hair:~# btrfs check --force --backup /dev/nvme0n1p3
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/nvme0n1p3
UUID: f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
[1/7] checking root items
[2/7] checking extents
super bytes used 222982537216 mismatches actual used 222982561792
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 222982561792 bytes used, error(s) found
total csum bytes: 195223260
total tree bytes: 430473216
total fs tree bytes: 155009024
total extent tree bytes: 50462720
btree space waste bytes: 60067299
file data blocks allocated: 2684188442624
 referenced 179438419968
 ```
エラーがありますねぇ。

nvme1n1p1 もチェックしてみる
```
root@hair:~# btrfs check --force --backup /dev/nvme1n1p1
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/nvme1n1p1
UUID: f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 222982537216 bytes used, no error found
total csum bytes: 195223236
total tree bytes: 430473216
total fs tree bytes: 155009024
total extent tree bytes: 50462720
btree space waste bytes: 60067111
file data blocks allocated: 2684188418048
 referenced 179438428160
root@hair:~#
```
流石に追加したばかりではエラー無し

## デバイス一覧の表示
```
root@hair:~# btrfs device scan
Scanning for Btrfs filesystems
registered: /dev/nvme0n1p3
registered: /dev/nvme1n1p1
```

nvme0n1p3 の方にエラーがありそうなのでディスクを取り外してみる。
```
root@hair:~# btrfs device delete /dev/nvme0n1p3 /
ERROR: error removing device '/dev/nvme0n1p3': Input/output error
```
読めないので、nvme1n1p1 側へコピーできずエラーになるっぽい。
残念。


# その後、気づいたらディスクが治っていた

仮想マシンを消してエラー部分が未使用になって、適当に処理されたのでしょう。
Peoxmox 自体が使っている部分だとそういうことは起きないですから運がよかったですね。
```
root@hair:~# btrfs scrub status /
UUID:             f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
Scrub started:    Fri Mar 21 02:35:29 2025
Status:           finished
Duration:         0:02:04
Total to scrub:   459.12GiB
Rate:             3.70GiB/s
Error summary:    no errors found
```

あれ？ディスクのデバイス名変わってますねぇ？？？
起動順番が変わった？
```
root@hair:~# btrfs filesystem usage /
Overall:
    Device size:                   4.29TiB
    Device allocated:            503.05GiB
    Device unallocated:            3.80TiB
    Device missing:                  0.00B
    Device slack:                  7.00KiB
    Used:                        459.90GiB
    Free (estimated):              3.84TiB      (min: 3.84TiB)
    Free (statfs, df):             3.84TiB
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:499.01GiB, Used:459.22GiB (92.03%)
   /dev/nvme1n1p3         63.01GiB
   /dev/nvme0n1p1        436.00GiB

Metadata,single: Size:4.01GiB, Used:693.86MiB (16.91%)
   /dev/nvme1n1p3          2.01GiB
   /dev/nvme0n1p1          2.00GiB

System,single: Size:32.00MiB, Used:80.00KiB (0.24%)
   /dev/nvme0n1p1         32.00MiB

Unallocated:
   /dev/nvme1n1p3        887.85GiB
   /dev/nvme0n1p1          2.94TiB
root@hair:~# btrfs filesystem show
Label: none  uuid: f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
        Total devices 2 FS bytes used 460.05GiB
        devid    1 size 952.87GiB used 65.02GiB path /dev/nvme1n1p3
        devid    2 size 3.36TiB used 438.03GiB path /dev/nvme0n1p1

root@hair:~#
```

一応チェックしてみると、何かエラーぽいのが見えますねぇ。うーむ。
```
root@hair:~# btrfs check --force --backup /dev/nvme1n1p3
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/nvme1n1p3
UUID: f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
[1/7] checking root items
[2/7] checking extents
super bytes used 494013734912 mismatches actual used 493974269952
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 493974269952 bytes used, error(s) found
total csum bytes: 453011616
total tree bytes: 727826432
total fs tree bytes: 188416000
total extent tree bytes: 48349184
btree space waste bytes: 74226903
file data blocks allocated: 4397997400064
 referenced 430474137600
```
```
root@hair:~# btrfs check --force --backup /dev/nvme0n1p1
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/nvme0n1p1
UUID: f21d1750-d9eb-4ea8-86ad-924c8bfdd0c8
[1/7] checking root items
[2/7] checking extents
super bytes used 494013734912 mismatches actual used 493974269952
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 493974269952 bytes used, error(s) found
total csum bytes: 453011616
total tree bytes: 727826432
total fs tree bytes: 188416000
total extent tree bytes: 48349184
btree space waste bytes: 74226903
file data blocks allocated: 4397997400064
 referenced 430474137600
root@hair:~#
```
どちらも btrfs として / を対象にチェックしてるみたいですね。両方のディスクで UUID が同じ、結果も同じになりました。