VMFS 上で削除された VMDK ファイル復旧の可能性を探った話

# VM の削除について

vSphere Client の操作をする際、右クリックのメニューは 
`インベントリから削除` と `ディスクから削除` が隣り合っている。

![vsphere_client_delete.png](https://qiita-image-store.s3.amazonaws.com/0/44206/21d2a27c-f766-83b2-e112-f52fcd42adf1.png)

一応どちらも警告は出るが、インベントリから削除するつもりでうっかり
ディスクから削除してしまうと VM のデータが葬られて使用できなくなる。

削除自体はすぐ終わるので 0 を上書きなどはされていないと推測する。
データ自体は残っていると思うので
消してしまったファイルは復旧可能なのか調べてみたい。


# 調査の流れ

VMFS は仕様が公開されていないので、
vmfs-tools のソースを参考にして調査を試みる。
VMFS5 を取り扱うには完璧じゃないようだが、とりあえず挑戦。

* ホストの vSphere ESXi にゲストとして vSphere ESXi をインストールする<br>
（このゲスト ESXi がインストールされた vmdk ファイルの VMFS 領域が調査対象になる）
* Nested VM として CentOS をインストールする<br>
（ゲスト ESXi に、さらにゲストとして CentOS を入れる）
* Netsed VM および ゲスト ESXi をシャットダウンして ゲスト ESXi の vmdk をコピー
* Nested VM のファイルが存在する状態で、事前に様子を見ておく
* Nested VM をディスクから削除した後、どう変わったか調べる<br>
(削除後の vmdk ファイルもコピーして VMFS 領域を調べてみる)

Nested 環境なら調査対象の VMFS フォーマットされたパーティションは vmdk ファイル内なので、
他のところに持っていって作業がし易い。

# 調査開始

`<vmname>.vmdk` ファイルは virtual disk descriptor file と呼ばれ、ディスクデータ本体ではないので、
`<vmname>-flat.vmdk` ファイルをコピーしてくる。
ここでのターゲットは __backup_vsphere-flat.vmdk という名前になっている。

さて、どう探せばいいだろうか。
まずは Nested VM が削除されていない状態で
Nested VM のデータを見つけられるか確認してみることにする。

捜索対象は、Nested VM の MBR にする。
ゲスト ESXi の VMDK を別の Linux に転送しておいて、そちらで調査開始。

と言いつつ MBR のことが良く分かっていないので、先に構造を確認しておく。
参考資料をもとに、ざっくりと下記を理解。

* 先頭から 446 byte が ブートローダ
* ブートローダの後ろ 64 byte がパーティションテーブル (16 byte ずつ 4 レコード)
* 最後の 2 byte がマジックナンバー


手元にある GRUB がインストールされた CentOS の MBR を読むと以下のものが見つかる

* GRUB が使うと思われる文字列
* パーティションテーブル
* マジックナンバー

これらを手掛かりにして MBR を探す。
Nested VM は 10GB の VMDK を作成して 2台インストール済み。
それぞれ異なるパーティションの切り方をした。
ゲスト ESXi の vmdk は 40GB で作っていて、とりあえず端から端まで探す。

```python:searchmbr.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, os, re, struct
#target = './mbr.dd'
target = './__backup_vsphere-flat.vmdk'
MAGIC_OF_MBR = '\x55\xaa'
MARK_OF_GRUB = 'GRUB \x00Geom\x00Hard Disk\x00Read\x00 Error'

try:
    if sys.argv[1] == '-d':
        DEBUG = 1
except:
    pass

def D(mesg):
    try:
        if DEBUG:
            import inspect
            print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
    except NameError:
        pass

def convert_num(num):
    if len(str(int(num))) > 12:
        return '%.1fT' % (num/(1024.0**4))
    elif len(str(int(num))) > 9:
        return '%.1fG' % (num/(1024.0**3))
    elif len(str(int(num))) > 6:
        return '%.1fM' % (num/(1024.0**2))
    elif len(str(int(num))) > 3:
        return '%.1fk' % (num/(1024.0**1))
    else:
        return str(num)


def check(data):
    bootloader = data[0:446]
    part_table = data[446:510]
    magic = data[510:512]
    if magic != MAGIC_OF_MBR:
        return False
    if re.search(MARK_OF_GRUB, bootloader):
        print 'This is maybe MBR !!'
    else:
        return False

    splitter = lambda s, n: [s[i:i+n] for i in range(0, len(s), n)]
    print "Partition    Boot    %s    %s    Id    Size" % ('Start'.rjust(10), 'End'.rjust(10))
    i=1
    for record in splitter(part_table, 16):
        boot = record[0]
        D('flag: %s' % hex(ord(boot)))
        if boot == '\x80':
            boot = '*'
        elif boot == '\x00':
            boot = ''
        else:
            print 'boot flag is wrong'
            boot = hex(ord(boot))
        id = record[4:5]
        # LBA 方式のデータのみ確認 (CHS 方式のデータを見ない)
        start_sector, num_sector = struct.unpack('II', record[8:16])
        print "%s    %s    %10d    %10d  %s    %sB" % (str(i).rjust(9), boot.rjust(4), start_sector, start_sector + num_sector, hex(ord(id)).rjust(4), convert_num(num_sector * 512))
        i+=1
    return True


with open(target , 'r') as f:
    try:
        fd = f.fileno()
    except Exception as e:
        print e
        raise Exception('cannot get file size')
    else:
        size = os.fstat(fd).st_size

    i=1
    data = f.read(512)
    while data:
        if check(data):
            print "position of above data: %d - %d" %  (f.tell() - 512, f.tell())
        data = f.read(512)
        if f.tell() > size / 100.0 * i:
            D("searching... %d / %d" % (f.tell(), size))
            i+=1
```

```console:MBR捜索
# python searchmbr.py
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
        1       *          2048       1026048  0x83    500.0MB
        2               1026048       2074624  0x82    512.0MB
        3               2074624      20971520  0x83    9.0GB
        4                     0             0   0x0    0B
position of above data: 38196477952 - 38196478464
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 38792327168 - 38792327680
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 39116325888 - 39116326400
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
        1       *          2048       1026048  0x83    500.0MB
        2               1026048      20971520  0x8e    9.5GB
        3                     0             0   0x0    0B
        4                     0             0   0x0    0B
position of above data: 39419117568 - 39419118080
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 40218379264 - 40218379776
This is maybe MBR !!
Partition    Boot         Start           End    Id    Size
boot flag is wrong
        1    0x24     332251185    2480556663   0x0    1.0TB
boot flag is wrong
        2    0xf9    2548826057    4435176463  0xbe    899.5GB
boot flag is wrong
        3    0x70    3053499650    4967399938   0x0    912.6GB
boot flag is wrong
        4    0xd7             0             0  0x4f    0B
position of above data: 40242606080 - 40242606592
```

無事探し出せた。
フラグがおかしなものも見つかるが、Nested VM は
10GB の vmdk で作ったので、フラグが正しい 2つで間違いないだろう。

## VMFS の探索

それでは VMFS を読んでいく。ここからが本番だ。

```console
# parted __backup_vsphere-flat.vmdk unit s print
モデル:  (file)
ディスク /root/recover/__backup_vsphere-flat.vmdk: 83886080s
セクタサイズ (論理/物理): 512B/512B
パーティションテーブル: gpt

番号  開始       終了       サイズ     ファイルシステム  名前  フラグ
 1    64s        8191s      8128s      fat16                   boot
 5    8224s      520191s    511968s    fat16
 6    520224s    1032191s   511968s    fat16
 7    1032224s   1257471s   225248s
 8    1257504s   1843199s   585696s    fat16
 2    1843200s   10229759s  8386560s   fat16
 3    10229760s  83886046s  73656287s
```

パーティションテーブルは GPT 
一番サイズの大きい領域が VMFS のはずだが、parted では良く分からない。

GPT において、VMFS の GUID は `AA31E02A400F11DB9590000C2911D1B8` が使われるそうだ。
表記を変えると `AA31E02A-400F-11DB-9590-000C2911D1B8`
サイズの大きい パーティション 3番に相当する場所を見てみると、
先頭の 3項目 はリトルエンディアンだそうだ。確かに GUID が一致した。

```console:GPTの調査
# hexdump -C -s `expr 512 \* 2 + 128 \* 2` -n 128 vsphere-flat.vmdk | less
00000500  2a e0 31 aa 0f 40 db 11  95 90 00 0c 29 11 d1 b8  |*.1..@......)...|
00000510  45 01 35 2b 41 d6 59 45  aa 77 33 40 c0 94 f4 0d  |E.5+A.YE.w3@....|
00000520  00 18 9c 00 00 00 00 00  de ff ff 04 00 00 00 00  |................|
00000530  00 00 00 00 00 00 00 00  00 00 ff ff ff ff ff ff  |................|
00000540  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
```

VMFS に関する情報を読み進めていきたい。

```c:vmfs_fs.h
#define VMFS_FSINFO_MAGIC  0x2fabf15e

struct vmfs_fsinfo_raw {
   uint32_t magic;
   uint32_t volver;
   u_char ver;
   uuid_t uuid;
   uint32_t mode;
   char label[128];
   uint32_t dev_blocksize;
   uint64_t blocksize;
   uint32_t ctime; /* ctime? in seconds */
   uint32_t _unknown3;
   uuid_t lvm_uuid;
   u_char _unknown4[16];
   uint32_t fdc_header_size;
   uint32_t fdc_bitmap_count;
   uint32_t subblock_size;
} __attribute__((packed));
```

vmfs-tools を見るとマジックナンバーがあるようなので、MBR の時と同じ要領で探してみる。

```python:searchvmfsinfo.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, os, re, struct
target = './__backup_vsphere-flat.vmdk'

MAGIC_OF_VMFSINFO = '\x5e\xf1\xab\x2f'

try:
    if sys.argv[1] == '-d':
        DEBUG = 1
except:
    pass

def D(mesg):
    try:
        if DEBUG:
            import inspect
            print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
    except NameError:
        pass

def check(data):
    if MAGIC_OF_VMFSINFO in data:
        print 'Magic Number found. This is maybe VMFS !!'
        return True
    else:
        return False

with open(target , 'r') as f:
    try:
        fd = f.fileno()
    except Exception as e:
        print e
        raise Exception('cannot get file size')
    else:
        size = os.fstat(fd).st_size

    i=1
    data = f.read(512)
    while data:
        if check(data):
            print "position of above data: %d - %d" %  (f.tell() - 512, f.tell())
        data = f.read(512)
        if f.tell() > size / 100.0 * i:
            D("searching... %d / %d" % (f.tell(), size))
            i+=1
```

```console:vmfsinfo捜索
# python searchvmfs.py
Magic Number found. This is maybe VMFS !!
position of above data: 5257560064 - 5257560576
Magic Number found. This is maybe VMFS !!
position of above data: 37435702784 - 37435703296
Magic Number found. This is maybe VMFS !!
position of above data: 38707637760 - 38707638272
Magic Number found. This is maybe VMFS !!
position of above data: 39859326464 - 39859326976
Magic Number found. This is maybe VMFS !!
position of above data: 40373243392 - 40373243904
```

いくつか見つかった。VMFS パーティションの先頭は `10229760 * 512 = 5237637120` なので先頭からすぐ始まっているわけではなかったようだ。
間が空いているのはよく分からないが、多分最初に見つかった場所に VMFS の情報があるのだろう。
その他は偶然一致しただけだろうか？
128 バイト分の label が見られそうなので、調べてみよう。

```console:labelデータの確認
# hexdump -C -s `expr 5257560064 + 29` -n 128 __backup_vsphere-flat.vmdk
13960001d  64 61 74 61 73 74 6f 72  65 31 00 00 00 00 00 00  |datastore1......|
13960002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
# hexdump -C -s `expr 37435702784 + 29` -n 128 __backup_vsphere-flat.vmdk
8b7577e1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
8b7577e2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
8b7577e3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
8b7577e4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
8b7577e5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
8b7577e6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
8b7577e7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
8b7577e8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
8b7577e9d
# hexdump -C -s `expr 38707637760 + 29` -n 128 __backup_vsphere-flat.vmdk
90327ae1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
90327ae2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
90327ae3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
90327ae4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
90327ae5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
90327ae6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
90327ae7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
90327ae8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
90327ae9d
# hexdump -C -s `expr 39859326464 + 29` -n 128 __backup_vsphere-flat.vmdk
947cd0e1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
947cd0e2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
947cd0e3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
947cd0e4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
947cd0e5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
947cd0e6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
947cd0e7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
947cd0e8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
947cd0e9d
# hexdump -C -s `expr 40373243392 + 29` -n 128 __backup_vsphere-flat.vmdk
9666ece1d  00 73 71 75 61 73 68 66  73 00 68 73 71 73 00 73  |.squashfs.hsqs.s|
9666ece2d  68 73 71 00 25 75 2e 25  30 32 75 00 6e 73 73 00  |hsq.%u.%02u.nss.|
9666ece3d  53 50 42 35 00 78 65 6e  69 78 00 2b 55 44 00 44  |SPB5.xenix.+UD.D|
9666ece4d  55 2b 00 73 79 73 76 00  62 74 72 66 73 00 5f 42  |U+.sysv.btrfs._B|
9666ece5d  48 52 66 53 5f 4d 00 4c  41 42 45 4c 4f 4e 45 00  |HRfS_M.LABELONE.|
9666ece6d  4c 56 4d 32 5f 6d 65 6d  62 65 72 00 4c 56 4d 32  |LVM2_member.LVM2|
9666ece7d  20 30 30 31 00 4c 56 4d  31 5f 6d 65 6d 62 65 72  | 001.LVM1_member|
9666ece8d  00 48 4d 00 44 4d 5f 73  6e 61 70 73 68 6f 74 5f  |.HM.DM_snapshot_|
9666ece9d
```

最初に見つけた場所からデフォルトで作成された `datastore1` という label が確認できる。
他は何か分からないが、今は無視してよさそうだ。

ここからどうすべきか、正直まださっぱり分からないが、inode があるようだ。

```c:vmfs_inode.h
struct vmfs_inode_raw {
   struct vmfs_metadata_hdr_raw mdh;
   uint32_t id;
   uint32_t id2; /* seems to be VMFS_BLK_FD_ITEM(id) + 1 */
   uint32_t nlink;
   uint32_t type;
   uint32_t flags;
   uint64_t size;
   uint64_t blk_size;
   uint64_t blk_count;
   uint32_t mtime;
   uint32_t ctime;
   uint32_t atime;
   uint32_t uid;
   uint32_t gid;
   uint32_t mode;
   uint32_t zla;
   uint32_t tbz;
   uint32_t cow;
   u_char _unknown2[432];
   union {
      uint32_t blocks[VMFS_INODE_BLK_COUNT];
      uint32_t rdm_id;
      char content[VMFS_INODE_BLK_COUNT * sizeof(uint32_t)];
   };
} __attribute__((packed));
```

```c:vmfs_metadata.h
struct vmfs_metadata_hdr_raw {
   uint32_t magic; /* Magic number */
   uint64_t pos; /* Position in the volume */
   uint64_t hb_pos; /* Heartbeat position */
   uint64_t hb_seq; /* Heartbeat sequence */
   uint64_t obj_seq; /* Object sequence */
   uint32_t hb_lock; /* Heartbeat lock flag */
   uuid_t hb_uuid; /* UUID of locking server */
   uint64_t mtime;
   u_char pad1[0x1c0]; /* Padding/unknown */
} __attribute__((packed));
```

とにかく inode が見つかれば何とかなる気がする。

```c:vmfs_inode.h
#define VMFS_INODE_MAGIC  0x10c00001
```

inode もマジックナンバーがあるようなので、これを探してみる。
ついでに id と size を表示すれば手掛かりになりそうだ。

```python:searchinode.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, os, re, struct
target = './__backup_vsphere-flat.vmdk'

POS_OF_VMFSINFOMAGIC = 5257560064
MAGIC_OF_VMFSINODE = '\x01\x00\xc0\x10'

try:
    if sys.argv[1] == '-d':
        DEBUG = 1
except:
    pass

def D(mesg):
    try:
        if DEBUG:
            import inspect
            print "[DEBUG:%d] %s" % (inspect.currentframe().f_back.f_lineno, mesg)
    except NameError:
        pass

def convert_num(num):
    if len(str(int(num))) > 12:
        return '%.1fT' % (num/(1024.0**4))
    elif len(str(int(num))) > 9:
        return '%.1fG' % (num/(1024.0**3))
    elif len(str(int(num))) > 6:
        return '%.1fM' % (num/(1024.0**2))
    elif len(str(int(num))) > 3:
        return '%.1fk' % (num/(1024.0**1))
    else:
        return str(num)


with open(target , 'r') as f:
    def check(data):
        if MAGIC_OF_VMFSINODE in data:
            print 'Magic Number found. This is maybe inode !!'
            data = f.read(512)
            id, id2, size = struct.unpack('IIQ', data[0:8] + data[20:28])
            print 'id = %d, id2 = %d, filesize = %sB' % (id, id2, convert_num(size))
            return True
        else:
            return False

    try:
        fd = f.fileno()
    except Exception as e:
        print e
        raise Exception('cannot get file size')
    else:
        size = os.fstat(fd).st_size

    f.seek(POS_OF_VMFSINFOMAGIC)

    i=1
    data = f.read(512)
    while data:
        if check(data):
            print "position of above data: %d - %d" %  (f.tell() - 1024, f.tell() - 512)
        data = f.read(512)
        if f.tell() > size / 100.0 * i:
            D("searching... %d / %d" % (f.tell(), size))
            i+=1
```

```console:inode捜索
# python searchinode.py
Magic Number found. This is maybe inode !!
id = 4, id2 = 1, filesize = 1.4kB
position of above data: 5259788288 - 5259788800
Magic Number found. This is maybe inode !!
id = 4194308, id2 = 1, filesize = 256.0kB
position of above data: 5259790336 - 5259790848
Magic Number found. This is maybe inode !!
id = 8388612, id2 = 1, filesize = 254.7MB
position of above data: 5259792384 - 5259792896
Magic Number found. This is maybe inode !!
id = 12582916, id2 = 1, filesize = 256.0MB
position of above data: 5259794432 - 5259794944
Magic Number found. This is maybe inode !!
id = 16777220, id2 = 1, filesize = 250.6MB
position of above data: 5259796480 - 5259796992
Magic Number found. This is maybe inode !!
id = 20971524, id2 = 1, filesize = 4.0MB
position of above data: 5259798528 - 5259799040
Magic Number found. This is maybe inode !!
id = 25165828, id2 = 1, filesize = 1.1MB
position of above data: 5259800576 - 5259801088
Magic Number found. This is maybe inode !!
id = 32772, id2 = 1, filesize = 0B
position of above data: 5472124928 - 5472125440
Magic Number found. This is maybe inode !!
id = 4227076, id2 = 2, filesize = 0B
position of above data: 5472126976 - 5472127488
Magic Number found. This is maybe inode !!
id = 8421380, id2 = 3, filesize = 0B
position of above data: 5472129024 - 5472129536
Magic Number found. This is maybe inode !!
id = 12615684, id2 = 4, filesize = 0B
position of above data: 5472131072 - 5472131584
Magic Number found. This is maybe inode !!
id = 16809988, id2 = 5, filesize = 0B
position of above data: 5472133120 - 5472133632
Magic Number found. This is maybe inode !!
id = 21004292, id2 = 6, filesize = 0B
position of above data: 5472135168 - 5472135680
Magic Number found. This is maybe inode !!
id = 25198596, id2 = 7, filesize = 0B
position of above data: 5472137216 - 5472137728
Magic Number found. This is maybe inode !!
id = 29392900, id2 = 8, filesize = 0B
position of above data: 5472139264 - 5472139776
Magic Number found. This is maybe inode !!
id = 33587204, id2 = 9, filesize = 0B
position of above data: 5472141312 - 5472141824
Magic Number found. This is maybe inode !!
id = 37781508, id2 = 10, filesize = 0B
position of above data: 5472143360 - 5472143872
Magic Number found. This is maybe inode !!
id = 41975812, id2 = 11, filesize = 0B
position of above data: 5472145408 - 5472145920
Magic Number found. This is maybe inode !!
id = 46170116, id2 = 12, filesize = 0B
position of above data: 5472147456 - 5472147968
Magic Number found. This is maybe inode !!
id = 50364420, id2 = 14, filesize = 0B
position of above data: 5472149504 - 5472150016
Magic Number found. This is maybe inode !!
id = 54558724, id2 = 15, filesize = 0B
position of above data: 5472151552 - 5472152064
Magic Number found. This is maybe inode !!
id = 58753028, id2 = 17, filesize = 0B
position of above data: 5472153600 - 5472154112
Magic Number found. This is maybe inode !!
id = 62947332, id2 = 18, filesize = 0B
position of above data: 5472155648 - 5472156160
Magic Number found. This is maybe inode !!
id = 67141636, id2 = 20, filesize = 1.4kB
position of above data: 5472157696 - 5472158208
Magic Number found. This is maybe inode !!
id = 71335940, id2 = 21, filesize = 2.6kB
position of above data: 5472159744 - 5472160256
Magic Number found. This is maybe inode !!
id = 75530244, id2 = 22, filesize = 261B
position of above data: 5472161792 - 5472162304
Magic Number found. This is maybe inode !!
id = 79724548, id2 = 23, filesize = 0B
position of above data: 5472163840 - 5472164352
Magic Number found. This is maybe inode !!
id = 83918852, id2 = 24, filesize = 10.0GB
position of above data: 5472165888 - 5472166400
Magic Number found. This is maybe inode !!
id = 88113156, id2 = 25, filesize = 493B
position of above data: 5472167936 - 5472168448
Magic Number found. This is maybe inode !!
id = 92307460, id2 = 26, filesize = 0B
position of above data: 5472169984 - 5472170496
Magic Number found. This is maybe inode !!
id = 96501764, id2 = 27, filesize = 64.8kB
position of above data: 5472172032 - 5472172544
Magic Number found. This is maybe inode !!
id = 100696068, id2 = 28, filesize = 0B
position of above data: 5472174080 - 5472174592
Magic Number found. This is maybe inode !!
id = 104890372, id2 = 29, filesize = 8.5kB
position of above data: 5472176128 - 5472176640
Magic Number found. This is maybe inode !!
id = 109084676, id2 = 30, filesize = 0B
position of above data: 5472178176 - 5472178688
Magic Number found. This is maybe inode !!
id = 113278980, id2 = 31, filesize = 123.1kB
position of above data: 5472180224 - 5472180736
Magic Number found. This is maybe inode !!
id = 117473284, id2 = 33, filesize = 1.2kB
position of above data: 5472182272 - 5472182784
Magic Number found. This is maybe inode !!
id = 121667588, id2 = 34, filesize = 2.5kB
position of above data: 5472184320 - 5472184832
Magic Number found. This is maybe inode !!
id = 125861892, id2 = 35, filesize = 261B
position of above data: 5472186368 - 5472186880
Magic Number found. This is maybe inode !!
id = 130056196, id2 = 36, filesize = 0B
position of above data: 5472188416 - 5472188928
Magic Number found. This is maybe inode !!
id = 134250500, id2 = 37, filesize = 10.0GB
position of above data: 5472190464 - 5472190976
Magic Number found. This is maybe inode !!
id = 138444804, id2 = 38, filesize = 493B
position of above data: 5472192512 - 5472193024
Magic Number found. This is maybe inode !!
id = 142639108, id2 = 39, filesize = 0B
position of above data: 5472194560 - 5472195072
Magic Number found. This is maybe inode !!
id = 146833412, id2 = 40, filesize = 119.8kB
position of above data: 5472196608 - 5472197120
Magic Number found. This is maybe inode !!
id = 151027716, id2 = 41, filesize = 0B
position of above data: 5472198656 - 5472199168
Magic Number found. This is maybe inode !!
id = 155222020, id2 = 42, filesize = 8.5kB
position of above data: 5472200704 - 5472201216
^CTraceback (most recent call last):
  File "searchinode.py", line 66, in <module>
    data = f.read(512)
KeyboardInterrupt
```

それらしいものが見つかったので中断。
size を見ると 10.0GB を示すものがあるので、これが
`<vmname>-flat.vmdk` ファイルの inode と見てよさそうだ。
実際のブロックにどうマッピングされているのかは追いきれていないが、
この \<vmname\>-flat.vmdk の inode の 0x400 にあるのは
(vmfs-tools で言う所の) pointer block id で、その id の pointer block を読んで
色々処理した結果、実際のブロックが見つかる仕組みになっているようだ。

何にせよこのあたりを使ってマッピングしているようなので、
削除された後にそれらしい情報が残っていれば復旧できそうに思える。

いったんこの inode を ESXi shell から確認してみよう。
83918852 と 134250500 が `<vmname>-flat.vmdk` なら期待通りだ。

```console:inode表示
# pwd
/vmfs/volumes/datastore1
# ls -li */*vmdk
83918852 -rw-------    1 root     root        10737418240 Jun 13 13:31 test01/test01-flat.vmdk
88113156 -rw-------    1 root     root                493 Jun 13 13:03 test01/test01.vmdk
134250500 -rw-------    1 root     root        10737418240 Jun 13 13:57 test02/test02-flat.vmdk
138444804 -rw-------    1 root     root                493 Jun 13 13:38 test02/test02.vmdk
```


致命的な思い違いは今のところ無さそうだ。さっそく削除してデータを確認する。
vSphere Client で接続して `ディスクから削除` により削除する。
ゲスト ESXi が稼働中だと vmdk にアクセスできないようなので、シャットダウンして調べる。

```console:ディスクから削除した
# pwd
/vmfs/volumes/datastore1
# ls -a
.        ..       .fbb.sf  .fdc.sf  .pb2.sf  .pbc.sf  .sbc.sf  .vh.sf
```

削除前のデータと比較する。

```console:削除前のinode情報
## <vmname>-flat.vmdk を示す inode
# hexdump -C -s 5472165888 -n 2048 __backup_vsphere-flat.vmdk
1462aa000  01 00 c0 10 00 a0 ea 0c  00 00 00 00 00 00 3e 00  |..............>.|
1462aa010  00 00 00 00 1f 00 00 00  00 00 00 00 20 00 00 00  |............ ...|
1462aa020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
1462aa030  00 00 00 00 00 00 00 00  91 05 00 00 00 00 00 00  |................|
1462aa040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1462aa200  04 80 00 05 18 00 00 00  01 00 00 00 03 00 00 00  |................|
1462aa210  00 00 00 00 00 00 00 80  02 00 00 00 00 00 10 00  |................|
1462aa220  00 00 00 00 52 04 00 00  00 00 00 00 4d fd 9a 53  |....R.......M..S|
1462aa230  f6 f8 9a 53 4d fd 9a 53  00 00 00 00 00 00 00 00  |...SM..S........|
1462aa240  80 01 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
1462aa250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1462aa400  03 ed 02 10 03 ed 02 30  03 ed 02 40 03 ed 02 50  |.......0...@...P|
1462aa410  03 ed 02 60 03 ed 02 70  03 ed 02 80 03 ed 02 90  |...`...p........|
1462aa420  03 ed 02 a0 03 ed 02 20  00 00 00 00 00 00 00 00  |....... ........|
1462aa430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1462aa800
```

```console:削除後のinode情報
## vSphere のシェル上で直接 hexdump で skip できなかったので dd で skip してから渡す

## <vmname>-flat.vmdk を示す inode
# dd if=vsphere-flat.vmdk bs=512 skip=10687824 count=4
 2> /dev/null | hexdump -C
00000000  01 00 c0 10 00 a0 ea 0c  00 00 00 00 00 f0 3d 00  |..............=.|
00000010  00 00 00 00 11 00 00 00  00 00 00 00 24 00 00 00  |............$...|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  d3 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  04 80 00 05 18 00 00 00  00 00 00 00 03 00 00 00  |................|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 10 00  |................|
00000220  00 00 00 00 00 00 00 00  00 00 00 00 4d fd 9a 53  |............M..S|
00000230  e8 a9 9d 53 e1 7b 9d 53  00 00 00 00 00 00 00 00  |...S.{.S........|
00000240  80 01 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800
```

マッピングに使われると思われる場所が 0 で上書きされている。
・・・こうなると私の手には負えない、諦めよう。そしてもう寝よう。

小さいファイルは inode に直接収まっていたようなので
そちらも確認してみたが、このファイルはデータがそのまま見つかった。

```console:削除後のvmdkから小さいファイルのinodeを確認
## <vmname>.vmdk (descriptor の方) を示す inode
# dd if=vsphere-flat.vmdk bs=512 skip=10687828 count=4 2> /dev/null | hexdump -C
00000000  01 00 c0 10 00 a8 ea 0c  00 00 00 00 00 f0 3d 00  |..............=.|
00000010  00 00 00 00 11 00 00 00  00 00 00 00 2b 00 00 00  |............+...|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  d9 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 3e 00  00 00 00 00 1f 00 00 00  |......>.........|
00000050  00 00 00 00 ff e6 9a 53  cf 54 c0 67 9d 62 00 0c  |.......S.T.g.b..|
00000060  29 d5 43 a8 00 00 00 00  00 00 00 00 00 00 00 00  |).C.............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  04 80 40 05 19 00 00 00  00 00 00 00 03 00 00 00  |..@.............|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 20 00 00  |............. ..|
00000220  00 00 00 00 00 00 00 00  00 00 00 00 a9 f6 9a 53  |...............S|
00000230  8e f5 9a 53 a9 f6 9a 53  00 00 00 00 00 00 00 00  |...S...S........|
00000240  80 01 00 00 d1 10 00 00  00 00 00 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  23 20 44 69 73 6b 20 44  65 73 63 72 69 70 74 6f  |# Disk Descripto|
00000410  72 46 69 6c 65 0a 76 65  72 73 69 6f 6e 3d 31 0a  |rFile.version=1.|
00000420  65 6e 63 6f 64 69 6e 67  3d 22 55 54 46 2d 38 22  |encoding="UTF-8"|
00000430  0a 43 49 44 3d 35 62 61  64 65 30 63 31 0a 70 61  |.CID=5bade0c1.pa|
00000440  72 65 6e 74 43 49 44 3d  66 66 66 66 66 66 66 66  |rentCID=ffffffff|
00000450  0a 69 73 4e 61 74 69 76  65 53 6e 61 70 73 68 6f  |.isNativeSnapsho|
00000460  74 3d 22 6e 6f 22 0a 63  72 65 61 74 65 54 79 70  |t="no".createTyp|
00000470  65 3d 22 76 6d 66 73 22  0a 0a 23 20 45 78 74 65  |e="vmfs"..# Exte|
00000480  6e 74 20 64 65 73 63 72  69 70 74 69 6f 6e 0a 52  |nt description.R|
00000490  57 20 32 30 39 37 31 35  32 30 20 56 4d 46 53 20  |W 20971520 VMFS |
000004a0  22 74 65 73 74 30 31 2d  66 6c 61 74 2e 76 6d 64  |"test01-flat.vmd|
000004b0  6b 22 0a 0a 23 20 54 68  65 20 44 69 73 6b 20 44  |k"..# The Disk D|
000004c0  61 74 61 20 42 61 73 65  20 0a 23 44 44 42 0a 0a  |ata Base .#DDB..|
000004d0  64 64 62 2e 61 64 61 70  74 65 72 54 79 70 65 20  |ddb.adapterType |
000004e0  3d 20 22 6c 73 69 6c 6f  67 69 63 22 0a 64 64 62  |= "lsilogic".ddb|
000004f0  2e 74 68 69 6e 50 72 6f  76 69 73 69 6f 6e 65 64  |.thinProvisioned|
00000500  20 3d 20 22 31 22 0a 64  64 62 2e 67 65 6f 6d 65  | = "1".ddb.geome|
00000510  74 72 79 2e 73 65 63 74  6f 72 73 20 3d 20 22 36  |try.sectors = "6|
00000520  33 22 0a 64 64 62 2e 67  65 6f 6d 65 74 72 79 2e  |3".ddb.geometry.|
00000530  68 65 61 64 73 20 3d 20  22 32 35 35 22 0a 64 64  |heads = "255".dd|
00000540  62 2e 67 65 6f 6d 65 74  72 79 2e 63 79 6c 69 6e  |b.geometry.cylin|
00000550  64 65 72 73 20 3d 20 22  31 33 30 35 22 0a 64 64  |ders = "1305".dd|
00000560  62 2e 75 75 69 64 20 3d  20 22 36 30 20 30 30 20  |b.uuid = "60 00 |
00000570  43 32 20 39 64 20 33 35  20 63 37 20 30 39 20 63  |C2 9d 35 c7 09 c|
00000580  66 2d 63 66 20 64 34 20  30 65 20 61 33 20 32 37  |f-cf d4 0e a3 27|
00000590  20 36 35 20 39 38 20 62  62 22 0a 64 64 62 2e 6c  | 65 98 bb".ddb.l|
000005a0  6f 6e 67 43 6f 6e 74 65  6e 74 49 44 20 3d 20 22  |ongContentID = "|
000005b0  37 64 35 30 66 36 36 32  33 30 37 32 30 38 37 61  |7d50f6623072087a|
000005c0  37 31 64 36 61 63 63 39  35 62 61 64 65 30 63 31  |71d6acc95bade0c1|
000005d0  22 0a 64 64 62 2e 76 69  72 74 75 61 6c 48 57 56  |".ddb.virtualHWV|
000005e0  65 72 73 69 6f 6e 20 3d  20 22 38 22 0a 00 00 00  |ersion = "8"....|
000005f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800
```

削除後のゲスト ESXi の vmdk をコピーして調査
inode のマジックナンバーは見つけられるので、サイズは 0 byte になっているものの
小さいファイルは片っ端から探せば復旧できそうだ。

```console:ディスクから削除後のinode捜索
# python searchinode.py
Magic Number found. This is maybe inode !!
id = 4, id2 = 1, filesize = 1.1kB
position of above data: 5259788288 - 5259788800
Magic Number found. This is maybe inode !!
id = 4194308, id2 = 1, filesize = 256.0kB
position of above data: 5259790336 - 5259790848
Magic Number found. This is maybe inode !!
id = 8388612, id2 = 1, filesize = 254.7MB
position of above data: 5259792384 - 5259792896
Magic Number found. This is maybe inode !!
id = 12582916, id2 = 1, filesize = 256.0MB
position of above data: 5259794432 - 5259794944
Magic Number found. This is maybe inode !!
id = 16777220, id2 = 1, filesize = 250.6MB
position of above data: 5259796480 - 5259796992
Magic Number found. This is maybe inode !!
id = 20971524, id2 = 1, filesize = 4.0MB
position of above data: 5259798528 - 5259799040
Magic Number found. This is maybe inode !!
id = 25165828, id2 = 1, filesize = 1.1MB
position of above data: 5259800576 - 5259801088
Magic Number found. This is maybe inode !!
id = 32772, id2 = 1, filesize = 0B
position of above data: 5472124928 - 5472125440
Magic Number found. This is maybe inode !!
id = 4227076, id2 = 2, filesize = 0B
position of above data: 5472126976 - 5472127488
Magic Number found. This is maybe inode !!
id = 8421380, id2 = 3, filesize = 0B
position of above data: 5472129024 - 5472129536
Magic Number found. This is maybe inode !!
id = 12615684, id2 = 4, filesize = 0B
position of above data: 5472131072 - 5472131584
Magic Number found. This is maybe inode !!
id = 16809988, id2 = 5, filesize = 0B
position of above data: 5472133120 - 5472133632
Magic Number found. This is maybe inode !!
id = 21004292, id2 = 6, filesize = 0B
position of above data: 5472135168 - 5472135680
Magic Number found. This is maybe inode !!
id = 25198596, id2 = 7, filesize = 0B
position of above data: 5472137216 - 5472137728
Magic Number found. This is maybe inode !!
id = 29392900, id2 = 8, filesize = 0B
position of above data: 5472139264 - 5472139776
Magic Number found. This is maybe inode !!
id = 33587204, id2 = 9, filesize = 0B
position of above data: 5472141312 - 5472141824
Magic Number found. This is maybe inode !!
id = 37781508, id2 = 10, filesize = 0B
position of above data: 5472143360 - 5472143872
Magic Number found. This is maybe inode !!
id = 41975812, id2 = 11, filesize = 0B
position of above data: 5472145408 - 5472145920
Magic Number found. This is maybe inode !!
id = 46170116, id2 = 12, filesize = 0B
position of above data: 5472147456 - 5472147968
Magic Number found. This is maybe inode !!
id = 50364420, id2 = 14, filesize = 0B
position of above data: 5472149504 - 5472150016
Magic Number found. This is maybe inode !!
id = 54558724, id2 = 15, filesize = 0B
position of above data: 5472151552 - 5472152064
Magic Number found. This is maybe inode !!
id = 58753028, id2 = 17, filesize = 0B
position of above data: 5472153600 - 5472154112
Magic Number found. This is maybe inode !!
id = 62947332, id2 = 18, filesize = 0B
position of above data: 5472155648 - 5472156160
Magic Number found. This is maybe inode !!
id = 67141636, id2 = 20, filesize = 0B
position of above data: 5472157696 - 5472158208
Magic Number found. This is maybe inode !!
id = 71335940, id2 = 21, filesize = 0B
position of above data: 5472159744 - 5472160256
Magic Number found. This is maybe inode !!
id = 75530244, id2 = 22, filesize = 0B
position of above data: 5472161792 - 5472162304
Magic Number found. This is maybe inode !!
id = 79724548, id2 = 23, filesize = 0B
position of above data: 5472163840 - 5472164352
Magic Number found. This is maybe inode !!
id = 83918852, id2 = 24, filesize = 0B
position of above data: 5472165888 - 5472166400
Magic Number found. This is maybe inode !!
id = 88113156, id2 = 25, filesize = 0B
position of above data: 5472167936 - 5472168448
Magic Number found. This is maybe inode !!
id = 92307460, id2 = 26, filesize = 0B
position of above data: 5472169984 - 5472170496
Magic Number found. This is maybe inode !!
id = 96501764, id2 = 27, filesize = 0B
position of above data: 5472172032 - 5472172544
Magic Number found. This is maybe inode !!
id = 100696068, id2 = 28, filesize = 0B
position of above data: 5472174080 - 5472174592
Magic Number found. This is maybe inode !!
id = 104890372, id2 = 29, filesize = 0B
position of above data: 5472176128 - 5472176640
Magic Number found. This is maybe inode !!
id = 109084676, id2 = 30, filesize = 0B
position of above data: 5472178176 - 5472178688
Magic Number found. This is maybe inode !!
id = 113278980, id2 = 31, filesize = 0B
position of above data: 5472180224 - 5472180736
Magic Number found. This is maybe inode !!
id = 117473284, id2 = 33, filesize = 0B
position of above data: 5472182272 - 5472182784
Magic Number found. This is maybe inode !!
id = 121667588, id2 = 34, filesize = 0B
position of above data: 5472184320 - 5472184832
Magic Number found. This is maybe inode !!
id = 125861892, id2 = 35, filesize = 0B
position of above data: 5472186368 - 5472186880
Magic Number found. This is maybe inode !!
id = 130056196, id2 = 36, filesize = 0B
position of above data: 5472188416 - 5472188928
Magic Number found. This is maybe inode !!
id = 134250500, id2 = 37, filesize = 0B
position of above data: 5472190464 - 5472190976
Magic Number found. This is maybe inode !!
id = 138444804, id2 = 38, filesize = 0B
position of above data: 5472192512 - 5472193024
Magic Number found. This is maybe inode !!
id = 142639108, id2 = 39, filesize = 0B
position of above data: 5472194560 - 5472195072
Magic Number found. This is maybe inode !!
id = 146833412, id2 = 40, filesize = 0B
position of above data: 5472196608 - 5472197120
Magic Number found. This is maybe inode !!
id = 151027716, id2 = 41, filesize = 0B
position of above data: 5472198656 - 5472199168
Magic Number found. This is maybe inode !!
id = 155222020, id2 = 42, filesize = 0B
position of above data: 5472200704 - 5472201216
Magic Number found. This is maybe inode !!
id = 159416324, id2 = 43, filesize = 0B
position of above data: 5472202752 - 5472203264
Magic Number found. This is maybe inode !!
id = 163610628, id2 = 44, filesize = 0B
position of above data: 5472204800 - 5472205312
Magic Number found. This is maybe inode !!
id = 167804932, id2 = 45, filesize = 0B
position of above data: 5472206848 - 5472207360
Magic Number found. This is maybe inode !!
id = 171999236, id2 = 46, filesize = 0B
position of above data: 5472208896 - 5472209408
Magic Number found. This is maybe inode !!
id = 176193540, id2 = 47, filesize = 0B
position of above data: 5472210944 - 5472211456
Magic Number found. This is maybe inode !!
id = 180387844, id2 = 49, filesize = 0B
position of above data: 5472212992 - 5472213504
Magic Number found. This is maybe inode !!
id = 184582148, id2 = 50, filesize = 0B
position of above data: 5472215040 - 5472215552
Magic Number found. This is maybe inode !!
id = 188776452, id2 = 51, filesize = 0B
position of above data: 5472217088 - 5472217600
Magic Number found. This is maybe inode !!
id = 192970756, id2 = 52, filesize = 0B
position of above data: 5472219136 - 5472219648
Magic Number found. This is maybe inode !!
id = 197165060, id2 = 53, filesize = 0B
position of above data: 5472221184 - 5472221696
Magic Number found. This is maybe inode !!
id = 201359364, id2 = 54, filesize = 0B
position of above data: 5472223232 - 5472223744
^CTraceback (most recent call last):
  File "searchinode.py", line 64, in <module>
    data = f.read(512)
KeyboardInterrupt
```


# まとめ

* データ自体は（削除時には）多分上書きされていない
* inode の情報は部分的に抹消される
* サイズが小さくて (1kB 以下?) inode 内にデータが収まるようなファイルは取り出せる<br>
(.vmxf や virtual disk descriptor file の方の .vmdk くらいか)
* `<vmname>-flat.vmdk` の方の VMDK ファイルは inode の対応するブロックの情報が消されているのでほぼ無理
* MBR くらいだったら inode に関係なく直接探し出して復旧できる
* 大事なデータはバックアップを取りましょう


# 参考資料

VMFS 関連

* http://glandium.org/projects/vmfs-tools/
* http://buildvirtual.net/using-vmkfstools-to-manage-vmfs-datastores/

MBR 関連

* http://ja.wikipedia.org/wiki/マスターブートレコード
* http://thestarman.narod.ru/asm/mbr/index.html

GPT 関連

* http://ja.wikipedia.org/wiki/GUIDパーティションテーブル