OpenStackでのLXCのrootfsについて

OpenStackでLXCをやってみようと、devstackを設定してみた。
http://qiita.com/makisyu/items/a76174234c4106aa7fe5

が、いざコンテナ内でdfするとこんな感じになる。

```text
$ sudo virsh --connect lxc:/// lxc-enter-namespace instance-00000001 --noseclabel /bin/df -h
Filesystem                Size      Used Available Use% Mounted on
devfs                    64.0K         0     64.0K   0% /dev
/dev/nbd7                22.2M     17.0M      4.1M  81% /
tmpfs                    64.0K         0     64.0K   0% /sys/fs/cgroup
tmpfs                     7.8G         0      7.8G   0% /dev/shm
tmpfs                   200.0K     64.0K    136.0K  32% /run
```

ん？何か思ってたのと違うような……。

# ってか、NBD?

```text
$ ps -eFH | grep qemu-nbd
ubuntu   20148 20778  0  2615   936   0 08:25 pts/25   00:00:00           grep --color=auto qemu-nbd
root       402     1  0 200128 19432  2 02:00 ?        00:00:00   /usr/bin/qemu-nbd -c /dev/nbd7 /opt/stack/data/nova/instances/072123b6-f342-4944-990b-02e0b9147978/disk
```

**とつぜんのNBD!**

## これもしや`disk`って

```text
$ qemu-img info /opt/stack/data/nova/instances/072123b6-f342-4944-990b-02e0b9147978/disk
image: /opt/stack/data/nova/instances/072123b6-f342-4944-990b-02e0b9147978/disk
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 1.7M
cluster_size: 65536
backing file: /opt/stack/data/nova/instances/_base/733137ff1173257fbd2dc691611bc44398018c66
Format specific information:
    compat: 1.1
    lazy refcounts: false
```

節子、それrootfsやない！QCOW2や！

なんということでしょう。
コンテナのrootfsは、QCOW2を一旦qemu-nbdでNBDとしてホストにデバイスとして見せて、ホストがNBDをrootfsにマウントしなおすことで実現されている。

どうりでドメインのXMLにないNBDが急に登場するわけだ。

```xml
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/opt/stack/data/nova/instances/072123b6-f342-4944-990b-02e0b9147978/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='fa:16:3e:41:10:67'/>
      <source bridge='qbr37347393-fe'/>
      <target dev='tap37347393-fe'/>
    </interface>
    <console type='pty' tty='/dev/pts/26'>
      <source path='/dev/pts/26'/>
      <target type='lxc' port='0'/>
      <alias name='console0'/>
    </console>
  </devices>
```

いや、「コンテナでGlanceのイメージってどうやるんだろうなー」とは思ってたけど、こういうことだったとは……。
ま、まぁ**これならGlanceで管理できるし、ディスクのサイズも制限できるし、スナップショットも取れるし、マイグレーションもできそう**なわけだが……。

いや、僕が欲しいのはホストのデバイスなんや！結局QCOW2じゃ意味ないんや！それならKVMでええんや！

ちなみにこのQCOW2のbacking fileは、

```text
$ qemu-img info  /opt/stack/data/nova/instances/_base/733137ff1173257fbd2dc691611bc44398018c66
image: /opt/stack/data/nova/instances/_base/733137ff1173257fbd2dc691611bc44398018c66
file format: raw
virtual size: 24M (25165824 bytes)
disk size: 24M
```

raw! devstackがダウンロードしてきたのは、Cirrosのrawイメージだったようだ。

# Novaにきいてみよう

ソースコードを読むと、

```text
LibvirtDriver.spawn
  -> LibvirtDriver._create_domain_and_network
    -> libvirtDriver._lxc_disk_handler
      -> LibvirtDriver._create_domain_setup_lxc
```

ここまで降りてきて、

```python:nova/virt/disk/api.py
def setup_container(image, container_dir):
    """Setup the LXC container.
    :param image: instance of nova.virt.image.model.Image
    :param container_dir: directory to mount the image at
    It will mount the loopback image to the container directory in order
    to create the root filesystem for the container.
    Returns path of image device which is mounted to the container directory.
    """
    img = _DiskImage(image=image, mount_dir=container_dir)
    dev = img.mount()
    if dev is None:
        LOG.error(_LE("Failed to mount container filesystem '%(image)s' "
                    "on '%(target)s': %(errors)s"),
                  {"image": img, "target": container_dir,
                   "errors": img.errors})
        raise exception.NovaException(img.errors)

    return dev
```

どうも、やはりQCOW2ディスクイメージをNBD経由でrootfsにマウントしているようだ。

## なんとかならんのか

```python:nova/virt/libvirt/driver.py
    def _create_domain_setup_lxc(self, instance, image_meta,
                                 block_device_info, disk_info):
        inst_path = libvirt_utils.get_instance_path(instance)
        disk_info = disk_info or {}
        disk_mapping = disk_info.get('mapping', {})

        if self._is_booted_from_volume(instance, disk_mapping):
            block_device_mapping = driver.block_device_info_get_mapping(
                                                            block_device_info)
            root_disk = block_device.get_root_bdm(block_device_mapping)
            disk_info = blockinfo.get_info_from_bdm(
                instance, CONF.libvirt.virt_type, image_meta, root_disk)
            self._connect_volume(root_disk['connection_info'], disk_info)
            disk_path = root_disk['connection_info']['data']['device_path']

            # NOTE(apmelton) - Even though the instance is being booted from a
            # cinder volume, it is still presented as a local block device.
            # LocalBlockImage is used here to indicate that the instance's
            # disk is backed by a local block device.
            image_model = imgmodel.LocalBlockImage(disk_path)
        else:
            image = self.image_backend.image(instance, 'disk')
            image_model = image.get_model(self._conn)

        container_dir = os.path.join(inst_path, 'rootfs')
        fileutils.ensure_tree(container_dir)
        rootfs_dev = disk.setup_container(image_model,
                                          container_dir=container_dir)
...<snip>...
```

# つまりcinder volume bootすればいいのでは？

Cinderのボリュームをcirrosのイメージから作成して、そこからLXCを起動する。

と、エラー(・´з`・)

```text
[instance: f7a87019-c8cd-458c-9ae1-19badac1e08d] Instance failed to spawn
 Traceback (most recent call last):
   File "/opt/stack/nova/nova/compute/manager.py", line 2155, in _build_resources
     yield resources
   File "/opt/stack/nova/nova/compute/manager.py", line 2009, in _build_and_run_instance
     block_device_info=block_device_info)
   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
     block_device_info=block_device_info)
   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4514, in _create_domain_and_network
     block_device_info, disk_info):
   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
     return self.gen.next()
   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4423, in _lxc_disk_handler
     block_device_info, disk_info)
   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4375, in _create_domain_setup_lxc
     container_dir=container_dir)
   File "/opt/stack/nova/nova/virt/disk/api.py", line 446, in setup_container
     dev = img.mount()
   File "/opt/stack/nova/nova/virt/disk/api.py", line 355, in mount
     self.partition)
   File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 78, in instance_for_format
     image.__class__.__name__)
 UnsupportedImageModel: LocalBlockImage
```

Unsupported!そんなばかな！

```python:nova/virt/disk/mount/api.py
    def instance_for_format(image, mountdir, partition):
        """Get a Mount instance for the image type
        :param image: instance of nova.virt.image.model.Image
        :param mountdir: path to mount the image at
        :param partition: partition number to mount
        """
        LOG.debug("Instance for format image=%(image)s "
                  "mountdir=%(mountdir)s partition=%(partition)s",
                  {'image': image, 'mountdir': mountdir,
                   'partition': partition})

        if isinstance(image, imgmodel.LocalFileImage):
            if image.format == imgmodel.FORMAT_RAW:
                LOG.debug("Using LoopMount")
                return importutils.import_object(
                    "nova.virt.disk.mount.loop.LoopMount",
                    image, mountdir, partition)
            else:
                LOG.debug("Using NbdMount")
                return importutils.import_object(
                    "nova.virt.disk.mount.nbd.NbdMount",
                    image, mountdir, partition)
        else:
            # TODO(berrange) we could mount images of
            # type LocalBlockImage directly without
            # involving loop or nbd devices
            #
            # We could also mount RBDImage directly
            # using kernel RBD block dev support.
            #
            # This is left as an enhancement for future
            # motivated developers todo, since raising
            # an exception is on par with what this
            # code did historically
            raise exception.UnsupportedImageModel(
                image.__class__.__name__)
```

(｀・ω・´)　＜ 息子よ、あとは任せた！

# パッチがあります

https://review.openstack.org/#/c/215311/