Repository: openvstorage/framework

Issue:
ovs setup crashed when disk is broken
When a disk is broken on an ovs node. The ovs setup crashed with error:

```
 if disk_name not in disk_names and configuration[disk_name]['state'] not in ['MISSING']:
KeyError: 'state'", "msg": "MODULE FAILURE", "parsed": false}
```

This error doesn't tell us what is wrong and which disk has this issue.

I think the setup should not crash by broken disks.
ovs setup needs only the root partitions to install all the packages, not the disks where we will install the roles.
We will see 1 missing disk on the ovs node when we want to add the roles.
We can rescan the disks when we replaced the disk.

Full Error:

```
2016-06-29 11:00:26 11300 +0200 - cmp01 - 2747/140005685561152 - lib/setup - 138 - INFO - \r\n\r\n2016-06-29 11:00:26 11300 +0200 - cmp01 - 2747/140005685561152 - lib/setup - 139 - ERROR - An unexpected error occurred:\r\nTraceback (most recent call last):\r\n  File \"/opt/OpenvStorage/ovs/lib/setup.py\", line 456, in setup_node\r\n    rdma=rdma)\r\n  File \"/opt/OpenvStorage/ovs/lib/setup.py\", line 908, in _setup_first_node\r\n    storagerouter = SetupController._finalize_setup(target_client, node_name, 'MASTER', hypervisor_info, unique_id)\r\n  File \"/opt/OpenvStorage/ovs/lib/setup.py\", line 1681, in _finalize_setup\r\n    DiskController.sync_with_reality(storagerouter.guid)\r\n  File \"/usr/local/lib/python2.7/dist-packages/celery/local.py\", line 188, in __call__\r\n    return self._get_current_object()(*a, **kw)\r\n  File \"/usr/local/lib/python2.7/dist-packages/celery/app/task.py\", line 420, in __call__\r\n    return self.run(*args, **kwargs)\r\n  File \"/opt/OpenvStorage/ovs/lib/helpers/decorators.py\", line 297, in new_function\r\n    output = function(*args, **kwargs)\r\n  File \"/opt/OpenvStorage/ovs/lib/disk.py\", line 214, in sync_with_reality\r\n    if disk_name not in disk_names and configuration[disk_name]['state'] not in ['MISSING']:\r\nKeyError: 'state'\r\n2016-06-29 11:00:26 11400 +0200 - cmp01 - 2747/140005685561152 - lib/setup - 140 - ERROR - 'state'\r\nTraceback (most recent call last):\r\n  File \"/opt/OpenvStorage/ovs/lib/setup.py\", line 456, in setup_node\r\n    rdma=rdma)\r\n  File \"/opt/OpenvStorage/ovs/lib/setup.py\", line 908, in _setup_first_node\r\n    storagerouter = SetupController._finalize_setup(target_client, node_name, 'MASTER', hypervisor_info, unique_id)\r\n  File \"/opt/OpenvStorage/ovs/lib/setup.py\", line 1681, in _finalize_setup\r\n    DiskController.sync_with_reality(storagerouter.guid)\r\n  File \"/usr/local/lib/python2.7/dist-packages/celery/local.py\", line 188, in __call__\r\n    return self._get_current_object()(*a, **kw)\r\n  File \"/usr/local/lib/python2.7/dist-packages/celery/app/task.py\", line 420, in __call__\r\n    return self.run(*args, **kwargs)\r\n  File \"/opt/OpenvStorage/ovs/lib/helpers/decorators.py\", line 297, in new_function\r\n    output = function(*args, **kwargs)\r\n  File \"/opt/OpenvStorage/ovs/lib/disk.py\", line 214, in sync_with_reality\r\n    if disk_name not in disk_names and configuration[disk_name]['state'] not in ['MISSING']:\r\nKeyError: 'state'\r\n", "msg": "MODULE FAILURE", "parsed": false}
```

configuration output:

```
 In [71]: configuration
Out[71]:
{u'sda': {'partitions': {u'1': {'filesystem': u'ext4',
    'offset': 1048576,
    'path': u'/dev/disk/by-id/ata-ZNSM20A4120M4_ZN1144A00002333-part1',
    'size': 120032591872,
    'state': 'OK'}}},
 u'sdb': {'is_ssd': False,
  'model': u'WDC_WD10EALS-002BA0',
  'name': u'sdb',
  'partitions': {u'1': {'filesystem': u'vfat',
    'inode': 2065L,
    'mountpoint': u'/boot',
    'offset': 1048576,
    'path': u'/dev/disk/by-id/wwn-0x50014ee204dd32c7-part1',
    'size': 1073741824,
    'state': 'OK'},
   u'2': {'filesystem': u'swap',
    'offset': 1074790400,
    'path': u'/dev/disk/by-id/wwn-0x50014ee204dd32c7-part2',
    'size': 32212254720,
    'state': 'OK'},
   u'3': {'filesystem': u'ext4',
    'inode': 2067L,
    'mountpoint': u'/',
    'offset': 33287045120,
    'path': u'/dev/disk/by-id/wwn-0x50014ee204dd32c7-part3',
    'size': 107374182400,
    'state': 'OK'}},
  'path': u'/dev/disk/by-id/wwn-0x50014ee204dd32c7',
  'size': 1000204886016,
  'state': 'OK',
  'vendor': None},
 u'sdc': {'is_ssd': True,
  'model': u'DENRSTE251M45-0100',
  'name': u'sdc',
  'partitions': {},
  'path': u'/dev/disk/by-id/wwn-0x5e83a970100089ff',
  'size': 100030242816,
  'state': 'OK',
  'vendor': None},
 u'sdd': {'is_ssd': True,
  'model': u'DENRSTE251M45-0100',
  'name': u'sdd',
  'partitions': {},
  'path': u'/dev/disk/by-id/wwn-0x5e83a970100067d6',
  'size': 100030242816,
  'state': 'OK',
  'vendor': None}}
In [84]: mount_mapping
Out[84]: {u'0bb96cc0-4124-47ba-b52e-49cad0ec277a': u'/', u'F89A-A601': u'/boot'}
```



Discussion:

Resolving patch:
diff --git a/ovs/lib/disk.py b/ovs/lib/disk.py
--- a/ovs/lib/disk.py
+++ b/ovs/lib/disk.py
@@ -211,7 +211,7 @@ def sync_with_reality(storagerouter_guid=None):
                         if partition_id not in partitions:
                             DiskController._create_partition(partition_id, partition_info[partition_id], disk)
             for disk_name in configuration:
-                if disk_name not in disk_names and configuration[disk_name]['state'] not in ['MISSING']:
+                if disk_name not in disk_names and configuration[disk_name].get('state', 'MISSING') not in ['MISSING']:
                     disk = Disk()
                     disk.storagerouter = storagerouter
                     disk.name = disk_name
diff --git a/ovs/lib/setup.py b/ovs/lib/setup.py
--- a/ovs/lib/setup.py
+++ b/ovs/lib/setup.py
@@ -1557,7 +1557,10 @@ def _finalize_setup(client, node_name, node_type, unique_id):
         storagerouter.save()
 
         StorageRouterController.set_rdma_capability(storagerouter.guid)
-        DiskController.sync_with_reality(storagerouter.guid)
+        try:
+            DiskController.sync_with_reality(storagerouter.guid)
+        except Exception as ex:
+            SetupController._logger.exception('Error syncing disks: {0}'.format(ex))
 
         return storagerouter
 


Test patch: