Unfreezing dependencies
HIP-669 do_scsi_cdb uses gipc where available instead of stdlib's multiprocessing

================================================================================

Host Power Tools / HIP-669
infinihost volume list indicates "IOError: handle out of range in select()" when 50 volumes are exposed

Type:              Bug                               Status:        Open                       Assignee: Guy Rozendorn
Priority:          Minor                             Resolution:    Unresolved                 Reporter: Koby Mordechai
Affects Version/s: 1.0.0                             Fix Version/s:
Components: infinihost volume list                                                             Created: 2013-07-29 17:44
Labels:                                                                                        Updated: 2013-07-29 17:44

Issue Links:

Sub-Tasks:

Description:
io016 is a CentOS 6.4 machine with infinihost 1.0.post1.g8ce0119
1. host is connected to ibox003
2. 50 volumes (100G each) are exposed to the host
3. running infinihost volume list indicates "OSError: Too many open files" (HIP-668 was opened)
4. After raising the number of open files limit from 1024  to 8192 <infinishell volume list> indicates a different new error message "OError: handle out of range in select()" :
{noformat}
[root@io016 log17]# ulimit -n 8192
[root@io016 log17]# infinihost volume list
Querying devices... /Process _GProcess-1:
Traceback (most recent call last):
  File "/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  Local variables:
    'util': <module 'multiprocessing.util' from '/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/util.pyc'>
    'self': <_GProcess(_GProcess-1, started)>
    'traceback': <module 'traceback' from '/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/traceback.pyc'>
    'exitcode': 1

  File "/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  Local variables:
    'self': <_GProcess(_GProcess-1, started)>

  File "/opt/infinidat/host-power-tools/eggs/gipc-0.3.2-py2.7.egg/gipc/gipc.py", line 278, in _child
    target(*args, **kwargs)
  Local variables:
    'target': <function main at 0x1ed2140>
    'hub': <Hub at 0x1ee8730 destroyed>
    'h': <Hub at 0x1ee8910 epoll default pending=0 ref=0 fileno=4>
    'args': ([2, 1],)
    'childhandles': []
    'kwargs': {}

  File "/opt/infinidat/host-power-tools/eggs/infi.traceback-0.3.5-py2.7.egg/infi/traceback/__init__.py", line 50, in callee
    return func(*args, **kwargs)
  Local variables:
    'args': ([2, 1],)
    'func': <function main at 0x1ed20c8>
    'kwargs': {}
                                                                                                                                                                                                       [679/97612]
  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': ([2, 1],)
    'item': [2, 1]
    'func': <function main at 0x1ed1ed8>
    'kwargs': {}
    'format_args': '[2, 1]'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/__init__.py", line 19, in main
    rescan_scsi_hosts(host_numbers)
  Local variables:
    'host_numbers': [2, 1]
    'reinit': <function reinit at 0x1d749b0>
    'err': IOError('handle out of range in select()',)

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': ([2, 1],)
    'item': [2, 1]
    'func': <function rescan_scsi_hosts at 0x1ed1de8>
    'kwargs': {}
    'format_args': '[2, 1]'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/logic.py", line 96, in rescan_scsi_hosts
    rescan_scsi_host(host_number)
  Local variables:
    'host_number': 1
    'host_numbers': [2, 1]
  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator                                [645/97612]
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': (1,)
    'item': 1
    'func': <function rescan_scsi_host at 0x1ed1cf8>
    'kwargs': {}
    'format_args': '1'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/logic.py", line 91, in rescan_scsi_host
    target_scan(host, channel, target)
  Local variables:
    'host': 1
    'target': 1
    'channel': 0
    'targets': set([0, 1, 2])

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': (1, 0, 1)
    'item': 1
    'func': <function target_scan at 0x1ed1c08>
    'kwargs': {}
    'format_args': '1, 0, 1'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/logic.py", line 84, in target_scan
    lun_scan(host, channel, target, lun)
  Local variables:
    'missing_luns': set([])
    'target': 1
    'existing_luns': set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48
, 49, 50])
    'host': 1
    'actual_luns': set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
49, 50])
    'channel': 0
    'expected_luns': set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48
, 49, 50])
    'unmapped_luns': set([])
    'lun': 42                                                                                                                                                                                          [603/97612]

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': (1, 0, 1, 42)
    'item': 42
    'func': <function lun_scan at 0x1ed1938>
    'kwargs': {}
    'format_args': '1, 0, 1, 42'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/logic.py", line 43, in lun_scan
    if not is_scsi_generic_device_online(sg_device):
  Local variables:
    'host': 1
    'channel': 0
    'lun': 42
    'sg_device': 'sg249'
    'target': 1

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': ('sg249',)
    'item': 'sg249'
    'func': <function is_scsi_generic_device_online at 0x1ed1848>
    'kwargs': {}
    'format_args': 'sg249'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/logic.py", line 28, in is_scsi_generic_device_online
    standard_inquiry = do_standard_inquiry(sg_device)
  Local variables:
    'sg_device': 'sg249'
                                                                                                                                                                                                       [568/97612]
  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': ('sg249',)
    'item': 'sg249'
    'func': <function do_standard_inquiry at 0x1eca668>
    'kwargs': {}
    'format_args': 'sg249'

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/scsi.py", line 125, in do_standard_inquiry
    return do_scsi_cdb(sg_device, cdb)
  Local variables:
    'StandardInquiryCommand': <class 'infi.asi.cdb.inquiry.standard.StandardInquiryCommand'>
    'sg_device': 'sg249'
    'cdb': StandardInquiryCommand(opcode=OperationCode(command_code=18, group_code=0) (const), evpd=0, <7 bits padding>, page_code=0, allocation_length=96, control=Control(<2 bits padding>, naca=0, <3 bits padd
ing>, vendor_specific=0))

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/utils.py", line 27, in decorator
    result = func(*args, **kwargs)
  Local variables:
    'format_kwargs': ''
    'args': ('sg249', StandardInquiryCommand(opcode=OperationCode(command_code=18, group_code=0) (const), evpd=0, <7 bits padding>, page_code=0, allocation_length=96, control=Control(<2 bits padding>, naca=0, <
3 bits padding>, vendor_specific=0)))
    'item': StandardInquiryCommand(opcode=OperationCode(command_code=18, group_code=0) (const), evpd=0, <7 bits padding>, page_code=0, allocation_length=96, control=Control(<2 bits padding>, naca=0, <3 bits pad
ding>, vendor_specific=0))
    'func': <function do_scsi_cdb at 0x1eca410>
    'kwargs': {}
    'format_args': 'sg249, \x12\x00\x00\x00`\x00'
                                                                                                                                                                                                       [538/97612]
  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post1.gf8180a8-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/scsi.py", line 88, in do_scsi_cdb
    return_value = queue.get(timeout=TIMEOUT_IN_SEC)
  Local variables:
    'Queue': <function Queue at 0x1cb3848>
    'cdb': StandardInquiryCommand(opcode=OperationCode(command_code=18, group_code=0) (const), evpd=0, <7 bits padding>, page_code=0, allocation_length=96, control=Control(<2 bits padding>, naca=0, <3 bits padd
ing>, vendor_specific=0))
    'Process': <class 'multiprocessing.process.Process'>
    'error': IOError('handle out of range in select()',)
    'sg_device': 'sg249'
    'subprocess': <Process(Process-1:509, started)>
    'queue': <multiprocessing.queues.Queue object at 0x26779d0>
    'return_value': None
    'EINTR': 4
    'Empty': <class 'Queue.Empty'>

  File "/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/queues.py", line 129, in get
    if not self._poll(block and (deadline-time.time()) or 0.0):
  Local variables:
    'self': <multiprocessing.queues.Queue object at 0x26779d0>
    'deadline': 1375107185.724053
    'block': True
    'timeout': 3

IOError: handle out of range in select()
System (serial=20026, name=ibox003):
...
...
{noformat}

Comments:
HIP-669 code review changes with @arnony
HIP-672 removing call to logger from do_scsi_cdb_with_in_process as it is in the child process

================================================================================

Host Power Tools / HIP-672
system register and other commands say "No handlers could be found for logger "infi.storagemodel.linux""

Type:              Bug                               Status:        Open                       Assignee: Guy Rozendorn
Priority:          Minor                             Resolution:    Unresolved                 Reporter: Rick Pelleg
Affects Version/s: 1.0.0                             Fix Version/s:
Components: system registration                                                                Created: 2013-07-31 23:04
Labels:                                                                                        Updated: 2013-07-31 23:10

Issue Links:

Sub-Tasks:

Description:
Version 1.0.post3.g9ea4ced.
{noformat}
# infinihost system register
Created host 'io-virt-001.infinidat.com'
Registered initiator port 10000000c9f21ddb in host 'io-virt-001.infinidat.com' on system 30026 (box-cl005)
Updated metadata in system 30026 for host id 102
    hostname = 'io-virt-001.infinidat.com'
    system = 'Linux'
    platform = 'Linux-2.6.32-122.el6.x86_64-x86_64-with-redhat-6.1-Santiago'
    powertools_version = '1.0.post3.g9ea4ced'
No handlers could be found for logger "infi.storagemodel.linux"
{noformat}

Comments:
HIP-673 do_scsi_cdb_with_in_process: handling failures in queue.put

================================================================================

Host Power Tools / HIP-673
volume unmap unmaps but prints big traceback

Type:              Bug                               Status:        Open                       Assignee: Guy Rozendorn
Priority:          Major                             Resolution:    Unresolved                 Reporter: Rick Pelleg
Affects Version/s: 1.0.0                             Fix Version/s:
Components: volume provissioning                                                               Created: 2013-07-31 23:07
Labels:                                                                                        Updated: 2013-07-31 23:27

Issue Links:

Sub-Tasks:

Description:
Version 1.0.post3.g9ea4ced.

In addtion to "No handlers ... " like "system register" in HIP-672, also:
{noformat}
[root@io-virt-001 to-box-cl005]# infinihost volume unmap dm-0 --yes
No handlers could be found for logger "infi.storagemodel.linux"
Process _GProcess-1:5:
Traceback (most recent call last):
  File "/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  Local variables:
    'util': <module 'multiprocessing.util' from '/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/util.pyc'>
    'self': <_GProcess(_GProcess-1:5, started)>
    'traceback': <module 'traceback' from '/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/traceback.pyc'>
    'exitcode': 1

  File "/opt/infinidat/host-power-tools/parts/python/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  Local variables:
    'self': <_GProcess(_GProcess-1:5, started)>

  File "/opt/infinidat/host-power-tools/eggs/gipc-0.3.2-py2.7.egg/gipc/gipc.py", line 278, in _child
    target(*args, **kwargs)
  Local variables:
    'target': <function do_scsi_cdb_with_in_process at 0x1ccccf8>
    'hub': <Hub at 0x1cbc9b0 destroyed>
    'h': <_GIPCReader_7e4f74 fd: 11>
    'args': (<_GIPCDuplexHandle(<_GIPCReader_7e4f74 fd: 11>, <_GIPCWriter_a820ae fd: 10>)>, 'sg2', StandardInquiryCommand(opcode=OperationCode(command_code=18, group_code=0) (const), evpd=0, <7 bits padding>, page_code=0, allocation_length=96, control=Control(<2 bits padding>, naca=0, <3 bits padding>, vendor_specific=0)))
    'childhandles': [<_GIPCWriter_a820ae fd: 10>, <_GIPCReader_7e4f74 fd: 11>]
    'kwargs': {}

  File "/opt/infinidat/host-power-tools/eggs/infi.storagemodel-0.1.60.post3.ge35c02e-py2.7.egg/infi/storagemodel/linux/rescan_scsi_bus/scsi.py", line 68, in do_scsi_cdb_with_in_process
    queue.put(ScsiCommandFailed())
  Local variables:
    'queue': <_GIPCDuplexHandle(<_GIPCReader_7e4f74 fd: 11>, <_GIPCWriter_a820ae fd: 10>)>
    'sync_wait': <function sync_wait at 0x170ed70>
    'sg_device': 'sg2'
    'cdb': StandardInquiryCommand(opcode=OperationCode(command_code=18, group_code=0) (const), evpd=0, <7 bits padding>, page_code=0, allocation_length=96, control=Control(<2 bits padding>, naca=0, <3 bits padding>, vendor_specific=0))
    'func': <function func at 0x1d95500>

  File "/opt/infinidat/host-power-tools/eggs/gipc-0.3.2-py2.7.egg/gipc/gipc.py", line 679, in put
    self._write(struct.pack("!i", len(bindata)) + bindata)
  Local variables:
    'self': <_GIPCWriter_a820ae fd: 10>
    'bindata': '\x80\x02cinfi.storagemodel.linux.rescan_scsi_bus.utils\nScsiCommandFailed\nq\x01)Rq\x02.'
    'o': ScsiCommandFailed()

  File "/opt/infinidat/host-power-tools/eggs/gipc-0.3.2-py2.7.egg/gipc/gipc.py", line 659, in _write
    diff = len(bindata) - _WRITE_NB(self._fd, bindata)
  Local variables:
    'self': <_GIPCWriter_a820ae fd: 10>
    'bindata': '\x00\x00\x00J\x80\x02cinfi.storagemodel.linux.rescan_scsi_bus.utils\nScsiCommandFailed\nq\x01)Rq\x02.'

  File "/opt/infinidat/host-power-tools/eggs/gevent-1.0rc2-py2.7-linux-x86_64.egg/gevent/os.py", line 69, in nb_write
    return _write(fd, buf)
  Local variables:
    'buf': '\x00\x00\x00J\x80\x02cinfi.storagemodel.linux.rescan_scsi_bus.utils\nScsiCommandFailed\nq\x01)Rq\x02.'
    'fd': 10
    'e': OSError(32, 'Broken pipe')
    'event': None
    'hub': None

OSError: [Errno 32] Broken pipe
{noformat}

Comments:
Guy Rozendorn added a comment - 2013-07-31 23:15
I copied a proposed fix for this and HIP-672 to io-virt-001, can you map some volumes and verify it fixed these two?

Rick Pelleg added a comment - 2013-07-31 23:24
No more "No handlers..." so far.
First unmap -- OK.
Second unmap (reproducable; try it):
{noformat}
infinihost volume unmap dm-8 --yes
> /opt/infinidat/host-power-tools/src/infi/vendata/powertools/volume_unmap/engine.py(43)_umount_partition()
-> remove_mount()
(Pdb)
{noformat}

Guy Rozendorn added a comment - 2013-07-31 23:25
that's me playing with HIP-674

Rick Pelleg added a comment - 2013-07-31 23:27
'K
HIP-674 LinuxMountRepository.get_all_persistent_mounts now returns entries from fstab and not mtab

================================================================================

Host Power Tools / HIP-674
volume unmap does not remove fstab entry

Type:              Bug                               Status:        Open                       Assignee: Guy Rozendorn
Priority:          Major                             Resolution:    Unresolved                 Reporter: Rick Pelleg
Affects Version/s: 1.0.0                             Fix Version/s:
Components: volume provissioning                                                               Created: 2013-07-31 23:13
Labels:                                                                                        Updated: 2013-07-31 23:36

Issue Links:

Sub-Tasks:

Description:
For example:
{noformat}
# infinihost volume unmap dm-2 --yes
No handlers could be found for logger "infi.storagemodel.linux"
# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Nov 19 12:09:02 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=836989d9-3e3c-4539-a194-326d59b24b71 /                       ext3    defaults        1 1
UUID=2dcc378d-c846-4420-9c60-55e4ba11f899 swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/mapper/mpathbhp1   /mnt/io-virt-001.infinidat.com-vol-20130731-2253        ext3    rw      0       0
/dev/mapper/mpathclp1   /mnt/io-virt-001.infinidat.com-vol-20130731-2253-1      ext3    rw      0       0
{noformat}

Comments:
Guy Rozendorn added a comment - 2013-07-31 23:19
All our VMs are down ATM due to storage faults, I'm experimenting on io-virt-001

Guy Rozendorn added a comment - 2013-07-31 23:35
okay found the problem(s); one ticket is enough though; done with io-virt-001, it now has code that remove the entries from fstab;

thanks!

Guy Rozendorn added a comment - 2013-07-31 23:36
we'll make sure these are tested in our regression
minor change
Freezing dependencies
Finished release v0.1.61.