blockVolume: remove inexplicable fixme comment

There's no apparent reason for this fixme. Since the line is raising an
exception the "fix me" string appears in the tracebacks making our code
appear sloppy.

I searched for other inexplicable fixmes but it seems that this was the
only one.

Change-Id: I39c5a2034fdc7e5300259a2c5a25b0aed7f679b6
Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/33949
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
fixing pyflakes errors

Change-Id: I54dad2edc73c3922bdb8c97e6fa036a3dfcf683b
Signed-off-by: Yaniv Bronhaim <ybronhei@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35026
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
vdsm-reg: Fix closed log file descriptors

When calling createDaemon(), all file descriptors are closed, including
the file descriptors held by log handlers. This patch moves logging
configuration below the daemonization to make sure the file descriptors
stay open.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1150238
Change-Id: Ida11e61aea18fbb2990176889423d2c4b6292a09
Signed-off-by: Dima Kuznetsov <dkuznets@redhat.com>
Signed-off-by: Yaniv Bronhaim <ybronhei@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/34981
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
tests: Avoid printing objects that reference None

One of the current ioprocess tests checks who holds reference to the
ioprocess object, but since only a weak reference is stored, if the
object is freed, we check who holds reference to None, and apparently
quite a lot things do.

Change-Id: I9770873d70ec962837574fe32e8b4cb613e7c696
Signed-off-by: Dima Kuznetsov <dkuznets@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/34978
Reviewed-by: Yaniv Bronhaim <ybronhei@redhat.com>
Reviewed-by: Yeela Kaplan <ykaplan@redhat.com>
Tested-by: Yeela Kaplan <ykaplan@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
clean dhclient processes that are left after functional tests

using a context manager to kill dhclient process that
outlives the functional test

Change-Id: I458aa38415c697d3863e173444ff921d759166a2
Signed-off-by: ibarkan <ibarkan@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/34366
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
Fix ballooning behaviour of freshly received VM

There is an issue with freshly received VMs. MOM might ask for
the current balloon status before the stats threads collect all
data. The target will be reported as 0 in that case and the VM
will be requested to return all its free memory.

The balloon info method won't return any data before it
has everything ready after this fix is applied.

This also fixes the unittest that was using balloon for checking
whether the stats system works.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1123274
Change-Id: If44c8e9d65b7221b2dec5fdd1f2d7d4343f7aeaa
Signed-off-by: Martin Sivak <msivak@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/34887
Reviewed-by: Francesco Romani <fromani@redhat.com>
Reviewed-by: Adam Litke <alitke@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
Makefile: add whitelist files to pyflakes and pep8 check

We should run pyflakes and pep8 check in vdsm-reg-setup.in for example.
We already have whitelist for pep8. this adds the files also to pyflakes
check.

Change-Id: I6be99965f3249374c99c1d4ab71145d571c13921
Signed-off-by: Douglas Schilling Landgraf <dougsland@redhat.com>
Signed-off-by: Yaniv Bronhaim <ybronhei@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/27025
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
spec: bump policycoreutils-python version to 2.0.83-19.47.

New version contains fix for:
seobject.booleanRecords().modify(...)
AttributeError: booleanRecords instance has no attribute 'load'

The bz for the policycoreutils bug is:
https://bugzilla.redhat.com/show_bug.cgi?id=1148800

Change-Id: Idc1c30a157d6763674874113e62add82833bd070
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1139873
Signed-off-by: Mooli Tayer <mtayer@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/33771
Reviewed-by: Yaniv Bronhaim <ybronhei@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
Tested-by: Dan Kenigsberg <danken@redhat.com>
vm: state change missing return value

Return arguments where ignored in schema for pause and resume verbs.
We had to update command_info with following verbs to returned
'vmList' dictionary required by the engine.


Change-Id: If0737403dcfa0ccd0a3c2813b215b4d5fa958eca
Signed-off-by: pkliczewski <piotr.kliczewski@gmail.com>
Bug-Url: https://bugzilla.redhat.com/1163089
Reviewed-on: http://gerrit.ovirt.org/35105
Reviewed-by: Francesco Romani <fromani@redhat.com>
Tested-by: Francesco Romani <fromani@redhat.com>
Reviewed-by: Omer Frenkel <ofrenkel@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
Change clientIF to get port as string and not as int from config.

Change-Id: I3021dc9b25581d1e0078e480764eb1202ee517b5
Bug-Url: https://bugzilla.redhat.com/1163771
Signed-off-by: Artyom Lukianov <alukiano@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35128
Reviewed-by: Francesco Romani <fromani@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
qemuimg: add backing file to convert

This patch adds the support for the backing file in the convert qemu-img
command.

Change-Id: I2d6f254f5750fa249809469ecc764a73c91914d4
Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/33354
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
multipath: add specific fail entry for emc-vnx

As few other storage arrays also the emc-vnx is forcing the
queue_if_no_path option on the multipath device.

This patch adds a specific section for the device maintaining
the emc defaults but forcing to fail the io instead of queuing.

In case of queuing we observed that the processes were blocked
on io and among other things it prevented the storage pool to
be reconstructed.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1098769
Change-Id: Icf5f4c89eac1273bf3a8e1afa300fec7b5b68eae
Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35072
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
jsonrpc: Wrong key name for getDevicesVisibility


Bug-Url: https://bugzilla.redhat.com/1159637
Change-Id: I63180a87e40e48134e6da8b9996f849362fb5501
Signed-off-by: pkliczewski <piotr.kliczewski@gmail.com>
Reviewed-on: http://gerrit.ovirt.org/35014
Reviewed-by: Allon Mureinik <amureini@redhat.com>
Tested-by: Maor Lipchuk <mlipchuk@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
jsonrpc: diskSizeExtend missing in command_info

Bug-Url: https://bugzilla.redhat.com/1161115
Change-Id: I099b83defd9fc3f0280ddeb28f3c4b706f8998c2
Signed-off-by: pkliczewski <piotr.kliczewski@gmail.com>
Reviewed-on: http://gerrit.ovirt.org/35218
Reviewed-by: Oved Ourfali <oourfali@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
protocoldetector: Fix polling timeout

The event loop was calculating timeout in seconds, but poll timeout
should be milliseconds. This caused poll to wait up to 30 milliseconds
instead of 30 seconds where there are no events to handle.

Profiling a system running one vm for 5 mintues show _process_events is
dominating the profile, taking 55% of cpu time:

    Mon Nov 17 00:22:02 2014    vdsmd-master.prof

         14440725 function calls (14721061 primitive calls) in 16.907 CPU seconds

    Ordered by: internal time
    List reduced from 3079 to 10 due to restriction <10>

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 2241981    5.685    0.000    9.445    0.000 protocoldetector.py:108(MultiProtocolAcceptor._process_events)
 4501127    1.442    0.000    1.442    0.000 time:0(time)
 2246162    1.396    0.000    1.396    0.000 __builtin__:0(poll)
 2242305    0.737    0.000    0.737    0.000 __builtin__:0(max)
    6566    0.352    0.000    0.536    0.000 spark.py:211(Parser.buildState)
   50705    0.170    0.001    0.373    0.000 copy.py:144(deepcopy)
     127    0.158    0.001    0.158    0.001 cpopen.cpopen:0(createProcess)
   10394    0.154    0.000    0.154    0.000 __builtin__:0(open)
  154006    0.132    0.000    0.269    0.000 encoder.py:284(JSONEncoder._iterencode)
    2533    0.124    0.000    0.141    0.000 pthread.py:133(Cond.timedwait)

With this patch we can see that number of functions calls decreased by 76% and
total cpu time decreased by 57%:

    Mon Nov 17 00:33:44 2014    vdsmd-poll-timeout.prof

         3344648 function calls (3624721 primitive calls) in 7.340 CPU seconds

    Ordered by: internal time
    List reduced from 3069 to 10 due to restriction <10>

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      6435    0.346    0.000    0.518    0.000 spark.py:211(Parser.buildState)
       122    0.166    0.001    0.166    0.001 cpopen.cpopen:0(createProcess)
     10202    0.157    0.000    0.157    0.000 __builtin__:0(open)
     50705    0.148    0.001    0.329    0.000 copy.py:144(deepcopy)
        98    0.140    0.001    0.140    0.001 posix:0(fork)
    154178    0.115    0.000    0.236    0.000 encoder.py:284(JSONEncoder._iterencode)
      2496    0.113    0.000    0.131    0.000 pthread.py:133(Cond.timedwait)
      5870    0.112    0.000    0.153    0.000 spark.py:103(Parser.addRule)
      3325    0.109    0.000    0.194    0.000 __init__.py:226(LogRecord.__init__)
     28264    0.103    0.000    0.384    0.000 protocoldetector.py:108(MultiProtocolAcceptor._process_events)

Now we calculate the the timeout correctly, and improved variable names
make this code unlikely to break again.

Change-Id: If676b543bff30d08067d2cff3e0639c7e415f61f
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35212
Reviewed-by: Francesco Romani <fromani@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
Reviewed-by: Piotr Kliczewski <piotr.kliczewski@gmail.com>
Reviewed-by: Saggi Mizrahi <smizrahi@redhat.com>
oopTests: replace 'if None' with try except clause

When asserting if the instance is None at the end of the test,
we already assume that it should be None,
or else the test fails.
This means that the 'if None' check is redundant and
can be moved into a try except clause because in case
of test failure we still want the relevant logs.

Change-Id: I8589ec591aad370cb390c0548ec52ded155bf085
Signed-off-by: Yeela Kaplan <ykaplan@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35243
Reviewed-by: Dima Kuznetsov <dkuznets@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
pep8: fix pep8 on generated vdsm.py

Change-Id: I83014e4443276975528429cb54c33decd0b45ab9
Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35240
Reviewed-by: Douglas Schilling Landgraf <dougsland@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
vm: replace try/finally with 'with'

The creation of VM is throttled using a bounded semaphore.
This patch
* replaces the try/finally construct with 'with' to make
  code (a tiny bit) nicer ad smaller
* drops the log lines around the lock, because
  - we are now guaranteed the code is correct and always
    release the lock
  - if we care about performances and timings (e.g.time spent
    in locked region) we need more precise instrumentation
    anyway.

Change-Id: I1bf75d64ab4e838957cbd2b0890b8cbc6f9fc9b0
Signed-off-by: Francesco Romani <fromani@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/34529
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
clientIF: strip unneeded logging

Everything we log inside clientIF.createVm is either
redundant or with little value.

Highlights
* if we need to debug vmContainerLock, there are better
  and more comprehensive ways to do that
* no need to log error, it should be reported anyway.

So this patch strips all the logging from this method.

Change-Id: I0c5bf3c85d105885ae5eee5024ae2f0b9f69f4f7
Signed-off-by: Francesco Romani <fromani@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/34872
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
jsonrpc: switch to the simplejson module

During testing, we discovered that on RHEL 6.x
and Centos 6.x we unnecessarily lose performance
because the JSON module shipped with python 2.6
does not include speedups found in python 2.7
and onwards.

The standard library module is based on the
simplejson package, and RHEL 6.x/Centos 6.x already
provide an updated simplejson rpm.

Then, this patches makes yajsonrpc use simplejson,
to recover speed at nearly zero cost.

To ease development and deployment, this patch
always make VDSM require simplejson, even
on platforms shipping a modern enough python
distribution.

Change-Id: Ic17aa6d75cef933ee36e002c03732aab14b39648
Bug-Url: https://bugzilla.redhat.com/1155146
Signed-off-by: Francesco Romani <fromani@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/35024
Reviewed-by: Piotr Kliczewski <piotr.kliczewski@gmail.com>
Reviewed-by: Dima Kuznetsov <dkuznets@redhat.com>
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Dan Kenigsberg <danken@redhat.com>