qemu.tests: Use qemu-kvm -M ? to get machine type for physical resource check

The old method to get expect machine type can not fit machine type check for
q35 and pc. So update it by get the machine type string from the output
of qemu-kvm -M ?.

Signed-off-by: Yiqiao Pu <ypu@redhat.com>
Merge pull request #1084 from cevich/default_set

virt-libvirt: Extend test-runner default tests
Merge pull request #1069 from ypu/machine_type

qemu.tests: Use qemu-kvm -M ? to get machine type for physical resource ...
virttest.utils_env_unittest: Add clean reproducer for the 'RuntimeError' issue

Sometimes, while virt-test is saving Env to a file, the
tcpdump thread writes to the Env dictionary "address_cache",
generating a:

RuntimeError: dictionary changed size during iteration

exception. Of course, any thread that modifies env data
while saving is taking place is going to create this
problem. So I implemented a similar situation in the
Env unittest code, so we have a reproducer, that can
be later fixed to use new Env locking mechanisms.

So, in case it is not clear, this unittest is supposed
to fail right now, and after said locking mechanism
is implemented, the test will start passing.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.utils_env: Solve the env["address_cache"] RuntimeError issue

There is a background tcpdump thread that records DHCP leases
in the env["address_cache"] dictionary. Sometimes, this thread
might write to this dictionary while the environment object is
being saved, triggering a RuntimeError in virt-test.

So the solution involves:

1) Creating a save lock, (a threading reentrant lock object),
that is acquired by the env.save() method when it wants to
save env.data to a pickle file, and for methods that alter
env.data.

2) Move the tcpdump thread to a env object attribute, rather
than one of the keys of env.data, given that the lock object
can't be pickled.

With the implementation of env.save_lock, we can also fix
the problem identified by the locking unittest. The base of
the solution was Pavel's idea, including a good deal of the
implementation, so kudos to him.

Signed-off-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.utils_env: Add @utils_env.lock_safe decorator

Add a utility method to wrap functions that modify the
contents of env.data. In order to avoid the boring
repetition of code that gets an environment save lock,
perform data change and then release the lock, put that
into the decorator and use it.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
Merge pull request #1091 from autotest/environment-fix-3

Environment RuntimeError fix
qemu.tests: physical_resources_check - Fix undefined variable bug

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virt: Regex strings must begin with r

Signed-off-by: Chris Evich <cevich@redhat.com>
virttest.aexpect: Close tail thread fd on thread end

When the aexpect "tail" thread is requested to stop, take the chance
and close the filedescriptor being used by that thread. This makes up
for a big decrease in the number of open fds over the course of a
default virt test set. Before the patch, in 10 migrations tests
executed, the evolution of fds open is:

26 -> 31 -> 36 -> 41 -> 46 -> 51 -> 56 -> 61 -> 66 -> 71

After the patch, the evolution goes:

23 -> 25 -> 27 -> 29 -> 31 -> 33 -> 35 -> 37 -> 39 -> 40

That's quite a nice cleanup.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.qemu_vm: Make the qemu process a Tail object rather than Expect

Checking up the qemu vm code, I noticed that none of the
uses of vm.process actually requires it to be an Expect
object. By using a Tail object, we obtain the same features
we've been using the code for, and we get rid of an extra
expect reader pipe, that ends up accumulating over the course
of the tests.

So make a function aexpect.run_tail(), nearly identical to
aexpect.run_bg(), and make vm.process to be a tail object.
With this change,  we further reduce the open number of open
fds from:

23 -> 25 -> 27 -> 29 -> 31 -> 33 -> 35 -> 37 -> 39 -> 40

to:

22 -> 23 -> 24 -> 25 -> 26 -> 26 -> 28 -> 29 -> 30 -> 31

If you guys can see any disadvantages of simplifying things
here, let me know.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.aexpect: Don't remove inpipe server side

The clients will do that on close() anyway.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.aexpect: Introduce server side logging

Make the aexpect server able to log into a file. As it
is, we don't really have an easy way to tell what's going
on in there.

On later patches, we'll implement better ways to manage
files generated by aexpect, as well as debugging mechanisms.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.aexpect: Change file/log directory structure

Instead of having a flat structure of files

BASE_DIR/[filename]-[aexpect_instance]

Make the aexpect instance the parent dir of the files

BASE_DIR/[aexpect_instance]/filename

That makes it easy to see the number of open aexpect
controlled process at any given point in time, while
keeping information available.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
Merge pull request #1098 from autotest/fd-leak

virttest.aexpect: FD leak fixes
virttest.env_process: Tcpdump is stopped too early

Tcpdump is required to get ARP ip when calling wait_for_login.
Stopping it before wait_for_login will make wait_for_login hang for 240s

Signed-off-by: Hao Liu <hliu@redhat.com>
libvirt-tests: Backup inactive domain XML

* Added option to toggle saving active or inactive domain XML for
  libvirt_vm.backup_xml

* Added '--inactive' option when dumping xml for backup when calling
  virsh.dumpxml or libvirt_vm.backup_xml. Without this option, the new
  domain restored will change its inactive xml like following diff permanently.

* using vmXML.new_from_inactive_xml instead of vmXML.new_from_xml to
  backup domain.

```diff
@@ -41,7 +41,9 @@
       <target type='serial' port='0'/>
     </console>
     <input type='mouse' bus='ps2'/>
-    <graphics type='vnc' port='-1' autoport='yes'/>
+    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
+      <listen type='address' address='127.0.0.1'/>
+    </graphics>
     <video>
       <model type='cirrus' vram='9216' heads='1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
@@ -50,4 +52,5 @@
       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
     </memballoon>
   </devices>
+  <seclabel type='dynamic' model='selinux' relabel='yes'/>
 </domain>
```

Signed-off-by: Hao Liu <hliu@redhat.com>
virttest.utils_env: Check the whether the key is none before using it.

When get vm objects from Env object, the key of the data maybe None.
if the key is None, will raise error like:
   'NoneType' object has no attribute 'startswith'.

Signed-off-by: Yunping Zheng <yunzheng@redhat.com>
Merge pull request #1109 from humanux/env_issue

virttest.utils_env: Check the whether the key is none before using it.
virsh_secret_set_get: Prepare for upstream change

An upstream change to libvirt:

https://www.redhat.com/archives/libvir-list/2013-November/msg00400.html

results in the output starting with a single space rather than
at the beginning of the line - this change will account for that
with an appropriately placed '.lstrip()'.

Also, if fail the check for the existence of the base64_file, then
indicate that's what the failure is rather than what would be a
false message of "The secret value mismatch with result" because
the secret_decoded_string is empty.