joyent/illumos-kvm-cmd — Issue #8: qemu-kvm-sys 100% SYS CPU, guest hangs
Author: estibi
Created: 2012-03-01 16:23:09+00:00
State: open

Host machine: oi_151a, oi_151a2
Both kvm and kvm-cmd are built by hand from github - the latest version.

When multiple guests are booted at the same time, some of them hangs.

VMs is are Linux hosts.

# mpstat 1 3
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 97 0 24 3821 1847 1499 1 38 464 0 813 0 11 0 89
1 56 0 61 2192 1054 1502 3 33 441 0 1224 0 20 0 79
2 43 0 24 181 19 1434 2 22 600 0 469 0 7 0 93
3 40 0 13 227 28 1215 2 24 674 0 913 0 22 0 78
4 97 0 47 199 17 1097 1 23 226 0 784 0 3 0 97
5 41 0 40 204 22 999 1 20 138 0 1180 0 4 0 96
6 50 0 41 1819 912 1535 2 17 222 0 764 0 4 0 96
7 94 0 96 3759 1842 1828 3 22 202 0 904 0 6 0 94
8 26 0 22 469 156 1646 1 24 450 0 442 0 9 0 91
9 24 0 11 190 19 741 1 17 222 0 497 0 9 0 90
10 30 0 33 1789 865 1667 2 19 993 0 397 0 10 0 90
11 19 0 15 163 17 977 1 13 291 0 270 0 8 0 92
12 56 0 59 1069 482 1678 2 21 186 0 1117 0 7 0 93
13 34 0 68 2327 1127 1770 2 17 468 0 310 0 6 0 94
14 39 0 60 2461 1186 1685 2 19 209 0 651 0 4 0 96
15 73 0 41 190 21 958 1 16 127 0 333 0 2 0 98
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 2399 1147 135 0 18 9 0 347 0 0 0 100
1 0 0 0 4 0 0 3 0 0 0 3 0 100 0 0
2 0 0 0 113 10 167 0 6 1 0 216 1 0 0 99
3 0 0 0 64 14 2051 4 4 301 0 121 0 1 0 99
4 21 0 1 128 9 211 2 9 2 0 336 0 0 0 100
5 0 0 4 74 14 1636 2 16 210 0 97 0 33 0 67
6 0 0 5 3979 1982 3300 2 15 174 0 44 0 3 0 97
7 0 0 4 2177 1075 2321 3 20 98 0 730 0 2 0 98
8 0 0 0 2006 1000 0 4 0 0 0 4 0 100 0 0
9 0 0 0 44 14 22 0 1 0 0 95 0 0 0 100
10 0 0 0 100 7 2273 0 21 286 0 315 0 1 0 99
11 0 0 0 223 72 146 0 3 19 0 665 0 0 0 100
12 0 0 5 2078 1025 3248 6 25 242 0 342 1 2 0 97
13 0 0 2 4076 2031 1740 1 15 31 0 51 0 14 0 86
14 0 0 2 83 19 72 6 12 1 0 73 0 55 0 45
15 0 0 0 64 14 50 0 11 3 0 18 0 0 0 100
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 2413 1157 183 0 15 20 0 662 1 1 0 98
1 0 0 0 5 1 0 3 0 0 0 3 0 100 0 0
2 0 0 0 100 2 158 0 2 0 0 142 0 0 0 100
3 0 0 0 33 2 1982 3 0 257 0 5 0 2 0 98
4 0 0 0 80 20 50 0 7 0 0 153 0 1 0 99
5 0 0 1 90 31 4798 2 13 637 0 620 0 4 0 96
6 0 0 1 4009 2001 208 0 10 10 0 10 0 2 0 98
7 8 0 1 2098 1034 2505 8 19 158 0 652 0 2 0 98
8 0 0 0 2004 1000 0 3 0 0 0 3 0 100 0 0
9 0 0 0 14 6 8 0 1 0 0 0 0 0 0 100
10 0 0 0 115 4 2310 0 8 353 0 298 0 2 0 98
11 0 0 0 217 72 144 0 2 11 0 665 0 0 0 100
12 0 0 2 2047 1010 4036 2 14 52 0 125 0 2 0 98
13 0 0 3 4079 2031 871 3 15 23 0 76 0 1 0 99
14 0 0 0 18 10 0 4 0 0 0 4 0 100 0 0
15 0 0 0 103 40 99 0 14 1 0 31 1 0 0 99

PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
2317 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 0 18 18 0 qemu-kvm-sys/6
2317 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 0 16 16 0 qemu-kvm-sys/5
2317 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 0 16 16 0 qemu-kvm-sys/4
2320 root 0.0 3.0 0.0 0.0 0.0 0.0 97 0.4 9K 15 917 6 qemu-kvm-sys/3
2323 root 0.0 2.5 0.0 0.0 0.0 0.0 97 0.4 9K 2 1K 0 qemu-kvm-sys/3
2317 root 0.0 1.5 0.0 0.0 0.0 0.0 98 0.4 9K 17 17 0 qemu-kvm-sys/3

# pstack 2317
2317:	qemu-kvm -enable-kvm -nographic -smp 4 -m 12288 -vnc 0.0.0.0:51 -no-hp
----------------- lwp# 1 / thread# 1 --------------------
fffffd7fff2cfeda pollsys (fffffd7fffdf9650, 4, fffffd7fffdf9760, 0)
fffffd7fff25c154 pselect () + 18c
fffffd7fff25c668 select () + 70
00000000004bcf1f main_loop_wait () + 19f
00000000004cf70b kvm_main_loop () + bb
00000000004be406 main () + 9a6
00000000004b031c _start () + 6c
----------------- lwp# 2 / thread# 2 --------------------
fffffd7fff2cf54a sigtimedwait (f75cc0, fffffd7ffee9ecb0, 0)
fffffd7fff2b8704 sigwaitinfo () + c
00000000004db088 sigwait_compat () + 48
fffffd7fff2c6fd4 _thrp_setup () + bc
fffffd7fff2c72a0 _lwp_start ()
----------------- lwp# 3 / thread# 3 --------------------
fffffd7fff2cfc5a ioctl (b, 2000ae80, 0)
00000000004cecd9 kvm_cpu_exec () + 9
00000000004cf33a ap_main_loop () + 22a
fffffd7fff2c6fd4 _thrp_setup () + bc
fffffd7fff2c72a0 _lwp_start ()
----------------- lwp# 4 / thread# 4 --------------------
fffffd7fff2cfc5a ioctl () + a
00000000004cecd9 kvm_cpu_exec () + 9
00000000004cf33a ap_main_loop () + 22a
fffffd7fff2c6fd4 _thrp_setup () + bc
fffffd7fff2c72a0 _lwp_start ()
----------------- lwp# 5 / thread# 5 --------------------
fffffd7fff2cfc5a ioctl () + a
00000000004cecd9 kvm_cpu_exec () + 9
00000000004cf33a ap_main_loop () + 22a
fffffd7fff2c6fd4 _thrp_setup () + bc
fffffd7fff2c72a0 _lwp_start ()
----------------- lwp# 6 / thread# 6 --------------------
fffffd7fff2cfc5a ioctl () + a
00000000004cecd9 kvm_cpu_exec () + 9
00000000004cf33a ap_main_loop () + 22a
fffffd7fff2c6fd4 _thrp_setup () + bc
fffffd7fff2c72a0 _lwp_start ()
----------------- lwp# 7 / thread# 7 --------------------
fffffd7fff2cf54a sigtimedwait (3011f4530, fffffd7ffe162cb0, 0)
fffffd7fff2b8704 sigwaitinfo () + c
00000000004db088 sigwait_compat () + 48
fffffd7fff2c6fd4 _thrp_setup () + bc
fffffd7fff2c72a0 _lwp_start ()