Issue 1:
The new ansible_ssh_common_args lets me add a ProxyCommand

argument to the ssh invocation for entire inventory groups, which

simplifies use of an ssh jump host.

My understanding (from reading tickets) is that this was implemented

to permit use of ProxyCommand: (1) without having to override the

whole ssh_args, or alternatively (2) without needing to use the

.ssh/config of the invoking user.

However, when using ProxyCommand this way, the control socket

in the filesystem is keyed off the far hop (inner tunnel) of the jump,

which means the near hop does not use a persistent socket. If

another host needs to be reached via the jump gate, an entirely

new socket will be set up, even though we could reuse it for the

inner portion (first hop) of the jump. If reaching 1000 remote hosts

via a jump gate, we will make 1000 separate ssh invocations, each

with its own control socket, and no re-use (it only reuses if same

far-host is destination)

This could be optimized by using ControlPath to describe only the

near hop, and then use another ssh with ControlPath on the far

side. I don't know if this is a real issue or not, but thought it was

worth submitting a ticket for. I think it would involve making a

jump-host scenario a "first class" connection property that was

implemented in code that could then be "nested" and duplicated

on the remote end (rather than just a string-append to ssh args).

If this is just totally crazy, by all means close ticket. I think it could

help some scenarios though, it would avoid doing the first-half of the

jump circuit 999 times in the case of 1000 jump targets on the far

side. So it would have to do only 1001 total ssh setups, rather than

2000, which could really be a big win...

Issue 2:
While running an Ansible playbook, it will intermittently hang during a yum install. It doesn't happen all the time, but it happens somewhat frequently. It looks like it is waiting on a socket response that never comes.

The environment is

Boot2Docker version 1.7.0, build master : 7960f90

Docker version 1.7.0, build 0baf609

Container is based on centos:centos6

Ansible version is 1.9.2

When I go into the actual box that Ansible is trying to install on, this is what I see:

[root@um-apache tmp]# ps -efL | grep repoquery

root 637 512 637 0 2 21:17 pts/0 00:00:00 /usr/bin/python -tt /usr/bin/repoquery --show-duplicates --plugins --quiet -q --disablerepo --enablerepo --qf %{name}-%{version}-%{release}.%{arch} unzip

root 637 512 638 0 2 21:17 pts/0 00:00:00 /usr/bin/python -tt /usr/bin/repoquery --show-duplicates --plugins --quiet -q --disablerepo --enablerepo --qf %{name}-%{version}-%{release}.%{arch} unzip

That just sits around forever unless I manually kill it.

When trying to debug the problem, first I was checking to see if repoquery might be conflicted over a lock, and this is what I see:

[root@um-apache tmp]# lsof | grep 637

repoquery 637 root cwd DIR 0,66 4096 206 /home/callidus

repoquery 637 root rtd DIR 0,66 4096 2 /

repoquery 637 root txt REG 0,66 4864 236 /usr/bin/python

repoquery 637 root mem REG 0,66 27424 186 /lib64/libnss_dns-2.12.so

repoquery 637 root mem REG 0,66 50280 661 /usr/lib64/python2.6/lib-dynload/pyexpat.so

repoquery 637 root mem REG 0,66 165264 656 /lib64/libexpat.so.1.5.2

repoquery 637 root mem REG 0,66 42392 653 /usr/lib64/python2.6/lib-dynload/_elementtree.so

repoquery 637 root mem REG 0,66 25160 565 /usr/lib64/python2.6/lib-dynload/termios.so

repoquery 637 root mem REG 0,66 1382496 558 /usr/lib64/libxml2.so.2.7.6

repoquery 637 root mem REG 0,66 1140496 556 /lib64/libglib-2.0.so.0.2800.8

repoquery 637 root mem REG 0,66 45320 271 /usr/lib64/python2.6/site-packages/_sqlitecache.so

repoquery 637 root mem REG 0,66 77104 554 /usr/lib64/python2.6/lib-dynload/_sqlite3.so

repoquery 637 root mem REG 0,66 81256 553 /usr/lib64/python2.6/lib-dynload/datetime.so

repoquery 637 root mem REG 0,66 7208 546 /usr/lib64/python2.6/lib-dynload/_weakref.so

repoquery 637 root mem REG 0,66 9080 527 /usr/lib64/python2.6/lib-dynload/grpmodule.so

repoquery 637 root mem REG 0,66 14288 512 /lib64/libgpg-error.so.0.5.0

repoquery 637 root mem REG 0,66 197792 510 /usr/lib64/libgpgme.so.11.6.6

repoquery 637 root mem REG 0,66 78016 508 /usr/lib64/python2.6/site-packages/gpgme/_gpgme.so

repoquery 637 root mem REG 0,66 35696 503 /usr/lib64/python2.6/lib-dynload/bz2.so

repoquery 637 root mem REG 0,66 23784 496 /usr/lib64/python2.6/lib-dynload/zlibmodule.so

repoquery 637 root mem REG 0,66 10312 144 /lib64/libfreebl3.so

repoquery 637 root mem REG 0,66 40400 131 /lib64/libcrypt-2.12.so

repoquery 637 root mem REG 0,66 106160 465 /usr/lib64/libsasl2.so.2.0.23

repoquery 637 root mem REG 0,66 60512 463 /lib64/liblber-2.4.so.2.10.3

repoquery 637 root mem REG 0,66 161856 461 /usr/lib64/libssh2.so.1.0.1

repoquery 637 root mem REG 0,66 181176 459 /usr/lib64/libsmime3.so

repoquery 637 root mem REG 0,66 254328 458 /usr/lib64/libssl3.so

repoquery 637 root mem REG 0,66 326768 457 /lib64/libldap-2.4.so.2.10.3

repoquery 637 root mem REG 0,66 206672 455 /lib64/libidn.so.11.6.1

repoquery 637 root mem REG 0,66 347536 453 /usr/lib64/libcurl.so.4.1.1

repoquery 637 root mem REG 0,66 69504 268 /usr/lib64/python2.6/site-packages/pycurl.so

repoquery 637 root mem REG 0,66 9872 451 /usr/lib64/python2.6/lib-dynload/_bisectmodule.so

repoquery 637 root mem REG 0,66 41408 444 /usr/lib64/python2.6/lib-dynload/arraymodule.so

repoquery 637 root mem REG 0,66 15432 441 /usr/lib64/python2.6/lib-dynload/_hashlib.so

repoquery 637 root mem REG 0,66 14632 402 /usr/lib64/python2.6/lib-dynload/fcntlmodule.so

repoquery 637 root mem REG 0,66 12680 401 /usr/lib64/python2.6/lib-dynload/_randommodule.so

repoquery 637 root mem REG 0,66 20976 400 /usr/lib64/python2.6/lib-dynload/binascii.so

repoquery 637 root mem REG 0,66 26408 399 /usr/lib64/python2.6/lib-dynload/mathmodule.so

repoquery 637 root mem REG 0,66 24432 392 /usr/lib64/python2.6/lib-dynload/selectmodule.so

repoquery 637 root mem REG 0,66 75664 386 /usr/lib64/python2.6/lib-dynload/cPickle.so

repoquery 637 root mem REG 0,66 110960 133 /lib64/libresolv-2.12.so

repoquery 637 root mem REG 0,66 10192 148 /lib64/libkeyutils.so.1.3

repoquery 637 root mem REG 0,66 43728 146 /lib64/libkrb5support.so.0.1

repoquery 637 root mem REG 0,66 174840 139 /lib64/libk5crypto.so.3.1

repoquery 637 root mem REG 0,66 14664 141 /lib64/libcom_err.so.2.1

repoquery 637 root mem REG 0,66 946048 137 /lib64/libkrb5.so.3.3

repoquery 637 root mem REG 0,66 277704 135 /lib64/libgssapi_krb5.so.2.2

repoquery 637 root mem REG 0,66 1963296 123 /usr/lib64/libcrypto.so.1.0.1e

repoquery 637 root mem REG 0,66 441256 385 /usr/lib64/libssl.so.1.0.1e

repoquery 637 root mem REG 0,66 34112 383 /usr/lib64/python2.6/lib-dynload/_ssl.so

repoquery 637 root mem REG 0,66 60752 382 /usr/lib64/python2.6/lib-dynload/_socketmodule.so

repoquery 637 root mem REG 0,66 115768 339 /usr/lib64/libmagic.so.1.0.0

repoquery 637 root mem REG 0,66 155344 337 /usr/lib64/librpmbuild.so.1.0.0

repoquery 637 root mem REG 0,66 12560 335 /usr/lib64/python2.6/site-packages/rpm/_rpmbmodule.so

repoquery 637 root mem REG 0,66 477712 211 /lib64/libfreeblpriv3.so

repoquery 637 root mem REG 0,66 583552 320 /usr/lib64/libsqlite3.so.0.8.6

repoquery 637 root mem REG 0,66 262168 318 /usr/lib64/libsoftokn3.so

repoquery 637 root mem REG 0,66 65928 37 /lib64/libnss_files-2.12.so

repoquery 637 root mem REG 0,66 244624 151 /lib64/libnspr4.so

repoquery 637 root mem REG 0,66 14560 150 /lib64/libplds4.so

repoquery 637 root mem REG 0,66 18720 143 /lib64/libplc4.so

repoquery 637 root mem REG 0,66 179416 149 /usr/lib64/libnssutil3.so

repoquery 637 root mem REG 0,66 90880 108 /lib64/libgcc_s-4.4.7-20120601.so.1

repoquery 637 root mem REG 0,66 18712 231 /lib64/libattr.so.1.1.0

repoquery 637 root mem REG 0,66 43880 106 /lib64/librt-2.12.so

repoquery 637 root mem REG 0,66 183920 317 /usr/lib64/liblua-5.1.so

repoquery 637 root mem REG 0,66 134920 316 /usr/lib64/liblzma.so.0.0.0

repoquery 637 root mem REG 0,66 36360 314 /lib64/libpopt.so.0.0.0

repoquery 637 root mem REG 0,66 88440 312 /usr/lib64/libelf-0.161.so

repoquery 637 root mem REG 0,66 88600 127 /lib64/libz.so.1.2.3

repoquery 637 root mem REG 0,66 67592 310 /lib64/libbz2.so.1.0.4

repoquery 637 root mem REG 0,66 1300048 142 /usr/lib64/libnss3.so

repoquery 637 root mem REG 0,66 183968 308 /usr/lib64/librpmio.so.1.0.0

repoquery 637 root mem REG 0,66 1523064 306 /lib64/libdb-4.7.so

repoquery 637 root mem REG 0,66 31280 229 /lib64/libacl.so.1.1.0

repoquery 637 root mem REG 0,66 16600 305 /lib64/libcap.so.2.16

repoquery 637 root mem REG 0,66 122040 152 /lib64/libselinux.so.1

repoquery 637 root mem REG 0,66 435400 303 /usr/lib64/librpm.so.1.0.0

repoquery 637 root mem REG 0,66 92872 301 /usr/lib64/python2.6/site-packages/rpm/_rpmmodule.so

repoquery 637 root mem REG 0,66 19248 359 /usr/lib64/python2.6/lib-dynload/cStringIO.so

repoquery 637 root mem REG 0,66 37840 389 /usr/lib64/python2.6/lib-dynload/_struct.so

repoquery 637 root mem REG 0,66 21608 410 /usr/lib64/python2.6/lib-dynload/_localemodule.so

repoquery 637 root mem REG 0,66 12256 366 /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so

repoquery 637 root mem REG 0,66 25288 358 /usr/lib64/python2.6/lib-dynload/stropmodule.so

repoquery 637 root mem REG 0,66 54896 373 /usr/lib64/python2.6/lib-dynload/itertoolsmodule.so

repoquery 637 root mem REG 0,66 38608 370 /usr/lib64/python2.6/lib-dynload/operator.so

repoquery 637 root mem REG 0,66 28112 369 /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so

repoquery 637 root mem REG 0,66 20328 348 /usr/lib64/python2.6/lib-dynload/timemodule.so

repoquery 637 root mem REG 0,66 1920896 32 /lib64/libc-2.12.so

repoquery 637 root mem REG 0,66 596272 239 /lib64/libm-2.12.so

repoquery 637 root mem REG 0,66 14584 125 /lib64/libutil-2.12.so

repoquery 637 root mem REG 0,66 19536 30 /lib64/libdl-2.12.so

repoquery 637 root mem REG 0,66 142640 110 /lib64/libpthread-2.12.so

repoquery 637 root mem REG 0,66 1669840 237 /usr/lib64/libpython2.6.so.1.0

repoquery 637 root mem REG 0,66 154664 25 /lib64/ld-2.12.so

repoquery 637 root 0u CHR 136,0 0t0 3 /dev/pts/0

repoquery 637 root 1w FIFO 0,8 0t0 261194 pipe

repoquery 637 root 2w FIFO 0,8 0t0 261195 pipe

repoquery 637 root 3w REG 0,66 2531 628 /var/log/yum.log

repoquery 637 root 4u sock 0,6 0t0 261197 can't identify protocol

repoquery 637 root 7u sock 0,6 0t0 258410 can't identify protocol

When I look at what the process is actually doing at that moment, this is what I see

[root@um-apache tmp]# strace -p 637

Process 637 attached

futex(0x12410c0, FUTEX_WAIT_PRIVATE, 0, NULL

^CProcess 637 detached

<detached ...>

[root@um-apache tmp]# strace -p 638

Process 638 attached

recvmsg(7, ^CProcess 638 detached

<detached ...>