Repository: ansible/ansible

Issue:
OSError: [Errno 24] Too many open files -- when running from OSX 10.7 
It seems some recent changes has introduced this regression, this occurs on OSX 10.7, bumping up the limits by "ulimit -n 2048" works around the problem. There might just be some file descriptors/pipes not being closed. You could limit the loop (wherever its happening) for the subprocesses to be just 1024?

I might poke around if I have time. For now the above is a bug report!

<pre>
TASK: [clone {{host_group}} code - private head] ******************************
fatal: [10.0.1.200] => Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 368, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 455, in _executor_internal
    return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 612, in _executor_internal_inner
    tmp = self._make_tmp_path(conn)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 736, in _make_tmp_path
    result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 671, in _low_level_exec_command
    rc, stdin, stdout, stderr = conn.exec_command(cmd, tmp, sudo_user, sudoable=sudoable, executable=executable)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/connection_plugins/ssh.py", line 165, in exec_command
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 703, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1120, in _get_handles
    errread, errwrite = self.pipe_cloexec()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1153, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files


FATAL: all hosts have already failed -- aborting
</pre>



Discussion:
Are you familiar with git, and if so would you be interested in doing a "git bisect"? If you're not familiar with the process, I can walk you through it (either here or on IRC).

Yeah so this needs more information about _how_ to reproduce this before I'm willing to consider it a bug.   Perhaps you were using paramiko before and the smart defaulting to SSH with control persist combined with a large host pool is just doing what it is supposed to.

Get with James on git bisect info please.

It may be you just have enough hosts that you are doing that many different things, but we need more information.

As I recall, this was discussed on the mailing list, so let's continue there if need be.

FYI I'm experiencing this too with only one host but with a nested hierarchy of interdependent roles. Don't have an easy reproduce yet but I thought it would be useful to share.

@cayblood Can you tell me if you're using paramiko or ssh? And also, how many forks are you setting for the run?

ansible-playbook appears to be connecting with ssh. I'm not sure what you mean by the forks question. This is a single vagrant box that we are provisioning with ansible.

@cayblood from `ansible --help`:

<pre>
  -f FORKS, --forks=FORKS
                        specify number of parallel processes to use
                        (default=5)
</pre>

I ask because yesterday we noticed that for every worker, we're duping the STDIN file handle (which I believe would go towards the open file count). That was one of the things that changed from 1.2 -> 1.2.2 and we were looking at it as a potential cause of the problem.

For a vagrant box, I'd assume you're not setting that and there is code that reduces the worker pool to the number of hosts. 

I'm not sure what the OSX equivalent of `lsof` is, but you might want to try and capture the list of open files and see what it is that's actually opening them all - it could be a mix of vagrant, ansible, or any other number of processes on the system. The ulimit is tied to the uid, so unless we can definitively tie this to something ansible is doing we can't really troubleshoot it.

I've done an lsof before running ansible and right before the failure takes place, then diffed the result. Here is the lion's share of the new open files that happen during provisioning. Looks like there are a lot of TTY sockets. Don't know why offhand, but maybe this is useful.

<pre>
> bash      11368 carl    0u     CHR               16,1    0t34857    11597 /dev/ttys001
> bash      11368 carl    1u     CHR               16,1    0t34857    11597 /dev/ttys001
> bash      11368 carl    2u     CHR               16,1    0t34857    11597 /dev/ttys001
> bash      11368 carl  255u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12462 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> python    12462 carl  txt      REG                1,2       8656 50109722 /Users/carl/Source/work/ttvm/.env/bin/python
> python    12462 carl  txt      REG                1,2    1543064  1839801 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python
> python    12462 carl  txt      REG                1,2      20824  1844694 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
> python    12462 carl  txt      REG                1,2      25124  1844690 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
> python    12462 carl  txt      REG                1,2      14368  1844672 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
> python    12462 carl  txt      REG                1,2      32588  1844654 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
> python    12462 carl  txt      REG                1,2      19552  1844662 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
> python    12462 carl  txt      REG                1,2      20732  1844669 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
> python    12462 carl  txt      REG                1,2      31736  1844611 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
> python    12462 carl  txt      REG                1,2      32952  1844684 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
> python    12462 carl  txt      REG                1,2      54820  1844678 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
> python    12462 carl  txt      REG                1,2       9836  1844600 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
> python    12462 carl  txt      REG                1,2      18468  1844627 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
> python    12462 carl  txt      REG                1,2      28648  1844691 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
> python    12462 carl  txt      REG                1,2      37768  1844659 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
> python    12462 carl  txt      REG                1,2      80076  1844670 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
> python    12462 carl  txt      REG                1,2     204728 50113116 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/_yaml.so
> python    12462 carl  txt      REG                1,2     118252 31530320 /usr/local/Cellar/libyaml/0.1.4/lib/libyaml-0.2.dylib
> python    12462 carl  txt      REG                1,2      14968  1844625 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
> python    12462 carl  txt      REG                1,2      15444  1844636 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
> python    12462 carl  txt      REG                1,2      72644  1844667 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
> python    12462 carl  txt      REG                1,2      55460  1844651 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
> python    12462 carl  txt      REG                1,2      30796  1844653 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
> python    12462 carl  txt      REG                1,2      11292  1844648 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
> python    12462 carl  txt      REG                1,2      10532 50113456 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/markupsafe/_speedups.so
> python    12462 carl  txt      REG                1,2      35648  1844680 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
> python    12462 carl  txt      REG                1,2      16724  1844626 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
> python    12462 carl  txt      REG                1,2      15188  1844646 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
> python    12462 carl  txt      REG                1,2      31004  1844633 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
> python    12462 carl  txt      REG                1,2      19152  1844693 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/termios.so
> python    12462 carl  txt      REG                1,2      27148  1844641 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
> python    12462 carl  txt      REG                1,2      69932  1844616 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
> python    12462 carl  txt      REG                1,2      67636 50114110 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/PublicKey/_fastmath.so
> python    12462 carl  txt      REG                1,2     433484 50540203 /usr/local/Cellar/gmp/5.1.2/lib/libgmp.10.dylib
> python    12462 carl  txt      REG                1,2      14544 50114205 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Util/_counter.so
> python    12462 carl  txt      REG                1,2      31872 50114062 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Cipher/_AES.so
> python    12462 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> python    12462 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> python    12462 carl    0u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12462 carl    1u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12462 carl    2u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12462 carl    3u     CHR              5,107        0t0      531 /dev/ptyvb
> python    12462 carl    4u     CHR               16,2        0t0    11849 /dev/ttys002
> python    12462 carl    5u     REG                1,2          0 51372895 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpqw21Zf
> python    12462 carl    6w     REG                1,2          0 50980754 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/.ansible-lock.501
> python    12462 carl    7u     REG                1,2          0 51372896 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpAVnQOf
> python    12462 carl    8u     CHR               16,3        0t0    11851 /dev/ttys003
> python    12462 carl    9u     CHR               16,4        0t0    11853 /dev/ttys004
> python    12462 carl   10u     CHR               16,5        0t0    11855 /dev/ttys005
> python    12462 carl   11u     CHR               16,6        0t0    11857 /dev/ttys006
> python    12462 carl   12u     CHR               16,7        0t0    11859 /dev/ttys007
> python    12462 carl   13u     CHR               16,8        0t0    11861 /dev/ttys008
> python    12462 carl   14u     CHR               16,9        0t0    11863 /dev/ttys009
> python    12462 carl   15u     CHR              16,10        0t0    11865 /dev/ttys010
> python    12462 carl   16u     CHR              16,11        0t0    11867 /dev/ttys011
> python    12462 carl   17u     CHR              16,12        0t0    11869 /dev/ttys012
> python    12462 carl   18u     CHR              16,13        0t0    11871 /dev/ttys013
> python    12462 carl   19u     CHR              16,14        0t0    11873 /dev/ttys014
> python    12462 carl   20u     CHR              16,15        0t0    11875 /dev/ttys015
> python    12462 carl   21u     CHR              16,16        0t0    11877 /dev/ttys016
> python    12462 carl   22u     CHR              16,17        0t0    11879 /dev/ttys017
> python    12462 carl   23u     CHR              16,18        0t0    11881 /dev/ttys018
> python    12462 carl   24u     CHR              16,19        0t0    11883 /dev/ttys019
> python    12462 carl   25u     CHR              16,20        0t0    11885 /dev/ttys020
> python    12462 carl   26u     CHR              16,21        0t0    11887 /dev/ttys021
> python    12462 carl   27u     CHR              16,22        0t0    11889 /dev/ttys022
> python    12462 carl   28u     CHR              16,23        0t0    11891 /dev/ttys023
> python    12462 carl   29u     CHR              16,24        0t0    11893 /dev/ttys024
> python    12462 carl   30u     CHR              16,25        0t0    11895 /dev/ttys025
> python    12462 carl   31u     CHR              16,26        0t0    11897 /dev/ttys026
> python    12462 carl   32u     CHR              16,27        0t0    11899 /dev/ttys027
> python    12462 carl   33u     CHR              16,28        0t0    11901 /dev/ttys028
> python    12462 carl   34u     CHR              16,29        0t0    11903 /dev/ttys029
> python    12462 carl   35u     CHR              16,30        0t0    11905 /dev/ttys030
> python    12462 carl   36u     CHR              16,31        0t0    11907 /dev/ttys031
> python    12462 carl   37u     CHR              16,32        0t0    11909 /dev/ttys032
> python    12462 carl   38u     CHR              16,33        0t0    11911 /dev/ttys033
> python    12462 carl   39u     CHR              16,34        0t0    11913 /dev/ttys034
> python    12462 carl   40u     CHR              16,35        0t0    11915 /dev/ttys035
> python    12462 carl   41u     CHR              16,36        0t0    11917 /dev/ttys036
> python    12462 carl   42u     CHR              16,37        0t0    11919 /dev/ttys037
> python    12462 carl   43u     CHR              16,38        0t0    11921 /dev/ttys038
> python    12462 carl   44u     CHR              16,39        0t0    11923 /dev/ttys039
> python    12462 carl   45u     CHR              16,40        0t0    11925 /dev/ttys040
> python    12462 carl   46u     CHR              16,41        0t0    11927 /dev/ttys041
> python    12462 carl   47u     CHR              16,42        0t0    11929 /dev/ttys042
> python    12462 carl   48u     CHR              16,43        0t0    11931 /dev/ttys043
> python    12462 carl   49u     CHR              16,44        0t0    11933 /dev/ttys044
> python    12462 carl   50u     CHR              16,45        0t0    11935 /dev/ttys045
> python    12462 carl   51u     CHR              16,46        0t0    11937 /dev/ttys046
> python    12462 carl   52u     CHR              16,47        0t0    11939 /dev/ttys047
> python    12462 carl   53u     CHR              16,48        0t0    11941 /dev/ttys048
> python    12462 carl   54u     CHR              16,49        0t0    11943 /dev/ttys049
> python    12462 carl   55u     CHR              16,50        0t0    11945 /dev/ttys050
> python    12462 carl   56u     CHR              16,51        0t0    11947 /dev/ttys051
> python    12462 carl   57u     CHR              16,52        0t0    11949 /dev/ttys052
> python    12462 carl   58u     CHR              16,53        0t0    11951 /dev/ttys053
> python    12462 carl   59u     CHR              16,54        0t0    11953 /dev/ttys054
> python    12462 carl   60u     CHR              16,55        0t0    11955 /dev/ttys055
> python    12462 carl   61u     CHR              16,56        0t0    11957 /dev/ttys056
> python    12462 carl   62u     CHR              16,57        0t0    11959 /dev/ttys057
> python    12462 carl   63u     CHR              16,58        0t0    11961 /dev/ttys058
> python    12462 carl   64u     CHR              16,59        0t0    11963 /dev/ttys059
> python    12462 carl   65u     CHR              16,60        0t0    11965 /dev/ttys060
> python    12462 carl   66u     CHR              16,61        0t0    11967 /dev/ttys061
> python    12462 carl   67u     CHR              16,62        0t0    11969 /dev/ttys062
> python    12462 carl   68u     CHR              16,63        0t0    11971 /dev/ttys063
> python    12462 carl   69u     CHR              16,64        0t0    11973 /dev/ttys064
> python    12462 carl   70u     CHR              16,65        0t0    11975 /dev/ttys065
> python    12462 carl   71u     CHR              16,66        0t0    11977 /dev/ttys066
> python    12462 carl   72u     CHR              16,67        0t0    11979 /dev/ttys067
> python    12462 carl   73u     CHR              16,68        0t0    11981 /dev/ttys068
> python    12462 carl   74u     CHR              16,69        0t0    11983 /dev/ttys069
> python    12462 carl   75u     CHR              16,70        0t0    11985 /dev/ttys070
> python    12462 carl   76u     CHR              16,71        0t0    11987 /dev/ttys071
> python    12462 carl   77u     CHR              16,72        0t0    11989 /dev/ttys072
> python    12462 carl   78u     CHR              16,73        0t0    11991 /dev/ttys073
> python    12462 carl   79u     CHR              16,74        0t0    11993 /dev/ttys074
> python    12462 carl   80u     CHR              16,75        0t0    11995 /dev/ttys075
> python    12462 carl   81u     CHR              16,76        0t0    11997 /dev/ttys076
> python    12462 carl   82u     CHR              16,77        0t0    11999 /dev/ttys077
> python    12462 carl   83u     CHR              16,78        0t0    12001 /dev/ttys078
> python    12462 carl   84u     CHR              16,79        0t0    12003 /dev/ttys079
> python    12462 carl   85u     CHR              16,80        0t0    12005 /dev/ttys080
> python    12462 carl   86u     CHR              16,81        0t0    12007 /dev/ttys081
> python    12462 carl   87u     CHR              16,82        0t0    12009 /dev/ttys082
> python    12462 carl   88u     CHR              16,83        0t0    12011 /dev/ttys083
> python    12462 carl   89u     CHR              16,84        0t0    12013 /dev/ttys084
> python    12462 carl   90u     CHR              16,85        0t0    12015 /dev/ttys085
> python    12462 carl   91u     CHR              16,86        0t0    12017 /dev/ttys086
> python    12462 carl   92u     CHR              16,87        0t0    12019 /dev/ttys087
> python    12462 carl   93u     CHR              16,88        0t0    12021 /dev/ttys088
> python    12462 carl   94u     CHR              16,89        0t0    12023 /dev/ttys089
> python    12462 carl   95u     CHR              16,90        0t0    12025 /dev/ttys090
> python    12462 carl   96u     CHR              16,91        0t0    12027 /dev/ttys091
> python    12462 carl   97u     CHR              16,92        0t0    12029 /dev/ttys092
> python    12462 carl   98u     CHR              16,93        0t0    12031 /dev/ttys093
> python    12462 carl   99u     CHR              16,94        0t0    12033 /dev/ttys094
> python    12462 carl  100u     CHR              16,95        0t0    12035 /dev/ttys095
> python    12462 carl  101u     CHR              16,96        0t0    12037 /dev/ttys096
> python    12462 carl  102u     CHR              16,97        0t0    12039 /dev/ttys097
> python    12462 carl  103u     CHR              16,98        0t0    12041 /dev/ttys098
> python    12462 carl  104u     CHR              16,99        0t0    12043 /dev/ttys099
> python    12462 carl  105u     CHR             16,100        0t0    12045 /dev/ttys100
> python    12462 carl  106u     CHR             16,101        0t0    12047 /dev/ttys101
> python    12462 carl  107u     CHR             16,102        0t0    12049 /dev/ttys102
> python    12462 carl  108u     CHR             16,103        0t0    12051 /dev/ttys103
> python    12462 carl  109u     CHR             16,104        0t0    12053 /dev/ttys104
> python    12462 carl  110u     CHR             16,105        0t0    12055 /dev/ttys105
> python    12462 carl  111u     CHR             16,106        0t0    12057 /dev/ttys106
> python    12462 carl  112u     CHR             16,107        0t0    12059 /dev/ttys107
> python    12462 carl  113u     CHR             16,108        0t0    12061 /dev/ttys108
> python    12462 carl  114u     CHR             16,109        0t0    12063 /dev/ttys109
> python    12462 carl  115u     CHR             16,110        0t0    12065 /dev/ttys110
> python    12462 carl  116u     CHR             16,111        0t0    12067 /dev/ttys111
> python    12462 carl  117u     CHR             16,112        0t0    12069 /dev/ttys112
> python    12462 carl  118u     CHR             16,113        0t0    12071 /dev/ttys113
> python    12462 carl  119u     CHR             16,114        0t0    12073 /dev/ttys114
> python    12462 carl  120u     CHR             16,115        0t0    12075 /dev/ttys115
> python    12462 carl  121u     CHR             16,116        0t0    12077 /dev/ttys116
> python    12462 carl  122u     CHR             16,117        0t0    12079 /dev/ttys117
> python    12462 carl  123u     CHR             16,118        0t0    12081 /dev/ttys118
> python    12462 carl  124u     CHR             16,119        0t0    12083 /dev/ttys119
> python    12462 carl  125u     CHR             16,120        0t0    12085 /dev/ttys120
> python    12462 carl  126u     CHR             16,121        0t0    12087 /dev/ttys121
> python    12462 carl  127u     CHR             16,122        0t0    12089 /dev/ttys122
> python    12462 carl  128u     CHR             16,123        0t0    12091 /dev/ttys123
> python    12462 carl  129u     CHR             16,124        0t0    12093 /dev/ttys124
> python    12462 carl  130u     CHR             16,125        0t0    12095 /dev/ttys125
> python    12462 carl  131u     CHR             16,126        0t0    12097 /dev/ttys126
> python    12462 carl  132u     CHR                4,0        0t0      316 /dev/ttyp0
> python    12462 carl  133u     CHR                4,1        0t0      318 /dev/ttyp1
> python    12462 carl  134u     CHR                4,2        0t0      320 /dev/ttyp2
> python    12462 carl  135u     CHR                4,3        0t0      322 /dev/ttyp3
> python    12462 carl  136u     CHR                4,4        0t0      324 /dev/ttyp4
> python    12462 carl  137u     CHR                4,5        0t0      326 /dev/ttyp5
> python    12462 carl  138u     CHR                4,6        0t0      328 /dev/ttyp6
> python    12462 carl  139u     CHR                4,7        0t0      330 /dev/ttyp7
> python    12462 carl  140u     CHR                4,8        0t0      332 /dev/ttyp8
> python    12462 carl  141u     CHR                4,9        0t0      334 /dev/ttyp9
> python    12462 carl  142u     CHR               4,10        0t0      336 /dev/ttypa
> python    12462 carl  143u     CHR               4,11        0t0      338 /dev/ttypb
> python    12462 carl  144u     CHR               4,12        0t0      340 /dev/ttypc
> python    12462 carl  145u     CHR               4,13        0t0      342 /dev/ttypd
> python    12462 carl  146u     CHR               4,14        0t0      344 /dev/ttype
> python    12462 carl  147u     CHR               4,15        0t0      346 /dev/ttypf
> python    12462 carl  148u     CHR               4,16        0t0      348 /dev/ttyq0
> python    12462 carl  149u     CHR               4,17        0t0      350 /dev/ttyq1
> python    12462 carl  150u     CHR               4,18        0t0      352 /dev/ttyq2
> python    12462 carl  151u     CHR               4,19        0t0      354 /dev/ttyq3
> python    12462 carl  152u     CHR               4,20        0t0      356 /dev/ttyq4
> python    12462 carl  153u     CHR               4,21        0t0      358 /dev/ttyq5
> python    12462 carl  154u     CHR               4,22        0t0      360 /dev/ttyq6
> python    12462 carl  155u     CHR               4,23        0t0      362 /dev/ttyq7
> python    12462 carl  156u     CHR               4,24        0t0      364 /dev/ttyq8
> python    12462 carl  157u     CHR               4,25        0t0      366 /dev/ttyq9
> python    12462 carl  158u     CHR               4,26        0t0      368 /dev/ttyqa
> python    12462 carl  159u     CHR               4,27        0t0      370 /dev/ttyqb
> python    12462 carl  160u     CHR               4,28        0t0      372 /dev/ttyqc
> python    12462 carl  161u     CHR               4,29        0t0      374 /dev/ttyqd
> python    12462 carl  162u     CHR               4,30        0t0      376 /dev/ttyqe
> python    12462 carl  163u     CHR               4,31        0t0      378 /dev/ttyqf
> python    12462 carl  164u     CHR               4,32        0t0      380 /dev/ttyr0
> python    12462 carl  165u     CHR               4,33        0t0      382 /dev/ttyr1
> python    12462 carl  166u     CHR               4,34        0t0      384 /dev/ttyr2
> python    12462 carl  167u     CHR               4,35        0t0      386 /dev/ttyr3
> python    12462 carl  168u     CHR               4,36        0t0      388 /dev/ttyr4
> python    12462 carl  169u     CHR               4,37        0t0      390 /dev/ttyr5
> python    12462 carl  170u     CHR               4,38        0t0      392 /dev/ttyr6
> python    12462 carl  171u     CHR               4,39        0t0      394 /dev/ttyr7
> python    12462 carl  172u     CHR               4,40        0t0      396 /dev/ttyr8
> python    12462 carl  173u     CHR               4,41        0t0      398 /dev/ttyr9
> python    12462 carl  174u     CHR               4,42        0t0      400 /dev/ttyra
> python    12462 carl  175u     CHR               4,43        0t0      402 /dev/ttyrb
> python    12462 carl  176u     CHR               4,44        0t0      404 /dev/ttyrc
> python    12462 carl  177u     CHR               4,45        0t0      406 /dev/ttyrd
> python    12462 carl  178u     CHR               4,46        0t0      408 /dev/ttyre
> python    12462 carl  179u     CHR               4,47        0t0      410 /dev/ttyrf
> python    12462 carl  180u     CHR               4,48        0t0      412 /dev/ttys0
> python    12462 carl  181u     CHR               4,49        0t0      414 /dev/ttys1
> python    12462 carl  182u     CHR               4,50        0t0      416 /dev/ttys2
> python    12462 carl  183u     CHR               4,51        0t0      418 /dev/ttys3
> python    12462 carl  184u     CHR               4,52        0t0      420 /dev/ttys4
> python    12462 carl  185u     CHR               4,53        0t0      422 /dev/ttys5
> python    12462 carl  186u     CHR               4,54        0t0      424 /dev/ttys6
> python    12462 carl  187u     CHR               4,55        0t0      426 /dev/ttys7
> python    12462 carl  188u     CHR               4,56        0t0      428 /dev/ttys8
> python    12462 carl  189u     CHR               4,57        0t0      430 /dev/ttys9
> python    12462 carl  190u     CHR               4,58        0t0      432 /dev/ttysa
> python    12462 carl  191u     CHR               4,59        0t0      434 /dev/ttysb
> python    12462 carl  192u     CHR               4,60        0t0      436 /dev/ttysc
> python    12462 carl  193u     CHR               4,61        0t0      438 /dev/ttysd
> python    12462 carl  194u     CHR               4,62        0t0      440 /dev/ttyse
> python    12462 carl  195u     CHR               4,63        0t0      442 /dev/ttysf
> python    12462 carl  196u     CHR               4,64        0t0      444 /dev/ttyt0
> python    12462 carl  197u     CHR               4,65        0t0      446 /dev/ttyt1
> python    12462 carl  198u     CHR               4,66        0t0      448 /dev/ttyt2
> python    12462 carl  199u     CHR               4,67        0t0      450 /dev/ttyt3
> python    12462 carl  200u     CHR               4,68        0t0      452 /dev/ttyt4
> python    12462 carl  201u     CHR               4,69        0t0      454 /dev/ttyt5
> python    12462 carl  202u     CHR               4,70        0t0      456 /dev/ttyt6
> python    12462 carl  203u     CHR               4,71        0t0      458 /dev/ttyt7
> python    12462 carl  204u     CHR               4,72        0t0      460 /dev/ttyt8
> python    12462 carl  205u     CHR               4,73        0t0      462 /dev/ttyt9
> python    12462 carl  206u     CHR               4,74        0t0      464 /dev/ttyta
> python    12462 carl  207u     CHR               4,75        0t0      466 /dev/ttytb
> python    12462 carl  208u     CHR               4,76        0t0      468 /dev/ttytc
> python    12462 carl  209u     CHR               4,77        0t0      470 /dev/ttytd
> python    12462 carl  210u     CHR               4,78        0t0      472 /dev/ttyte
> python    12462 carl  211u     CHR               4,79        0t0      474 /dev/ttytf
> python    12462 carl  212u     CHR               4,80        0t0      476 /dev/ttyu0
> python    12462 carl  213u     CHR               4,81        0t0      478 /dev/ttyu1
> python    12462 carl  214u     CHR               4,82        0t0      480 /dev/ttyu2
> python    12462 carl  215u     CHR               4,83        0t0      482 /dev/ttyu3
> python    12462 carl  216u     CHR               4,84        0t0      484 /dev/ttyu4
> python    12462 carl  217u     CHR               4,85        0t0      486 /dev/ttyu5
> python    12462 carl  218u     CHR               4,86        0t0      488 /dev/ttyu6
> python    12462 carl  219u     CHR               4,87        0t0      490 /dev/ttyu7
> python    12462 carl  220u     CHR               4,88        0t0      492 /dev/ttyu8
> python    12462 carl  221u     CHR               4,89        0t0      494 /dev/ttyu9
> python    12462 carl  222u     CHR               4,90        0t0      496 /dev/ttyua
> python    12462 carl  223u     CHR               4,91        0t0      498 /dev/ttyub
> python    12462 carl  224u     CHR               4,92        0t0      500 /dev/ttyuc
> python    12462 carl  225u     CHR               4,93        0t0      502 /dev/ttyud
> python    12462 carl  226u     CHR               4,94        0t0      504 /dev/ttyue
> python    12462 carl  227u     CHR               4,95        0t0      506 /dev/ttyuf
> python    12462 carl  228u     CHR               4,96        0t0      508 /dev/ttyv0
> python    12462 carl  229u     CHR               4,97        0t0      510 /dev/ttyv1
> python    12462 carl  230u     CHR               4,98        0t0      512 /dev/ttyv2
> python    12462 carl  231u     CHR               4,99        0t0      514 /dev/ttyv3
> python    12462 carl  232u     CHR              4,100        0t0      516 /dev/ttyv4
> python    12462 carl  233u     CHR              4,101        0t0      518 /dev/ttyv5
> python    12462 carl  234u     CHR              4,102        0t0      520 /dev/ttyv6
> python    12462 carl  235u     CHR              4,103        0t0      522 /dev/ttyv7
> python    12462 carl  236u     CHR              4,104        0t0      524 /dev/ttyv8
> python    12462 carl  237u     CHR              4,105        0t0      526 /dev/ttyv9
> python    12462 carl  238u     CHR              4,106        0t0      528 /dev/ttyva
> python    12462 carl  239u     CHR              4,107        0t0      530 /dev/ttyvb
> python    12462 carl  240     PIPE 0xffffff803d885b80      16384          ->0xffffff80263cc370
> python    12462 carl  242     PIPE 0xffffff803d884dc0      16384          ->0xffffff803d885970
> python    12463 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> python    12463 carl  txt      REG                1,2       8656 50109722 /Users/carl/Source/work/ttvm/.env/bin/python
> python    12463 carl  txt      REG                1,2    1543064  1839801 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python
> python    12463 carl  txt      REG                1,2      20824  1844694 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
> python    12463 carl  txt      REG                1,2      25124  1844690 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
> python    12463 carl  txt      REG                1,2      14368  1844672 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
> python    12463 carl  txt      REG                1,2      32588  1844654 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
> python    12463 carl  txt      REG                1,2      19552  1844662 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
> python    12463 carl  txt      REG                1,2      20732  1844669 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
> python    12463 carl  txt      REG                1,2      31736  1844611 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
> python    12463 carl  txt      REG                1,2      32952  1844684 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
> python    12463 carl  txt      REG                1,2      54820  1844678 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
> python    12463 carl  txt      REG                1,2       9836  1844600 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
> python    12463 carl  txt      REG                1,2      18468  1844627 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
> python    12463 carl  txt      REG                1,2      28648  1844691 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
> python    12463 carl  txt      REG                1,2      37768  1844659 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
> python    12463 carl  txt      REG                1,2      80076  1844670 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
> python    12463 carl  txt      REG                1,2     204728 50113116 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/_yaml.so
> python    12463 carl  txt      REG                1,2     118252 31530320 /usr/local/Cellar/libyaml/0.1.4/lib/libyaml-0.2.dylib
> python    12463 carl  txt      REG                1,2      14968  1844625 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
> python    12463 carl  txt      REG                1,2      15444  1844636 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
> python    12463 carl  txt      REG                1,2      72644  1844667 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
> python    12463 carl  txt      REG                1,2      55460  1844651 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
> python    12463 carl  txt      REG                1,2      30796  1844653 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
> python    12463 carl  txt      REG                1,2      11292  1844648 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
> python    12463 carl  txt      REG                1,2      10532 50113456 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/markupsafe/_speedups.so
> python    12463 carl  txt      REG                1,2      35648  1844680 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
> python    12463 carl  txt      REG                1,2      16724  1844626 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
> python    12463 carl  txt      REG                1,2      15188  1844646 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
> python    12463 carl  txt      REG                1,2      31004  1844633 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
> python    12463 carl  txt      REG                1,2      19152  1844693 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/termios.so
> python    12463 carl  txt      REG                1,2      27148  1844641 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
> python    12463 carl  txt      REG                1,2      69932  1844616 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
> python    12463 carl  txt      REG                1,2      67636 50114110 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/PublicKey/_fastmath.so
> python    12463 carl  txt      REG                1,2     433484 50540203 /usr/local/Cellar/gmp/5.1.2/lib/libgmp.10.dylib
> python    12463 carl  txt      REG                1,2      14544 50114205 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Util/_counter.so
> python    12463 carl  txt      REG                1,2      31872 50114062 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Cipher/_AES.so
> python    12463 carl  txt      REG                1,2       9992  1844676 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
> python    12463 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> python    12463 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> python    12463 carl    0u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12463 carl    1u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12463 carl    2u     CHR               16,1    0t34857    11597 /dev/ttys001
> python    12463 carl    3r     REG                1,2      24397 50295415 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/ansible/playbook/__init__.py
> python    12463 carl    4r     REG                1,2      40644 50295419 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/ansible/runner/__init__.py
> python    12463 carl    5u     REG                1,2          0 51372895 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpqw21Zf
> python    12463 carl    6w     REG                1,2          0 50980754 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/.ansible-lock.501
> python    12463 carl    7u     REG                1,2          0 51372896 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpAVnQOf
> python    12463 carl    8     PIPE 0xffffff8032197b50      16384          
> python    12463 carl   10r     CHR                3,2        0t0      305 /dev/null
> python    12463 carl   11u    unix  0x336b99d9d006edb        0t0          /var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/pymp-puX5bo/listener-PhcADh
> ssh       12468 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> ssh       12468 carl  txt      REG                1,2     732128     6172 /usr/bin/ssh
> ssh       12468 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> ssh       12468 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> ssh       12468 carl    0u     CHR                3,2        0t0      305 /dev/null
> ssh       12468 carl    1u     CHR                3,2        0t0      305 /dev/null
> ssh       12468 carl    2     PIPE 0xffffff80321958f0      16384          
> ssh       12468 carl    3u    IPv4  0x336b99da50d1753        0t0      TCP 192.168.33.1:58001->192.168.33.10:ssh (ESTABLISHED)
> ssh       12468 carl    4u    unix  0x336b99da1eefdb3        0t0          /tmp/ansible-ssh-192.168.33.10-22-vagrant.sCNYDqlIP6bpEAqX
> ssh       12468 carl    5u    unix  0x336b99d9cf653eb        0t0          /tmp/ansible-ssh-192.168.33.10-22-vagrant.sCNYDqlIP6bpEAqX
> ssh       12468 carl    6u     CHR              4,107        0t0      530 /dev/ttyvb
> ssh       12468 carl    7     PIPE 0xffffff80263cc370      16384          ->0xffffff803d885b80
> ssh       12468 carl    8     PIPE 0xffffff803d885970      16384          ->0xffffff803d884dc0
> ssh       13084 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> ssh       13084 carl  txt      REG                1,2     732128     6172 /usr/bin/ssh
> ssh       13084 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> ssh       13084 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> ssh       13084 carl    0u     CHR              4,107        0t0      530 /dev/ttyvb
> ssh       13084 carl    1     PIPE 0xffffff80263cc370      16384          ->0xffffff803d885b80
> ssh       13084 carl    2     PIPE 0xffffff803d885970      16384          ->0xffffff803d884dc0
> ssh       13084 carl    3u    unix  0x336b99db3c86ee3        0t0          ->0x336b99d9cf653eb
> lsof      13085 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> lsof      13085 carl  txt      REG                1,2     113328    10566 /usr/sbin/lsof
> lsof      13085 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> lsof      13085 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> lsof      13085 carl    0u     CHR               16,0   0t692290    11343 /dev/ttys000
> lsof      13085 carl    1w     REG                1,2          0 51372286 /Users/carl/Source/work/ttvm/after.txt
> lsof      13085 carl    2u     CHR               16,0   0t692290    11343 /dev/ttys000
> lsof      13085 carl    4     PIPE 0xffffff803d884fd0      16384          ->0xffffff803d885fa0
> lsof      13085 carl    5     PIPE 0xffffff8032195000      16384          ->0xffffff803d887a70
</pre>

To elaborate on what James has said,

We are _intentionally_ duping stdin, in order to allow for prompting about
SSH keys from multiple processes.

Perhaps there's something about them not being well closed on OS X that we
need to investigate, though the number of reports about this is still
extremely low.

On Fri, Aug 23, 2013 at 7:43 AM, Carl Youngblood
notifications@github.comwrote:

> I've done an lsof before running ansible and right before the failure
> takes place, then diffed the result. Here is the lion's share of the new
> open files that happen during provisioning. Looks like there are a lot of
> TTY sockets. Don't know why offhand, but maybe this is useful.
> 
> > bash      11368 carl    0u     CHR               16,1    0t34857    11597 /dev/ttys001
> > bash      11368 carl    1u     CHR               16,1    0t34857    11597 /dev/ttys001
> > bash      11368 carl    2u     CHR               16,1    0t34857    11597 /dev/ttys001
> > bash      11368 carl  255u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12462 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> > python    12462 carl  txt      REG                1,2       8656 50109722 /Users/carl/Source/work/ttvm/.env/bin/python
> > python    12462 carl  txt      REG                1,2    1543064  1839801 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python
> > python    12462 carl  txt      REG                1,2      20824  1844694 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
> > python    12462 carl  txt      REG                1,2      25124  1844690 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
> > python    12462 carl  txt      REG                1,2      14368  1844672 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
> > python    12462 carl  txt      REG                1,2      32588  1844654 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
> > python    12462 carl  txt      REG                1,2      19552  1844662 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
> > python    12462 carl  txt      REG                1,2      20732  1844669 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
> > python    12462 carl  txt      REG                1,2      31736  1844611 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
> > python    12462 carl  txt      REG                1,2      32952  1844684 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
> > python    12462 carl  txt      REG                1,2      54820  1844678 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
> > python    12462 carl  txt      REG                1,2       9836  1844600 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
> > python    12462 carl  txt      REG                1,2      18468  1844627 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
> > python    12462 carl  txt      REG                1,2      28648  1844691 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
> > python    12462 carl  txt      REG                1,2      37768  1844659 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
> > python    12462 carl  txt      REG                1,2      80076  1844670 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
> > python    12462 carl  txt      REG                1,2     204728 50113116 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/_yaml.so
> > python    12462 carl  txt      REG                1,2     118252 31530320 /usr/local/Cellar/libyaml/0.1.4/lib/libyaml-0.2.dylib
> > python    12462 carl  txt      REG                1,2      14968  1844625 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
> > python    12462 carl  txt      REG                1,2      15444  1844636 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
> > python    12462 carl  txt      REG                1,2      72644  1844667 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
> > python    12462 carl  txt      REG                1,2      55460  1844651 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
> > python    12462 carl  txt      REG                1,2      30796  1844653 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
> > python    12462 carl  txt      REG                1,2      11292  1844648 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
> > python    12462 carl  txt      REG                1,2      10532 50113456 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/markupsafe/_speedups.so
> > python    12462 carl  txt      REG                1,2      35648  1844680 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
> > python    12462 carl  txt      REG                1,2      16724  1844626 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
> > python    12462 carl  txt      REG                1,2      15188  1844646 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
> > python    12462 carl  txt      REG                1,2      31004  1844633 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
> > python    12462 carl  txt      REG                1,2      19152  1844693 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/termios.so
> > python    12462 carl  txt      REG                1,2      27148  1844641 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
> > python    12462 carl  txt      REG                1,2      69932  1844616 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
> > python    12462 carl  txt      REG                1,2      67636 50114110 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/PublicKey/_fastmath.so
> > python    12462 carl  txt      REG                1,2     433484 50540203 /usr/local/Cellar/gmp/5.1.2/lib/libgmp.10.dylib
> > python    12462 carl  txt      REG                1,2      14544 50114205 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Util/_counter.so
> > python    12462 carl  txt      REG                1,2      31872 50114062 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Cipher/_AES.so
> > python    12462 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> > python    12462 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> > python    12462 carl    0u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12462 carl    1u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12462 carl    2u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12462 carl    3u     CHR              5,107        0t0      531 /dev/ptyvb
> > python    12462 carl    4u     CHR               16,2        0t0    11849 /dev/ttys002
> > python    12462 carl    5u     REG                1,2          0 51372895 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpqw21Zf
> > python    12462 carl    6w     REG                1,2          0 50980754 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/.ansible-lock.501
> > python    12462 carl    7u     REG                1,2          0 51372896 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpAVnQOf
> > python    12462 carl    8u     CHR               16,3        0t0    11851 /dev/ttys003
> > python    12462 carl    9u     CHR               16,4        0t0    11853 /dev/ttys004
> > python    12462 carl   10u     CHR               16,5        0t0    11855 /dev/ttys005
> > python    12462 carl   11u     CHR               16,6        0t0    11857 /dev/ttys006
> > python    12462 carl   12u     CHR               16,7        0t0    11859 /dev/ttys007
> > python    12462 carl   13u     CHR               16,8        0t0    11861 /dev/ttys008
> > python    12462 carl   14u     CHR               16,9        0t0    11863 /dev/ttys009
> > python    12462 carl   15u     CHR              16,10        0t0    11865 /dev/ttys010
> > python    12462 carl   16u     CHR              16,11        0t0    11867 /dev/ttys011
> > python    12462 carl   17u     CHR              16,12        0t0    11869 /dev/ttys012
> > python    12462 carl   18u     CHR              16,13        0t0    11871 /dev/ttys013
> > python    12462 carl   19u     CHR              16,14        0t0    11873 /dev/ttys014
> > python    12462 carl   20u     CHR              16,15        0t0    11875 /dev/ttys015
> > python    12462 carl   21u     CHR              16,16        0t0    11877 /dev/ttys016
> > python    12462 carl   22u     CHR              16,17        0t0    11879 /dev/ttys017
> > python    12462 carl   23u     CHR              16,18        0t0    11881 /dev/ttys018
> > python    12462 carl   24u     CHR              16,19        0t0    11883 /dev/ttys019
> > python    12462 carl   25u     CHR              16,20        0t0    11885 /dev/ttys020
> > python    12462 carl   26u     CHR              16,21        0t0    11887 /dev/ttys021
> > python    12462 carl   27u     CHR              16,22        0t0    11889 /dev/ttys022
> > python    12462 carl   28u     CHR              16,23        0t0    11891 /dev/ttys023
> > python    12462 carl   29u     CHR              16,24        0t0    11893 /dev/ttys024
> > python    12462 carl   30u     CHR              16,25        0t0    11895 /dev/ttys025
> > python    12462 carl   31u     CHR              16,26        0t0    11897 /dev/ttys026
> > python    12462 carl   32u     CHR              16,27        0t0    11899 /dev/ttys027
> > python    12462 carl   33u     CHR              16,28        0t0    11901 /dev/ttys028
> > python    12462 carl   34u     CHR              16,29        0t0    11903 /dev/ttys029
> > python    12462 carl   35u     CHR              16,30        0t0    11905 /dev/ttys030
> > python    12462 carl   36u     CHR              16,31        0t0    11907 /dev/ttys031
> > python    12462 carl   37u     CHR              16,32        0t0    11909 /dev/ttys032
> > python    12462 carl   38u     CHR              16,33        0t0    11911 /dev/ttys033
> > python    12462 carl   39u     CHR              16,34        0t0    11913 /dev/ttys034
> > python    12462 carl   40u     CHR              16,35        0t0    11915 /dev/ttys035
> > python    12462 carl   41u     CHR              16,36        0t0    11917 /dev/ttys036
> > python    12462 carl   42u     CHR              16,37        0t0    11919 /dev/ttys037
> > python    12462 carl   43u     CHR              16,38        0t0    11921 /dev/ttys038
> > python    12462 carl   44u     CHR              16,39        0t0    11923 /dev/ttys039
> > python    12462 carl   45u     CHR              16,40        0t0    11925 /dev/ttys040
> > python    12462 carl   46u     CHR              16,41        0t0    11927 /dev/ttys041
> > python    12462 carl   47u     CHR              16,42        0t0    11929 /dev/ttys042
> > python    12462 carl   48u     CHR              16,43        0t0    11931 /dev/ttys043
> > python    12462 carl   49u     CHR              16,44        0t0    11933 /dev/ttys044
> > python    12462 carl   50u     CHR              16,45        0t0    11935 /dev/ttys045
> > python    12462 carl   51u     CHR              16,46        0t0    11937 /dev/ttys046
> > python    12462 carl   52u     CHR              16,47        0t0    11939 /dev/ttys047
> > python    12462 carl   53u     CHR              16,48        0t0    11941 /dev/ttys048
> > python    12462 carl   54u     CHR              16,49        0t0    11943 /dev/ttys049
> > python    12462 carl   55u     CHR              16,50        0t0    11945 /dev/ttys050
> > python    12462 carl   56u     CHR              16,51        0t0    11947 /dev/ttys051
> > python    12462 carl   57u     CHR              16,52        0t0    11949 /dev/ttys052
> > python    12462 carl   58u     CHR              16,53        0t0    11951 /dev/ttys053
> > python    12462 carl   59u     CHR              16,54        0t0    11953 /dev/ttys054
> > python    12462 carl   60u     CHR              16,55        0t0    11955 /dev/ttys055
> > python    12462 carl   61u     CHR              16,56        0t0    11957 /dev/ttys056
> > python    12462 carl   62u     CHR              16,57        0t0    11959 /dev/ttys057
> > python    12462 carl   63u     CHR              16,58        0t0    11961 /dev/ttys058
> > python    12462 carl   64u     CHR              16,59        0t0    11963 /dev/ttys059
> > python    12462 carl   65u     CHR              16,60        0t0    11965 /dev/ttys060
> > python    12462 carl   66u     CHR              16,61        0t0    11967 /dev/ttys061
> > python    12462 carl   67u     CHR              16,62        0t0    11969 /dev/ttys062
> > python    12462 carl   68u     CHR              16,63        0t0    11971 /dev/ttys063
> > python    12462 carl   69u     CHR              16,64        0t0    11973 /dev/ttys064
> > python    12462 carl   70u     CHR              16,65        0t0    11975 /dev/ttys065
> > python    12462 carl   71u     CHR              16,66        0t0    11977 /dev/ttys066
> > python    12462 carl   72u     CHR              16,67        0t0    11979 /dev/ttys067
> > python    12462 carl   73u     CHR              16,68        0t0    11981 /dev/ttys068
> > python    12462 carl   74u     CHR              16,69        0t0    11983 /dev/ttys069
> > python    12462 carl   75u     CHR              16,70        0t0    11985 /dev/ttys070
> > python    12462 carl   76u     CHR              16,71        0t0    11987 /dev/ttys071
> > python    12462 carl   77u     CHR              16,72        0t0    11989 /dev/ttys072
> > python    12462 carl   78u     CHR              16,73        0t0    11991 /dev/ttys073
> > python    12462 carl   79u     CHR              16,74        0t0    11993 /dev/ttys074
> > python    12462 carl   80u     CHR              16,75        0t0    11995 /dev/ttys075
> > python    12462 carl   81u     CHR              16,76        0t0    11997 /dev/ttys076
> > python    12462 carl   82u     CHR              16,77        0t0    11999 /dev/ttys077
> > python    12462 carl   83u     CHR              16,78        0t0    12001 /dev/ttys078
> > python    12462 carl   84u     CHR              16,79        0t0    12003 /dev/ttys079
> > python    12462 carl   85u     CHR              16,80        0t0    12005 /dev/ttys080
> > python    12462 carl   86u     CHR              16,81        0t0    12007 /dev/ttys081
> > python    12462 carl   87u     CHR              16,82        0t0    12009 /dev/ttys082
> > python    12462 carl   88u     CHR              16,83        0t0    12011 /dev/ttys083
> > python    12462 carl   89u     CHR              16,84        0t0    12013 /dev/ttys084
> > python    12462 carl   90u     CHR              16,85        0t0    12015 /dev/ttys085
> > python    12462 carl   91u     CHR              16,86        0t0    12017 /dev/ttys086
> > python    12462 carl   92u     CHR              16,87        0t0    12019 /dev/ttys087
> > python    12462 carl   93u     CHR              16,88        0t0    12021 /dev/ttys088
> > python    12462 carl   94u     CHR              16,89        0t0    12023 /dev/ttys089
> > python    12462 carl   95u     CHR              16,90        0t0    12025 /dev/ttys090
> > python    12462 carl   96u     CHR              16,91        0t0    12027 /dev/ttys091
> > python    12462 carl   97u     CHR              16,92        0t0    12029 /dev/ttys092
> > python    12462 carl   98u     CHR              16,93        0t0    12031 /dev/ttys093
> > python    12462 carl   99u     CHR              16,94        0t0    12033 /dev/ttys094
> > python    12462 carl  100u     CHR              16,95        0t0    12035 /dev/ttys095
> > python    12462 carl  101u     CHR              16,96        0t0    12037 /dev/ttys096
> > python    12462 carl  102u     CHR              16,97        0t0    12039 /dev/ttys097
> > python    12462 carl  103u     CHR              16,98        0t0    12041 /dev/ttys098
> > python    12462 carl  104u     CHR              16,99        0t0    12043 /dev/ttys099
> > python    12462 carl  105u     CHR             16,100        0t0    12045 /dev/ttys100
> > python    12462 carl  106u     CHR             16,101        0t0    12047 /dev/ttys101
> > python    12462 carl  107u     CHR             16,102        0t0    12049 /dev/ttys102
> > python    12462 carl  108u     CHR             16,103        0t0    12051 /dev/ttys103
> > python    12462 carl  109u     CHR             16,104        0t0    12053 /dev/ttys104
> > python    12462 carl  110u     CHR             16,105        0t0    12055 /dev/ttys105
> > python    12462 carl  111u     CHR             16,106        0t0    12057 /dev/ttys106
> > python    12462 carl  112u     CHR             16,107        0t0    12059 /dev/ttys107
> > python    12462 carl  113u     CHR             16,108        0t0    12061 /dev/ttys108
> > python    12462 carl  114u     CHR             16,109        0t0    12063 /dev/ttys109
> > python    12462 carl  115u     CHR             16,110        0t0    12065 /dev/ttys110
> > python    12462 carl  116u     CHR             16,111        0t0    12067 /dev/ttys111
> > python    12462 carl  117u     CHR             16,112        0t0    12069 /dev/ttys112
> > python    12462 carl  118u     CHR             16,113        0t0    12071 /dev/ttys113
> > python    12462 carl  119u     CHR             16,114        0t0    12073 /dev/ttys114
> > python    12462 carl  120u     CHR             16,115        0t0    12075 /dev/ttys115
> > python    12462 carl  121u     CHR             16,116        0t0    12077 /dev/ttys116
> > python    12462 carl  122u     CHR             16,117        0t0    12079 /dev/ttys117
> > python    12462 carl  123u     CHR             16,118        0t0    12081 /dev/ttys118
> > python    12462 carl  124u     CHR             16,119        0t0    12083 /dev/ttys119
> > python    12462 carl  125u     CHR             16,120        0t0    12085 /dev/ttys120
> > python    12462 carl  126u     CHR             16,121        0t0    12087 /dev/ttys121
> > python    12462 carl  127u     CHR             16,122        0t0    12089 /dev/ttys122
> > python    12462 carl  128u     CHR             16,123        0t0    12091 /dev/ttys123
> > python    12462 carl  129u     CHR             16,124        0t0    12093 /dev/ttys124
> > python    12462 carl  130u     CHR             16,125        0t0    12095 /dev/ttys125
> > python    12462 carl  131u     CHR             16,126        0t0    12097 /dev/ttys126
> > python    12462 carl  132u     CHR                4,0        0t0      316 /dev/ttyp0
> > python    12462 carl  133u     CHR                4,1        0t0      318 /dev/ttyp1
> > python    12462 carl  134u     CHR                4,2        0t0      320 /dev/ttyp2
> > python    12462 carl  135u     CHR                4,3        0t0      322 /dev/ttyp3
> > python    12462 carl  136u     CHR                4,4        0t0      324 /dev/ttyp4
> > python    12462 carl  137u     CHR                4,5        0t0      326 /dev/ttyp5
> > python    12462 carl  138u     CHR                4,6        0t0      328 /dev/ttyp6
> > python    12462 carl  139u     CHR                4,7        0t0      330 /dev/ttyp7
> > python    12462 carl  140u     CHR                4,8        0t0      332 /dev/ttyp8
> > python    12462 carl  141u     CHR                4,9        0t0      334 /dev/ttyp9
> > python    12462 carl  142u     CHR               4,10        0t0      336 /dev/ttypa
> > python    12462 carl  143u     CHR               4,11        0t0      338 /dev/ttypb
> > python    12462 carl  144u     CHR               4,12        0t0      340 /dev/ttypc
> > python    12462 carl  145u     CHR               4,13        0t0      342 /dev/ttypd
> > python    12462 carl  146u     CHR               4,14        0t0      344 /dev/ttype
> > python    12462 carl  147u     CHR               4,15        0t0      346 /dev/ttypf
> > python    12462 carl  148u     CHR               4,16        0t0      348 /dev/ttyq0
> > python    12462 carl  149u     CHR               4,17        0t0      350 /dev/ttyq1
> > python    12462 carl  150u     CHR               4,18        0t0      352 /dev/ttyq2
> > python    12462 carl  151u     CHR               4,19        0t0      354 /dev/ttyq3
> > python    12462 carl  152u     CHR               4,20        0t0      356 /dev/ttyq4
> > python    12462 carl  153u     CHR               4,21        0t0      358 /dev/ttyq5
> > python    12462 carl  154u     CHR               4,22        0t0      360 /dev/ttyq6
> > python    12462 carl  155u     CHR               4,23        0t0      362 /dev/ttyq7
> > python    12462 carl  156u     CHR               4,24        0t0      364 /dev/ttyq8
> > python    12462 carl  157u     CHR               4,25        0t0      366 /dev/ttyq9
> > python    12462 carl  158u     CHR               4,26        0t0      368 /dev/ttyqa
> > python    12462 carl  159u     CHR               4,27        0t0      370 /dev/ttyqb
> > python    12462 carl  160u     CHR               4,28        0t0      372 /dev/ttyqc
> > python    12462 carl  161u     CHR               4,29        0t0      374 /dev/ttyqd
> > python    12462 carl  162u     CHR               4,30        0t0      376 /dev/ttyqe
> > python    12462 carl  163u     CHR               4,31        0t0      378 /dev/ttyqf
> > python    12462 carl  164u     CHR               4,32        0t0      380 /dev/ttyr0
> > python    12462 carl  165u     CHR               4,33        0t0      382 /dev/ttyr1
> > python    12462 carl  166u     CHR               4,34        0t0      384 /dev/ttyr2
> > python    12462 carl  167u     CHR               4,35        0t0      386 /dev/ttyr3
> > python    12462 carl  168u     CHR               4,36        0t0      388 /dev/ttyr4
> > python    12462 carl  169u     CHR               4,37        0t0      390 /dev/ttyr5
> > python    12462 carl  170u     CHR               4,38        0t0      392 /dev/ttyr6
> > python    12462 carl  171u     CHR               4,39        0t0      394 /dev/ttyr7
> > python    12462 carl  172u     CHR               4,40        0t0      396 /dev/ttyr8
> > python    12462 carl  173u     CHR               4,41        0t0      398 /dev/ttyr9
> > python    12462 carl  174u     CHR               4,42        0t0      400 /dev/ttyra
> > python    12462 carl  175u     CHR               4,43        0t0      402 /dev/ttyrb
> > python    12462 carl  176u     CHR               4,44        0t0      404 /dev/ttyrc
> > python    12462 carl  177u     CHR               4,45        0t0      406 /dev/ttyrd
> > python    12462 carl  178u     CHR               4,46        0t0      408 /dev/ttyre
> > python    12462 carl  179u     CHR               4,47        0t0      410 /dev/ttyrf
> > python    12462 carl  180u     CHR               4,48        0t0      412 /dev/ttys0
> > python    12462 carl  181u     CHR               4,49        0t0      414 /dev/ttys1
> > python    12462 carl  182u     CHR               4,50        0t0      416 /dev/ttys2
> > python    12462 carl  183u     CHR               4,51        0t0      418 /dev/ttys3
> > python    12462 carl  184u     CHR               4,52        0t0      420 /dev/ttys4
> > python    12462 carl  185u     CHR               4,53        0t0      422 /dev/ttys5
> > python    12462 carl  186u     CHR               4,54        0t0      424 /dev/ttys6
> > python    12462 carl  187u     CHR               4,55        0t0      426 /dev/ttys7
> > python    12462 carl  188u     CHR               4,56        0t0      428 /dev/ttys8
> > python    12462 carl  189u     CHR               4,57        0t0      430 /dev/ttys9
> > python    12462 carl  190u     CHR               4,58        0t0      432 /dev/ttysa
> > python    12462 carl  191u     CHR               4,59        0t0      434 /dev/ttysb
> > python    12462 carl  192u     CHR               4,60        0t0      436 /dev/ttysc
> > python    12462 carl  193u     CHR               4,61        0t0      438 /dev/ttysd
> > python    12462 carl  194u     CHR               4,62        0t0      440 /dev/ttyse
> > python    12462 carl  195u     CHR               4,63        0t0      442 /dev/ttysf
> > python    12462 carl  196u     CHR               4,64        0t0      444 /dev/ttyt0
> > python    12462 carl  197u     CHR               4,65        0t0      446 /dev/ttyt1
> > python    12462 carl  198u     CHR               4,66        0t0      448 /dev/ttyt2
> > python    12462 carl  199u     CHR               4,67        0t0      450 /dev/ttyt3
> > python    12462 carl  200u     CHR               4,68        0t0      452 /dev/ttyt4
> > python    12462 carl  201u     CHR               4,69        0t0      454 /dev/ttyt5
> > python    12462 carl  202u     CHR               4,70        0t0      456 /dev/ttyt6
> > python    12462 carl  203u     CHR               4,71        0t0      458 /dev/ttyt7
> > python    12462 carl  204u     CHR               4,72        0t0      460 /dev/ttyt8
> > python    12462 carl  205u     CHR               4,73        0t0      462 /dev/ttyt9
> > python    12462 carl  206u     CHR               4,74        0t0      464 /dev/ttyta
> > python    12462 carl  207u     CHR               4,75        0t0      466 /dev/ttytb
> > python    12462 carl  208u     CHR               4,76        0t0      468 /dev/ttytc
> > python    12462 carl  209u     CHR               4,77        0t0      470 /dev/ttytd
> > python    12462 carl  210u     CHR               4,78        0t0      472 /dev/ttyte
> > python    12462 carl  211u     CHR               4,79        0t0      474 /dev/ttytf
> > python    12462 carl  212u     CHR               4,80        0t0      476 /dev/ttyu0
> > python    12462 carl  213u     CHR               4,81        0t0      478 /dev/ttyu1
> > python    12462 carl  214u     CHR               4,82        0t0      480 /dev/ttyu2
> > python    12462 carl  215u     CHR               4,83        0t0      482 /dev/ttyu3
> > python    12462 carl  216u     CHR               4,84        0t0      484 /dev/ttyu4
> > python    12462 carl  217u     CHR               4,85        0t0      486 /dev/ttyu5
> > python    12462 carl  218u     CHR               4,86        0t0      488 /dev/ttyu6
> > python    12462 carl  219u     CHR               4,87        0t0      490 /dev/ttyu7
> > python    12462 carl  220u     CHR               4,88        0t0      492 /dev/ttyu8
> > python    12462 carl  221u     CHR               4,89        0t0      494 /dev/ttyu9
> > python    12462 carl  222u     CHR               4,90        0t0      496 /dev/ttyua
> > python    12462 carl  223u     CHR               4,91        0t0      498 /dev/ttyub
> > python    12462 carl  224u     CHR               4,92        0t0      500 /dev/ttyuc
> > python    12462 carl  225u     CHR               4,93        0t0      502 /dev/ttyud
> > python    12462 carl  226u     CHR               4,94        0t0      504 /dev/ttyue
> > python    12462 carl  227u     CHR               4,95        0t0      506 /dev/ttyuf
> > python    12462 carl  228u     CHR               4,96        0t0      508 /dev/ttyv0
> > python    12462 carl  229u     CHR               4,97        0t0      510 /dev/ttyv1
> > python    12462 carl  230u     CHR               4,98        0t0      512 /dev/ttyv2
> > python    12462 carl  231u     CHR               4,99        0t0      514 /dev/ttyv3
> > python    12462 carl  232u     CHR              4,100        0t0      516 /dev/ttyv4
> > python    12462 carl  233u     CHR              4,101        0t0      518 /dev/ttyv5
> > python    12462 carl  234u     CHR              4,102        0t0      520 /dev/ttyv6
> > python    12462 carl  235u     CHR              4,103        0t0      522 /dev/ttyv7
> > python    12462 carl  236u     CHR              4,104        0t0      524 /dev/ttyv8
> > python    12462 carl  237u     CHR              4,105        0t0      526 /dev/ttyv9
> > python    12462 carl  238u     CHR              4,106        0t0      528 /dev/ttyva
> > python    12462 carl  239u     CHR              4,107        0t0      530 /dev/ttyvb
> > python    12462 carl  240     PIPE 0xffffff803d885b80      16384          ->0xffffff80263cc370
> > python    12462 carl  242     PIPE 0xffffff803d884dc0      16384          ->0xffffff803d885970
> > python    12463 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> > python    12463 carl  txt      REG                1,2       8656 50109722 /Users/carl/Source/work/ttvm/.env/bin/python
> > python    12463 carl  txt      REG                1,2    1543064  1839801 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python
> > python    12463 carl  txt      REG                1,2      20824  1844694 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
> > python    12463 carl  txt      REG                1,2      25124  1844690 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/select.so
> > python    12463 carl  txt      REG                1,2      14368  1844672 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
> > python    12463 carl  txt      REG                1,2      32588  1844654 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
> > python    12463 carl  txt      REG                1,2      19552  1844662 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
> > python    12463 carl  txt      REG                1,2      20732  1844669 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
> > python    12463 carl  txt      REG                1,2      31736  1844611 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
> > python    12463 carl  txt      REG                1,2      32952  1844684 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
> > python    12463 carl  txt      REG                1,2      54820  1844678 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
> > python    12463 carl  txt      REG                1,2       9836  1844600 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_bisect.so
> > python    12463 carl  txt      REG                1,2      18468  1844627 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
> > python    12463 carl  txt      REG                1,2      28648  1844691 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
> > python    12463 carl  txt      REG                1,2      37768  1844659 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
> > python    12463 carl  txt      REG                1,2      80076  1844670 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
> > python    12463 carl  txt      REG                1,2     204728 50113116 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/_yaml.so
> > python    12463 carl  txt      REG                1,2     118252 31530320 /usr/local/Cellar/libyaml/0.1.4/lib/libyaml-0.2.dylib
> > python    12463 carl  txt      REG                1,2      14968  1844625 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
> > python    12463 carl  txt      REG                1,2      15444  1844636 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
> > python    12463 carl  txt      REG                1,2      72644  1844667 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
> > python    12463 carl  txt      REG                1,2      55460  1844651 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so
> > python    12463 carl  txt      REG                1,2      30796  1844653 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
> > python    12463 carl  txt      REG                1,2      11292  1844648 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_scproxy.so
> > python    12463 carl  txt      REG                1,2      10532 50113456 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/markupsafe/_speedups.so
> > python    12463 carl  txt      REG                1,2      35648  1844680 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
> > python    12463 carl  txt      REG                1,2      16724  1844626 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
> > python    12463 carl  txt      REG                1,2      15188  1844646 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
> > python    12463 carl  txt      REG                1,2      31004  1844633 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_json.so
> > python    12463 carl  txt      REG                1,2      19152  1844693 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/termios.so
> > python    12463 carl  txt      REG                1,2      27148  1844641 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_multiprocessing.so
> > python    12463 carl  txt      REG                1,2      69932  1844616 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_curses.so
> > python    12463 carl  txt      REG                1,2      67636 50114110 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/PublicKey/_fastmath.so
> > python    12463 carl  txt      REG                1,2     433484 50540203 /usr/local/Cellar/gmp/5.1.2/lib/libgmp.10.dylib
> > python    12463 carl  txt      REG                1,2      14544 50114205 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Util/_counter.so
> > python    12463 carl  txt      REG                1,2      31872 50114062 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/Crypto/Cipher/_AES.so
> > python    12463 carl  txt      REG                1,2       9992  1844676 /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
> > python    12463 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> > python    12463 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> > python    12463 carl    0u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12463 carl    1u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12463 carl    2u     CHR               16,1    0t34857    11597 /dev/ttys001
> > python    12463 carl    3r     REG                1,2      24397 50295415 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/ansible/playbook/__init__.py
> > python    12463 carl    4r     REG                1,2      40644 50295419 /Users/carl/Source/work/ttvm/.env/lib/python2.7/site-packages/ansible/runner/**init**.py
> > python    12463 carl    5u     REG                1,2          0 51372895 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpqw21Zf
> > python    12463 carl    6w     REG                1,2          0 50980754 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/.ansible-lock.501
> > python    12463 carl    7u     REG                1,2          0 51372896 /private/var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/tmpAVnQOf
> > python    12463 carl    8     PIPE 0xffffff8032197b50      16384
> > python    12463 carl   10r     CHR                3,2        0t0      305 /dev/null
> > python    12463 carl   11u    unix  0x336b99d9d006edb        0t0          /var/folders/h_/8m3k_yvx1zvdbc97x47g4g080000gn/T/pymp-puX5bo/listener-PhcADh
> > ssh       12468 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> > ssh       12468 carl  txt      REG                1,2     732128     6172 /usr/bin/ssh
> > ssh       12468 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> > ssh       12468 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> > ssh       12468 carl    0u     CHR                3,2        0t0      305 /dev/null
> > ssh       12468 carl    1u     CHR                3,2        0t0      305 /dev/null
> > ssh       12468 carl    2     PIPE 0xffffff80321958f0      16384
> > ssh       12468 carl    3u    IPv4  0x336b99da50d1753        0t0      TCP 192.168.33.1:58001->192.168.33.10:ssh (ESTABLISHED)
> > ssh       12468 carl    4u    unix  0x336b99da1eefdb3        0t0          /tmp/ansible-ssh-192.168.33.10-22-vagrant.sCNYDqlIP6bpEAqX
> > ssh       12468 carl    5u    unix  0x336b99d9cf653eb        0t0          /tmp/ansible-ssh-192.168.33.10-22-vagrant.sCNYDqlIP6bpEAqX
> > ssh       12468 carl    6u     CHR              4,107        0t0      530 /dev/ttyvb
> > ssh       12468 carl    7     PIPE 0xffffff80263cc370      16384          ->0xffffff803d885b80
> > ssh       12468 carl    8     PIPE 0xffffff803d885970      16384          ->0xffffff803d884dc0
> > ssh       13084 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> > ssh       13084 carl  txt      REG                1,2     732128     6172 /usr/bin/ssh
> > ssh       13084 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> > ssh       13084 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> > ssh       13084 carl    0u     CHR              4,107        0t0      530 /dev/ttyvb
> > ssh       13084 carl    1     PIPE 0xffffff80263cc370      16384          ->0xffffff803d885b80
> > ssh       13084 carl    2     PIPE 0xffffff803d885970      16384          ->0xffffff803d884dc0
> > ssh       13084 carl    3u    unix  0x336b99db3c86ee3        0t0          ->0x336b99d9cf653eb
> > lsof      13085 carl  cwd      DIR                1,2       1190 31525169 /Users/carl/Source/work/ttvm
> > lsof      13085 carl  txt      REG                1,2     113328    10566 /usr/sbin/lsof
> > lsof      13085 carl  txt      REG                1,2     600576    10192 /usr/lib/dyld
> > lsof      13085 carl  txt      REG                1,2  302141440 38883722 /private/var/db/dyld/dyld_shared_cache_x86_64
> > lsof      13085 carl    0u     CHR               16,0   0t692290    11343 /dev/ttys000
> > lsof      13085 carl    1w     REG                1,2          0 51372286 /Users/carl/Source/work/ttvm/after.txt
> > lsof      13085 carl    2u     CHR               16,0   0t692290    11343 /dev/ttys000
> > lsof      13085 carl    4     PIPE 0xffffff803d884fd0      16384          ->0xffffff803d885fa0
> > lsof      13085 carl    5     PIPE 0xffffff8032195000      16384          ->0xffffff803d887a70
> 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/ansible/ansible/issues/3877#issuecomment-23158523
> .

@cayblood can you hop on IRC? I'd like to look at this in realtime, and know what process those PID's belong to. It does look like they'd be ansible, since I see the pipe connecting from one to the SSH process, just want to be sure.

I'm on the #ansible channel now. What is your IRC handle?

James is jimi_c.

On Fri, Aug 23, 2013 at 8:56 AM, Carl Youngblood
notifications@github.comwrote:

> I'm on the #ansible channel now. What is your IRC handle?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ansible/ansible/issues/3877#issuecomment-23161836
> .

We already connected on IRC and I confirmed that ansible is the culprit for all of these TTY sockets.

@cayblood I created a patch which may fix this issue, if you'd like to give it a try:

https://gist.github.com/jimi1283/3d7033c6886b6d20c58b

Hunk #1 FAILED at 367

I seem to have different content there. Pretty sure what I have has been installed from git. Here is the content around that area:

<pre>
        def get_flags():
            # flags are a way of passing arbitrary event information
            # back up the chain, since multiprocessing forks and doesn't
            # allow state exchange
            flags = []
            if template.Flags.LEGACY_TEMPLATE_WARNING:
                flags.append('LEGACY_TEMPLATE_WARNING')
            return flags
        try:
            self._new_stdin = new_stdin
            exec_rc = self._executor_internal(host, new_stdin)
            if type(exec_rc) != ReturnData:
                raise Exception("unexpected return type: %s" % type(exec_rc))
            exec_rc.flags = get_flags()
            # redundant, right?
            if not exec_rc.comm_ok:
                self.callbacks.on_unreachable(host, exec_rc.result)
            return exec_rc
</pre>

I added a patch the other day to fix an issue at that position, and this modifies that change. You probably want to do a "git pull --rebase" and then apply my patch.

OK, I hope to get to this next Tuesday when I am back at work.

@cayblood great, just let me know.

@jimi1283 that didn't fix the problem, but here is a (rather clunky) patch that does. Hopefully it will give you enough info to be able to fix the issue properly.

https://gist.github.com/cayblood/6351639

I just wanted to add that I am seeing this as well.

Same here. With only one host but with many roles played.

Ran into this one as well with a single host looping a template task using a large with_items list.

edit:
I reproduced this on a virtualbox host, as well as an LXC container.  Upping ulimit just produced a different error.  Was able to successfully work around this using the patch from cayblood.  Thanks!

Forgot to re-open this when we were discussing it a few weeks ago. @cayblood I will give that patch a review and test today.


Resolving patch:
diff --git a/lib/ansible/runner/connection_plugins/ssh.py b/lib/ansible/runner/connection_plugins/ssh.py
--- a/lib/ansible/runner/connection_plugins/ssh.py
+++ b/lib/ansible/runner/connection_plugins/ssh.py
@@ -26,6 +26,7 @@
 import hmac
 import pwd
 import gettext
+import pty
 from hashlib import sha1
 import ansible.constants as C
 from ansible.callbacks import vvv
@@ -175,11 +176,11 @@ def exec_command(self, cmd, tmp_path, sudo_user,sudoable=False, executable='/bin
 
         try:
             # Make sure stdin is a proper (pseudo) pty to avoid: tcgetattr errors
-            import pty
             master, slave = pty.openpty()
             p = subprocess.Popen(ssh_cmd, stdin=slave,
                                  stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             stdin = os.fdopen(master, 'w', 0)
+            os.close(slave)
         except:
             p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
                                  stdout=subprocess.PIPE, stderr=subprocess.PIPE)


Test patch: