saltstack/salt — pull request #2072 by teftin (2012-09-23 22:10:49+00:00)
Title: minion not closing extra sockets when running external commands

It seems, that when running external command (through salt.cmd.run/retcode/etc) salt doesn't close file descriptors (apart from setting up stdin/stdout/stderr) and it seems to be causing issues when command being run spawns child processes.

Simple test:

```
master% salt '*compute.internal' test.ping
ip-X-X-X-X.eu-west-1.compute.internal: True
```
```
master% salt '*compute.internal' cmd.retcode '/etc/init.d/agraph restart'
{'ip-X-X-X-X.eu-west-1.compute.internal': 0}
```

Let's now see what descriptors this fresh process has:

```
minion% lsof -p `pidof -s agraph`
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
...
agraph 21524 agraph txt REG 202,1 26477 137014 /usr/lib64/agraph/agraph
...
agraph 21524 agraph 0u CHR 1,3 0t0 3908 /dev/null
agraph 21524 agraph 1w REG 202,2 41167 4702220 /media/ephemeral0/agraph/logs/agraph.log
agraph 21524 agraph 2w REG 202,2 41167 4702220 /media/ephemeral0/agraph/logs/agraph.log
agraph 21524 agraph 3w REG 202,1 4737 136619 /var/log/salt/minion
agraph 21524 agraph 4u unix 0xffff88001150ed80 0t0 76883 socket
agraph 21524 agraph 5u unix 0xffff88001150dd40 0t0 76884 socket
agraph 21524 agraph 6u unix 0xffff88001150f0c0 0t0 76885 socket
agraph 21524 agraph 7u unix 0xffff88001150f740 0t0 76886 socket
agraph 21524 agraph 8u 0000 0,9 0 3906 anon_inode
agraph 21524 agraph 9u unix 0xffff88001150fa80 0t0 76887 socket
agraph 21524 agraph 10u unix 0xffff88001150e700 0t0 76888 socket
agraph 21524 agraph 11u 0000 0,9 0 3906 anon_inode
agraph 21524 agraph 12u unix 0xffff88001150e080 0t0 76889 socket
agraph 21524 agraph 13u unix 0xffff88001150e3c0 0t0 76890 socket
agraph 21524 agraph 14u unix 0xffff88001150d6c0 0t0 76891 socket
agraph 21524 agraph 15u unix 0xffff88001150d380 0t0 76892 socket
agraph 21524 agraph 16r REG 202,1 34717696 137028 /usr/lib64/agraph/agraph.dxl
agraph 21524 agraph 17u unix 0xffff88001150c000 0t0 76893 socket
agraph 21524 agraph 18u unix 0xffff88001150c340 0t0 76894 socket
agraph 21524 agraph 19u unix 0xffff88001150ea40 0t0 76895 /tmp/.salt-unix/minion_event_pub.ipc
agraph 21524 agraph 20u unix 0xffff88001150c680 0t0 76896 /tmp/.salt-unix/minion_event_pull.ipc
agraph 21524 agraph 21u unix 0xffff880012c296c0 0t0 77100 socket
agraph 21524 agraph 22u unix 0xffff880012c28680 0t0 77101 socket
agraph 21524 agraph 23u IPv4 77103 0t0 TCP ip-X-X-X-X.eu-west-1.compute.internal:53907->master:4505 (ESTABLISHED)
agraph 21524 agraph 24r CHR 1,3 0t0 3908 /dev/null
agraph 21524 agraph 25r REG 202,1 21152160 137030 /usr/lib64/agraph/agraph.pll
agraph 21524 agraph 26w REG 202,2 41167 4702220 /media/ephemeral0/agraph/logs/agraph.log
agraph 21524 agraph 27u REG 202,2 5 4702222 /media/ephemeral0/agraph/db/settings/lock
agraph 21524 agraph 28u unix 0xffff880012c29380 0t0 77239 /media/ephemeral0/agraph/db/settings/shm-server
agraph 21524 agraph 33w FIFO 0,8 0t0 77235 pipe
agraph 21524 agraph 35r FIFO 0,8 0t0 77236 pipe
```

So, we can see both mix of own sockets created/openned by agraph daemon and 'leftovers' from salt minion, including open connection to salt master server....

And now, lets try ..

```
master% salt '*compute.internal' test.ping
... (timeouts)
```

But let's restart agraph directly on minion (which will close all the extra/copies of sockets from salt minion):
```
minion% /etc/init.d/agraph restart
...
```

and lets try again:

```
% salt '*compute.internal' test.ping
ip-X-X-X-X.eu-west-1.compute.internal: True
```

My guess would be that, because new daemon still holds connection to master (although it doesn't do anything with it), it 'consumes' commands from master (effectively blackholing them). This affects not only 'cmd.run' but in service handling as well.

Going through the code, it looks like all service/command handling is routed through module/cmdmod.py (which makes life easier), so this (should) fix it ..