opened
aronneagu
username_0
### Description of Issue/Question
I am trying to run `salt <target> state.highstate` but I get the error "Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased. " or "Minion did not return. [No response]". We have worker_threads the default value (5).

We are running 15 salt-minion EC2 instances with a salt-master running on t2.medium EC2 instance.
I tried restarting the salt-master service but I still receive this error. When I run command `salt '*' test.ping` only some of the minions return, the others return the error above (No response), and never the same minions fail to reply on consecutive calls. Even after I connect to the minion and restart the service, test.ping returns "Minion did not return. [No response]".

When I do manage to run state.highstate the return is 
```
 The minion function caused an exception: Traceback (most recent call last):
                  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1332, in _thread_return
                    return_data = executor.execute()
                  File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 28, in execute
                    return self.func(*self.args, **self.kwargs)
                  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 707, in highstate
                    mocked=kwargs.get('mock', False))
                  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3350, in __init__
                    loader=loader)
                  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 657, in __init__
                    self.opts['pillar'] = self._gather_pillar()
                  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 699, in _gather_pillar
                    ret = pillar.compile_pillar()
                  File "/usr/lib/python2.7/dist-packages/salt/pillar/__init__.py", line 170, in compile_pillar
                    dictkey='pillar',
                  File "/usr/lib/python2.7/dist-packages/salt/utils/async.py", line 75, in wrap
                    ret = self._block_future(ret)
                  File "/usr/lib/python2.7/dist-packages/salt/utils/async.py", line 85, in _block_future
                    return future.result()
                  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
                    raise_exc_info(self._exc_info)
                  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
                    yielded = self.gen.throw(*exc_info)
                  File "/usr/lib/python2.7/dist-packages/salt/transport/zeromq.py", line 188, in crypted_transfer_decode_dictentry
                    tries=tries,
                  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
                    value = future.result()
                  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
                    raise_exc_info(self._exc_info)
                  File "<string>", line 3, in raise_exc_info
                SaltReqTimeoutError: Message timed out
```
What could be causing this, or how can further troubleshoot this?

### Setup
(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

### Steps to Reproduce Issue
(Include debug logs if possible and relevant.)

### Versions Report
(Provided by running `salt --versions-report`. Please also mention any differences in master/minion versions.)
```
Salt Version:
           Salt: 2016.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 2.5.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-48-generic
         system: Linux
        version: Ubuntu 14.04 trusty
```
Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.
issue
created
aronneagu
username_0
I forgot to mention that I tried increasing the worker_threads up to 10 and it didn't help.
`salt '*' test.ping` returned the error
`Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.`
comment
created
Ch3LL
username_1
@username_0 For the `"Minion did not return. [No response]"` error please ensure there is no network slowness or issues between the minion and masters. Also please set `log_level: debug` in the master and minion that will give us some more information in general. But the minion did not return a lot of times is telling of a network issue.

Also can you check `top` to see if maybe there is a resource issue? Thanks
comment
created
buildmaster-nyansa
username_2
+1 

This seems like the same [issue](https://github.com/saltstack/salt/issues/36932) I am facing. I am using Boron as well for the master and slave. And after the upgrade, slave cannot connect to master. I am also completely **blocked!**

## Salt Master Version
```shell
# salt --versions-report
Salt Version:
           Salt: 2016.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-92-generic
         system: Linux
        version: Ubuntu 14.04 trusty

```

## Salt Minion Version
```bash
$ sudo salt-minion --version
salt-minion 2015.5.3 (Lithium)
```

All of the minions not able to establish contact with master over ZMQ. Even the ones spun up using the upgraded salt:

```bash
$ sudo salt-minion -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Configuration file path: /etc/salt/minion
[INFO    ] Setting up the Salt Minion "minion1"
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Attempting to authenticate with the Salt Master at 172.31.48.39
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'minion1', 'tcp://172.31.48.39:4506')
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 1 of 7)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 2 of 7)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 3 of 7)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 4 of 7)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 5 of 7)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 6 of 7)
[INFO    ] SaltReqTimeoutError: after 60 seconds. (Try 7 of 7)
[INFO    ] The salt minion is shut down
[ERROR   ] Attempt to authenticate with the salt master failed
[WARNING ] ** Restarting minion **
```
comment
created
buildmaster-nyansa
username_2
Minions were able to attach when I upgraded salt master from Ubuntu 14.04 to 16.04.

@username_0 Please double check your master version.

```bash
$ salt --versions-report
Salt Version:
           Salt: 2016.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jul  1 2016, 15:12:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-43-generic
         system: Linux
        version: Ubuntu 16.04 xenial
```
comment
created
Ch3LL
username_1
ahh thanks for that information @username_2  

i wonder if you had upgraded your minions to 2016.3.3 as well if you still would see that error? but that might be too much to ask since you alreay moved to ubuntu16.

@username_0 are you also using older minions?
comment
created
rehmanzile
username_3
@username_1 I did upgrade the minions to 2016.3.3 but didn't work for me. It turns out the problem is somehow related to Ubuntu 14.04 when you upgrade salt-master from 2015.5.3 to 2016.3.3. 

Tell your QA team to try this combination.
comment
created
aronneagu
username_0
Hi guys,
Indeed I am using older version of the salt-minion
```
salt-minion 2016.3.0 (Boron)
```

The problem is that when this happened 2 days ago, the problem appeared out of nowhere, and some of the minions did not reply (always different ones), and then abruptly as it started it the issue disappeared. I have no way of reproducing the issue. I will go ahead and upgrade the minions to 2016.3.3.
@username_2 @username_1 thank you for your support and suggestions
comment
created
rehmanzile
username_3
@username_0 What is your Salt Master version? Use ```$ salt --versions-report``` to find out. Also, if you Salt Master server is using Ubuntu Trusty 14.04, then these intermittent connectivity issues do occur. I have experienced it in my environment and had to upgrade to 16.04.
comment
created
aronneagu
username_0
@username_3 I am using 2016.3.3 for the salt-master (see my first post). Also the I am using Ubuntu 14.04. I am considering upgrading to 16.04, but it won't be something I can do in the immediate future, before I determine the impact on our application. 
I have created a script that will do a ping from the master to the minions, and that will show if there are losses in the network. However, the servers are EC2 instances running in a VPC, so I highly doubt that underlying network to be the cause.
How did you determine that the OS, and which part of it, was causing the connectivity issue?
comment
created
rehmanzile
username_3
@username_0 I ruled out connectivity because I was using 2015.8.12 version of Salt Master on Ubuntu 14.04 originally in AWS. The master and minions were operating correctly. 

Then I upgraded the Salt Master and Minion version to 2016.3.3. Minions were having a hard time connecting to master and pings were failing. Same setup, different Salt version. Then I upgraded my OS to Ubuntu 16.04. All connectivity issues went away.

I agree with you, I doubt your network is an issue in AWS. You can take a snap shot of the current salt and try to upgrade to resolve these issues. You can always revert back to the previous version.
comment
created
aronneagu
username_0
@username_3 I will try to upgrade then the OS to Ubuntu 16.04 as soon as possible. Did you upgrade only the master OS, or also the minions? And just to be clear, you are using version 2016.3.3 for both the master and minion? (I appreciate all the effort and comments you added to the thread, they have been invaluable)

Today it happened that I had a virtual machine running locally on my laptop connecting to the salt-master on AWS, and it was still timing out, so low chances that AWS infrastructure is at fault here.
comment
created
rehmanzile
username_3
@username_0 You only need to upgrade the master OS. I believe the problem is with the Master OS. All of my minions are still running Ubuntu 14.04 with salt-minion version 2016.3.3. Hope this works out for you!
comment
created
rehmanzile
username_3
@username_0 Just curious. Were you able to resolve the issue by upgrading salt master?
comment
created
bkruger99
username_4
I have run into similar with ubuntu 14.04 doing a fresh salt master install.  I've had to reboot the instance for it to work correctly (after all updates from apt, etc).   16.04 works without issue.
comment
created
aronneagu
username_0
@username_3 I have successfully upgraded to 16.04, and the first time I ran salt '*' test.ping (after reboot), more than half of the instances failed to reply, with error "Minion not connected [No response] (or [Not connected]". The second time I issued the same command they all replied. So far they all seem to accept commands from the master.

I encountered the same behavior on my test stack, the first salt command (usually test.ping) after upgrading to 16.04 fails.

If I won't encounter any issues in the next week, I will close the ticket
comment
created
Ch3LL
username_1
Okay i went ahead and tried to replicate this issue and I am able to successfully replicate it. I even tried upgrading a bunch of the dependencies and did not see a difference. Here is my versions-report after the upgrade of versions:

```
root@ip-10-63-221-160:~# salt --versions-report
Salt Version:
           Salt: 2016.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 16.0.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.5
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-36-generic
         system: Linux
        version: Ubuntu 14.04 trusty
 
```

Weirdly enough when I installed the salt-minion on the same host as the master and was running the salt-minon processes everything worked as expected. I could ping remote minions and the local minion. But when I stop that minion process I can no longer ping and I get the `The master is not responding` error. 

I'm currently bringing up a test case to see if this affects clean installs of ubuntu14 as well. Will report results when done.
comment
created
Ch3LL
username_1
Oh and if anyone still has their ubuntu14 instance up can they see if running a minion the master helps? or mabye someone can clarify if they had a local minion running?
comment
created
Ch3LL
username_1
38d37
```
comment
created
Ch3LL
username_1
Okay I can also confirm that upgrading the minions to 2016.3.3 resolves the issue well and this is my versions report for that upgraded minion:

```
Salt Version:
           Salt: 2016.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-36-generic
         system: Linux
        version: Ubuntu 14.04 trusty
```

So to sum everything up:

1. following situation creates issue:
     - upgrade 2015.5.3 master to 2016.3.3
     - minion stays at 2015.5.3

WORKAROUND: fix this issue by installnig a 2016.3.3 minion on the master and weirdly enough you can ping the 2015.5.3 minion at that point)

SOLUTION: upgrade minion to 2016.3.3 (although I think you guys pointed out that it did not work for you when upgrading the minions to 2016.3.3., which i canot replicate, so i'm wondering if you guys did not have a salt-minion on your master?)
comment
created
cachedout
username_5
@username_1 Given your summary, do you think it would be sufficient to document those caveats and close this issue?
comment
created
aronneagu
username_0
@username_1 I can confirm I did not have a minion on the master at any point. Both the master and minions were all at version 2016.3.3 when the issue happened, but I did upgrade the versions from 2015.x.x, so it wasn't a clean install.
comment
created
rehmanzile
username_3
@username_1 My main concern with the solution you pointed out is the fact that salt-master cannot communicate to minions. And upgrading minions will be a daunting task, especially if ssh port is not available.
comment
created
Ch3LL
username_1
@username_3 agreed its not the best solution and it looks like an engineer has been assigned this so i'm sure there will be additional eyes on this. Did upgrading the minions to 2016.3.3 resolve the issue for you though? I think you mentioned it did not so I wanted to clear that up.

I think the easier solution for someone would be to install a 2016.3.3 minion on the master and run that. That way they don't have to upgrade all minions. That should work as a temporary workaround.
comment
created
rehmanzile
username_3
@username_1 I did not try upgrading the minions. With 50+ minions in the environment, it was not a feasible option for me.
comment
created
Ch3LL
username_1
@username_3  Thanks for clarifying
comment
created
DmitryKuzmenko
username_6
**The problem here is the same as #35480.**
Salt master in 2016.3.3 has an issue in the shutdown procedure that causes some old MWorker processes could still alive after the master restart. In this case those workers could fetch jobs from the queue, but they aren't connected to the new Event Bus instance so master doesn't receive messages about job done by those workers.

**How to check am I right:**
Issue the following command: `ps aux | grep salt-master` and check MWorker processes PIDs. They all should have close numbers and all should be greater than `MWorkerQueue`, i.e. should be at the bottom of the list. For instance:
```bash
$ ps aux | grep salt-master
root     21229  0.0  4.1 243484 41916 ?        Ss   11:34   0:00 /usr/bin/python /usr/bin/salt-master ProcessManager
root     21235  0.0  1.9 149500 20180 ?        S    11:34   0:00 /usr/bin/python /usr/bin/salt-master MultiprocessingLoggingQueue
root     21238  0.0  3.5 320852 36072 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master ZeroMQPubServerChannel
root     21239  0.0  3.3 243204 34500 ?        S    11:34   0:00 /usr/bin/python /usr/bin/salt-master EventPublisher
root     21242  0.0  4.0 247504 41372 ?        S    11:34   0:00 /usr/bin/python /usr/bin/salt-master Maintenance
root     21304  0.3  3.8 850344 38824 ?        Sl   11:34   0:01 /usr/bin/python /usr/bin/salt-master Reactor
root     21305  0.0  3.3 243484 33928 ?        S    11:34   0:00 /usr/bin/python /usr/bin/salt-master ReqServer_ProcessManager
root     21306  0.0  3.7 620340 38024 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master MWorkerQueue
root     21314  0.0  4.3 477456 43860 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master MWorker-0
root     21318  0.0  4.3 329996 44212 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master MWorker-1
root     21321  0.0  4.3 329992 44536 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master MWorker-2
root     21324  0.0  4.2 328788 43596 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master MWorker-3
root     21327  0.0  4.2 328792 43604 ?        Sl   11:34   0:00 /usr/bin/python /usr/bin/salt-master MWorker-4
```

**"What can I do?"**
You can do one of the following to fix the issue:
1 Upgrade to 2016.3.4. It has all the master shutdown fixes applied. The only master update is needed.
2 Manually apply ca8eb7e0760297c2bb8abc49fe58af57637de3fb, 6ad299871505e5b3adcfbe582855ae509a1879e7, b6937ebaa80e95a1f5edc89011d2119c0ab6a4cf to 2016.3.3 master. This could be done by cloning salt git repo to somewhere and execute these commands:
```bash
$ git show ca8eb7e0760297c2bb8abc49fe58af57637de3fb 6ad299871505e5b3adcfbe582855ae509a1879e7 b6937ebaa80e95a1f5edc89011d2119c0ab6a4cf >/tmp/patch
$ cd /usr/lib/python2.7/dist-packages/salt
$ sudo patch -p2 < /tmp/patch
```
3 Cleanup on stop via the init system. 14.04 uses upstart, you can add post-stop script to `/etc/init/salt-master.conf`:
```bash
post-stop script
  exec /bin/salt-kill.sh
end script
```
where `/bin/salt-kill.sh` is executable file with the following contents:
```bash
#!/bin/bash
pkill -9 -f "/usr/bin/python /usr/bin/salt-master"
exit 0
```
Ubuntu 16.04 uses systemd and it looks systemd do the stop job better than upstart. But if it's needed the systemd script could be updated too by adding `ExecStopPost=pkill -9 -f "/usr/bin/python /usr/bin/salt-master"` to the `[Service]` section of `/lib/systemd/system/salt-master.service`
comment