Repository: ansible/ansible

Issue:
Sporadic WinRMTransportError: 500 WinRMTransport
##### Issue Type:

Bug Report
##### Ansible Version:

ansible 1.8.2
  configured module search path = None
##### Environment:

Ansible runner: Ubuntu 14.04
Managed machine: Windows 2008 R2 SP1
##### Summary:

From time to time we get the following Ansible error:

```
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/ansible/runner/connection_plugins/winrm.py", line 149, in exec_command
    result = self._winrm_exec(cmd_parts[0], cmd_parts[1:], from_exec=True)
   File "/usr/local/lib/python2.7/dist-packages/ansible/runner/connection_plugins/winrm.py", line 114, in _winrm_exec
     self.shell_id = self.protocol.open_shell()
   File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 103, in open_shell
     rs = self.send_message(xmltodict.unparse(rq))
   File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 173, in send_message
     return self.transport.send_message(message)
   File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 101, in send_message
     raise WinRMTransportError('http', error_message)
WinRMTransportError: 500 WinRMTransport. Bad HTTP response returned from server. Code 500
```

It does not seem that this is caused by a timeout because the previous task finished just a few seconds before the failing one (for example, 7 seconds in the case I am looking at the moment).

We use latest version of pywinrm.
##### Steps To Reproduce:

Unfortunately, none. It is a sporadic issue that we see regularly... Let's say once per day.
##### Expected Results:

The task to execute successfully.
##### Actual Results:

The described error.



Discussion:
It seems this is related to this comment in the pywinrm implementation:
https://github.com/diyan/pywinrm/blob/master/winrm/transport.py#L101 

This happens for me 100% of the time when I use copy to transfer a large file to the windows guest. After it happens, the VM is completely lost to vagrant and ansible. Nothing works via them anymore. I can access the machine and it's still working via VirtualBox, but the vagrant and ansible tools are cactus. For instance, "vagrant rdp mymachine" just hangs indefinitely.

This is using Windows 2012 R2. Also Ubuntu 14.04 as the runner.

For me, it is just sporadic. It happens from time to time and I still can't find the reason for it. Here is the latest error log:

```
 File "/usr/local/lib/python2.7/dist-packages/ansible/runner/connection_plugins/winrm.py", line 149, in exec_command
     result = self._winrm_exec(cmd_parts[0], cmd_parts[1:], from_exec=True)
   File "/usr/local/lib/python2.7/dist-packages/ansible/runner/connection_plugins/winrm.py", line 128, in _winrm_exec
     self.protocol.cleanup_command(self.shell_id, command_id)
   File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 287, in cleanup_command
     rs = self.send_message(xmltodict.unparse(rq))
   File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 190, in send_message
     return self.transport.send_message(message)
   File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 112, in send_message
     raise WinRMTransportError('http', ex.reason)
 WinRMTransportError: 500 WinRMTransport. [Errno 104] Connection reset by peer
```

I'm also having this issue. When transfering a large file with the unreleased win_copy module it takes a lot of time with the action (runned with ansible-playbook with -vvv) `PUT /x/webinterface/www.zip TO C:\Users\Administratror\AppData\Local\Temp\ansible-tmp-1431003458.4-23654771504711\source` and eventually I get the following error:

```
  File "/usr/lib/pymodules/python2.7/ansible/runner/connection_plugins/winrm.py", line 198, in put_f
ile
    result = self._winrm_exec(cmd_parts[0], cmd_parts[1:])
  File "/usr/lib/pymodules/python2.7/ansible/runner/connection_plugins/winrm.py", line 125, in _winr
m_exec
    command_id = self.protocol.run_command(self.shell_id, command, args)
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 256, in run_command
    rs = self.send_message(xmltodict.unparse(rq))
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 190, in send_message
    return self.transport.send_message(message)
  File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 110, in send_message
    raise WinRMTransportError('http', error_message)
WinRMTransportError: 500 WinRMTransport. Bad HTTP response returned from server.  Code 500
fatal: [jenkins] => failed to transfer file to C:\Users\Administrator\AppData\Local\Temp\ansible-
tmp-1431003458.4-23654771504711\source
```

After this happens I get the 500 error with every win_ping action I execute, until I restart the server.

@Gigha I don't have a fix but I work around this problem by setting up an http server and using win_get_url to fetch big files.  I still use win_copy for small things like configuration files but fetching from an http server is faster than using win_copy for larger files. 

@Gigha what is the size of a large file in your case?  What do you consider a "lot of time" before you hit this issue?

Due to the less than ideal way files are transferred over winrm, it may be possible you're hitting one of the [remote shell quotas](https://msdn.microsoft.com/en-us/library/ee309367%28v=vs.85%29.aspx).

Thanks for the suggestions, I will look into in monday since I cant access my computer from here. Will comment again on monday!

I just checked it, the file size is 30MB, and the copy task takes 8 minutes and 24 seconds before it fails.  Is there a solution to make the file copy succeed and go faster? 8 Minutes is too long for us to give deployment with Ansible to Windows machines a serious chance. Thanks in advance! ;)

i would advise using samba/cifs to copy large files, winrm is really designed for tiny scripts

I did some more observations. For me it happens regularly after installing a lot of features with the win_feature module. The feature installation goes well, but one of the next 1-2-3 tasks fails with this error; if the features are already installed - all is fine. I am currently testing with 1.9.0.1 and it happens there as well.
As a possible fix for my scenario would be a global retry configuration; which actually can help in other scenarios as well.
@cchurch is working on different transport layer using requests, maybe that can help. Btw, let me know if I can help with that.
Another fix would be using something like accelerated mode for Windows.
If I continue to have these issues I will need to implement one of the above fixes. I really don't want to fork the project so what do you think makes most sense and has highest chance of getting merged?

Can you observe the performance on the machine straight after you have
installed features?

I wonder if ngen is running or something else is taking up resources on
your machine.

I use win_feature a bit and have not encountered the problem you describe,
although typically I will only add one role or feature.

Jon

On Mon, May 11, 2015 at 10:08 PM, Ivaylo Bratoev notifications@github.com
wrote:

> I did some more observations. For me it happens regularly after installing
> a lot of features with the win_feature module. The feature installation
> goes well, but one of the next 1-2-3 tasks fails with this error; if the
> features are already installed - all if fine. I am currently testing with
> 1.9.0.1 and it happens there as well.
> As a possible fix for my scenario would be a global retry configuration;
> which actually can help in other scenarios as well.
> @cchurch https://github.com/cchurch is working on different transport
> layer using requests, maybe that can help. btw, let me know if I can help
> with that.
> Another fix would be using something like accelerated mode for Windows.
> If we continue to have these issues I will need to implement one of the
> above fixes. I really don't want to fork the project so what do you think
> makes most sense and has highest chance of getting merged?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9980#issuecomment-101047370.

Thanks for this.  Worth knowing its possible.  I really only use copy to
push out templated files, anything bigger I fetch using win_get_url.

On Fri, May 29, 2015 at 10:46 AM, Loona91 notifications@github.com wrote:

> I had the same problem when sending files with greater size than defaults
> setting of MaxEnvelopeSizekb property in wirm/config. According to msdn it
> is default set on 150 kB. Go to C:\Windows\System32 and change it with
> 'winrm set winrm/config @{MaxEnvelopeSizekb= "X"}' where X is maximum size
> you need.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9980#issuecomment-106760110.

I had the same problem. After I set "300000" for MaxEnvelopeSizekb, I run a playbook however I had the same result. 
When I want to send to the windows systems any file than XXXMB should I use win_get_url ?

I did some testing last night and the largest file I was able to send via
win_copy was 3Mb (3145728 bytes).  A 4Mb file resulted in 500 response from
the server and I was unable to reconnect via winrm until I had restarted
the windows remote management service on the host.
I have documented the above and created a PR with this information today so
others can know about this limitation.

On Fri, Jun 19, 2015 at 10:28 AM, Takeshi Kuramochi <
notifications@github.com> wrote:

> I had the same problem. After I set "300000" for MaxEnvelopeSizekb, I run
> a playbook however I had the same result.
> When I want to send to the windows systems any file than XXXMB should I
> use win_get_url ?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9980#issuecomment-113445033.

Thank you! I could understand this issue.

@jhawkesworth could you mention the associated PR # here so the issues are linked? Thanks!

Sure, the documentation PR was https://github.com/ansible/ansible-modules-core/pull/1557 and has already been merged.

I am on Ansible version: v2.0.0-0.2.alpha2

I have also been getting this error after installing numerous win_features and when it moves on to chocolately installs, it usually throws this error within a few tasks.  It was alleviated some by forcing a restart after the win_features have been installed, but it still happens sometimes.   In fact, it just happened in the middle of the restart task!

![2015-10-01 08_44_29-ansible-host running - oracle vm virtualbox](https://cloud.githubusercontent.com/assets/12701030/10220861/61147404-6818-11e5-85b5-6722bd7bda45.png)

@luis-echegaray I would hope that a 500 (server error) from the winrm service would log something to the event logs.  Could you run eventvwr on the affected windows machine and search for anything from the time the failure occurred?
Also can you share a complete playbook that demonstrates this?

the playbook is here: https://gist.github.com/luis-echegaray/aabb370bd507d3331449
(I just started building playbooks after working with adhoc commands for a while! sorry for the noobiness) 

I searched the Event Viewer and came up empty handed, but I am new to this tool. I will look further into it.

I am terminating my current servers and trying a rebuild, and I will pay attention to the errors at the time they occur.

Thanks for any help you can provide!

@luis-echegaray can I suggest passing the list of features you want to install in as a comma separated list, rather than with lots of calls to win_feature:
name: Web-Server,Web-Common-Http, etc
This would reduce the length of your playbook and perhaps speed things up a bit.  Also it would mean you are only going to have to wait for a reboot if required once, not after (potentially) each feature that needs a reboot.  I think maybe in your case it is failing with 500 because a reboot is taking place ( I don't think that module guarantees to wait until the reboot has completed).  Can you try and report back?

@jhawkesworth thanks for the feedback and solution!
- I like the individual feedback I get when I explicitly declare each win_feature with a name, although it does make for a long playbook (Once I put this in a role, it might alleviate the overall length). But I ended up taking your advice and ending up with: (w/ my weird syntax highlighting)
  ![2015-10-02 10_32_35-ansible-host running - oracle vm virtualbox](https://cloud.githubusercontent.com/assets/12701030/10248943/a49b1470-68f0-11e5-8b6f-7f853a58f186.png)
  I ran it and it seems to have eliminated the random failures (spun up 5 machines multiple times, and reran the script on already provisioned boxes x4 times. This always reproduced the issue several times in my testing).  This is very weird.  Does it have to do with how many times i'm calling win_feature? Is there some timeout I am not aware of? I guess its time to be as terse as possible!
- In regards to the restart module... it is pretty awesome, I got it from Ansible Galaxy ( https://galaxy.ansible.com/list#/roles/4576 -the only windows entry! ), and it works properly, it does wait for the server to come back up and you can tweak the vars for your specific case. In any case, I was getting the exact same error before I started using the restart. Here is an example of the module pinging 3 servers while it waits from winRM to respond:
  ![2015-10-02 11_03_28-ansible-host running - oracle vm virtualbox](https://cloud.githubusercontent.com/assets/12701030/10249750/f003f7de-68f4-11e5-8875-c0c6318a917c.png)

Thanks so much!  I will report back if the issue pops back up.  

FYI, after some research the sporadic issue by retrying in the pywinrm library itself. You can check our changes here:
https://github.com/geyanev/pywinrm/compare/master...geyanev:feature/retry-on-http-error-500

In simple words - if we get WinRMTransportError 500 when sending a WINRM message - we wait a second and try while logging all this for debugging. This has fixed our sporadic WinRMTransportError 500 and has not caused any other issues so far.

I believe the problem exists with the processes WinrsHost.exe and conhost.exe not exiting.  These processes are not exiting/closing when multiple winrm commands are executed in succession from a single Ansible task.  These processes can be seen growing in the process table on the Windows host.  Eventually you will hit the "MaxShellsPerUser" maximum and when this occurs all future connections (until others close from another task, etc) will be ignored and generate the error: WinRMTransportError: 500 WinRMTransport. Bad HTTP response returned from server. Code 500

A workaround is to increase the max shells per user:  winrm set winrm/config/winrs '@{MaxShellsPerUser="val"}', but this must be done with caution as it will consume a great deal of memory if set too high.

In researching further, there is a known issue with winrm processes not closing that has been closed. Unfortunately, this is the version I am currently using and I am still seeing the same issue with connections not closing.

https://github.com/ansible/ansible/issues/12275

In analyzing this more closely I can see the connection closing code is not playing nicely with multiple items (EX: when using with_items) in Ansible tasks.   Its only closing after all items have been iterated, thus leaving many open connections.  I have added a CLOSING CONNECTION statement in task_executor.py to display this behavior in action.  Three of the four connections made will not close from the example below.

<10.1.233.159> ESTABLISH WINRM CONNECTION FOR USER: Administrator on PORT 5986 TO 10.1.233.159
<10.1.233.159> ESTABLISH WINRM CONNECTION FOR USER: Administrator on PORT 5986 TO 10.1.233.159
<10.1.233.159> ESTABLISH WINRM CONNECTION FOR USER: Administrator on PORT 5986 TO 10.1.233.159
<10.1.233.159> ESTABLISH WINRM CONNECTION FOR USER: Administrator on PORT 5986 TO 10.1.233.159
CLOSING CONNECTION

The fix for this issue is to close only winrm connections from task_executor inside _execute, not from the run method.  Closing from here only closes the LAST connection after all items in a task have completed.  There is a major assumption that ssh is being used here without taking into account the current winrm connection limitations.   I have a patch that I am currently testing to submit.


Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -67,6 +67,7 @@ def __init__(self, host, task, job_vars, play_context, new_stdin, loader, shared
         self._new_stdin         = new_stdin
         self._loader            = loader
         self._shared_loader_obj = shared_loader_obj
+        self._connection        = None
 
     def run(self):
         '''
@@ -361,8 +362,9 @@ def _execute(self, variables=None):
                 self._task.args = variable_params
 
         # get the connection and the handler for this execution
-        self._connection = self._get_connection(variables=variables, templar=templar)
-        self._connection.set_host_overrides(host=self._host)
+        if not self._connection or not getattr(self._connection, '_connected', False):
+            self._connection = self._get_connection(variables=variables, templar=templar)
+            self._connection.set_host_overrides(host=self._host)
 
         self._handler = self._get_action_handler(connection=self._connection, templar=templar)
 
diff --git a/lib/ansible/plugins/connection/winrm.py b/lib/ansible/plugins/connection/winrm.py
--- a/lib/ansible/plugins/connection/winrm.py
+++ b/lib/ansible/plugins/connection/winrm.py
@@ -169,14 +169,16 @@ def _winrm_send_input(self, protocol, shell_id, command_id, stdin, eof=False):
         rs = protocol.send_message(xmltodict.unparse(rq))
 
     def _winrm_exec(self, command, args=(), from_exec=False, stdin_iterator=None):
-        if from_exec:
-            display.vvvvv("WINRM EXEC %r %r" % (command, args), host=self._winrm_host)
-        else:
-            display.vvvvvv("WINRM EXEC %r %r" % (command, args), host=self._winrm_host)
         if not self.protocol:
             self.protocol = self._winrm_connect()
+            self._connected = True
         if not self.shell_id:
             self.shell_id = self.protocol.open_shell(codepage=65001) # UTF-8
+            display.vvvvv('WINRM OPEN SHELL: %s' % self.shell_id, host=self._winrm_host)
+        if from_exec:
+            display.vvvvv("WINRM EXEC %r %r" % (command, args), host=self._winrm_host)
+        else:
+            display.vvvvvv("WINRM EXEC %r %r" % (command, args), host=self._winrm_host)
         command_id = None
         try:
             stdin_push_failed = False
@@ -211,6 +213,7 @@ def _winrm_exec(self, command, args=(), from_exec=False, stdin_iterator=None):
     def _connect(self):
         if not self.protocol:
             self.protocol = self._winrm_connect()
+            self._connected = True
         return self
 
     def exec_command(self, cmd, in_data=None, sudoable=True):
@@ -387,5 +390,8 @@ def fetch_file(self, in_path, out_path):
 
     def close(self):
         if self.protocol and self.shell_id:
+            display.vvvvv('WINRM CLOSE SHELL: %s' % self.shell_id, host=self._winrm_host)
             self.protocol.close_shell(self.shell_id)
-            self.shell_id = None
+        self.shell_id = None
+        self.protocol = None
+        self._connected = False


Test patch:
diff --git a/test/integration/roles/test_win_raw/tasks/main.yml b/test/integration/roles/test_win_raw/tasks/main.yml
--- a/test/integration/roles/test_win_raw/tasks/main.yml
+++ b/test/integration/roles/test_win_raw/tasks/main.yml
@@ -101,3 +101,16 @@
   assert:
     that:
       - "raw_result2.stdout_lines[0] == '--% icacls D:\\\\somedir\\\\ /grant \"! ЗАО. Руководство\":F'"
+
+# Assumes MaxShellsPerUser == 30 (the default)
+
+- name: test raw + with_items to verify that winrm connection is reused for each item
+  raw: echo "{{item}}"
+  with_items: "{{range(32)|list}}"
+  register: raw_with_items_result
+
+- name: check raw + with_items result
+  assert:
+    that:
+      - "not raw_with_items_result|failed"
+      - "raw_with_items_result.results|length == 32"