Repository: sublimelsp/LSP

Issue:
LSP very often causes deadlocks in cooperation with pyls
* OS and language server : Windows 10 Pro 1709 / pyls 0.14.0
* How you installed LSP (Package Control or from git?) : current master (d6da33ba061820c15b99d4f912d09d5f323dceef)

As the title says, ST 3157 gets locked up completely quite often when editing python files. Killing the pyls.exe via TaskManager resolves the issue. LSP detects the broken pipe error then and asks to restart pyls. Even though it is possible to recover from that situation it is very annoying.

Tried to do some logging/debugging, but can't really find the reason for freezing. It just looks like some mutex/lock not being released correctly causing a deadlock at some point.

The issue happens with both `stdio` and `tcp` interface, while `stdio` seems to suffer from that more often.

LSP config

```
{
	"auto_show_diagnostics_panel": false,
	"clients":
	{
		"pyls":
		{
			"enabled": true,
			 "command": ["pyls", "--log-file", "C:\\data\\pyls.log", "-vv"]
			// "command": ["pyls", "--tcp", "--log-file", "C:\\data\\pyls.log", "-vv"],
			// "tcp_port": 2087,
		}
	},
	"show_diagnostics_in_view_status": false,
	"only_show_lsp_completions": false,
	"log_debug": true,
	"log_server": true,
	// "log_payloads": true,
}
```

Console log 
```
LSP: initialize on open 2 C:\Apps\Sublime\Data\Packages\GitGutter\modules\temp.py
LSP: initialize on open 2 C:\Apps\Sublime\Data\Packages\GitGutter\modules\handler.py
LSP:  --> textDocument/hover
LSP:  --> textDocument/didChange
LSP: <--  textDocument/publishDiagnostics
LSP:  --> textDocument/didChange
LSP:  --> textDocument/completion
LSP: <--  textDocument/publishDiagnostics
LSP:  --> textDocument/didChange
LSP: <--  textDocument/publishDiagnostics
LSP:  --> textDocument/didChange
LSP: <--  textDocument/publishDiagnostics
LSP:  --> textDocument/didChange
LSP:  --> textDocument/signatureHelp
LSP: <--  textDocument/publishDiagnostics
LSP:  --> textDocument/didChange
LSP:  --> textDocument/signatureHelp
LSP:  --> textDocument/didChange
LSP: <--  textDocument/publishDiagnostics
LSP:  --> textDocument/hover
Failure writing to stdout
Traceback (most recent call last):
  File "C:\Apps\Sublime\Data\Packages\LSP\plugin\core\rpc.py", line 229, in send
    self.process.stdin.flush()
OSError: [Errno 22] Invalid argument

Failure writing to stdout
LSP: LSP stdout process ended.
Traceback (most recent call last):
  File "C:\Apps\Sublime\Data\Packages\LSP\plugin\core\rpc.py", line 228, in send
    self.process.stdin.write(bytes(message, 'UTF-8'))
BrokenPipeError: [Errno 32] Broken pipe

LSP:  --> textDocument/signatureHelp
LSP: initialize on open 2 C:\Apps\Sublime\Data\Packages\GitGutter\modules\handler.py
LSP: initialize on open 2 C:\Apps\Python33\Lib\tempfile.py
LSP:  --> textDocument/didChange
LSP: initialize on open 2 C:\Apps\Python33\Lib\tempfile.py
LSP: initialize on open 2 C:\Apps\Sublime\Data\Packages\GitGutter\modules\handler.py
```

At some point ST froze and I killed pyls.exe. As the console was not open at that point of time, I can only guess the Errno32 being a result of killing pyls.


Discussion:
Thanks for so many details, I had hoped that `pyls`'s switch to async rpc would help. There is some discussion at https://github.com/palantir/python-language-server/issues/257 suggesting the situation has become worse.

I see you were logging `pyls`'s output as well, any chance we can correlate what was happening on the server side? 
Unfortunately I accidentally deleted the file catching this special situation. As soon as I find myself in such a situation again, I'll carefully copy and paste the log of both sides.
I'm also experiencing quite frequent deadlocks on ubuntu 17.10 with release 0.50 at work.
And also with the master branch. Not sure what's going on because I don't experience this on my mac at home.
I guess it is some kind of race condition. While enabling all the logging features on client and server side the issue appears to occur much less often. Makes debugging a bit like tricky.
Please keep sharing data points, eventually we'll narrow it down. It also can't hurt to review what sync handlers we have. Finally, there's another issue here where we had a bit more discussion about sublime's threading with plugins.
OK, I managed to reproduce this (I think?) on my macbook when turning off all logging. Here's a stack trace from plugin_host:

<details><summary>Stacktrace</summary><p>

```
Process:         plugin_host [12264]
Path:            /Applications/Sublime Text.app/Contents/MacOS/plugin_host
Architecture:    x86_64
Parent:          Sublime Text [12263]
Responsible:     Sublime Text [12263]
UID:             501
Task size:       70.40 MB
Note:            1 idle work queue threads omitted

  Thread 0x2bc1d            DispatchQueue 1           10 samples (1-10)         priority 31 (base 31)
  <thread QoS default (requested user interactive), IO tier 0>
  10  start + 52 (plugin_host + 5940) [0x1004a2734] 1-10
    10  main + 579 (plugin_host + 158673) [0x1004c7bd1] 1-10
      10  handle_protocol_message(string_buffer const&) + 885 (plugin_host + 150666) [0x1004c5c8a] 1-10
        10  void decode_message<int, object_id, int>(int (*)(object_id, int), binary_input_stream&) + 97 (plugin_host + 157879) [0x1004c78b7] 1-10
          10  plugin_call_int(object_id, int) + 86 (plugin_host + 66549) [0x1004b13f5] 1-10
            10  PyEval_CallObjectWithKeywords + 93 (plugin_host + 1079287) [0x1005a87f7] 1-10
              10  PyObject_Call + 103 (plugin_host + 462070) [0x100511cf6] 1-10
                10  function_call + 365 (plugin_host + 590889) [0x100531429] 1-10
                  10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                    10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                      10  fast_function + 321 (plugin_host + 1082318) [0x1005a93ce] 1-10
                        10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                          10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                            10  fast_function + 321 (plugin_host + 1082318) [0x1005a93ce] 1-10
                              10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                                10  PyEval_EvalFrameEx + 20546 (plugin_host + 1066761) [0x1005a5709] 1-10
                                  10  _object* call_function_release_gil<unsigned int, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, int>(unsigned int (*)(unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, int), _object*) + 183 (plugin_host + 44781) [0x1004abeed] 1-10
                                    10  api_window_open_file(unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, int) + 34 (plugin_host + 44882) [0x1004abf52] 1-10
                                      10  unsigned int send_helper<unsigned int>::send_message<165u, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, unsigned int, int>(unsigned int const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int const&, int const&) + 277 (plugin_host + 130915) [0x1004c0f63] 1-10
                                        10  void wait_for_reply<unsigned int>(reply_entry*, unsigned int*) + 141 (plugin_host + 182815) [0x1004cda1f] 1-10
                                          10  handle_protocol_message(string_buffer const&) + 424 (plugin_host + 150205) [0x1004c5abd] 1-10
                                            10  void decode_message<int, unsigned int>(int (*)(unsigned int), binary_input_stream&) + 75 (plugin_host + 151498) [0x1004c5fca] 1-10
                                              10  plugin_on_close(unsigned int) + 34 (plugin_host + 58331) [0x1004af3db] 1-10
                                                10  void py_call<unsigned int>(char const*, char const*, unsigned int const&) + 127 (plugin_host + 57008) [0x1004aeeb0] 1-10
                                                  10  PyEval_CallObjectWithKeywords + 93 (plugin_host + 1079287) [0x1005a87f7] 1-10
                                                    10  PyObject_Call + 103 (plugin_host + 462070) [0x100511cf6] 1-10
                                                      10  function_call + 365 (plugin_host + 590889) [0x100531429] 1-10
                                                        10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                                                          10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                            10  fast_function + 203 (plugin_host + 1082200) [0x1005a9358] 1-10
                                                              10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                                10  fast_function + 321 (plugin_host + 1082318) [0x1005a93ce] 1-10
                                                                  10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                                                                    10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                                      10  fast_function + 203 (plugin_host + 1082200) [0x1005a9358] 1-10
                                                                        10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                                          10  fast_function + 321 (plugin_host + 1082318) [0x1005a93ce] 1-10
                                                                            10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                                                                              10  PyEval_EvalFrameEx + 22167 (plugin_host + 1068382) [0x1005a5d5e] 1-10
                                                                                10  PyObject_Call + 103 (plugin_host + 462070) [0x100511cf6] 1-10
                                                                                  10  function_call + 365 (plugin_host + 590889) [0x100531429] 1-10
                                                                                    10  PyEval_EvalCodeEx + 1622 (plugin_host + 1045776) [0x1005a0510] 1-10
                                                                                      10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                                                        10  fast_function + 203 (plugin_host + 1082200) [0x1005a9358] 1-10
                                                                                          10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                                                            10  fast_function + 203 (plugin_host + 1082200) [0x1005a9358] 1-10
                                                                                              10  PyEval_EvalFrameEx + 20335 (plugin_host + 1066550) [0x1005a5636] 1-10
                                                                                                10  fast_function + 203 (plugin_host + 1082200) [0x1005a9358] 1-10
                                                                                                  10  PyEval_EvalFrameEx + 20546 (plugin_host + 1066761) [0x1005a5709] 1-10
                                                                                                    10  bufferedwriter_write + 189 (plugin_host + 1938860) [0x10067a5ac] 1-10
                                                                                                      10  _enter_buffered_busy + 90 (plugin_host + 1940022) [0x10067aa36] 1-10
                                                                                                        10  PyThread_acquire_lock_timed + 272 (plugin_host + 1251392) [0x1005d2840] 1-10
                                                                                                          10  __psynch_cvwait + 10 (libsystem_kernel.dylib + 113902) [0x7fff778accee] 1-10
                                                                                                           *10  psynch_cvcontinue + 0 (pthread + 38976) [0xffffff7f83abe840] 1-10
```

</p></details>

After this, I did a git-bisect between `master` and `0.4.0`. What I did here was: use sublime text for a while, and if it deadlocks, declare this commit as "bad". Otherwise declare it as "good". It usually deadlocks after 1 or 2 minutes.

<details><summary>git-bisect details</summary><p>

```
~/L/A/S/P/LSP (master=) git fetch upstream --tags
From https://github.com/tomv564/LSP
 * [new tag]         0.5.0      -> 0.5.0
~/L/A/S/P/LSP (master=) git bisect start
~/L/A/S/P/LSP (master|BISECTING=) git bisect bad
~/L/A/S/P/LSP (master|BISECTING=) git bisect good 0.4.0
Bisecting: 45 revisions left to test after this (roughly 6 steps)
[be315dc3ac02c064d74dbccdca6f931044f32fe5] Fix variable name
~/L/A/S/P/LSP ((be315dc3…)|BISECTING) git bisect good
Bisecting: 22 revisions left to test after this (roughly 5 steps)
[67158b3f2b7c9b3009fdb6192a28332679a6593a] Squashed commit of the following:
~/L/A/S/P/LSP ((67158b3f…)|BISECTING) git bisect bad
Bisecting: 11 revisions left to test after this (roughly 4 steps)
[fea99e1ae4ea0df654cfcf5f53ae02a03a2bc955] cancel current completion if the previous input is a space
~/L/A/S/P/LSP ((fea99e1a…)|BISECTING) git bisect good
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[6dcd17dd314ed2d07766617d885a8fc8b2a90d9d] Rename R settings key to prevent it overwriting Rust settings (#265)
~/L/A/S/P/LSP ((6dcd17dd…)|BISECTING) git bisect good
Bisecting: 2 revisions left to test after this (roughly 2 steps)
[b6a3b9dca2cf04fd14fe6602d6cb1fb22ae51679] Expose hover as command
~/L/A/S/P/LSP ((b6a3b9dc…)|BISECTING) git bisect bad
Bisecting: 0 revisions left to test after this (roughly 1 step)
[f665329cf976effac34ab61c757646d109b2ee03] Add issue template
~/L/A/S/P/LSP ((f665329c…)|BISECTING) git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[7ef826805d51eb76fbabd07506d5ec53bc329a3c] TCP support (#248)
~/L/A/S/P/LSP ((7ef82680…)|BISECTING) git bisect bad
7ef826805d51eb76fbabd07506d5ec53bc329a3c is the first bad commit
commit 7ef826805d51eb76fbabd07506d5ec53bc329a3c
Author: Tom van Ommeren <tom.vanommeren@gmail.com>
Date:   Fri Feb 2 08:05:30 2018 +0100

    TCP support (#248)

    * Add socket-based client activated when tcp_port is specified

    Tested against a manually started pyls --tcp

    * Extract Transport and TCP Transport

    * Update tcp parser for PHP language server

    * Also start server if tcp specified, wait for socket to open

    * Add stdio handler and narrow down error handling

    * Use same client for stdio and tcp

    * Re-enable log_stderr, reading stdout for TCP servers

    * Move startup and shutdown state into clients.py

    Properly recognize startup and shutdown state

    * Wait with restart until clients have received exit notification

    * didopen_after_initialize needs to be by window

    So we don't send didOpens from two starting windows to same server

    * Clean debug logging
```

</p></details>


According to my digging, 7ef826805d51eb76fbabd07506d5ec53bc329a3c is the first bad commit (it's always the big commits eh? :P). Can this be of any use?
It looks like you deleted the feature branch in https://github.com/tomv564/LSP/pull/248, do you still have it locally so that I can do a git bisect in that branch?
I seem to have a local copy of the feature-tcp-client branch, but it only goes up to adf08371006bbc565e72662265edc69a59716aa6. This commit seems to work OK! I also don't think f9117fa64e1889133ec7ee4fe7fd32ec7d2ab8bc would cause any problems, so the problem is somewhere between 430578a and b81ffb1.
Sorry this took me so long to get back to, amazing detective work! How did you tell bisect that a given commit was bad?
I just pushed the branch with the missing commits added, hope it helps!
OK, I'm fairly certain I found the bad commit.

```
~/L/A/S/P/LSP (feature-tcp-client=) git bisect start
~/L/A/S/P/LSP (feature-tcp-client|BISECTING=) git bisect bad
~/L/A/S/P/LSP (feature-tcp-client|BISECTING=) git bisect good 78f8b7afb8dabd8a408c86e773f4532bb6b51b53
Bisecting: 6 revisions left to test after this (roughly 3 steps)
[2eb4cb9a03b01bfa17caaadd0b0f0dbc66620fb8] Also start server if tcp specified, wait for socket to open
~/L/A/S/P/LSP ((2eb4cb9a…)|BISECTING) git bisect good
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[225ca5f53fda241a97ccff1f6160974dc66d2e34] Re-enable log_stderr, reading stdout for TCP servers
~/L/A/S/P/LSP ((225ca5f5…)|BISECTING) git bisect good
Bisecting: 1 revision left to test after this (roughly 1 step)
[7fb5bc817b89903f2074de6222349b3aaa57a538] Wait with restart until clients have received exit notification
~/L/A/S/P/LSP ((7fb5bc81…)|BISECTING) git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[ce77ada0ddd99743c88d8df1fcbdc7f44488260e] Move startup and shutdown state into clients.py
~/L/A/S/P/LSP ((ce77ada0…)|BISECTING) git bisect bad
ce77ada0ddd99743c88d8df1fcbdc7f44488260e is the first bad commit
commit ce77ada0ddd99743c88d8df1fcbdc7f44488260e
Author: Tom van Ommeren <tom.vanommeren@gmail.com>
Date:   Tue Jan 30 01:38:20 2018 +0100

    Move startup and shutdown state into clients.py

    Properly recognize startup and shutdown state
```
It's still a pretty large commit, so I cannot see what might be wrong with it at first glance.

> How did you tell bisect that a given commit was bad?

I just start typing a few completions and switch between a few files. LSP will reliably lock up if I'm at a bad commit after about 30 seconds.
This must be related to document-related events being fired at startup - I can confirm I had similar lockups with my typescript server last week (couldn't restart until all other documents were closed) so it's likely a fundamental issue with LSP.
Will have a closer look at that commit !
In case it helps somehow: LSP (with python) locked up ST recently most likely when adding/editing/removing function call parameters. Maybe the reason is related to the parameter hint popup? 
I looked closer at the suspicious commit, the intended change was to introduce a proper "starting" state for clients that are waiting for their language server to start up.
The condition `if config_state.client:` in `_client_for_view_and_window` should prevent LSP features accessing the server too early. This doesn't feel like it should cause deadlocks.

Are there built-in locks in the dicts used for managing state for loading and unloading clients? Changes to `unload_old_clients` might inadvertently compete with starting / ready client lookups?
I don't know. What is a strategy to debug this? This is still a fairly serious issue IMO.
I narrowed down a repro where lsp-tsserver locks up when documentHighlight requests are sent to it while it is still starting up. Some additional logging might reveal if the issue persists all the way to the transport (trying TCP mode might provide some insight as well)

Resolving patch:
diff --git a/plugin/core/rpc.py b/plugin/core/rpc.py
--- a/plugin/core/rpc.py
+++ b/plugin/core/rpc.py
@@ -119,14 +119,8 @@ def handle_transport_failure(self) -> None:
             self._crash_handler()
 
     def send_payload(self, payload: 'Dict[str, Any]') -> None:
-        if self.transport:
-            try:
-                message = format_request(payload)
-                self.transport.send(message)
-            except Exception as err:
-                self._error_display_handler("Failure sending LSP server message, exiting")
-                exception_log("Failure writing payload", err)
-                self.handle_transport_failure()
+        message = format_request(payload)
+        self.transport.send(message)
 
     def receive_payload(self, message: str) -> None:
         payload = None
diff --git a/plugin/core/transports.py b/plugin/core/transports.py
--- a/plugin/core/transports.py
+++ b/plugin/core/transports.py
@@ -2,6 +2,7 @@
 import threading
 import time
 import socket
+from queue import Queue
 import subprocess
 from .logging import exception_log, debug
 
@@ -59,14 +60,18 @@ def start_tcp_transport(port: int):
 class TCPTransport(Transport):
     def __init__(self, socket: 'Any') -> None:
         self.socket = socket  # type: 'Optional[Any]'
+        self.send_queue = Queue()  # type: Queue[Optional[str]]
 
     def start(self, on_receive: 'Callable[[str], None]', on_closed: 'Callable[[], None]') -> None:
         self.on_receive = on_receive
         self.on_closed = on_closed
         self.read_thread = threading.Thread(target=self.read_socket)
         self.read_thread.start()
+        self.write_thread = threading.Thread(target=self.write_socket)
+        self.write_thread.start()
 
     def close(self) -> None:
+        self.send_queue.put(None)  # kill the write thread as it's blocked on send_queue
         self.socket = None
         self.on_closed()
 
@@ -118,28 +123,37 @@ def read_socket(self) -> None:
                         remaining_data = data
 
     def send(self, message: str) -> None:
-        try:
-            if self.socket:
-                # debug('socket send')
-                self.socket.sendall(bytes(message, 'UTF-8'))
-        except Exception as err:
-            exception_log("Failure writing to socket", err)
-            self.socket = None
-            self.on_closed()
+        self.send_queue.put(message)
+
+    def write_socket(self) -> None:
+        while self.socket:
+            message = self.send_queue.get()
+            if message is None:
+                break
+            else:
+                try:
+                    self.socket.sendall(bytes(message, 'UTF-8'))
+                except Exception as err:
+                    exception_log("Failure writing to socket", err)
+                    self.close()
 
 
 class StdioTransport(Transport):
     def __init__(self, process: 'subprocess.Popen') -> None:
         self.process = process  # type: Optional[subprocess.Popen]
+        self.send_queue = Queue()  # type: Queue[Optional[str]]
 
     def start(self, on_receive: 'Callable[[str], None]', on_closed: 'Callable[[], None]') -> None:
         self.on_receive = on_receive
         self.on_closed = on_closed
-        self.stdout_thread = threading.Thread(target=self.read_stdout)
-        self.stdout_thread.start()
+        self.write_thread = threading.Thread(target=self.write_stdin)
+        self.write_thread.start()
+        self.read_thread = threading.Thread(target=self.read_stdout)
+        self.read_thread.start()
 
     def close(self) -> None:
         self.process = None
+        self.send_queue.put(None)  # kill the write thread as it's blocked on send_queue
         self.on_closed()
 
     def read_stdout(self) -> None:
@@ -176,10 +190,17 @@ def read_stdout(self) -> None:
         debug("LSP stdout process ended.")
 
     def send(self, message: str) -> None:
-        if self.process:
-            try:
-                self.process.stdin.write(bytes(message, 'UTF-8'))
-                self.process.stdin.flush()
-            except (BrokenPipeError, OSError) as err:
-                exception_log("Failure writing to stdout", err)
-                self.close()
+        self.send_queue.put(message)
+
+    def write_stdin(self) -> None:
+        while self.process:
+            message = self.send_queue.get()
+            if message is None:
+                break
+            else:
+                try:
+                    self.process.stdin.write(bytes(message, 'UTF-8'))
+                    self.process.stdin.flush()
+                except (BrokenPipeError, OSError) as err:
+                    exception_log("Failure writing to stdout", err)
+                    self.close()


Test patch:
diff --git a/plugin/core/test_rpc.py b/plugin/core/test_rpc.py
--- a/plugin/core/test_rpc.py
+++ b/plugin/core/test_rpc.py
@@ -138,20 +138,6 @@ def test_response_error(self):
         self.assertEqual(len(responses), 0)
         self.assertGreater(len(errors), 0)
 
-    def test_forwards_transport_error(self):
-        set_exception_logging(False)
-        transport = TestTransport(raise_error)
-        settings = TestSettings()
-        client = Client(transport, settings)
-        errors = []
-        client.set_transport_failure_handler(lambda: errors.append(""))
-        self.assertTrue(transport.has_started)
-        responses = []
-        req = Request.initialize(dict())
-        client.send_request(req, lambda resp: responses.append(resp))
-        self.assertEqual(len(responses), 0)
-        self.assertGreater(len(errors), 0)
-
     def test_handles_transport_closed_unexpectedly(self):
         set_exception_logging(False)
         transport = TestTransport(raise_error)