Repository: palantir/python-language-server

Issue:
Newline issues on Windows
Hi there,

Thanks for the excellent work you've done on the language server! Overall, it's working great on OSX and Linux - I'm working on incorporating it into Oni: https://github.com/extr0py/oni

There is one blocking issue on Windows, however:

**Issue:** The LSP protocol expects there to be `\r\n\r\n` following the Content-Header. This works as expected on OSX and Linux. However, on Windows, we actually get `\r\r\n\r\r\n`, which some LSP clients will not handle this. `vscode-jsonrpc` is strict on matching this and therefore never realizes the stream is complete, and never completes initialization. I've tested this using the _stdio_ strategy - I haven't validated with the _tcp_ strategy.

**Defect:** It turns out Python has some 'magic' behavior with newlines - see https://stackoverflow.com/questions/2536545/how-to-write-unix-end-of-line-characters-in-windows-using-python. It looks like Python is converting `\n` on Windows to `os.linesep` - which is `\r\n`. 

This is the impacted code (`server.py`):
```
    def _write_message(self, msg):
        body = json.dumps(msg, separators=(",", ":"))
        content_length = len(body)
        response = (
            "Content-Length: {}\r\n"
            "Content-Type: application/vscode-jsonrpc; charset=utf8\r\n\r\n"
            "{}".format(content_length, body)
        )
        self.wfile.write(response)
        self.wfile.flush()
```

In this case, the `\n` in the `\r\n` is getting expanded - so we actually end up with `\r\n`.

**Proposed Fix**: The `os.linesep` should be checked. If it is `\n`, we should use `\r\n` as the line ending to conform to the protocol. If it is `\r\n`, that means `\n` will be expanded to `\r\n`, so we should use `\n` as the terminator above. 

I'm not an expert in Python, so there may be a cleaner way to fix this. It looks like when reading from a file, there are options in terms of handling newlines, so maybe there is an option to set when writing to the output. I'm not sure as well if this would cause problems with the tcp server.

Let me know if this isn't clear - happy to give more info.

Discussion:

Patch:
diff --git a/pyls/__main__.py b/pyls/__main__.py
index fe9f564..b88ae07 100644
--- a/pyls/__main__.py
+++ b/pyls/__main__.py
@@ -53,4 +53,29 @@ def main():
     if args.tcp:
         language_server.start_tcp_lang_server(args.host, args.port, PythonLanguageServer)
     else:
-        language_server.start_io_lang_server(sys.stdin, sys.stdout, PythonLanguageServer)
+        stdin, stdout = _binary_stdio()
+        language_server.start_io_lang_server(stdin, stdout, PythonLanguageServer)
+
+
+def _binary_stdio():
+    """Construct binary stdio streams (not text mode).
+
+    This seems to be different for Window/Unix Python2/3, so going by:
+        https://stackoverflow.com/questions/2850893/reading-binary-data-from-stdin
+    """
+    PY3K = sys.version_info >= (3, 0)
+
+    if PY3K:
+        stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
+    else:
+        # Python 2 on Windows opens sys.stdin in text mode, and
+        # binary data that read from it becomes corrupted on \r\n
+        if sys.platform == "win32":
+            # set sys.stdin to binary mode
+            import os
+            import msvcrt
+            msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
+            msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
+        stdin, stdout = sys.stdin, sys.stdout
+
+    return stdin, stdout
diff --git a/pyls/server.py b/pyls/server.py
index 391ca99..9e3fcac 100644
--- a/pyls/server.py
+++ b/pyls/server.py
@@ -49,8 +49,8 @@ class JSONRPCServer(object):
         self._write_message(req.data)
 
     def _content_length(self, line):
-        if line.startswith("Content-Length: "):
-            _, value = line.split("Content-Length: ")
+        if line.startswith(b'Content-Length: '):
+            _, value = line.split(b'Content-Length: ')
             value = value.strip()
             try:
                 return int(value)
@@ -83,5 +83,5 @@ class JSONRPCServer(object):
             "Content-Type: application/vscode-jsonrpc; charset=utf8\r\n\r\n"
             "{}".format(content_length, body)
         )
-        self.wfile.write(response)
+        self.wfile.write(response.encode('utf-8'))
         self.wfile.flush()


Test patch:
diff --git a/test/test_language_server.py b/test/test_language_server.py
index b1a68a5..7f0de1b 100644
--- a/test/test_language_server.py
+++ b/test/test_language_server.py
@@ -28,12 +28,12 @@ def client_server():
     scr, scw = os.pipe()
 
     server = Thread(target=start_io_lang_server, args=(
-        os.fdopen(csr), os.fdopen(scw, 'w'), PythonLanguageServer
+        os.fdopen(csr, 'rb'), os.fdopen(scw, 'wb'), PythonLanguageServer
     ))
     server.daemon = True
     server.start()
 
-    client = JSONRPCClient(os.fdopen(scr), os.fdopen(csw, 'w'))
+    client = JSONRPCClient(os.fdopen(scr, 'rb'), os.fdopen(csw, 'wb'))
 
     yield client, server
 
@@ -95,10 +95,10 @@ def test_linting(client_server):
 
 
 def _get_notification(client):
-    request = jsonrpc.jsonrpc.JSONRPCRequest.from_json(client._read_message())
+    request = jsonrpc.jsonrpc.JSONRPCRequest.from_json(client._read_message().decode('utf-8'))
     assert request.is_notification
     return request.data
 
 
 def _get_response(client):
-    return json.loads(client._read_message())
+    return json.loads(client._read_message().decode('utf-8'))