neovim/neovim — pull request #1399 by equalsraf (2014-11-05 14:06:25+00:00)
Title: [WIP] Better handling for invalid msgpack-rpc

Yesterday I [accidentaly](https://gist.github.com/equalsraf/62cb08bcdd654d7f12e2) crashed Neovim by having the python-client push invalid data (invalid msgpack-rpc but possibly valid msgpack) into the channel. Basically msgpack_rpc_validate treated the object as an array when there was none and then it segfaulted.

So far I only added a commit to return when an error occurs, this fixes the segfault, but what happens now is that Neovim hangs calling the python process i.e.

1. Neovim calls a function in the host
2. Python host sends some (invalid data) into the channel
3. Neovim sends an error message to the host
4. (Neovim is now blocked)

I was considering closing the channel after serializing an error message i.e.

```diff
diff --git a/src/nvim/msgpack_rpc/channel.c b/src/nvim/msgpack_rpc/channel.c
index c2d16d1..96c24da 100644
--- a/src/nvim/msgpack_rpc/channel.c
+++ b/src/nvim/msgpack_rpc/channel.c
@@ -418,6 +418,7 @@ static void handle_request(Channel *channel, msgpack_object *request)
&error,
NIL,
&out_buffer));
+ close_channel(channel);
return;
}
```

Or perhaps `call_set_error()` would be better. But I'm tripping over myself because I get

```#0 0x00000000005e72b0 in wstream_write (wstream=0x0, buffer=0x8be400) at ../src/nvim/os/wstream.c:159
#1 0x00000000005e3da5 in job_write (job=0x8be570, buffer=0x8be400) at ../src/nvim/os/job.c:365
#2 0x00000000005f2801 in channel_write (channel=0x8be200, buffer=0x8be400) at ../src/nvim/msgpack_rpc/channel.c:480
#3 0x00000000005f2798 in on_request_event (event=...) at ../src/nvim/msgpack_rpc/channel.c:465
#4 0x00000000005e826b in process_events_from (queue=0x899a70) at ../src/nvim/os/event.c:159
#5 0x00000000005e81c2 in event_poll (ms=-1) at ../src/nvim/os/event.c:135
#6 0x00000000005f1cd7 in channel_send_call (id=1, method_name=0x8bdbe0 "python_eval", args=..., err=0x7fffffffbb90)
at ../src/nvim/msgpack_rpc/channel.c:226
#7 0x000000000046d4ab in f_rpcrequest (argvars=0x7fffffffc0e0, rettv=0x7fffffffc770) at ../src/nvim/eval.c:12508
#8 0x0000000000462803 in call_func (
funcname=0x8be2f3 "rpcrequest(s:pyhost_id, 'python_eval', '\"o\"+\"k\"') != 'ok' || !has('python')", len=10, rettv=0x7fffffffc770,
argcount=3, argvars=0x7fffffffc0e0, firstline=1, lastline=1, doesrange=0x7fffffffc29c, evaluate=1, selfdict=0x0)
at ../src/nvim/eval.c:6909
```

Basically Neovim trying to write into the channel after I called close and failing hard.