Repository: ethereum/web3.py

Issue:
ValueError when using IPCProvider
* Version: 4.2.1
* Python: 2.7/3.4/3.5
* OS: osx/linux/win


### What was wrong?

When using IPCProvider sometimes ValueError is encountered if the response is incomplete. Looks like in this pull request https://github.com/ethereum/web3.py/pull/785/files the author forgot to add `except ValueError` to IPCProvider  https://github.com/ethereum/web3.py/blob/master/web3/providers/ipc.py#L176

### How can it be fixed?

Add `except ValueError` to IPCProvider.make_request https://github.com/ethereum/web3.py/blob/master/web3/providers/ipc.py#L176



Discussion:
Ah, thanks for tracking that down! Another solution would be to re-raise a `JSONDecodeError` here:
https://github.com/ethereum/web3.py/pull/785/files#diff-0260bdf984abdd4b8ab8987ea75d032cR258
@carver  i'll look at it tonight. i think the reason for not re-raising was to avoid the nested (double) error. let me have a look.
i'm having a bit of trouble reproducing the issue.
for, say,
```bash
curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"net_listening","params":[],"id":67}'
```
a complete result would be `{"jsonrpc":"2.0","result":true,"id":67}` and an incomplete result would be`{"result":true,"id":67}` assuming an incomplete response is one where one or more of the json-rpc required key-value pairs is missing.
``` python
>>> web3.__version__
'4.2.1'
>>> incomplete_res = b'{result":true,"id":67}'
>>> # passes the has_valid_json_rpc_ending check and
>>> IPCProvider().decode_rpc_response(incomplete_response)
{'result': True, 'id': 67}
```
which checks out. if instead we're looking at a structurally invalid json response other than invalid rpc endings:
``` python
>>> web3.__version__
'4.2.1'
>>> invalid_response = b'{"result":true,"id"}'
>>> IPCProvider().decode_rpc_response(invalid_response)
Traceback (most recent call last):
  File "/Users/93501/localDev/source-code/forks/web3.py/web3/utils/encoding.py", line 254, in json_decode
    decoded = json.loads(json_str)
  File "/Users/93501/.pyenv/versions/3.6.4/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Users/93501/.pyenv/versions/3.6.4/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/93501/.pyenv/versions/3.6.4/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 20 (char 19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/93501/localDev/source-code/forks/web3.py/web3/providers/base.py", line 63, in decode_rpc_response
    return FriendlyJsonSerde().json_decode(text_response)
  File "/Users/93501/localDev/source-code/forks/web3.py/web3/utils/encoding.py", line 258, in json_decode
    raise ValueError(err_msg)
ValueError: Could not decode '{"result":true,"id"}' because of Expecting ':' delimiter: line 1 column 20 (char 19).
>>>
```

is raising the stacked Exception the issue?
 
@boneyard93501 I meant the structurally invalid result. When the json rpc response is not fully received from the socket, the IPC provider will try to parse it anyway and if the parsing failed, it will continue to receive more chunks from the socket, until the json is valid. 

An example of incomplete response would be:

`'{"jsonrpc":"2.0","result":["test"]'`
k. thanks. appreciate the update.

at the surface, that looks more like a `has_valid_json_rpc_ending` error of omission since it's a little permissive. if we tighten that up, we don't even get to the json processing. at the mere minimum we could check for matching startswith/endswith. 
however, having PersistentSocket parse the complete response ought to be the preferred course of action. maybe  increase Timeout a tad and do a socket.reset before breaking from the while loop to at least prevent orphaned chunks to further pollute the results. that, in conjunction with a tighter valid rpc check, might give us much improved coverage.  deferring to @carver
`has_valid_json_rpc_ending` will always be optimistic, and fail sometimes. For example `'{"jsonrpc":"2.0","result":[{"key":"val"}'`. The provider's parser already handles that case by capturing `JSONDecodeError`, so all we need to do is raise a `JSONDecodeError` instead of a `ValueError` in our friendly decoder. It should be pretty quick, I'll take care of it.

Resolving patch:
diff --git a/web3/utils/encoding.py b/web3/utils/encoding.py
--- a/web3/utils/encoding.py
+++ b/web3/utils/encoding.py
@@ -255,7 +255,9 @@ def json_decode(self, json_str):
             return decoded
         except json.decoder.JSONDecodeError as exc:
             err_msg = 'Could not decode {} because of {}.'.format(repr(json_str), exc)
-            raise ValueError(err_msg)
+            # Calling code may rely on catching JSONDecodeError to recognize bad json
+            # so we have to re-raise the same type.
+            raise json.decoder.JSONDecodeError(err_msg, exc.doc, exc.pos)
 
     def json_encode(self, obj):
         try:


Test patch:
diff --git a/tests/core/providers/test_ipc_provider.py b/tests/core/providers/test_ipc_provider.py
--- a/tests/core/providers/test_ipc_provider.py
+++ b/tests/core/providers/test_ipc_provider.py
@@ -1,11 +1,73 @@
+import os
+import pytest
+import socket
+import tempfile
+from threading import (
+    Thread,
+)
+import time
+import uuid
+
 from web3.providers.ipc import (
     IPCProvider,
 )
 
 
+@pytest.fixture
+def jsonrpc_ipc_pipe_path():
+    with tempfile.TemporaryDirectory() as temp_dir:
+        ipc_path = os.path.join(temp_dir, '{0}.ipc'.format(uuid.uuid4()))
+        try:
+            yield ipc_path
+        finally:
+            if os.path.exists(ipc_path):
+                os.remove(ipc_path)
+
+
 def test_ipc_no_path():
     """
     IPCProvider.isConnected() returns False when no path is supplied
     """
     ipc = IPCProvider(None)
     assert ipc.isConnected() is False
+
+
+@pytest.fixture
+def simple_ipc_server(jsonrpc_ipc_pipe_path):
+    serv = socket.socket(socket.AF_UNIX)
+    serv.bind(jsonrpc_ipc_pipe_path)
+    serv.listen(1)
+    try:
+        yield serv
+    finally:
+        serv.close()
+
+
+@pytest.fixture
+def serve_empty_result(simple_ipc_server):
+    def reply():
+        connection, client_address = simple_ipc_server.accept()
+        try:
+            connection.recv(1024)
+            connection.sendall(b'{"id":1, "result": {}')
+            time.sleep(0.1)
+            connection.sendall(b'}')
+        finally:
+            # Clean up the connection
+            connection.close()
+            simple_ipc_server.close()
+
+    thd = Thread(target=reply, daemon=True)
+    thd.start()
+
+    try:
+        yield
+    finally:
+        thd.join()
+
+
+def test_sync_waits_for_full_result(jsonrpc_ipc_pipe_path, serve_empty_result):
+    provider = IPCProvider(jsonrpc_ipc_pipe_path, timeout=3)
+    result = provider.make_request("method", [])
+    assert result == {'id': 1, 'result': {}}
+    provider._socket.sock.close()