Repository: ethereum/web3.py

Issue:
Erorr in websockets.py: '<=' not supported between instances of 'int' and 'NoneType'
* web3 (4.3.0)
* websockets (4.0.1)
* Python: 3.6
* OS: osx HighSierra


### What was wrong?

`web3 = Web3(Web3.WebsocketProvider("ws://10.224.12.6:8546"))`
`web3.eth.syncing //returns data`

The websocket is clearly open but when I run a filter which is supposed to have many entries, I get the following error trace:

Upon running: `data = web3.eth.getFilterLogs(new_block_filter.filter_id)`, I get:

```
~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/web3/providers/websocket.py in make_request(self, method, params)
     81             WebsocketProvider._loop
     82         )
---> 83         return future.result()

/anaconda3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

/anaconda3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/web3/providers/websocket.py in coro_make_request(self, request_data)
     71         async with self.conn as conn:
     72             await conn.send(request_data)
---> 73             return json.loads(await conn.recv())
     74 
     75     def make_request(self, method, params):

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/websockets/protocol.py in recv(self)
    321             next_message.cancel()
    322             if not self.legacy_recv:
--> 323                 raise ConnectionClosed(self.close_code, self.close_reason)
    324 
    325     @asyncio.coroutine

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/websockets/exceptions.py in __init__(self, code, reason)
    145         self.reason = reason
    146         message = "WebSocket connection is closed: "
--> 147         if 3000 <= code < 4000:
    148             explanation = "registered"
    149         elif 4000 <= code < 5000:

TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```

The same filter runs fine (albeit a bit slow) using `Web3.HTTPProvider()`




Discussion:
I have confirmed that if I alter the filter to include very few blocks (around 10), and the result is a transaction list with around 10 entries and things are fine.

A filter which reproduces the error:
`new_block_filter = web3.eth.filter({"fromBlock":hex(5235798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})
`

A filter which works smoothly:
`new_block_filter = web3.eth.filter({"fromBlock":hex(5335798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})
`

And again, everything is fine with `HTTPProvider()`; it's only the `WebSocketProvider()`
The two lines that create a filter are the same. Can you update your comment @davehcker ?
cc @voith 
@carver my bad. I've updated the comment.

@davehcker I can replicate the issue with infura.
```python
from web3 import Web3

w3 = Web3(Web3.WebsocketProvider('wss://mainnet.infura.io/_ws'))
new_block_filter = w3.eth.filter({"fromBlock":hex(5235798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})
w3.eth.getFilterLogs(new_block_filter.filter_id)
```

Firstly, failing on `NoneType` is a bug in `websockets` :
```
~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/websockets/exceptions.py in __init__(self, code, reason)
    145         self.reason = reason
    146         message = "WebSocket connection is closed: "
--> 147         if 3000 <= code < 4000:
    148             explanation = "registered"
    149         elif 4000 <= code < 5000:

TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```
However, I need to investigate what is causing the connection to close!
@davehcker The problem was that the filter returned a lot of data(more than the default size that a websocket connection allows). On my re attempt to fetch  data from the filter I got the following error:
```
~/Projects/web3.py/.tox/py36-integration-parity-ws/lib/python3.6/site-packages/websockets/framing.py in read(cls, reader, mask, max_size, extensions)
    122                 "Payload length exceeds size limit ({} > {} bytes)"
--> 123                 .format(length, max_size))
    124         if mask:

PayloadTooBig: Payload length exceeds size limit (33572833 > 1048576 bytes)
```
The `max_size` has a default value of `2 ** 20`:
https://github.com/aaugustin/websockets/blob/b7a2bfe4ed5a14208ee3658eaad9be9f689cb6fb/websockets/client.py#L329
I tried increasing the value of `max_size` and it worked. However, the current implementation of `WebsocketProvider` does not allow passing extra config, for which I have opened #914.

@davehcker Can you help me check if increasing the value of `max_size` fixes the issue for you?
Install web3 from my branch: 
`pip install git+https://github.com/voith/web3.py.git@allow-passing-websocket-kwargs`
and then:
```python
from web3 import Web3

w3 = Web3(Web3.WebsocketProvider('wss://mainnet.infura.io/_ws', max_size=999999999))
new_block_filter = w3.eth.filter({"fromBlock":hex(5235798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})
w3.eth.getFilterLogs(new_block_filter.filter_id)
```
However, I have no idea why it was giving that random `NoneType` error earlier.

Also, A suggestion for you: Instead of fetching such big chunks of data, break `fromBlock` and `toBlock` in to smaller batches!
@voith  just guessing but ws client default timeout is ten seconds, i believe, and explains the None for large requests. the first time around, no infura cache in place, i presume, the client times out. second time around, infura cache is primed, you get the size issue. passing a (much) larger timeout value with the kwargs should solve the issue client side.
@boneyard93501 I know what happened. I have two different virtualenv's on my machine. One with with `websockets` v4 and one with `v5`. Unconsciously, I switched between the two while testing. 😆  
I get the `NoneType` error in `v4`. However, [a security fix](https://github.com/aaugustin/websockets/pull/407) in `websockets` released in `v5` errors earlier with a helpful error message. Interestingly, we bumped the `websockets` version to `v5` [just yesterday](https://github.com/ethereum/web3.py/pull/912). 
Also setting `max_size` to a large value fixes the issue in `v4` as well. 

> just guessing but ws client default timeout is ten seconds, , i believe, and explains the None for large requests.

That was not  the issue here. Here, data was being received continuously. It was just beyond the `max_size` limit. 

@carver I think It's time to cut a minor release if @davehcker can confirm that #914 fixes the issue for them.
Sounds good. Once we get #914 in, I'll cut a release.

Resolving patch:
diff --git a/web3/providers/websocket.py b/web3/providers/websocket.py
--- a/web3/providers/websocket.py
+++ b/web3/providers/websocket.py
@@ -8,10 +8,15 @@
 
 import websockets
 
+from web3.exceptions import (
+    ValidationError,
+)
 from web3.providers.base import (
     JSONBaseProvider,
 )
 
+RESTRICTED_WEBSOCKET_KWARGS = {'uri', 'loop'}
+
 
 def _start_event_loop(loop):
     asyncio.set_event_loop(loop)
@@ -32,14 +37,17 @@ def get_default_endpoint():
 
 class PersistentWebSocket:
 
-    def __init__(self, endpoint_uri, loop):
+    def __init__(self, endpoint_uri, loop, websocket_kwargs):
         self.ws = None
         self.endpoint_uri = endpoint_uri
         self.loop = loop
+        self.websocket_kwargs = websocket_kwargs
 
     async def __aenter__(self):
         if self.ws is None:
-            self.ws = await websockets.connect(uri=self.endpoint_uri, loop=self.loop)
+            self.ws = await websockets.connect(
+                uri=self.endpoint_uri, loop=self.loop, **self.websocket_kwargs
+            )
         return self.ws
 
     async def __aexit__(self, exc_type, exc_val, exc_tb):
@@ -55,13 +63,26 @@ class WebsocketProvider(JSONBaseProvider):
     logger = logging.getLogger("web3.providers.WebsocketProvider")
     _loop = None
 
-    def __init__(self, endpoint_uri=None):
+    def __init__(self, endpoint_uri=None, websocket_kwargs=None):
         self.endpoint_uri = endpoint_uri
         if self.endpoint_uri is None:
             self.endpoint_uri = get_default_endpoint()
         if WebsocketProvider._loop is None:
             WebsocketProvider._loop = _get_threaded_loop()
-        self.conn = PersistentWebSocket(self.endpoint_uri, WebsocketProvider._loop)
+        if websocket_kwargs is None:
+            websocket_kwargs = {}
+        else:
+            found_restricted_keys = set(websocket_kwargs.keys()).intersection(
+                RESTRICTED_WEBSOCKET_KWARGS
+            )
+            if found_restricted_keys:
+                raise ValidationError(
+                    '{0} are not allowed in websocket_kwargs, '
+                    'found: {1}'.format(RESTRICTED_WEBSOCKET_KWARGS, found_restricted_keys)
+                )
+        self.conn = PersistentWebSocket(
+            self.endpoint_uri, WebsocketProvider._loop, websocket_kwargs
+        )
         super().__init__()
 
     def __str__(self):


Test patch:
diff --git a/tests/core/providers/test_websocket_provider.py b/tests/core/providers/test_websocket_provider.py
new file mode 100644
--- /dev/null
+++ b/tests/core/providers/test_websocket_provider.py
@@ -0,0 +1,15 @@
+import pytest
+
+from web3.exceptions import (
+    ValidationError,
+)
+from web3.providers.websocket import (
+    WebsocketProvider,
+)
+
+
+def test_restricted_websocket_kwargs():
+    invalid_kwargs = {'uri': 'ws://127.0.0.1:8546'}
+    re_exc_message = r'.*found: {0}*'.format(set(invalid_kwargs.keys()))
+    with pytest.raises(ValidationError, match=re_exc_message):
+        WebsocketProvider(websocket_kwargs=invalid_kwargs)
diff --git a/tests/integration/common.py b/tests/integration/common.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/common.py
@@ -0,0 +1,17 @@
+import pytest
+
+from websockets.exceptions import (
+    ConnectionClosed,
+)
+
+from web3 import Web3
+
+
+class MiscWebsocketTest:
+
+    def test_websocket_max_size_error(self, web3, endpoint_uri):
+        w3 = Web3(Web3.WebsocketProvider(
+            endpoint_uri=endpoint_uri, websocket_kwargs={'max_size': 1})
+        )
+        with pytest.raises(ConnectionClosed):
+            w3.eth.getBlock(0)
diff --git a/tests/integration/go_ethereum/test_goethereum_ws.py b/tests/integration/go_ethereum/test_goethereum_ws.py
--- a/tests/integration/go_ethereum/test_goethereum_ws.py
+++ b/tests/integration/go_ethereum/test_goethereum_ws.py
@@ -1,5 +1,8 @@
 import pytest
 
+from tests.integration.common import (
+    MiscWebsocketTest,
+)
 from tests.integration.utils import (
     wait_for_ws,
 )
@@ -65,3 +68,7 @@ class TestGoEthereumNetModuleTest(GoEthereumNetModuleTest):
 
 class TestGoEthereumPersonalModuleTest(GoEthereumPersonalModuleTest):
     pass
+
+
+class TestMiscWebsocketTest(MiscWebsocketTest):
+    pass
diff --git a/tests/integration/parity/test_parity_ws.py b/tests/integration/parity/test_parity_ws.py
--- a/tests/integration/parity/test_parity_ws.py
+++ b/tests/integration/parity/test_parity_ws.py
@@ -1,6 +1,9 @@
 import os
 import pytest
 
+from tests.integration.common import (
+    MiscWebsocketTest,
+)
 from tests.integration.utils import (
     wait_for_ws,
 )
@@ -96,3 +99,7 @@ class TestParityPersonalModuleTest(ParityPersonalModuleTest):
 
 class TestParityTraceModuleTest(ParityTraceModuleTest):
     pass
+
+
+class TestMiscWebsocketTest(MiscWebsocketTest):
+    pass