Repository: ethereum/web3.py

Issue:
Extend Web3.toJSON json encoder (Web3JsonEncoder) to support bytes type
* Version: 5.30.0
* Python: 3.9.4
* OS:  Microsoft Windows 10 [Version 10.0.19042.1826]
* `pip freeze` output

```
aiohttp==3.7.4.post0
aiosignal==1.2.0
altgraph==0.17.2
ansicon==1.89.0
appdirs==1.4.4
async-timeout==3.0.1
atomicwrites==1.4.1
attrs==20.3.0
Automat==20.2.0
bankreader==0.2.3
base58==2.1.1
beautifulsoup4==4.9.3
bitarray==1.2.2
black==22.6.0
blessed==1.19.0
Brotli==1.0.9
bscscan-python==2.0.0
certifi==2020.12.5
cffi==1.14.5
cfscrape==2.1.1
chardet==4.0.0
charset-normalizer==2.0.9
click==8.0.3
cloudscraper==1.2.60
colorama==0.4.4
coloredlogs==15.0.1
colorlog==6.6.0
config-checker==0.0.1
constantly==15.1.0
cryptography==3.4.7
cssselect==1.1.0
cssselect2==0.4.1
cytoolz==0.11.2
db-ops==0.1.1
decompyle3==3.8.0
discord.py-self==1.9.2
docopt==0.6.2
et-xmlfile==1.1.0
eth-abi==2.1.1
eth-account==0.5.7
eth-hash==0.3.2
eth-keyfile==0.5.1
eth-keys==0.3.4
eth-rlp==0.2.1
eth-typing==2.2.2
eth-utils==1.10.0
etherscan-python==2.1.0
ethplorer-py==1.0
ezenv==0.92
Flask==2.1.3
fonttools==4.23.1
frozenlist==1.2.0
future==0.18.2
h2==3.2.0
halo==0.0.31
hexbytes==0.2.2
hpack==3.0.0
html5lib==1.1
humanfriendly==10.0
hyperframe==5.2.0
hyperlink==21.0.0
idna==2.10
img2pdf==0.4.1
importlib-metadata==4.12.0
incremental==21.3.0
influxdb==5.3.1
iniconfig==1.1.1
inquirer==2.8.0
install==1.3.5
ipfshttpclient==0.8.0a2
itemadapter==0.2.0
itemloaders==1.0.4
itsdangerous==2.1.2
Jinja2==3.1.2
jinxed==1.1.0
jmespath==0.10.0
jsonschema==3.2.0
keyboard==0.13.5
log-symbols==0.0.14
lru-dict==1.1.7
lxml==4.6.3
markdown2==2.4.0
MarkupSafe==2.1.1
md2pdf==0.5
morse-talk==0.2
msgpack==1.0.2
multiaddr==0.0.9
multidict==5.2.0
mypy-extensions==0.4.3
netaddr==0.8.0
numpy==1.20.3
oauthlib==3.1.1
openpyxl==3.0.7
packaging==21.3
pandas==1.2.4
panoramix-decompiler-abi==0.5.2
parsel==1.6.0
parsimonious==0.8.1
pathspec==0.9.0
pefile==2021.9.3
pika==1.3.0
pika-stubs==0.1.3
pikepdf==2.12.0
Pillow==8.2.0
platformdirs==2.5.2
pluggy==1.0.0
priority==1.3.0
prompt-toolkit==1.0.14
Protego==0.1.16
protobuf==3.19.1
py==1.11.0
pyaes==1.6.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycoingecko==1.4.1
pycparser==2.20
pycryptodome==3.12.0
PyDispatcher==2.0.5
pydyf==0.0.3
Pygments==2.10.0
PyInquirer==1.0.3
pyinstaller==4.9
pyinstaller-hooks-contrib==2022.2
pymongo==4.2.0
pyonepassword==2.0.0
pyOpenSSL==20.0.1
pyparsing==3.0.8
Pyphen==0.10.0
pyreadline3==3.3
pyRFC3339==1.1
pyrsistent==0.18.0
PySocks==1.7.1
pytest==7.1.2
pytest-mock==3.8.2
python-dateutil==2.8.1
python-editor==1.0.4
pytz==2021.1
pywin32==302
pywin32-ctypes==0.2.0
PyYAML==6.0
qrcode==7.3.1
queuelib==1.5.0
readchar==2.0.1
regex==2021.11.10
requests==2.26.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rlp==2.0.1
rsa==4.8
Scrapy==2.5.0
scribd-downloader==1.3.1
selenium==3.141.0
service-identity==18.1.0
six==1.15.0
soupsieve==2.2.1
spark-parser==1.8.9
spinners==0.0.24
Telethon==1.24.0
termcolor==1.1.0
timeout-decorator==0.5.0
tinycss2==1.1.0
tomli==2.0.1
toolz==0.11.2
tweepy==3.10.0
Twisted==21.2.0
twisted-iocpsupport==1.0.1
typing_extensions==4.0.1
uncompyle6==3.8.0
urllib3==1.26.4
varint==1.0.2
w3lib==1.22.0
wcwidth==0.2.5
weasyprint==53.0b1
web3==5.30.0
webencodings==0.5.1
websockets==9.1
Werkzeug==2.1.2
wget==3.2
whale-alert==0.0.4
xdis==6.0.3
xlrd==2.0.1
xortool==1.0.1
yarl==1.7.2
zipp==3.8.0
zope.interface==5.3.0
zopfli==0.1.8
```

### What was wrong?

While attempting to convert some filtered events of `LogReceipt` type (https://github.com/ethereum/web3.py/blob/master/web3/types.py#L164 as returned by https://web3py.readthedocs.io/en/latest/filters.html#web3.utils.filters.Filter.get_new_entries) to json using `Web3.toJSON`, I encountered events with `bytes` as type, that resulted in an `TypeError` exception being thrown:

```
Traceback (most recent call last):
  File "C:\Python\lib\site-packages\web3\_utils\encoding.py", line 206, in _friendly_json_encode
    encoded = json.dumps(obj, cls=cls)
  File "C:\Python\lib\json\__init__.py", line 234, in dumps
    return cls(
  File "C:\Python\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Python\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Python\lib\site-packages\web3\_utils\encoding.py", line 285, in default
    return json.JSONEncoder.default(self, obj)
  File "C:\Python\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
```

I would consider extending the Web3.toJSON used `JsonEncoder` (https://github.com/ethereum/web3.py/blob/master/web3/_utils/encoding.py#L286) to also support `bytes`.

* What type of node you were connecting to: infura for Ethereum network

### How can it be fixed?

Resolving this can be very easy just by reusing the available types and adding this check in https://github.com/ethereum/web3.py/blob/master/web3/_utils/encoding.py#L290 :
```
if isinstance(obj, bytes):
    return HexStr(HexBytes(obj).hex())
```

My reasoning is that, a specific `toJSON` function for web3py should be able to process anything that other web3py APIs return (TypeDicts or AttributeDict at least).

Can be reproduced by parsing the `OrderFulfilled` event from the (OpenSea: Seaport 1.1 - Etherscan contract 0x00000000006c3852cbEf3e08E8dF289169EdE581) for example.

---
**Note:** We prefer to use issues to track our work. If you think you've encountered a bug in web3py or 
have a feature request, you're in the right place. If you have implementation or usage questions, 
please refer to our [documentation](https://web3py.readthedocs.io/en/latest/) and/or join the conversation 
on [discord](https://discord.gg/GHryRvPB84).



Discussion:

Resolving patch:
diff --git a/web3/_utils/encoding.py b/web3/_utils/encoding.py
--- a/web3/_utils/encoding.py
+++ b/web3/_utils/encoding.py
@@ -290,9 +290,11 @@ def encode_single_packed(_type: TypeStr, value: Any) -> bytes:
 class Web3JsonEncoder(json.JSONEncoder):
     def default(self, obj: Any) -> Union[Dict[Any, Any], HexStr]:
         if isinstance(obj, AttributeDict):
-            return {k: v for k, v in obj.items()}
-        if isinstance(obj, HexBytes):
+            return obj.__dict__
+        elif isinstance(obj, HexBytes):
             return HexStr(obj.hex())
+        elif isinstance(obj, bytes):
+            return to_hex(obj)
         return json.JSONEncoder.default(self, obj)
 
 
diff --git a/web3/providers/async_base.py b/web3/providers/async_base.py
--- a/web3/providers/async_base.py
+++ b/web3/providers/async_base.py
@@ -17,6 +17,7 @@
 
 from web3._utils.encoding import (
     FriendlyJsonSerde,
+    Web3JsonEncoder,
 )
 from web3.exceptions import (
     ProviderConnectionError,
@@ -106,7 +107,7 @@ def encode_rpc_request(self, method: RPCEndpoint, params: Any) -> bytes:
             "params": params or [],
             "id": request_id,
         }
-        encoded = FriendlyJsonSerde().json_encode(rpc_dict)
+        encoded = FriendlyJsonSerde().json_encode(rpc_dict, cls=Web3JsonEncoder)
         return to_bytes(text=encoded)
 
     def decode_rpc_response(self, raw_response: bytes) -> RPCResponse:
diff --git a/web3/providers/base.py b/web3/providers/base.py
--- a/web3/providers/base.py
+++ b/web3/providers/base.py
@@ -15,6 +15,7 @@
 
 from web3._utils.encoding import (
     FriendlyJsonSerde,
+    Web3JsonEncoder,
 )
 from web3.exceptions import (
     ProviderConnectionError,
@@ -106,7 +107,7 @@ def encode_rpc_request(self, method: RPCEndpoint, params: Any) -> bytes:
             "params": params or [],
             "id": next(self.request_counter),
         }
-        encoded = FriendlyJsonSerde().json_encode(rpc_dict)
+        encoded = FriendlyJsonSerde().json_encode(rpc_dict, Web3JsonEncoder)
         return to_bytes(text=encoded)
 
     def is_connected(self, show_traceback: bool = False) -> bool:


Test patch:
diff --git a/tests/core/utilities/test_encoding.py b/tests/core/utilities/test_encoding.py
--- a/tests/core/utilities/test_encoding.py
+++ b/tests/core/utilities/test_encoding.py
@@ -11,6 +11,9 @@
     is_hex,
     to_hex,
 )
+from hexbytes import (
+    HexBytes,
+)
 from hypothesis import (
     example,
     given,
@@ -19,6 +22,7 @@
 
 from web3._utils.encoding import (
     FriendlyJsonSerde as FriendlyJson,
+    Web3JsonEncoder,
     hex_encode_abi_type,
     hexstr_if_str,
     text_if_str,
@@ -26,6 +30,9 @@
 from web3._utils.hypothesis import (
     hexstr_strategy,
 )
+from web3.datastructures import (
+    AttributeDict,
+)
 from web3.providers import (
     JSONBaseProvider,
 )
@@ -139,13 +146,85 @@ def test_text_if_str_on_text(val):
             TypeError,
             "Could not encode to JSON: .*'other_date'.*is not JSON serializable",
         ),
+        (
+            {
+                "bytes_obj": b"\x00\x01\x02\x03",
+                "hexbytes_obj": HexBytes(b"\x00\x01\x02\x03"),
+            },
+            None,
+            '{"bytes_obj": "0x00010203", "hexbytes_obj": "0x00010203"}',
+        ),
+        (
+            AttributeDict(
+                {
+                    "transactions": [
+                        AttributeDict(
+                            {
+                                "hash": HexBytes(
+                                    "0x142ab034696c09dcfb2a8b086b494f3f4c419e67b6c04d95882f87156a3b6f35"  # noqa: E501
+                                ),
+                                "nonce": 3,
+                                "blockHash": HexBytes(
+                                    "0xe14a0029f8ae6f41ab2287871d7f2f0658696ce0a842883147629cc1b300fc89"  # noqa: E501
+                                ),
+                                "blockNumber": 4322026,
+                                "transactionIndex": 2,
+                                "from": "0xb17473E95Cc2c37f88C56593Ff8767e10c972359",
+                                "to": "0x8daF62dF221D11b470Ca4531305470DaE4A65784",
+                                "value": 0,
+                                "gasPrice": 3459216019,
+                                "maxPriorityFeePerGas": 2500000000,
+                                "maxFeePerGas": 4421310622,
+                                "gas": 26856,
+                                "input": HexBytes("0x3857"),
+                                "chainId": 1337,
+                                "type": 2,
+                                "accessList": [],
+                                "v": 1,
+                                "s": HexBytes(
+                                    "0x6f5216fc207221a11efe2e4c3e3a881a0b5ca286ede538fc9dbc403b2009ea76"  # noqa: E501
+                                ),
+                                "r": HexBytes(
+                                    "0xd148ae70c8cbef3a038e70e6d1639f0951e60a2965820f33bad19d0a6c2b8116"  # noqa: E501
+                                ),
+                                "yParity": 1,
+                            },
+                        )
+                    ]
+                }
+            ),
+            None,
+            """{"transactions": [{
+                "hash": "0x142ab034696c09dcfb2a8b086b494f3f4c419e67b6c04d95882f87156a3b6f35",  # noqa: E501
+                "nonce": 3,
+                "blockHash": "0xe14a0029f8ae6f41ab2287871d7f2f0658696ce0a842883147629cc1b300fc89",  # noqa: E501
+                "blockNumber": 4322026,
+                "transactionIndex": 2,
+                "from": "0xb17473E95Cc2c37f88C56593Ff8767e10c972359",
+                "to": "0x8daF62dF221D11b470Ca4531305470DaE4A65784",
+                "value": 0,
+                "gasPrice": 3459216019,
+                "maxPriorityFeePerGas": 2500000000,
+                "maxFeePerGas": 4421310622,
+                "gas": 26856,
+                "input": "0x3857",
+                "chainId": 1337,
+                "type": 2,
+                "accessList": [],
+                "v": 1,
+                "s": "0x6f5216fc207221a11efe2e4c3e3a881a0b5ca286ede538fc9dbc403b2009ea76",  # noqa: E501
+                "r": "0xd148ae70c8cbef3a038e70e6d1639f0951e60a2965820f33bad19d0a6c2b8116",  # noqa: E501
+                "yParity": 1,
+            }]}""",
+        ),
     ),
+    ids=("datetime", "datetime_error", "bytes types", "nested attrdict with hexbytes"),
 )
-def test_friendly_json_encode(py_obj, exc_type, expected):
+def test_friendly_json_encode_with_web3_json_encoder(py_obj, exc_type, expected):
     if exc_type is None:
-        assert literal_eval(FriendlyJson().json_encode(py_obj)) == literal_eval(
-            expected
-        )
+        assert literal_eval(
+            FriendlyJson().json_encode(py_obj, Web3JsonEncoder)
+        ) == literal_eval(expected)
     else:
         with pytest.raises(exc_type, match=expected):
             FriendlyJson().json_encode(py_obj)