Repository: ethereum/web3.py (python)

Issue:
Difficult to attach one extra module to a web3 instance..
### What was wrong?

There is no simple way to include a custom module in a web3 instance while still also including all of the default modules.

### How can it be fixed?

Ideally it should be a one-liner

```python

class Foo(ModuleV2):
    ...

w3 = Web3(...)
w3.attach('foo', Foo)

# or

w3 = Web3(..., extra_modules=(('foo', Foo),)
```

Pull request:
Add support for external modules to web3 instance
### What was wrong?

- We don't have an easy way to attach modules to the `Web3` instance. This will help users customize their web3 experience but can also help to support things like attaching layer 2 modules from a separate library or any other modules that can prove useful.

- Closes #2231

### How was it fixed?

- Added support for attaching external modules at `Web3` instantiation. This is separate from the internal default modules already attached at instantiation.
- Added support for attaching a single module via a new `attach_module()` method
- Added support for passing in a module as the class itself if there are no submodules, rather than having to pass in a tuple with a single module class.
- Tests added

### Todo:
- [x] Add entry to the [release notes](https://github.com/ethereum/web3.py/blob/master/newsfragments/README.md)
- [x] Add documentation
- [x] Add cute animal picture

#### Cute Animal Picture

![20220104_155122](https://user-images.githubusercontent.com/3532824/148135056-06ae0b72-9e7c-4603-8688-7cb1cbc58d46.jpg)


Patch:
diff --git a/docs/providers.rst b/docs/providers.rst
index 8e9fff84..9cdcd190 100644
--- a/docs/providers.rst
+++ b/docs/providers.rst
@@ -387,7 +387,7 @@ AsyncHTTPProvider
         >>> from web3.net import AsyncNet
 
         >>> w3 = Web3(AsyncHTTPProvider("http://127.0.0.1:8545"),
-        ...           modules={'eth': (AsyncEth,), 'net': (AsyncNet,)},
+        ...           modules={'eth': AsyncEth, 'net': AsyncNet},
         ...           middlewares=[])  # See supported middleware section below for middleware options
 
     Under the hood, the ``AsyncHTTPProvider`` uses the python
diff --git a/docs/web3.main.rst b/docs/web3.main.rst
index 1c9da42c..6eae8850 100644
--- a/docs/web3.main.rst
+++ b/docs/web3.main.rst
@@ -11,7 +11,7 @@ Web3 API
 
 .. py:class:: Web3(provider)
 
-Each ``web3`` instance exposes the following APIs.
+Each ``Web3`` instance exposes the following APIs.
 
 Providers
 ~~~~~~~~~
@@ -388,7 +388,7 @@ Check Encodability
 RPC APIS
 ~~~~~~~~
 
-Each ``web3`` instance also exposes these namespaced APIs.
+Each ``Web3`` instance also exposes these namespaced APIs.
 
 
 .. py:attribute:: Web3.eth
@@ -410,3 +410,70 @@ Each ``web3`` instance also exposes these namespaced APIs.
 .. py:attribute:: Web3.parity
 
     See :doc:`./web3.parity`
+
+
+Attaching Modules
+~~~~~~~~~~~~~~~~~
+
+Modules that inherit from the ``web3.module.Module`` class may be attached to the ``Web3`` instance either at
+instantiation or by making use of the ``attach_modules()`` method.
+
+To instantiate the ``Web3`` instance with external modules:
+
+.. code-block:: python
+
+    >>> from web3 import Web3, EthereumTesterProvider
+    >>> w3 = Web3(
+    ...     EthereumTesterProvider(),
+    ...     external_modules={
+    ...         # ModuleClass objects in this example inherit from the `web3.module.Module` class
+    ...         'module1': ModuleClass1,
+    ...         'module2': (ModuleClass2, {
+    ...             'submodule1': ModuleClass3,
+    ...             'submodule2': (ModuleClass4, {
+    ...                 'submodule2a': ModuleClass5,  # submodule children may be nested further if necessary
+    ...             })
+    ...         })
+    ...     }
+    ... )
+
+    # `return_zero`, in this case, is an example attribute of the `ModuleClass1` object
+    >>> w3.module1.return_zero
+    0
+    >>> w3.module2.submodule1.return_one
+    1
+    >>> w3.module2.submodule2.submodule2a.return_two
+    2
+
+
+.. py:method:: w3.attach_modules(modules)
+
+    The ``attach_modules()`` method can be used to attach external modules after the ``Web3`` instance has been
+    instantiated.
+
+    Modules are attached via a `dict` with module names as the keys. The values can either be the module classes
+    themselves, if there are no submodules, or two-item tuples with the module class as the 0th index and a similarly
+    built `dict` containing the submodule information as the 1st index. This pattern may be repeated as necessary.
+
+    .. note:: Module classes must inherit from the ``web3.module.Module`` class.
+
+    .. code-block:: python
+
+        >>> from web3 import Web3, EthereumTesterProvider
+        >>> w3 = Web3(EthereumTesterProvider())
+
+        >>> w3.attach_modules({
+        ...     'module1': ModuleClass1,  # the module class itself may be used for a single module with no submodules
+        ...     'module2': (ModuleClass2, {  # a tuple with module class and corresponding submodule dict may be used for modules with submodules
+        ...         'submodule1': ModuleClass3,
+        ...         'submodule2': (ModuleClass4, {  # this pattern may be repeated as necessary
+        ...             'submodule2a': ModuleClass5,
+        ...         })
+        ...     })
+        ... })
+        >>> w3.module1.return_zero
+        0
+        >>> w3.module2.submodule1.return_one
+        1
+        >>> w3.module2.submodule2.submodule2a.return_two
+        2
diff --git a/newsfragments/2288.feature.rst b/newsfragments/2288.feature.rst
new file mode 100644
index 00000000..102c5e75
--- /dev/null
+++ b/newsfragments/2288.feature.rst
@@ -0,0 +1,1 @@
+Support for attaching external modules to the ``Web3`` instance when instantiating the ``Web3`` instance, via the ``external_modules`` argument, or via the new ``attach_modules()`` method
\ No newline at end of file
diff --git a/web3/_utils/module.py b/web3/_utils/module.py
index afc0e46e..dd84e6e8 100644
--- a/web3/_utils/module.py
+++ b/web3/_utils/module.py
@@ -18,11 +18,13 @@ if TYPE_CHECKING:
 
 def attach_modules(
     parent_module: Union["Web3", "Module"],
-    module_definitions: Dict[str, Sequence[Any]],
+    module_definitions: Dict[str, Any],
     w3: Optional[Union["Web3", "Module"]] = None
 ) -> None:
     for module_name, module_info in module_definitions.items():
-        module_class = module_info[0]
+        module_info_is_list_like = isinstance(module_info, Sequence)
+
+        module_class = module_info[0] if module_info_is_list_like else module_info
 
         if hasattr(parent_module, module_name):
             raise AttributeError(
@@ -36,9 +38,10 @@ def attach_modules(
         else:
             setattr(parent_module, module_name, module_class(w3))
 
-        if len(module_info) == 2:
-            submodule_definitions = module_info[1]
-            module = getattr(parent_module, module_name)
-            attach_modules(module, submodule_definitions, w3)
-        elif len(module_info) != 1:
-            raise ValidationError("Module definitions can only have 1 or 2 elements.")
+        if module_info_is_list_like:
+            if len(module_info) == 2:
+                submodule_definitions = module_info[1]
+                module = getattr(parent_module, module_name)
+                attach_modules(module, submodule_definitions, w3)
+            elif len(module_info) != 1:
+                raise ValidationError("Module definitions can only have 1 or 2 elements.")
diff --git a/web3/eth.py b/web3/eth.py
index b8bad742..72879031 100644
--- a/web3/eth.py
+++ b/web3/eth.py
@@ -458,7 +458,7 @@ class AsyncEth(BaseEth):
         return await self._call(transaction, block_identifier, state_override)
 
 
-class Eth(BaseEth, Module):
+class Eth(BaseEth):
     account = Account()
     defaultContractFactory: Type[Union[Contract, ConciseContract, ContractCaller]] = Contract  # noqa: E704,E501
     iban = Iban
diff --git a/web3/main.py b/web3/main.py
index bc2a182c..ec39c3c2 100644
--- a/web3/main.py
+++ b/web3/main.py
@@ -28,6 +28,7 @@ from typing import (
     List,
     Optional,
     Sequence,
+    Type,
     TYPE_CHECKING,
     Union,
     cast,
@@ -65,7 +66,7 @@ from web3._utils.rpc_abi import (
     RPC,
 )
 from web3._utils.module import (
-    attach_modules,
+    attach_modules as _attach_modules,
 )
 from web3._utils.normalizers import (
     abi_ens_resolver,
@@ -86,6 +87,9 @@ from web3.iban import (
 from web3.manager import (
     RequestManager as DefaultRequestManager,
 )
+from web3.module import (
+    Module,
+)
 from web3.net import (
     AsyncNet,
     Net,
@@ -128,21 +132,21 @@ if TYPE_CHECKING:
     from web3.pm import PM  # noqa: F401
 
 
-def get_default_modules() -> Dict[str, Sequence[Any]]:
+def get_default_modules() -> Dict[str, Union[Type[Module], Sequence[Any]]]:
     return {
-        "eth": (Eth,),
-        "net": (Net,),
-        "version": (Version,),
+        "eth": Eth,
+        "net": Net,
+        "version": Version,
         "parity": (Parity, {
-            "personal": (ParityPersonal,),
+            "personal": ParityPersonal,
         }),
         "geth": (Geth, {
-            "admin": (GethAdmin,),
-            "miner": (GethMiner,),
-            "personal": (GethPersonal,),
-            "txpool": (GethTxPool,),
+            "admin": GethAdmin,
+            "miner": GethMiner,
+            "personal": GethPersonal,
+            "txpool": GethTxPool,
         }),
-        "testing": (Testing,),
+        "testing": Testing,
     }
 
 
@@ -232,7 +236,8 @@ class Web3:
         self,
         provider: Optional[BaseProvider] = None,
         middlewares: Optional[Sequence[Any]] = None,
-        modules: Optional[Dict[str, Sequence[Any]]] = None,
+        modules: Optional[Dict[str, Union[Type[Module], Sequence[Any]]]] = None,
+        external_modules: Optional[Dict[str, Union[Type[Module], Sequence[Any]]]] = None,
         ens: ENS = cast(ENS, empty)
     ) -> None:
         self.manager = self.RequestManager(self, provider, middlewares)
@@ -243,7 +248,10 @@ class Web3:
         if modules is None:
             modules = get_default_modules()
 
-        attach_modules(self, modules)
+        self.attach_modules(modules)
+
+        if external_modules is not None:
+            self.attach_modules(external_modules)
 
         self.ens = ens
 
@@ -323,6 +331,15 @@ class Web3:
         )))
         return cls.keccak(hexstr=hex_string)
 
+    def attach_modules(
+        self, modules: Optional[Dict[str, Union[Type[Module], Sequence[Any]]]]
+    ) -> None:
+        """
+        Attach modules to the `Web3` instance. Modules should inherit from the `web3.module.Module`
+        class.
+        """
+        _attach_modules(self, modules)
+
     def isConnected(self) -> bool:
         return self.provider.isConnected()
 
@@ -355,7 +372,7 @@ class Web3:
     def enable_unstable_package_management_api(self) -> None:
         from web3.pm import PM  # noqa: F811
         if not hasattr(self, '_pm'):
-            attach_modules(self, {'_pm': (PM,)})
+            self.attach_modules({'_pm': PM})
 
     def enable_strict_bytes_type_checking(self) -> None:
         self.codec = ABICodec(build_strict_registry())
diff --git a/web3/module.py b/web3/module.py
index bd9cebe1..c0dc6edb 100644
--- a/web3/module.py
+++ b/web3/module.py
@@ -79,7 +79,7 @@ def retrieve_async_method_call_fn(
 
 #  Module should no longer have access to the full web3 api.
 #  Only the calling functions need access to the request methods.
-#  Any "re-entrant" shinanigans can go in the middlewares, which do
+#  Any "re-entrant" shenanigans can go in the middlewares, which do
 #  have web3 access.
 class Module:
     is_async = False
diff --git a/web3/tools/benchmark/main.py b/web3/tools/benchmark/main.py
index 1f797a27..7146974e 100644
--- a/web3/tools/benchmark/main.py
+++ b/web3/tools/benchmark/main.py
@@ -75,7 +75,7 @@ async def build_async_w3_http(endpoint_uri: str) -> Web3:
     _web3 = Web3(
         AsyncHTTPProvider(endpoint_uri),  # type: ignore
         middlewares=[async_gas_price_strategy_middleware, async_buffered_gas_estimate_middleware],
-        modules={"eth": (AsyncEth,)},
+        modules={"eth": AsyncEth},
     )
     return _web3
 


Test patch:
diff --git a/tests/core/conftest.py b/tests/core/conftest.py
new file mode 100644
index 00000000..c2b9aaff
--- /dev/null
+++ b/tests/core/conftest.py
@@ -0,0 +1,41 @@
+import pytest
+
+from web3.module import (
+    Module,
+)
+
+
+@pytest.fixture(scope='module')
+def module1():
+    class Module1(Module):
+        a = 'a'
+
+        @property
+        def b(self):
+            return 'b'
+    return Module1
+
+
+@pytest.fixture(scope='module')
+def module2():
+    class Module2(Module):
+        c = 'c'
+
+        @staticmethod
+        def d():
+            return 'd'
+    return Module2
+
+
+@pytest.fixture(scope='module')
+def module3():
+    class Module3(Module):
+        e = 'e'
+    return Module3
+
+
+@pytest.fixture(scope='module')
+def module4():
+    class Module4(Module):
+        f = 'f'
+    return Module4
diff --git a/tests/core/method-class/test_method.py b/tests/core/method-class/test_method.py
index 594eaa65..28b7f953 100644
--- a/tests/core/method-class/test_method.py
+++ b/tests/core/method-class/test_method.py
@@ -265,7 +265,7 @@ class FakeModule(Module):
 def dummy_w3():
     return Web3(
         EthereumTesterProvider(),
-        modules={'fake': (FakeModule,)},
+        modules={'fake': FakeModule},
         middlewares=[])
 
 
diff --git a/tests/core/method-class/test_result_formatters.py b/tests/core/method-class/test_result_formatters.py
index 88880c02..ff798dc0 100644
--- a/tests/core/method-class/test_result_formatters.py
+++ b/tests/core/method-class/test_result_formatters.py
@@ -46,7 +46,7 @@ def dummy_w3():
     w3 = Web3(
         DummyProvider(),
         middlewares=[result_middleware],
-        modules={"module": (ModuleForTest,)})
+        modules={"module": ModuleForTest})
     return w3
 
 
diff --git a/tests/core/utilities/test_attach_modules.py b/tests/core/utilities/test_attach_modules.py
index c38eb403..36cb8a56 100644
--- a/tests/core/utilities/test_attach_modules.py
+++ b/tests/core/utilities/test_attach_modules.py
@@ -1,5 +1,9 @@
 import pytest
 
+from eth_utils import (
+    is_integer,
+)
+
 from web3 import Web3
 from web3._utils.module import (
     attach_modules,
@@ -37,10 +41,10 @@ class MockGethPersonal(Module):
 def test_attach_modules():
     mods = {
         "geth": (MockGeth, {
-            "personal": (MockGethPersonal,),
-            "admin": (MockGethAdmin,),
+            "personal": MockGethPersonal,
+            "admin": MockGethAdmin,
         }),
-        "eth": (MockEth,),
+        "eth": MockEth,
     }
     w3 = Web3(EthereumTesterProvider(), modules={})
     attach_modules(w3, mods)
@@ -49,12 +53,17 @@ def test_attach_modules():
     assert w3.geth.admin.start_ws() is True
 
 
+def test_attach_single_module_as_tuple():
+    w3 = Web3(EthereumTesterProvider(), modules={'eth': (MockEth,)})
+    assert w3.eth.block_number() == 42
+
+
 def test_attach_modules_multiple_levels_deep():
     mods = {
-        "eth": (MockEth,),
+        "eth": MockEth,
         "geth": (MockGeth, {
             "personal": (MockGethPersonal, {
-                "admin": (MockGethAdmin,),
+                "admin": MockGethAdmin,
             }),
         }),
     }
@@ -76,9 +85,45 @@ def test_attach_modules_with_wrong_module_format():
 
 def test_attach_modules_with_existing_modules():
     mods = {
-        "eth": (MockEth,),
+        "eth": MockEth,
     }
     w3 = Web3(EthereumTesterProvider, modules=mods)
     with pytest.raises(AttributeError,
                        match="The web3 object already has an attribute with that name"):
         attach_modules(w3, mods)
+
+
+def test_attach_external_modules_multiple_levels_deep(module1, module2, module3, module4):
+    w3 = Web3(
+        EthereumTesterProvider(),
+        external_modules={
+            'module1': module1,
+            'module2': (module2, {
+                'submodule1': (module3, {
+                    'submodule2': module4,
+                }),
+            })
+        }
+    )
+
+    assert w3.isConnected()
+
+    # assert instantiated with default modules
+    assert hasattr(w3, 'geth')
+    assert hasattr(w3, 'eth')
+    assert is_integer(w3.eth.chain_id)
+
+    # assert instantiated with module1
+    assert hasattr(w3, 'module1')
+    assert w3.module1.a == 'a'
+    assert w3.module1.b == 'b'
+
+    # assert instantiated with module2 + submodules
+    assert hasattr(w3, 'module2')
+    assert w3.module2.c == 'c'
+    assert w3.module2.d() == 'd'
+
+    assert hasattr(w3.module2, 'submodule1')
+    assert w3.module2.submodule1.e == 'e'
+    assert hasattr(w3.module2.submodule1, 'submodule2')
+    assert w3.module2.submodule1.submodule2.f == 'f'
diff --git a/tests/core/web3-module/test_attach_modules.py b/tests/core/web3-module/test_attach_modules.py
new file mode 100644
index 00000000..1b26c6d8
--- /dev/null
+++ b/tests/core/web3-module/test_attach_modules.py
@@ -0,0 +1,34 @@
+from eth_utils import (
+    is_integer,
+)
+
+
+def test_attach_modules(web3, module1, module2, module3, module4):
+    web3.attach_modules({
+        'module1': module1,
+        'module2': (module2, {
+            'submodule1': (module3, {
+                'submodule2': module4,
+            }),
+        })
+    })
+
+    # assert module1 attached
+    assert hasattr(web3, 'module1')
+    assert web3.module1.a == 'a'
+    assert web3.module1.b == 'b'
+
+    # assert module2 + submodules attached
+    assert hasattr(web3, 'module2')
+    assert web3.module2.c == 'c'
+    assert web3.module2.d() == 'd'
+
+    assert hasattr(web3.module2, 'submodule1')
+    assert web3.module2.submodule1.e == 'e'
+    assert hasattr(web3.module2.submodule1, 'submodule2')
+    assert web3.module2.submodule1.submodule2.f == 'f'
+
+    # assert default modules intact
+    assert hasattr(web3, 'geth')
+    assert hasattr(web3, 'eth')
+    assert is_integer(web3.eth.chain_id)