Repository: ethereum/web3.py

Issue:
Move usage of `eth_maxPriorityFeePerGas` to `eth_feeHistory`
Since `eth_maxPriorityFeePerGas` is Geth-specific, we should stop using it to fill in gas values. We should use `eth_feeHistory` instead. I think it still makes sense to keep the RPC endpoint in `web3/eth.py`, but all places we call `eth_maxPriorityFeePerGas` should be changed.

Comment from #2237: 

> Hi, I'd like to add that `eth_maxPriorityFeePerGas` isn't part of [ETH JSON-RPC spec](https://playground.open-rpc.org/?schemaUrl=https://raw.githubusercontent.com/ethereum/eth1.0-apis/assembled-spec/openrpc.json&uiSchema%5BappBar%5D%5Bui:splitView%5D=true&uiSchema%5BappBar%5D%5Bui:input%5D=false&uiSchema%5BappBar%5D%5Bui:examplesDropdown%5D=false). It is an endpoint Geth added on its own. Clearly some ETH providers don't have a plan to add it https://github.com/nomiclabs/hardhat/issues/1664. Maybe using `eth_feeHistory` is a better idea since it's part of the spec?

> Also, it would be great if this can be a built-in middleware so that users could choose between `eth_maxPriorityFeePerGas` (or `eth_feeHistory`) and `eth_gasPrice` based on the chain they want to use.

_Originally posted by @guoyiteng in https://github.com/ethereum/web3.py/issues/2237#issuecomment-993133677_


Discussion:
Hi @kclowes, good excuse to dig into some more web3.py code. I will put together a PR for this.
Awesome, thanks @broper2!
Hi @kclowes, wanted to run something by you before submitting PR. I do think the underlying data used in  eth_maxPriorityFeePerGas implementation is available in eth_feeHistory, and we could add a simple function to estimate a reasonable "tip" default.

However, as I read about this more, I found this discussion:
https://github.com/ethereum/pm/issues/328#issuecomment-853510466

Seems we may be able to leverage relation between gas_price and get_block('latest')['baseFeePerGas'] to get an exact value of what eth_maxPriorityFeePerGas would return. The relationship (web3.eth.gas_price - web3.eth.get_block('latest')['baseFeePerGas']) == web3.eth.max_priority_fee seems to hold in some quick functional testing I have done.

I don't claim to be ETH expert, so please tell me if I am missing something, or if this looks ok as a no-impact change. Thanks!
That looks good to me, but @fselmo has done way more of the dynamic fee work so I'll defer to him. Do you have any thoughts @fselmo? Thanks for looking into it, @broper2!
 > Seems we may be able to leverage relation between gas_price and get_block('latest')['baseFeePerGas'] to get an exact value of what eth_maxPriorityFeePerGas would return. The relationship (web3.eth.gas_price - web3.eth.get_block('latest')['baseFeePerGas']) == web3.eth.max_priority_fee seems to hold in some quick functional testing I have done.

I don't think this is good enough. Some testing on my end suggests that at every new block transition (most likely) there is inconsistency in the values for `eth_maxPriorityFeePerGas` and (`eth_gasPrice` - `latest_base_fee`), therefore this is simply not good enough to hold.

I did some quick-and-dirty testing calling the below method on an infura connection multiple times until I got `False` and the inequality printed. It seems to happen about once every few seconds or so and then re-stabilizes to `True` which makes me think it's at the new block transition but I haven't really inspected it too much. Only enough to know this is not ideal.

``` python
def max_prio_calc():
    gp = w3.eth.gas_price
    bf = w3.eth.get_block('latest')['baseFeePerGas']
    mpf = w3.eth.max_priority_fee

    print(f'gas_price: {gp}  |  base_fee: {bf}  |  max_priority_fee: {mpf}')
    
    theory_holds = mpf == gp - bf
    print(theory_holds)

    if not theory_holds:
        print(f'{mpf} != {gp - bf}')
```

e.g.

``` python
>>> max_prio_calc()
gas_price: 78216696028  |  base_fee: 77168141801  |  max_priority_fee: 1250000000
False
1250000000 != 1048554227
```

Again, I haven't looked into this too much yet (other than testing the suggestion) but I would be good with building a calculation from scratch that is equal to what `geth` is doing or some other method that is more robust using `eth_feeHistory`. I also wouldn't mind putting the responsibility of specifying the `maxPriorityFeePerGas` on the user if the `eth_maxPriorityFeePerGas` RPC method isn't available (though this is less ideal).

Thoughts?
Yeah its a good point. From reading about this, I am not convinced we will be able to replicate behavior of RPC endpoint eth_maxPriorityFeePerGas perfectly either way, but if that is accepted I can put together a calculation and see if it gets closer over number of tries (I am actually guessing that it will not be as good, but don't mind trying).

If the goal is really to never deviate from what eth_maxPriorityFeePerGas returns, I think it may make better sense to put responsibility on user.

Let me know what you think @fselmo 
@broper2 yeah I hear you... I don't necessarily care that we replicate what geth is doing... something else that is reasonable would work well here too. The reason I don't like the `eth_gasPrice` approach is that sometimes it doesn't make sense at all and thus isn't reliable. I just ran it for less than a minute and I got a negative value because the `base_fee` was higher than the `gas_price` at that moment.

e.g.
``` python
>>> max_prio_calc()
gas_price: 67953109912  |  base_fee: 74750863316  |  max_priority_fee: 1500000000
False
1500000000 != -6797753404
```

Also, the more I think about it, the more I don't think I'm ok with putting the responsibility on the user. The barrier to entry is still pretty high for web3 and one of our main goals is to make the user experience better and make development more inviting.

In the end I think this ended up being a bit more complicated of a task than we expected but thank you for starting the conversation on it. Let me know if you are still interested in pursuing this. No sweat if not. I wouldn't mind scooping this up after the holidays. I would probably start looking at what geth is doing and any other examples (if they exist) and look to implement a similar internal approach for when the `eth_maxPriorityFeePerGas` method is not available.
Good catch with the negative value, should have considered that. And no problem, I'll give it a go creating an estimate to this endpoint in the next few days. I am just trying to dig into some web3 code so don't mind the added complexity at all. 

When I was researching this before, I came across this Alchemy blog which gives a lot of info on best way to do exactly this calculation. Will circle back when I have that PR updated. Thanks @fselmo 

https://docs.alchemy.com/alchemy/guides/eip-1559/gas-estimator

Resolving patch:
diff --git a/web3/_utils/fee_utils.py b/web3/_utils/fee_utils.py
new file mode 100644
--- /dev/null
+++ b/web3/_utils/fee_utils.py
@@ -0,0 +1,53 @@
+from typing import (
+    TYPE_CHECKING,
+)
+
+from web3.types import (
+    FeeHistory,
+    Wei,
+)
+
+if TYPE_CHECKING:
+    from web3.eth import (
+        AsyncEth,  # noqa: F401
+        Eth,  # noqa: F401
+    )
+
+PRIORITY_FEE_MAX = Wei(1500000000)  # 1.5 gwei
+PRIORITY_FEE_MIN = Wei(1000000000)  # 1 gwei
+
+# 5th percentile fee history from the last 10 blocks
+PRIORITY_FEE_HISTORY_PARAMS = (10, 'pending', [5.0])
+
+
+def _fee_history_priority_fee_estimate(fee_history: FeeHistory) -> Wei:
+    # grab only non-zero fees and average against only that list
+    non_empty_block_fees = [fee[0] for fee in fee_history['reward'] if fee[0] != 0]
+
+    # prevent division by zero in the extremely unlikely case that all fees within the polled fee
+    # history range for the specified percentile are 0
+    divisor = len(non_empty_block_fees) if len(non_empty_block_fees) != 0 else 1
+
+    priority_fee_average_for_percentile = Wei(
+        round(sum(non_empty_block_fees) / divisor)
+    )
+
+    return (  # keep estimated priority fee within a max / min range
+        PRIORITY_FEE_MAX if priority_fee_average_for_percentile > PRIORITY_FEE_MAX else
+        PRIORITY_FEE_MIN if priority_fee_average_for_percentile < PRIORITY_FEE_MIN else
+        priority_fee_average_for_percentile
+    )
+
+
+def fee_history_priority_fee(eth: "Eth") -> Wei:
+    # This is a tested internal call so no need for type hinting. We can keep better consistency
+    # between the sync and async calls by unpacking PRIORITY_FEE_HISTORY_PARAMS as constants here.
+    fee_history = eth.fee_history(*PRIORITY_FEE_HISTORY_PARAMS)  # type: ignore
+    return _fee_history_priority_fee_estimate(fee_history)
+
+
+async def async_fee_history_priority_fee(async_eth: "AsyncEth") -> Wei:
+    # This is a tested internal call so no need for type hinting. We can keep better consistency
+    # between the sync and async calls by unpacking PRIORITY_FEE_HISTORY_PARAMS as constants here.
+    fee_history = await async_eth.fee_history(*PRIORITY_FEE_HISTORY_PARAMS)  # type: ignore
+    return _fee_history_priority_fee_estimate(fee_history)
diff --git a/web3/eth.py b/web3/eth.py
--- a/web3/eth.py
+++ b/web3/eth.py
@@ -49,6 +49,10 @@
 from web3._utils.encoding import (
     to_hex,
 )
+from web3._utils.fee_utils import (
+    async_fee_history_priority_fee,
+    fee_history_priority_fee,
+)
 from web3._utils.filters import (
     select_filter_method,
 )
@@ -356,7 +360,18 @@ async def hashrate(self) -> int:
 
     @property
     async def max_priority_fee(self) -> Wei:
-        return await self._max_priority_fee()  # type: ignore
+        """
+        Try to use eth_maxPriorityFeePerGas but, since this is not part of the spec and is only
+        supported by some clients, fall back to an eth_feeHistory calculation with min and max caps.
+        """
+        try:
+            return await self._max_priority_fee()  # type: ignore
+        except ValueError:
+            warnings.warn(
+                "There was an issue with the method eth_maxPriorityFeePerGas. Calculating using "
+                "eth_feeHistory."
+            )
+            return await async_fee_history_priority_fee(self)
 
     @property
     async def mining(self) -> bool:
@@ -604,7 +619,18 @@ def chainId(self) -> int:
 
     @property
     def max_priority_fee(self) -> Wei:
-        return self._max_priority_fee()
+        """
+        Try to use eth_maxPriorityFeePerGas but, since this is not part of the spec and is only
+        supported by some clients, fall back to an eth_feeHistory calculation with min and max caps.
+        """
+        try:
+            return self._max_priority_fee()
+        except ValueError:
+            warnings.warn(
+                "There was an issue with the method eth_maxPriorityFeePerGas. Calculating using "
+                "eth_feeHistory."
+            )
+            return fee_history_priority_fee(self)
 
     def get_storage_at_munger(
         self,
diff --git a/web3/middleware/fixture.py b/web3/middleware/fixture.py
--- a/web3/middleware/fixture.py
+++ b/web3/middleware/fixture.py
@@ -2,6 +2,7 @@
     TYPE_CHECKING,
     Any,
     Callable,
+    Coroutine,
     Dict,
 )
 
@@ -21,7 +22,7 @@ def construct_fixture_middleware(fixtures: Dict[RPCEndpoint, Any]) -> Middleware
     which is found in the provided fixtures.
     """
     def fixture_middleware(
-        make_request: Callable[[RPCEndpoint, Any], Any], web3: "Web3"
+        make_request: Callable[[RPCEndpoint, Any], Any], _: "Web3"
     ) -> Callable[[RPCEndpoint, Any], RPCResponse]:
         def middleware(method: RPCEndpoint, params: Any) -> RPCResponse:
             if method in fixtures:
@@ -43,7 +44,7 @@ def construct_result_generator_middleware(
     functions with the signature `fn(method, params)`.
     """
     def result_generator_middleware(
-        make_request: Callable[[RPCEndpoint, Any], Any], web3: "Web3"
+        make_request: Callable[[RPCEndpoint, Any], Any], _: "Web3"
     ) -> Callable[[RPCEndpoint, Any], RPCResponse]:
         def middleware(method: RPCEndpoint, params: Any) -> RPCResponse:
             if method in result_generators:
@@ -65,7 +66,7 @@ def construct_error_generator_middleware(
     functions with the signature `fn(method, params)`.
     """
     def error_generator_middleware(
-        make_request: Callable[[RPCEndpoint, Any], Any], web3: "Web3"
+        make_request: Callable[[RPCEndpoint, Any], Any], _: "Web3"
     ) -> Callable[[RPCEndpoint, Any], RPCResponse]:
         def middleware(method: RPCEndpoint, params: Any) -> RPCResponse:
             if method in error_generators:
@@ -75,3 +76,25 @@ def middleware(method: RPCEndpoint, params: Any) -> RPCResponse:
                 return make_request(method, params)
         return middleware
     return error_generator_middleware
+
+
+async def async_construct_error_generator_middleware(
+    error_generators: Dict[RPCEndpoint, Any]
+) -> Middleware:
+    """
+    Constructs a middleware which intercepts requests for any method found in
+    the provided mapping of endpoints to generator functions, returning
+    whatever error message the generator function returns.  Callbacks must be
+    functions with the signature `fn(method, params)`.
+    """
+    async def error_generator_middleware(
+        make_request: Callable[[RPCEndpoint, Any], Any], _: "Web3"
+    ) -> Callable[[RPCEndpoint, Any], Coroutine[Any, Any, RPCResponse]]:
+        async def middleware(method: RPCEndpoint, params: Any) -> RPCResponse:
+            if method in error_generators:
+                error_msg = error_generators[method](method, params)
+                return {'error': error_msg}
+            else:
+                return await make_request(method, params)
+        return middleware
+    return error_generator_middleware


Test patch:
diff --git a/tests/core/utilities/test_fee_utils.py b/tests/core/utilities/test_fee_utils.py
new file mode 100644
--- /dev/null
+++ b/tests/core/utilities/test_fee_utils.py
@@ -0,0 +1,65 @@
+import pytest
+
+from eth_utils import (
+    is_integer,
+)
+
+from web3.middleware import (
+    construct_error_generator_middleware,
+    construct_result_generator_middleware,
+)
+from web3.types import (
+    RPCEndpoint,
+)
+
+
+@pytest.mark.parametrize(
+    'fee_history_rewards,expected_max_prio_calc',
+    (
+        (
+            [[10 ** 20], [10 ** 20], [10 ** 20], [10 ** 20]],
+            15 * (10 ** 8),
+        ),
+        (
+            [[10 ** 2], [10 ** 2], [10 ** 2], [10 ** 2], [10 ** 2]],
+            10 ** 9,
+        ),
+        (
+            [[0], [0], [0], [0], [0]],
+            10 ** 9,
+        ),
+        (
+            [[1223344455], [1111111111], [1222777777], [0], [1000222111], [0], [0]],
+            round(sum([1223344455, 1111111111, 1222777777, 1000222111]) / 4),
+        ),
+    ),
+    ids=[
+        'test-max', 'test-min', 'test-min-all-zero-fees', 'test-non-zero-average'
+    ]
+)
+# Test fee_utils indirectly by mocking eth_feeHistory results and checking against expected output
+def test_fee_utils_indirectly(
+    web3, fee_history_rewards, expected_max_prio_calc
+) -> None:
+    fail_max_prio_middleware = construct_error_generator_middleware(
+        {RPCEndpoint("eth_maxPriorityFeePerGas"): lambda *_: ''}
+    )
+    fee_history_result_middleware = construct_result_generator_middleware(
+        {RPCEndpoint('eth_feeHistory'): lambda *_: {'reward': fee_history_rewards}}
+    )
+
+    web3.middleware_onion.add(fail_max_prio_middleware, 'fail_max_prio')
+    web3.middleware_onion.inject(fee_history_result_middleware, 'fee_history_result', layer=0)
+
+    with pytest.warns(
+        UserWarning,
+        match="There was an issue with the method eth_maxPriorityFeePerGas. Calculating using "
+              "eth_feeHistory."
+    ):
+        max_priority_fee = web3.eth.max_priority_fee
+        assert is_integer(max_priority_fee)
+        assert max_priority_fee == expected_max_prio_calc
+
+    # clean up
+    web3.middleware_onion.remove('fail_max_prio')
+    web3.middleware_onion.remove('fee_history_result')
diff --git a/tests/integration/test_ethereum_tester.py b/tests/integration/test_ethereum_tester.py
--- a/tests/integration/test_ethereum_tester.py
+++ b/tests/integration/test_ethereum_tester.py
@@ -239,6 +239,10 @@ def func_wrapper(self, eth_tester, *args, **kwargs):
 
 
 class TestEthereumTesterEthModule(EthModuleTest):
+    test_eth_max_priority_fee_with_fee_history_calculation = not_implemented(
+        EthModuleTest.test_eth_max_priority_fee_with_fee_history_calculation,
+        ValueError
+    )
     test_eth_sign = not_implemented(EthModuleTest.test_eth_sign, ValueError)
     test_eth_sign_ens_names = not_implemented(
         EthModuleTest.test_eth_sign_ens_names, ValueError
diff --git a/web3/_utils/module_testing/eth_module.py b/web3/_utils/module_testing/eth_module.py
--- a/web3/_utils/module_testing/eth_module.py
+++ b/web3/_utils/module_testing/eth_module.py
@@ -54,11 +54,16 @@
     TransactionNotFound,
     TransactionTypeMismatch,
 )
+from web3.middleware.fixture import (
+    async_construct_error_generator_middleware,
+    construct_error_generator_middleware,
+)
 from web3.types import (  # noqa: F401
     BlockData,
     FilterParams,
     LogReceipt,
     Nonce,
+    RPCEndpoint,
     SyncStatus,
     TxParams,
     Wei,
@@ -423,6 +428,25 @@ async def test_eth_max_priority_fee(self, async_w3: "Web3") -> None:
         max_priority_fee = await async_w3.eth.max_priority_fee  # type: ignore
         assert is_integer(max_priority_fee)
 
+    @pytest.mark.asyncio
+    async def test_eth_max_priority_fee_with_fee_history_calculation(
+        self, async_w3: "Web3"
+    ) -> None:
+        fail_max_prio_middleware = await async_construct_error_generator_middleware(
+            {RPCEndpoint("eth_maxPriorityFeePerGas"): lambda *_: ''}
+        )
+        async_w3.middleware_onion.add(fail_max_prio_middleware, name='fail_max_prio_middleware')
+
+        with pytest.warns(
+            UserWarning,
+            match="There was an issue with the method eth_maxPriorityFeePerGas. Calculating using "
+                  "eth_feeHistory."
+        ):
+            max_priority_fee = await async_w3.eth.max_priority_fee  # type: ignore
+            assert is_integer(max_priority_fee)
+
+        async_w3.middleware_onion.remove('fail_max_prio_middleware')  # clean up
+
     @pytest.mark.asyncio
     async def test_eth_getBlockByHash(
         self, async_w3: "Web3", empty_block: BlockData
@@ -1136,6 +1160,22 @@ def test_eth_max_priority_fee(self, web3: "Web3") -> None:
         max_priority_fee = web3.eth.max_priority_fee
         assert is_integer(max_priority_fee)
 
+    def test_eth_max_priority_fee_with_fee_history_calculation(self, web3: "Web3") -> None:
+        fail_max_prio_middleware = construct_error_generator_middleware(
+            {RPCEndpoint("eth_maxPriorityFeePerGas"): lambda *_: ''}
+        )
+        web3.middleware_onion.add(fail_max_prio_middleware, name='fail_max_prio_middleware')
+
+        with pytest.warns(
+            UserWarning,
+            match="There was an issue with the method eth_maxPriorityFeePerGas. Calculating using "
+                  "eth_feeHistory."
+        ):
+            max_priority_fee = web3.eth.max_priority_fee
+            assert is_integer(max_priority_fee)
+
+        web3.middleware_onion.remove('fail_max_prio_middleware')  # clean up
+
     def test_eth_accounts(self, web3: "Web3") -> None:
         accounts = web3.eth.accounts
         assert is_list_like(accounts)