Repository: ethereum/web3.py

Issue:
Crash in time based gas price strategies
* Version: 4.5.0
* Python: 3.6
* OS: osx


### What was wrong?

My script to check the gas price calculation crashes after the update to web3 4.5:
```
python test_gas_price.py                                                                     0:13:21
RPC: 252000000
Traceback (most recent call last):
  File "test_gas_price.py", line 14, in <module>
    print('Slow', w3.eth.generateGasPrice())
  File "/Users/paul/Work/venv-raiden/lib/python3.6/site-packages/web3/eth.py", line 385, in generateGasPrice
    return self.gasPriceStrategy(self.web3, transaction_params)
  File "/Users/paul/Work/venv-raiden/lib/python3.6/site-packages/web3/gas_strategies/time_based.py", line 160, in time_based_gas_price_strategy
    sample_size=sample_size,
  File "/Users/paul/Work/venv-raiden/lib/python3.6/site-packages/eth_utils/functional.py", line 22, in inner
    return callback(fn(*args, **kwargs))
  File "/Users/paul/Work/venv-raiden/lib/python3.6/site-packages/web3/gas_strategies/time_based.py", line 79, in _compute_probabilities
    reverse=True,
  File "/Users/paul/Work/venv-raiden/lib/python3.6/site-packages/web3/gas_strategies/time_based.py", line 67, in _aggregate_miner_data
    percentile(gas_prices, percentile=15))
  File "/Users/paul/Work/venv-raiden/lib/python3.6/site-packages/web3/utils/math.py", line 5, in percentile
    raise ValueError("Expected a sequence of at least 3 integers, got {0}".format(values))
ValueError: Expected a sequence of at least 3 integers, got {41000000000, 26000000000}
```

The code is the following:
```py
from web3.auto import w3

from web3 import middleware
from web3.gas_strategies.time_based import medium_gas_price_strategy, fast_gas_price_strategy, slow_gas_price_strategy
from web3.gas_strategies.rpc import rpc_gas_price_strategy

w3.middleware_stack.add(middleware.time_based_cache_middleware)
w3.middleware_stack.add(middleware.latest_block_based_cache_middleware)
w3.middleware_stack.add(middleware.simple_cache_middleware)

w3.eth.setGasPriceStrategy(rpc_gas_price_strategy)
print('RPC:', w3.eth.generateGasPrice())
w3.eth.setGasPriceStrategy(slow_gas_price_strategy)
print('Slow', w3.eth.generateGasPrice())
w3.eth.setGasPriceStrategy(medium_gas_price_strategy)
print('Medium:', w3.eth.generateGasPrice())
w3.eth.setGasPriceStrategy(fast_gas_price_strategy)
print('Fast:', w3.eth.generateGasPrice())
```


### How can it be fixed?

It should crash and return a proper gas price.



Discussion:
The timed_based gas strategies will throw this exception when a block has less than 3 transactions to use to make an estimate.  This could change to just return an average of the 2 transactions or the price of the single transaction, if only 1 or 2 transactions occurred.  But in that case, the estimate isn't going to be very useful.

@carver Do you think that the gas price strategy should be kept to throw an error in this case?  IMO, I would want to handle this case separately, when there is not enough data to make a good estimate.  If I understand correctly, either the block gas limit is being reached with a few transactions, or the network traffic is really low.  Both would be on the edges of the price bell curve, and should probably be dealt with intentionally.
@palango I updated the web3 version in the issue from 3.5.0 -> 4.5.0, based on the context you provided.

> Do you think that the gas price strategy should be kept to throw an error in this case? 

Well, I suppose you might want to embed the gas strategy to your app, and then test your app against a private network that only has one transaction in a block. Sure, they could disable gas estimation during testing, but why make them.

>  IMO, I would want to handle this case separately, when there is not enough data to make a good estimate.

Yeah, but also it's only an estimate. We could still calculate something with one data point right? It will just be a really rough estimate. The cutoff at 3 seems sort of arbitrary. I'm in favor of giving an answer if we have any data.
> I updated the web3 version in the issue from 3.5.0 -> 4.5.0, based on the context you provided.

Thanks!

> Sure, they could disable gas estimation during testing, but why make them.

I agree. I tested on ropsten and had lots of blocks with just one or two transactions. I think there should be a fallback, even `eth_gasPrice` would be enough.

Resolving patch:
diff --git a/web3/exceptions.py b/web3/exceptions.py
--- a/web3/exceptions.py
+++ b/web3/exceptions.py
@@ -99,3 +99,11 @@ class NoABIEventsFound(AttributeError):
     Raised when an ABI doesn't contain any events.
     """
     pass
+
+
+class InsufficientData(Exception):
+    """
+    Raised when there are insufficient data points to
+    complete a calculation
+    """
+    pass
diff --git a/web3/gas_strategies/time_based.py b/web3/gas_strategies/time_based.py
--- a/web3/gas_strategies/time_based.py
+++ b/web3/gas_strategies/time_based.py
@@ -7,6 +7,7 @@
 )
 
 from web3.exceptions import (
+    InsufficientData,
     ValidationError,
 )
 from web3.utils.math import (
@@ -60,11 +61,15 @@ def _aggregate_miner_data(raw_data):
 
     for miner, miner_data in data_by_miner.items():
         _, block_hashes, gas_prices = map(set, zip(*miner_data))
+        try:
+            price_percentile = percentile(gas_prices, percentile=20)
+        except InsufficientData:
+            price_percentile = min(gas_prices)
         yield MinerData(
             miner,
             len(set(block_hashes)),
             min(gas_prices),
-            percentile(gas_prices, percentile=15))
+            price_percentile)
 
 
 @to_tuple
diff --git a/web3/utils/math.py b/web3/utils/math.py
--- a/web3/utils/math.py
+++ b/web3/utils/math.py
@@ -1,8 +1,14 @@
+from web3.exceptions import (
+    InsufficientData,
+)
+
+
 def percentile(values=None, percentile=None):
     """Calculates a simplified weighted average percentile
     """
-    if values in [None, tuple(), []] or len(values) < 3:
-        raise ValueError("Expected a sequence of at least 3 integers, got {0}".format(values))
+    if values in [None, tuple(), []] or len(values) < 1:
+        raise InsufficientData(
+            "Expected a sequence of at least 1 integers, got {0!r}".format(values))
     if percentile is None:
         raise ValueError("Expected a percentile choice, got {0}".format(percentile))
 
@@ -11,6 +17,8 @@ def percentile(values=None, percentile=None):
     rank = len(values) * percentile / 100
     if rank > 0:
         index = rank - 1
+        if index < 0:
+            return sorted_values[0]
     else:
         index = rank
 


Test patch:
diff --git a/tests/core/gas-strategies/test_time_based_gas_price_strategy.py b/tests/core/gas-strategies/test_time_based_gas_price_strategy.py
--- a/tests/core/gas-strategies/test_time_based_gas_price_strategy.py
+++ b/tests/core/gas-strategies/test_time_based_gas_price_strategy.py
@@ -28,6 +28,11 @@ def _get_block_by_something(method, params):
             'transactions': [
                 {'gasPrice': 70},
                 {'gasPrice': 60},
+                {'gasPrice': 60},
+                {'gasPrice': 60},
+                {'gasPrice': 15},
+                {'gasPrice': 5},
+                {'gasPrice': 50},
             ],
             'miner': '0xA',
             'timestamp': 100,
@@ -51,8 +56,6 @@ def _get_block_by_something(method, params):
             'number': 3,
             'parentHash': '0x0000000000000000000000000000000000000000000000000000000000000002',
             'transactions': [
-                {'gasPrice': 10},
-                {'gasPrice': 50},
                 {'gasPrice': 100},
             ],
             'miner': '0xC',
@@ -64,9 +67,6 @@ def _get_block_by_something(method, params):
             'number': 2,
             'parentHash': '0x0000000000000000000000000000000000000000000000000000000000000001',
             'transactions': [
-                {'gasPrice': 50},
-                {'gasPrice': 70},
-                {'gasPrice': 100},
             ],
             'miner': '0xB',
             'timestamp': 40,
@@ -96,6 +96,15 @@ def _get_block_by_something(method, params):
                 {'gasPrice': 30},
                 {'gasPrice': 50},
                 {'gasPrice': 60},
+                {'gasPrice': 30},
+                {'gasPrice': 50},
+                {'gasPrice': 60},
+                {'gasPrice': 30},
+                {'gasPrice': 50},
+                {'gasPrice': 60},
+                {'gasPrice': 30},
+                {'gasPrice': 54},
+                {'gasPrice': 10000000000000000000000},
             ],
             'miner': '0xA',
             'timestamp': 0,
@@ -108,21 +117,21 @@ def _get_block_by_something(method, params):
     'strategy_params,expected',
     (
         # 120 second wait times
-        (dict(max_wait_seconds=80, sample_size=5, probability=98), 61),
-        (dict(max_wait_seconds=80, sample_size=5, probability=90), 46),
-        (dict(max_wait_seconds=80, sample_size=5, probability=50), 40),
+        (dict(max_wait_seconds=80, sample_size=5, probability=98), 70),
+        (dict(max_wait_seconds=80, sample_size=5, probability=90), 25),
+        (dict(max_wait_seconds=80, sample_size=5, probability=50), 11),
         # 60 second wait times
-        (dict(max_wait_seconds=60, sample_size=5, probability=98), 62),
-        (dict(max_wait_seconds=60, sample_size=5, probability=90), 55),
-        (dict(max_wait_seconds=60, sample_size=5, probability=50), 40),
+        (dict(max_wait_seconds=60, sample_size=5, probability=98), 92),
+        (dict(max_wait_seconds=60, sample_size=5, probability=90), 49),
+        (dict(max_wait_seconds=60, sample_size=5, probability=50), 11),
         # 40 second wait times
-        (dict(max_wait_seconds=40, sample_size=5, probability=98), 63),
-        (dict(max_wait_seconds=40, sample_size=5, probability=90), 61),
-        (dict(max_wait_seconds=40, sample_size=5, probability=50), 40),
+        (dict(max_wait_seconds=40, sample_size=5, probability=98), 100),
+        (dict(max_wait_seconds=40, sample_size=5, probability=90), 81),
+        (dict(max_wait_seconds=40, sample_size=5, probability=50), 11),
         # 20 second wait times
-        (dict(max_wait_seconds=20, sample_size=5, probability=98), 63),
-        (dict(max_wait_seconds=20, sample_size=5, probability=90), 62),
-        (dict(max_wait_seconds=20, sample_size=5, probability=50), 50),
+        (dict(max_wait_seconds=20, sample_size=5, probability=98), 100),
+        (dict(max_wait_seconds=20, sample_size=5, probability=90), 100),
+        (dict(max_wait_seconds=20, sample_size=5, probability=50), 36),
     ),
 )
 def test_time_based_gas_price_strategy(strategy_params, expected):
@@ -141,7 +150,7 @@ def test_time_based_gas_price_strategy(strategy_params, expected):
     )
     w3.eth.setGasPriceStrategy(time_based_gas_price_strategy)
     actual = w3.eth.generateGasPrice()
-    assert actual == expected
+    assert int(actual) == expected
 
 
 @pytest.mark.parametrize(
diff --git a/tests/core/utilities/test_math.py b/tests/core/utilities/test_math.py
--- a/tests/core/utilities/test_math.py
+++ b/tests/core/utilities/test_math.py
@@ -5,6 +5,9 @@
     strategies as st,
 )
 
+from web3.exceptions import (
+    InsufficientData,
+)
 from web3.utils.math import (
     percentile,
 )
@@ -29,13 +32,21 @@ def test_percentiles_out_of_one_hundred(p, expected):
     assert percentile(values, p) == expected
 
 
-def test_percentiles_with_single_values():
-    with pytest.raises(ValueError):
-        percentile([0], 1)
+def test_percentiles_with_no_values():
+    with pytest.raises(InsufficientData):
+        percentile([], 1)
+
+
+def test_percentiles_with_out_of_bounds_fractions():
+    assert 1 == percentile([1, 2, 3, 4], percentile=10)
+    assert 1 == percentile([1, 2, 3, 4], percentile=15)
+    assert 1 == percentile([1, 2, 3, 4], percentile=20)
+    assert 1 == percentile([1, 2, 3, 4], percentile=25)
+    assert 1 < percentile([1, 2, 3, 4], percentile=30)
 
 
 @given(
-    values=st.lists(elements=st.integers(), min_size=3, max_size=200),
+    values=st.lists(elements=st.integers(), min_size=1, max_size=200),
     p=st.integers(max_value=100, min_value=0))
 def test_fuzz_test_percentiles(values, p):
     if not values: