Repository: liampauling/flumine
Pull request #422 (2021-05-06 07:20:12)

black upgrade


Patch:
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 0000000..491deae
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,7 @@
+version: 2
+updates:
+- package-ecosystem: pip
+  directory: "/"
+  schedule:
+    interval: daily
+  open-pull-requests-limit: 10
diff --git a/HISTORY.rst b/HISTORY.rst
index 417d10b..9c1ae21 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -3,6 +3,20 @@
 Release History
 ---------------
 
+1.18.6 (2021-05-06)
++++++++++++++++++++
+
+**Improvements**
+
+- Stream retry timeout bumped to 60s
+- Docs improved
+- get_file_md tuple handing (race stream)
+
+**Libraries**
+
+- black upgraded to 21.5b0
+- dependabot native added
+
 1.18.5 (2021-04-26)
 +++++++++++++++++++
 
diff --git a/docs/advanced.md b/docs/advanced.md
index 17f6802..00ce562 100644
--- a/docs/advanced.md
+++ b/docs/advanced.md
@@ -26,43 +26,6 @@ The Flumine class can be adapted by overriding the following functions:
 - `_process_end_flumine()` called on Flumine termination
 
 ## Base Strategy
-### Parameters
-
-- `market_filter` Streaming market filter required
-- `market_data_filter` Streaming market data filter required
-- `streaming_timeout` Streaming timeout, will call snap() on cache every x seconds
-- `conflate_ms` Streaming conflate
-- `stream_class` MarketStream or RawDataStream
-- `name` Strategy name, if None will default to class name
-- `context` Dictionary object where any extra data can be stored here such as triggers
-- `max_selection_exposure` Max exposure per selection (including new order), note this does __not__ handle reduction in exposure due to laying another runner
-- `max_order_exposure` Max exposure per order
-- `client` Strategy client, half implemented when flumine will be migrated to multi clients
-- `max_trade_count` Max total number of trades per runner
-- `max_live_trade_count` Max live (with executable orders) trades per runner
-- `multi_order_trades` Allow multiple live orders per trade
-
-### Functions
-
-The following functions can be overridden dependent on the strategy:
-
-`add()` Function called when strategy is added to framework
-
-`start()` Function called when framework starts
-
-`check_market_book()` Function called with marketBook, `process_market_book` is only executed if this returns True
-
-`process_market_book()` Processes market book updates, called on every update that is received
-
-`process_raw_data()` As per `process_market_book` but handles raw data
-
-`process_orders()` Process list of Order objects for strategy and Market
-
-`process_closed_market()` Process Market after closure
-
-`finish()` Function called when framework ends
-
-### Runner Context
 
 
 ## Streams
@@ -85,58 +48,8 @@ Subscribes to all orders per running instance using the `config.hostname`
 
 ## Custom Event
 
-## Trading Controls
-
-Before placing an order flumine will check the client and trading controls, this allows validation to occur before execution. If an order does not meet any of these validations it is not executed and status is updated to `Violation`.
-
-### Client Controls
-
-- `MaxTransactionCount`: Checks transaction count is not over betfair transaction limit (5000 per hour) 
-
-### Trading Controls
-
-- `OrderValidation`: Checks order is valid (size/odds)
-- `StrategyExposure`: Checks order does not invalidate `strategy.validate_order`, `strategy.max_order_exposure` or `strategy.max_selection_exposure`
 
-## Logging Controls
 
-Custom logging is available using the `LoggingControl` class, the [base class](https://github.com/liampauling/flumine/blob/master/flumine/controls/loggingcontrols.py#L12) creates debug logs and can be used as follows:
-
-```python
-from flumine.controls.loggingcontrols import LoggingControl
-
-control = LoggingControl()
-
-framework.add_logging_control(control)
-```
-
-!!! tip
-    More than one control can be added, for example a csv logger and db logger.
-
-## Background Workers
-
-By default flumine adds the following workers:
- 
-- `keep_alive`: runs every 1200s (or session_timeout/2) to make sure the client is either logged in or kept alive
-- `poll_account_balance`: runs every 120s to poll account balance endpoint
-- `poll_market_catalogue`: runs every 60s to poll listMarketCatalogue endpoint
-- `poll_cleared_orders`: runs when closed market is added to `flumine.cleared_market_queue`
-
-Further workers can be added as per:
-
-```python
-from flumine.worker import BackgroundWorker
-
-def func(a): print(a)
-
-worker = BackgroundWorker(
-    framework, interval=10, function=func, func_args=("hello",), name="print_a"
-)
-
-framework.add_worker(
-    worker
-)
-```
 
 ## Error Handling
 
diff --git a/docs/controls.md b/docs/controls.md
new file mode 100644
index 0000000..f63e233
--- /dev/null
+++ b/docs/controls.md
@@ -0,0 +1,31 @@
+# Controls
+
+## BaseStrategy
+
+## Trading Controls
+
+Before placing an order flumine will check the client and trading controls, this allows validation to occur before execution. If an order does not meet any of these validations it is not executed and status is updated to `Violation`.
+
+### Client Controls
+
+- `MaxTransactionCount`: Checks transaction count is not over betfair transaction limit (5000 per hour) 
+
+### Trading Controls
+
+- `OrderValidation`: Checks order is valid (size/odds)
+- `StrategyExposure`: Checks order does not invalidate `strategy.validate_order`, `strategy.max_order_exposure` or `strategy.max_selection_exposure`
+
+## Logging Controls
+
+Custom logging is available using the `LoggingControl` class, the [base class](https://github.com/liampauling/flumine/blob/master/flumine/controls/loggingcontrols.py#L12) creates debug logs and can be used as follows:
+
+```python
+from flumine.controls.loggingcontrols import LoggingControl
+
+control = LoggingControl()
+
+framework.add_logging_control(control)
+```
+
+!!! tip
+    More than one control can be added, for example a csv logger and db logger.
diff --git a/docs/execution.md b/docs/execution.md
deleted file mode 100644
index 2a402a6..0000000
--- a/docs/execution.md
+++ /dev/null
@@ -1,71 +0,0 @@
-# Clients / Execution
-
-## Live - Betfair
-
-When using the `BetfairClient` flumine will default to using the integrated `BetfairExecution` class, this class can handle:
-
-- `placeOrders`
-- `cancelOrders`
-- `updateOrders`
-- `replaceOrders`
-
-## Simulated
-
-The `BacktestClient` will use the `SimulatedExecution` class and can be used for backtesting (betfair historical data) or paper trading where order matching is handled locally.
-
-Matching is calculated using the [simulated middleware](https://github.com/liampauling/flumine/blob/master/flumine/markets/middleware.py#L15) and can be overridden if required.
-
-## Custom
-
-Custom execution can be applied by creating your own client/overriding the `add_execution` function like so:
-
-```python
-from flumine.execution.baseexecution import BaseExecution
-from flumine.clients.baseclient import BaseClient
-
-
-class PrinterExecution(BaseExecution):
-    def execute_place(self, order_package, http_session):
-        print("execute_place")
-
-    def execute_cancel(self, order_package, http_session):
-        print("execute_cancel")
-
-    def execute_update(self, order_package, http_session):
-        print("execute_update")
-
-    def execute_replace(self, order_package, http_session):
-        print("execute_replace")
-
-
-class PrinterClient(BaseClient):
-    def login(self):
-        return
-
-    def keep_alive(self):
-        return
-
-    def logout(self):
-        return
-
-    def add_execution(self, flumine):
-        self.execution = PrinterExecution(flumine)
-
-    @property
-    def min_bet_size(self):
-        return 2.0
-
-    @property
-    def min_bet_payout(self):
-        return 10.0
-
-    @property
-    def min_bsp_liability(self):
-        return 10.0
-
-
-client = PrinterClient()
-```
-
-!!! tip
-    This can be used to handle other exchanges (betdaq) or API's that flumine cannot currently handle.
diff --git a/docs/markets.md b/docs/markets.md
index 41feecd..17fb82d 100644
--- a/docs/markets.md
+++ b/docs/markets.md
@@ -32,6 +32,24 @@ Within markets you have market objects which contains current up to date market
 - `elapsed_seconds_closed` Seconds since market was closed (543.21)
 - `market_start_datetime` Market scheduled start time
 
+## Transaction
+
+The transaction class is used by default when orders are executed, however it is possible to control the execution behaviour using the transaction class like so:
+
+```python
+with market.transaction() as t:
+    market.place_order(order)  # executed immediately in separate transaction
+    t.place_order(order)  # executed on transaction __exit__
+
+with market.transaction() as t:
+    t.place_order(order)
+    ..
+    t.execute()  # above order executed
+    ..
+    t.cancel_order(order)
+    t.place_order(order)  # both executed on transaction __exit__
+```
+
 ## Blotter
 
 The blotter is a simple and fast class to hold all orders for a particular market.
@@ -42,6 +60,11 @@ The blotter is a simple and fast class to hold all orders for a particular marke
 - `strategy_selection_orders(strategy, selection_id, handicap)` Returns all orders related to a strategy selection
 - `selection_exposure(strategy, lookup)` Returns strategy/selection exposure
 
+### Properties
+
+- `live_orders` List of live orders
+- `has_live_orders` Bool on live orders
+
 ## Middleware
 
 It is common that you want to carry about analysis on a market before passing through to strategies, similar to Django's middleware design flumine allows middleware to be executed.
diff --git a/docs/strategies.md b/docs/strategies.md
new file mode 100644
index 0000000..1ab0ed1
--- /dev/null
+++ b/docs/strategies.md
@@ -0,0 +1,46 @@
+# Strategies
+
+## BaseStrategy
+
+The base strategy class should be used for all strategies and contains the following parameters / functions for order/trade execution.
+
+### Parameters
+
+- `market_filter` Streaming market filter required
+- `market_data_filter` Streaming market data filter required
+- `streaming_timeout` Streaming timeout, will call snap() on cache every x seconds
+- `conflate_ms` Streaming conflate
+- `stream_class` MarketStream or RawDataStream
+- `name` Strategy name, if None will default to class name
+- `context` Dictionary object where any extra data can be stored here such as triggers
+- `max_selection_exposure` Max exposure per selection (including new order), note this does __not__ handle reduction in exposure due to laying another runner
+- `max_order_exposure` Max exposure per order
+- `client` Strategy client, half implemented when flumine will be migrated to multi clients
+- `max_trade_count` Max total number of trades per runner
+- `max_live_trade_count` Max live (with executable orders) trades per runner
+- `multi_order_trades` Allow multiple live orders per trade
+
+### Functions
+
+The following functions can be overridden dependent on the strategy:
+
+- `add()` Function called when strategy is added to framework
+- `start()` Function called when framework starts
+- `check_market_book()` Function called with marketBook, `process_market_book` is only executed if this returns True
+- `process_market_book()` Processes market book updates, called on every update that is received
+- `process_raw_data()` As per `process_market_book` but handles raw data
+- `process_orders()` Process list of Order objects for strategy and Market
+- `process_closed_market()` Process Market after closure
+- `finish()` Function called when framework ends
+
+### Runner Context
+
+Each strategy stores a `RunnerContext` object which contains the state of a runner based on all and current active trades. This is used by controls to calculate exposure and control the number of live or total trades.
+
+```python
+runner_context = self.get_runner_context(
+    market.market_id, runner.selection_id, runner.handicap
+)
+
+runner_context.live_trade_count
+```
diff --git a/docs/workers.md b/docs/workers.md
new file mode 100644
index 0000000..f15b94d
--- /dev/null
+++ b/docs/workers.md
@@ -0,0 +1,30 @@
+# Workers
+
+## Background Workers
+
+Background workers run in their own thread allowing cleanup / cron like workers to run in the background, by default flumine adds the following workers:
+
+- `keep_alive`: runs every 1200s (or session_timeout/2) to make sure the client is either logged in or kept alive
+- `poll_account_balance`: runs every 120s to poll account balance endpoint
+- `poll_market_catalogue`: runs every 60s to poll listMarketCatalogue endpoint
+- `poll_market_closure`: checks for closed markets to get cleared orders at order and market level
+
+## Custom Workers
+
+Further workers can be added as per:
+
+```python
+from flumine.worker import BackgroundWorker
+
+def func(context: dict, flumine, name=""):
+    print(name)
+
+    
+worker = BackgroundWorker(
+    framework, interval=10, function=func, func_args=("hello",)
+)
+
+framework.add_worker(
+    worker
+)
+```
diff --git a/flumine/__version__.py b/flumine/__version__.py
index 9f750ab..3c014b3 100644
--- a/flumine/__version__.py
+++ b/flumine/__version__.py
@@ -1,6 +1,6 @@
 __title__ = "flumine"
 __description__ = "Betfair trading framework"
 __url__ = "https://github.com/liampauling/flumine"
-__version__ = "1.18.5"
+__version__ = "1.18.6"
 __author__ = "Liam Pauling"
 __license__ = "MIT"
diff --git a/flumine/streams/basestream.py b/flumine/streams/basestream.py
index 2ec6e61..a1b4a9a 100644
--- a/flumine/streams/basestream.py
+++ b/flumine/streams/basestream.py
@@ -12,7 +12,7 @@ class BaseStream(threading.Thread):
 
     LISTENER = StreamListener
     MAX_LATENCY = 0.5
-    RETRY_WAIT = wait_exponential(multiplier=1, min=2, max=20)
+    RETRY_WAIT = wait_exponential(multiplier=1, min=2, max=60)
 
     def __init__(
         self,
diff --git a/flumine/utils.py b/flumine/utils.py
index cd0833d..4c14c0b 100644
--- a/flumine/utils.py
+++ b/flumine/utils.py
@@ -42,6 +42,8 @@ def file_line_count(file_path: str) -> int:
 
 def get_file_md(file_dir: str, value: str) -> Optional[str]:
     # get value from raw streaming file marketDefinition
+    if isinstance(file_dir, tuple):
+        file_dir = file_dir[0]
     with open(file_dir, "r") as f:
         first_line = f.readline()
         update = json.loads(first_line)
@@ -142,9 +144,6 @@ def price_ticks_away(price: float, n_ticks: int) -> float:
         return 1000
 
 
-# todo LRU cache?
-# JH: LRU cache does not work with list inputs as they are unhashable.
-#     So might need to refactor mb and ml into tuples.
 def calculate_matched_exposure(mb: list, ml: list) -> Tuple:
     """Calculates exposure based on list
     of (price, size)
@@ -152,10 +151,16 @@ def calculate_matched_exposure(mb: list, ml: list) -> Tuple:
     """
     if not mb and not ml:
         return 0.0, 0.0
-    back_exp = sum(-i[1] for i in mb)
-    back_profit = sum((i[0] - 1) * i[1] for i in mb)
-    lay_exp = sum((i[0] - 1) * -i[1] for i in ml)
-    lay_profit = sum(i[1] for i in ml)
+    if mb:
+        back_exp = sum(-i[1] for i in mb)
+        back_profit = sum((i[0] - 1) * i[1] for i in mb)
+    else:
+        back_exp, back_profit = 0, 0
+    if ml:
+        lay_exp = sum((i[0] - 1) * -i[1] for i in ml)
+        lay_profit = sum(i[1] for i in ml)
+    else:
+        lay_exp, lay_profit = 0, 0
     _win = back_profit + lay_exp
     _lose = lay_profit + back_exp
     return round(_win, 2), round(_lose, 2)
@@ -177,7 +182,6 @@ def calculate_unmatched_exposure(ub: list, ul: list) -> Tuple:
     return round(lay_exp, 2), round(back_exp, 2)
 
 
-# todo LRU cache?
 def wap(matched: list) -> Tuple[float, float]:
     if not matched:
         return 0, 0
diff --git a/mkdocs.yml b/mkdocs.yml
index 6c584ee..efa9444 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -8,19 +8,27 @@ theme:
       primary: 'blue grey'
       accent: 'indigo'
 
+    features:
+      - navigation.tabs
+      - navigation.tabs.sticky
+
 repo_name: liampauling/flumine
 repo_url: https://github.com/liampauling/flumine
 edit_uri: ""
 
 nav:
     - Introduction: 'index.md'
-    - QuickStart: 'quickstart.md'
-    - Markets: 'markets.md'
-    - Clients / Execution: 'execution.md'
-    - Trades / Orders: 'trades.md'
-    - Advanced Usage: 'advanced.md'
-    - Architecture: 'architecture.md'
-    - Help: 'help.md'
+    - Getting started:
+      - QuickStart: 'quickstart.md'
+      - Help: 'help.md'
+    - Advanced:
+      - Stategies: 'strategies.md'
+      - Trades / Orders: 'trades.md'
+      - Markets: 'markets.md'
+      - Controls: 'controls.md'
+      - Workers: 'workers.md'
+      - Advanced Usage: 'advanced.md'
+      - Architecture: 'architecture.md'
 
 markdown_extensions:
   - admonition


Test patch:
diff --git a/requirements-test.txt b/requirements-test.txt
index c8f26a0..7e1a600 100644
--- a/requirements-test.txt
+++ b/requirements-test.txt
@@ -1,5 +1,5 @@
 # Tests & Linting
-black==21.4b0
+black==21.5b0
 coverage
 
 # Documentation
diff --git a/tests/test_utils.py b/tests/test_utils.py
index f1578dc..959068f 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -20,6 +20,12 @@ class UtilsTest(unittest.TestCase):
             utils.get_file_md("tests/resources/PRO-1.170258213", "eventId"), "29761984"
         )
 
+    def test_get_file_event_id_tuple(self):
+        self.assertEqual(
+            utils.get_file_md(("tests/resources/PRO-1.170258213", "test"), "eventId"),
+            "29761984",
+        )
+
     def test_chunks(self):
         self.assertEqual([i for i in utils.chunks([1, 2, 3], 1)], [[1], [2], [3]])