Repository: hummingbot/hummingbot
Pull request: feat / continually export trades to CSV file
**Before submitting this PR, please make sure**:
- [x] Your code builds clean without any errors or warnings
- [x] You are using approved title ("feat/", "fix/", "docs/", "refactor/", etc)


**Optional**:
- Related Github issue:
- Related Clubhouse Story: [ch7715]


**A description of the changes proposed in the pull request**:


Problem statement:
Continuously save trades to CSV file
**Feature Suggestion**
// A succinct description of **how you want to achieve the desired behavior**.
// If you have looked at the code and know exactly what code changes are needed then please _consider submitting a pull request_ instead.
1) Scheduled auto Trade Data export
2) Capable of export trade data on a certain date

**Impact**
// A succinct description of **why you want the _desired behavior_** specified above.
The desired behavior will allow me to ...
1) So Users do not need to manually export the trade data constantly.  
*When the container stopped running, all trades data are wiped off if we did not export the trade data manually
2) When we export trade data, all trade data are exported to .csv.  If we manually exported the trade data every day, we are getting a lot of previous trade data in the same file.

**Additional context**
Add any other context or screenshots about the feature request here.
Trade data is valuable data for bot traders
So the above features mentioned is very important for us

Hints from issue discussion:
@sweetpotatoz as a workaround, I can show you how to get trade data from the local database.
@dennisocana can you describe the workaround briefly here?
@mifeng the trades are saved in the database in the `.sqlite` file inside the `/data` folder. I provided instructions how to access this file even if Hummingbot is installed via Docker.

<img width="960" alt="dbeaver_nlyGdfrW2H" src="https://user-images.githubusercontent.com/50150287/81458129-74ab6680-91cb-11ea-8fff-686905ff8142.png">

Then I created PR https://github.com/CoinAlpha/hummingbot/pull/1280 to edit the `create.sh` and `update.sh` scripts so it will always map the folder to `hummingbot_data` when creating an instance and this file can be easily accessed.
@dennisocana I meant describing the tool that you used to access the trades.

What software is it? How should users import the `.sqlite` file into the software?
@mifeng any database management software like [DBeaver](https://dbeaver.io/) or [DataGrip](https://www.jetbrains.com/datagrip/) or via web by simply dropping the file at https://inloop.github.io/sqlite-viewer/

![image](https://user-images.githubusercontent.com/50150287/81459334-f9997e80-91d1-11ea-879c-311f345015f4.png)

Importing the database file varies depending on the software. For DBeaver, create a new database connection by following the steps outlined [here](https://github.com/dbeaver/dbeaver/wiki/Create-Connection) and select **SQLite**.

![image](https://user-images.githubusercontent.com/50150287/81459436-7fb5c500-91d2-11ea-8d70-fa18b471336d.png)

Click **Next** and **Browse ...** to where the `.sqlite` file is saved, then click **FInish**.

![image](https://user-images.githubusercontent.com/50150287/81459455-9d832a00-91d2-11ea-9ec3-ae00e4e3a1b9.png)

DataGrip has different set of instructions to do this. As I recall, you can simply go to **File** -> **Open** but I can no longer confirm since my trial already expired.

Commit message: (feat) continually export trades to CSV file

Patch to review:
diff --git a/hummingbot/market/markets_recorder.py b/hummingbot/market/markets_recorder.py
index 0439e590010..9cfa3e6fba8 100644
--- a/hummingbot/market/markets_recorder.py
+++ b/hummingbot/market/markets_recorder.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python
 
+import pandas as pd
 import asyncio
 from sqlalchemy.orm import (
     Session,
@@ -15,6 +16,7 @@
     Union
 )
 
+from hummingbot.client.settings import CONF_FILE_PATH
 from hummingbot.core.event.events import (
     BuyOrderCreatedEvent,
     SellOrderCreatedEvent,
@@ -233,6 +235,13 @@ def _did_fill_order(self,
         session.add(trade_fill_record)
         self.save_market_states(self._config_file_path, market, no_commit=True)
         session.commit()
+        self.append_to_csv(trade_fill_record)
+
+    def append_to_csv(self, trade: TradeFill):
+        csv_file = CONF_FILE_PATH + "/trades_" + trade.config_file_path + ".csv"
+        df = pd.DataFrame([[trade.config_file_path, trade.strategy, trade.market, trade.symbol, trade.base_asset, trade.quote_asset, trade.timestamp,
+                            trade.order_id, trade.trade_type, trade.order_type, trade.price, trade.amount, trade.trade_fee, trade.exchange_trade_id]])
+        df.to_csv(csv_file, mode='a', header=False, index=False)
 
     def _update_order_status(self,
                              event_tag: int,


Code review comments:
diff_hunk: @@ -15,6 +16,7 @@
     Union
 )
 
+from hummingbot.client.settings import CONF_FILE_PATH
path: hummingbot/market/markets_recorder.py
text: @user1:
Would this save the file in the `conf` folder? If possible, we should save this in the `data` folder since that's where the sqlite db containing the trades are.

@author:
Done!

diff_hunk: @@ -233,6 +235,13 @@ def _did_fill_order(self,
         session.add(trade_fill_record)
         self.save_market_states(self._config_file_path, market, no_commit=True)
         session.commit()
+        self.append_to_csv(trade_fill_record)
+
+    def append_to_csv(self, trade: TradeFill):
+        csv_file = CONF_FILE_PATH + "/trades_" + trade.config_file_path + ".csv"
+        df = pd.DataFrame([[trade.config_file_path, trade.strategy, trade.market, trade.symbol, trade.base_asset, trade.quote_asset, trade.timestamp,
+                            trade.order_id, trade.trade_type, trade.order_type, trade.price, trade.amount, trade.trade_fee, trade.exchange_trade_id]])
path: hummingbot/market/markets_recorder.py
text: @user1:
Can you add the order age here also, so that everything we show in `history` is also in the exported CSV?

@author:
Done!