Repository: rotki/rotki

Issue:
pywsgi log format is duplicating timestamps
_Originally posted by @aspiers and @LefterisJP in https://github.com/rotki/rotki/issues/1187#issuecomment-662122516_:
> > (and BTW it seems that the pywsgi log format is duplicating timestamps somehow).
>
> Indeed. It's been like that forever. Perhaps need to have an issue for that to remind us too :)

OK here's the issue ;-)  Example:

```
21/07/2020 22:17:58 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:17:58] "GET /api/1/messages/ HTTP/1.1" 200 245 0.001985
21/07/2020 22:17:58 BST -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"last_balance_save": 1595365275, "eth_node_connection": false, "history_process_start_ts": -1, "history_process_current_ts": -1}, "message": ""}, status_code=200
21/07/2020 22:17:58 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:17:58] "GET /api/1/periodic/ HTTP/1.1" 200 345 0.001522
21/07/2020 22:18:03 BST -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"warnings": [], "errors": []}, "message": ""}, status_code=200
21/07/2020 22:18:03 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:18:03] "GET /api/1/messages/ HTTP/1.1" 200 245 0.001976
21/07/2020 22:18:03 BST -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"last_balance_save": 1595365275, "eth_node_connection": false, "history_process_start_ts": -1, "history_process_current_ts": -1}, "message": ""}, status_code=200
21/07/2020 22:18:03 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:18:03] "GET /api/1/periodic/ HTTP/1.1" 200 345 0.001935
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop start
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop end
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop start
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop end
```
pywsgi log format is duplicating timestamps
_Originally posted by @aspiers and @LefterisJP in https://github.com/rotki/rotki/issues/1187#issuecomment-662122516_:
> > (and BTW it seems that the pywsgi log format is duplicating timestamps somehow).
>
> Indeed. It's been like that forever. Perhaps need to have an issue for that to remind us too :)

OK here's the issue ;-)  Example:

```
21/07/2020 22:17:58 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:17:58] "GET /api/1/messages/ HTTP/1.1" 200 245 0.001985
21/07/2020 22:17:58 BST -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"last_balance_save": 1595365275, "eth_node_connection": false, "history_process_start_ts": -1, "history_process_current_ts": -1}, "message": ""}, status_code=200
21/07/2020 22:17:58 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:17:58] "GET /api/1/periodic/ HTTP/1.1" 200 345 0.001522
21/07/2020 22:18:03 BST -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"warnings": [], "errors": []}, "message": ""}, status_code=200
21/07/2020 22:18:03 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:18:03] "GET /api/1/messages/ HTTP/1.1" 200 245 0.001976
21/07/2020 22:18:03 BST -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"last_balance_save": 1595365275, "eth_node_connection": false, "history_process_start_ts": -1, "history_process_current_ts": -1}, "message": ""}, status_code=200
21/07/2020 22:18:03 BST -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-07-21 22:18:03] "GET /api/1/periodic/ HTTP/1.1" 200 345 0.001935
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop start
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop end
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop start
21/07/2020 22:18:06 BST -- DEBUG:rotkehlchen.rotkehlchen:Main loop end
```


Discussion:

Resolving patch:
diff --git a/rotkehlchen/api/rest.py b/rotkehlchen/api/rest.py
--- a/rotkehlchen/api/rest.py
+++ b/rotkehlchen/api/rest.py
@@ -969,6 +969,7 @@ def query_all_assets() -> Response:
         return api_response(
             _wrap_in_ok_result(assets),
             status_code=HTTPStatus.OK,
+            log_result=False,
         )
 
     @require_loggedin_user()
@@ -1097,7 +1098,8 @@ def get_eth_tokens(self) -> Response:
         return api_response(
             _wrap_in_ok_result(result_dict),
             status_code=HTTPStatus.OK,
-            log_result=False)
+            log_result=False,
+        )
 
     def _add_owned_eth_tokens(
             self,
diff --git a/rotkehlchen/logging.py b/rotkehlchen/logging.py
--- a/rotkehlchen/logging.py
+++ b/rotkehlchen/logging.py
@@ -1,5 +1,7 @@
+import argparse
 import logging
 import random
+import re
 import string
 import time
 from typing import Any, Dict, MutableMapping, Tuple
@@ -7,6 +9,8 @@
 from rotkehlchen.fval import FVal
 from rotkehlchen.typing import EthAddress
 
+PYWSGI_RE = re.compile(r'\[(.*)\] ')
+
 ANONYMIZABLE_BIGINT_VALUES = ('tx_value', 'gas_price', 'gas', 'gas_used')
 ANONYMIZABLE_BIG_VALUES = (
     'amount',
@@ -131,3 +135,81 @@ def process(self, msg: str, kwargs: MutableMapping[str, Any]) -> Tuple[str, Dict
 
         msg = msg + ','.join(' {}={}'.format(a[0], a[1]) for a in new_kwargs.items())
         return msg, {}
+
+
+class PywsgiFilter(logging.Filter):
+    def filter(self, record: logging.LogRecord) -> bool:
+        """Filter out the additional timestamp put in by pywsgi
+
+        This is really a hack to fix https://github.com/rotki/rotki/issues/1192
+
+        It seems that the way they do the logging in pywsgi they create the log
+        entry completely on their own. So the %message part of the entry contains
+        everything and is hence not properly customizale via normal python logging.
+
+        Other options apart from using this filter would be:
+        - Ignore it and just have the timestamp two times in the logs
+        - Completely disable pywsgi logging and perhaps move it all to the
+        rest api.
+        """
+        record.msg = PYWSGI_RE.sub('', record.msg)
+        return True
+
+
+def configure_logging(args: argparse.Namespace) -> None:
+    loglevel = args.loglevel.upper()
+    formatters = {
+        'default': {
+            'format': '[%(asctime)s] %(name)s: %(message)s',
+            'datefmt': '%d/%m/%Y %H:%M:%S %Z',
+        },
+    }
+    handlers = {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'level': loglevel,
+            'formatter': 'default',
+        },
+    }
+
+    if args.logtarget == 'file':
+        selected_handlers = ['file']
+        handlers['file'] = {
+            'class': 'logging.FileHandler',
+            'filename': args.logfile,
+            'mode': 'w',
+            'level': loglevel,
+            'formatter': 'default',
+        }
+    else:
+        selected_handlers = ['console']
+
+    filters = {
+        'pywsgi': {
+            '()': PywsgiFilter,
+        },
+    }
+    loggers = {
+        '': {  # root logger
+            'level': loglevel,
+            'handlers': selected_handlers,
+        },
+        'rotkehlchen.api.server.pywsgi': {
+            'level': loglevel,
+            'handlers': selected_handlers,
+            'filters': ['pywsgi'],
+            'propagate': False,
+        },
+    }
+    logging.config.dictConfig({
+        'version': 1,
+        'disable_existing_loggers': False,
+        'filters': filters,
+        'formatters': formatters,
+        'handlers': handlers,
+        'loggers': loggers,
+    })
+
+    if not args.logfromothermodules:
+        logging.getLogger('urllib3').setLevel(logging.CRITICAL)
+        logging.getLogger('urllib3.connectionpool').setLevel(logging.CRITICAL)
diff --git a/rotkehlchen/rotkehlchen.py b/rotkehlchen/rotkehlchen.py
--- a/rotkehlchen/rotkehlchen.py
+++ b/rotkehlchen/rotkehlchen.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 
 import argparse
-import logging
+import logging.config
 import os
 import time
 from pathlib import Path
@@ -37,7 +37,12 @@
 from rotkehlchen.greenlets import GreenletManager
 from rotkehlchen.history import PriceHistorian, TradesHistorian
 from rotkehlchen.inquirer import Inquirer
-from rotkehlchen.logging import DEFAULT_ANONYMIZED_LOGS, LoggingSettings, RotkehlchenLogsAdapter
+from rotkehlchen.logging import (
+    DEFAULT_ANONYMIZED_LOGS,
+    LoggingSettings,
+    RotkehlchenLogsAdapter,
+    configure_logging,
+)
 from rotkehlchen.premium.premium import Premium, PremiumCredentials, premium_create_and_verify
 from rotkehlchen.premium.sync import PremiumSyncManager
 from rotkehlchen.transactions import EthereumAnalyzer
@@ -74,35 +79,7 @@ def __init__(self, args: argparse.Namespace) -> None:
         # authenticate or premium server temporarily offline
         self.premium: Optional[Premium] = None
         self.user_is_logged_in = False
-
-        logfilename = None
-        if args.logtarget == 'file':
-            logfilename = args.logfile
-
-        if args.loglevel == 'debug':
-            loglevel = logging.DEBUG
-        elif args.loglevel == 'info':
-            loglevel = logging.INFO
-        elif args.loglevel == 'warn':
-            loglevel = logging.WARN
-        elif args.loglevel == 'error':
-            loglevel = logging.ERROR
-        elif args.loglevel == 'critical':
-            loglevel = logging.CRITICAL
-        else:
-            raise AssertionError('Should never get here. Illegal log value')
-
-        logging.basicConfig(
-            filename=logfilename,
-            filemode='w',
-            level=loglevel,
-            format='%(asctime)s -- %(levelname)s:%(name)s:%(message)s',
-            datefmt='%d/%m/%Y %H:%M:%S %Z',
-        )
-
-        if not args.logfromothermodules:
-            logging.getLogger('urllib3').setLevel(logging.CRITICAL)
-            logging.getLogger('urllib3.connectionpool').setLevel(logging.CRITICAL)
+        configure_logging(args)
 
         self.sleep_secs = args.sleep_secs
         if args.data_dir is None:


Test patch: