Repository: logzio/logzio-python-handler

Issue:
Endless loop when no connection to Logz.io
I'm sending Python application logs to Logz.io. If there is not connection to the remote server, `logzio-python-handler` sits in an endless loop retrying until the queue is empty (which it never is because the logs never get sent because there is no connection).

Is this desirable behaviour? I would imagine that there should be a limit on the number of retries. My application essentially hangs when closing, as it is waiting for the logs handler to flush.

```
Got exception while sending logs to Logz.io, Try (4/4). Message: HTTPSConnectionPool(host='listener-nl.logz.io', port=8071): Max retries exceeded with url: /?token=<<mytoken>>(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000214D43E8FD0>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
2022-01-19 15:45:01,623 - logzio.logger - WARNING - Got exception while sending logs to Logz.io, Try (4/4). Message: HTTPSConnectionPool(host='listener-nl.logz.io', port=8071): Max retries exceeded with url: /?token=<<mytoken>>(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000214D43E8FD0>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
Could not send logs to Logz.io after 4 tries, backing up to local file system
2022-01-19 15:45:17,640 - logzio.logger - ERROR - Could not send logs to Logz.io after 4 tries, backing up to local file system
Backing up your logs to logzio-failures-19012022-154517.txt
2022-01-19 15:45:17,642 - logzio.logger - INFO - Backing up your logs to logzio-failures-19012022-154517.txt
Got exception while sending logs to Logz.io, Try (1/4). Message: HTTPSConnectionPool(host='listener-nl.logz.io', port=8071): Max retries exceeded with url: /?token=<<mytoken>>(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000214D3DC4BE0>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
2022-01-19 15:45:19,741 - logzio.logger - WARNING - Got exception while sending logs to Logz.io, Try (1/4). Message: HTTPSConnectionPool(host='listener-nl.logz.io', port=8071): Max retries exceeded with url: /?token=<<mytoken>>(Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000214D3DC4BE0>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
```

Discussion:

Patch:
diff --git a/.github/workflows/auto-release.yml b/.github/workflows/auto-release.yml
new file mode 100644
index 0000000..9b08f61
--- /dev/null
+++ b/.github/workflows/auto-release.yml
@@ -0,0 +1,27 @@
+name: Pypi Release
+on:
+  workflow_run:
+    workflows: ["CI Tests"]
+    branches: [master]
+    types:
+      - completed
+  # Allows you to run this workflow manually from the Actions tab
+jobs:
+  build-and-release:
+    if: ${{ github.event.workflow_run.conclusion == 'success' }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v3.0.0
+        with:
+          python-version: '3.x'
+      - name: "Install dependencies"
+        run: |
+          python3 -m pip install setuptools wheel twine
+      - name: "Build and uploads to PyPI"
+        run: |
+          python3 setup.py sdist bdist_wheel
+          python3 -m twine upload dist/*
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_REPO_TOKEN }}
diff --git a/README.md b/README.md
index bad528b..282870b 100644
--- a/README.md
+++ b/README.md
@@ -32,6 +32,7 @@ Travis CI will build this handler and test against:
   - "3.6"
   - "3.7"
   - "3.8"
+  - "3.9"
 
 We can't ensure compatibility to any other version, as we can't test it automatically.
 
@@ -76,7 +77,7 @@ format={"additional_field": "value"}
  - Debug flag. Set to True, will print debug messages to stdout. (defaults to "False")
  - Backup logs flag. Set to False, will disable the local backup of logs in case of failure. (defaults to "True")
  - Network timeout, in seconds, int or float, for sending the logs to logz.io. (defaults to 10)
- - Retries number (retry_no), in seconds (defaults to 4).
+ - Retries number (retry_no, defaults to 4).
  - Retry timeout (retry_timeout) in seconds (defaults to 2).
 
  Please note, that you have to configure those parameters by this exact order.
@@ -213,15 +214,19 @@ LOGGING = {
 Please note that if you are using `python 3.8` it is preferred to use the `logging.config.dictConfig` method, as mentioned in [python's documentation](https://docs.python.org/3/library/logging.config.html#configuration-file-format).
 
 ## Release Notes
+- 3.1.1
+  - Bug fixes (issue #68, exception message formatting)
+  - Added CI: Tests and Auto release 
+
 - 3.1.0
     - Bug fixes
     - Retry number and timeout is now configurable
-- 3.0.0
-    - Deprecated `python2.7` & `python3.4`
-    - Changed log levels on `_flush_queue()` method (@hilsenrat)
     
 <details>
   <summary markdown="span"> Expand to check old versions </summary>
+- 3.0.0
+    - Deprecated `python2.7` & `python3.4`
+    - Changed log levels on `_flush_queue()` method (@hilsenrat)
 
 - 2.0.15
     - Added flusher decorator for serverless platforms(@mcmasty)
diff --git a/logzio/handler.py b/logzio/handler.py
index 9fca847..e0a4b45 100644
--- a/logzio/handler.py
+++ b/logzio/handler.py
@@ -98,13 +98,15 @@ class LogzioHandler(logging.Handler):
         if message.exc_info:
             return_json['exception'] = self.format_exception(message.exc_info)
 
-            # # We want to ignore default logging formatting on exceptions
-            # # As we handle those differently directly into exception field
+        # # We want to ignore default logging formatting on exceptions
+        # # As we handle those differently directly into exception field
 
         formatted_message = self.format(message)
+        # Exception with multiple fields, apply them to log json.
         if isinstance(formatted_message, dict):
             return_json.update(formatted_message)
-        else:
+        # No exception, apply default formatted message
+        elif not message.exc_info:
             return_json['message'] = formatted_message
 
         return_json.update(self.extra_fields(message))
diff --git a/logzio/logger.py b/logzio/logger.py
index f120485..529efaa 100644
--- a/logzio/logger.py
+++ b/logzio/logger.py
@@ -7,7 +7,9 @@ def get_logger(debug):
 
 
 def get_stdout_logger(debug):
-    return __get_logger(debug, __name__ + '_stdout', logging.StreamHandler(sys.stdout))
+    stdout_logger = __get_logger(debug, __name__ + '_stdout', logging.StreamHandler(sys.stdout))
+    stdout_logger.propagate = False
+    return stdout_logger
 
 
 def __get_logger(debug, name, handler=None):
diff --git a/logzio/sender.py b/logzio/sender.py
index ca73b73..5a7d16c 100644
--- a/logzio/sender.py
+++ b/logzio/sender.py
@@ -40,7 +40,6 @@ class LogzioSender:
         self.token = token
         self.url = '{}/?token={}'.format(url, token)
         self.logs_drain_timeout = logs_drain_timeout
-        self.logger = get_logger(debug)
         self.stdout_logger = get_stdout_logger(debug)
         self.backup_logs = backup_logs
         self.network_timeout = network_timeout
@@ -57,7 +56,6 @@ class LogzioSender:
         self._initialize_sending_thread()
 
     def __del__(self):
-        del self.logger
         del self.stdout_logger
         del self.backup_logs
         del self.queue
@@ -93,7 +91,7 @@ class LogzioSender:
             try:
                 self._flush_queue()
             except Exception as e:
-                self.logger.debug(
+                self.stdout_logger.debug(
                     'Unexpected exception while draining queue to Logz.io, '
                     'swallowing. Exception: %s', e)
 
@@ -165,7 +163,7 @@ class LogzioSender:
                 self.stdout_logger.error(
                     'Could not send logs to Logz.io after %s tries, '
                     'backing up to local file system', self.number_of_retries)
-                backup_logs(logs_list, self.logger)
+                backup_logs(logs_list, self.stdout_logger)
 
             del logs_list
 
diff --git a/setup.py b/setup.py
index fe0e999..0b6d2e1 100644
--- a/setup.py
+++ b/setup.py
@@ -3,7 +3,7 @@
 from setuptools import setup, find_packages
 setup(
     name="logzio-python-handler",
-    version='3.1.0',
+    version='3.1.1',
     description="Logging handler to send logs to your Logz.io account with bulk SSL",
     keywords="logging handler logz.io bulk https",
     author="roiravhon",
diff --git a/tox.ini b/tox.ini
index 066a58c..9fd1b6d 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
 [tox]
 minversion = 1.7.2
-envlist = flake8, py3flake8, py35, py36, py37, py38, pypy, pypy3
+envlist = flake8, py3flake8, py35, py36, py37, py38, py39, pypy, pypy3
 skip_missing_interpreters = true
 
 [testenv]
@@ -13,11 +13,14 @@ passenv = CI TRAVIS TRAVIS_*
 commands = pytest --cov-report term-missing --cov logzio tests -v
 
 [testenv:flake8]
-basepython = python3.8
+basepython = python3.9
 deps = flake8
 commands = flake8 logzio
 
 [testenv:py3flake8]
-basepython = python3.8
+basepython = python3.9
 deps = flake8
 commands = flake8 logzio
+
+[gh-actions]
+python = 3.9: py39


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
new file mode 100644
index 0000000..62d9a53
--- /dev/null
+++ b/.github/workflows/tests.yml
@@ -0,0 +1,14 @@
+name: CI Tests
+on: [push]
+jobs:
+  run-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v1
+      - uses: actions/setup-python@v3.0.0
+        with:
+          python-version: '3.x'
+      - name: Install tox
+        run: pip install tox
+      - name: Run tox
+        run: python -m tox
\ No newline at end of file
diff --git a/tests/test_logzioHandler.py b/tests/test_logzioHandler.py
index 60ec3ec..e2ae4a1 100644
--- a/tests/test_logzioHandler.py
+++ b/tests/test_logzioHandler.py
@@ -159,7 +159,6 @@ class TestLogzioHandler(TestCase):
 
         formatted_message["exception"] = formatted_message["exception"].replace(os.path.abspath(__file__), "")
         formatted_message["exception"] = re.sub(r", line \d+", "", formatted_message["exception"])
-        formatted_message["message"] = formatted_message["message"].replace(os.path.abspath(__file__), "")
 
         self.assertDictEqual(
             {
@@ -167,10 +166,8 @@ class TestLogzioHandler(TestCase):
                 'line_number': 10,
                 'log_level': 'NOTSET',
                 'logger': 'my-logger',
-                'message': f'exception test:\nTraceback (most recent call last):\n  File "", line 142, in test_exc\n    raise '
-                           'ValueError("oops.")\nValueError: oops.',
-                'exception': 'Traceback (most recent call last):\n\n  File "", in test_exc\n    raise ValueError('
-                             '"oops.")\n\nValueError: oops.\n',
+                'message': 'exception test:',
+                'exception': 'Traceback (most recent call last):\n\n  File "", in test_exc\n    raise ValueError("oops.")\n\nValueError: oops.\n',
                 'path_name': 'handler_test.py',
                 'type': 'python'
             },