Repository: fkie-cad/Logprep (language: python)

Issue:
issues initiating uvicorn with multiprocessing logqueue in python version 3.12.4 
When running the logprep acceptance test with python 3.12.4 the following two tests of the http input connector fail:
```bash
FAILED tests/acceptance/test_http_input.py::test_http_input_accepts_message_for_single_pipeline - TimeoutError: Timer expired
FAILED tests/acceptance/test_http_input.py::test_http_input_accepts_message_for_multiple_pipelines - TimeoutError: Timer expired
```
When running the acceptance tests with debugging information (adding print statements to the `tests/acceptance/util.py:wait_for_output` method), we get the following stacktraces:
```bash
2024-06-20 09:17:21 2620   Manager    INFO    : Created new pipeline
2024-06-20 09:17:21 2620   py.warnings WARNING : /opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=2620) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()

2024-06-20 09:17:21 2620   Manager    WARNING : Restarting failed pipeline on index 1 with exit code: 1
Process Process-14:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/logging/config.py", line 581, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/logging/config.py", line 792, in configure_handler
    proxy_queue = MM().Queue()
                  ^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/multiprocessing/managers.py", line 562, in start
    self._process.start()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/work/Logprep/Logprep/logprep/framework/pipeline.py", line 210, in run
    self._setup()
  File "/home/runner/work/Logprep/Logprep/logprep/framework/pipeline.py", line 50, in _inner
    return func(self)
           ^^^^^^^^^^
  File "/home/runner/work/Logprep/Logprep/logprep/framework/pipeline.py", line 186, in _setup
    self._input.setup()
  File "/home/runner/work/Logprep/Logprep/logprep/connector/http/input.py", line 466, in setup
    self.http_server = http.ThreadingHTTPServer(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/Logprep/Logprep/logprep/util/http.py", line 71, in __init__
    uvicorn_config = uvicorn.Config(**uvicorn_config, app=app, log_config=logprep_log_config)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/uvicorn/config.py", line 273, in __init__
    self.configure_logging()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/uvicorn/config.py", line 365, in configure_logging
    logging.config.dictConfig(self.log_config)
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/logging/config.py", line 920, in dictConfig
    dictConfigClass(config).configure()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/logging/config.py", line 588, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'queue'
```
This indicates that there is an issue when configuring the logqueue for the uvcorn.Config. In the logprep version 3.12.4 the logging configuration initiates a multiprocessing queue to check for the configured type. Initiating this queue inside a daemonic thread probably issues the `AssertionError: daemonic processes are not allowed to have children` exception.

Python 3.12.3: https://github.com/python/cpython/blob/v3.12.3/Lib/logging/config.py#L789
Python 3.12.4: https://github.com/python/cpython/blob/v3.12.4/Lib/logging/config.py#L792 (containing the `MM().Queue()` which causes the error).

The change was probably introduced by this pull request: https://github.com/python/cpython/pull/120030/files

When the logqueue is given to a uvicorn server that runs in the main process this issue does not arise. As reference following scratch python file can be used for validating this: 

```python
import multiprocessing as mp

import falcon.asgi
import uvicorn

logqueue = mp.Queue(-1)

log_config = {
    "version": 1,
    "formatters": {
        "logprep": {
            "class": "logprep.util.logging.LogprepFormatter",
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "logprep",
            "stream": "ext://sys.stdout",
        },
        "queue": {
            "class": "logging.handlers.QueueHandler",
            "queue": logqueue,
        },
    },
    "loggers": {
        "root": {"level": "INFO", "handlers": ["queue"]},
    },
}

app = falcon.asgi.App()
uvicorn.run(app=app, log_config=log_config)
```


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 85c4265fc..b580d89d5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -8,38 +8,8 @@ on:
     types: [opened, synchronize]
 
 jobs:
-
   test:
-    runs-on: ubuntu-22.04
-
-    strategy:
-      matrix:
-        python-version: ["3.10", "3.11", "3.12"]
-
-    steps:
-      - uses: actions/checkout@v4
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
-
-      - name: Install dependencies
-        run: |
-          pip install --upgrade pip wheel
-          pip install .[dev]
-
-      - name: Perform unit tests
-        env:
-          PYTEST_ADDOPTS: "--color=yes"
-        run: |
-          pytest  -vv tests/unit
-      
-      - name: Perform acceptance tests
-        env:
-          PYTEST_ADDOPTS: "--color=yes"
-        run: |
-          pytest  -vv tests/acceptance
+    uses: ./.github/workflows/testing.yml
 
   build-docs:
     runs-on: ubuntu-22.04
@@ -117,7 +87,7 @@ jobs:
   containerbuild:
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12.3"]
 
     runs-on: ubuntu-latest
     steps:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 1d6bc83b1..4c971e7bf 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -6,38 +6,8 @@ on:
     branches: [main]
 
 jobs:
-
   test:
-    runs-on: ubuntu-22.04
-
-    strategy:
-      matrix:
-        python-version: ["3.10", "3.11", "3.12"]
-
-    steps:
-      - uses: actions/checkout@v4
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
-
-      - name: Install dependencies
-        run: |
-          pip install --upgrade pip wheel
-          pip install .[dev]
-
-      - name: Perform unit tests
-        env:
-          PYTEST_ADDOPTS: "--color=yes"
-        run: |
-          pytest  -vv tests/unit
-
-      - name: Perform acceptance tests
-        env:
-          PYTEST_ADDOPTS: "--color=yes"
-        run: |
-          pytest  -vv tests/acceptance
+    uses: ./.github/workflows/testing.yml
 
   code-quality:
     runs-on: ubuntu-22.04
diff --git a/.github/workflows/publish-release-to-pypi.yml b/.github/workflows/publish-release-to-pypi.yml
index 17c7aff02..e9cfe1815 100644
--- a/.github/workflows/publish-release-to-pypi.yml
+++ b/.github/workflows/publish-release-to-pypi.yml
@@ -28,7 +28,7 @@ jobs:
         with:
           name: logprep-build
           path: dist
-    
+
   publish-latest-release-to-pypi:
     runs-on: ubuntu-latest
     name: Publish release to PyPi
@@ -52,7 +52,7 @@ jobs:
   containerbuild:
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12.3"]
 
     runs-on: ubuntu-latest
     needs: publish-latest-release-to-pypi
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c0c848b95..6960591ea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,9 +3,15 @@
 
 ## next release
 
+This release limits the maximum python version to `3.12.3` because of the issue
+[#612](https://github.com/fkie-cad/Logprep/issues/612).
+
 ### Breaking
 ### Features
 ### Improvements
+* a result object was added which is returned by every processor
+  * includes generated extra_data, warnings and errors
+
 ### Bugfix
 
 ## 12.0.0
diff --git a/pyproject.toml b/pyproject.toml
index 102d2279c..7a7714639 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -12,7 +12,7 @@ fallback_version = "unset"
 [project]
 name = "logprep"
 description = "Logprep allows to collect, process and forward log messages from various data sources."
-requires-python = ">=3.10"
+requires-python = ">=3.10,<3.12.4"
 readme = "README.md"
 dynamic = ["version"]
 license = { file = "LICENSE" }


Test patch:
diff --git a/.github/workflows/publish-latest-dev-release-to-pypi.yml b/.github/workflows/publish-latest-dev-release-to-pypi.yml
index f1e354c92..d4fa4ed5d 100644
--- a/.github/workflows/publish-latest-dev-release-to-pypi.yml
+++ b/.github/workflows/publish-latest-dev-release-to-pypi.yml
@@ -39,7 +39,7 @@ jobs:
   containerbuild:
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12.3"]
 
     runs-on: ubuntu-latest
     steps:
diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml
new file mode 100644
index 000000000..50606b9a9
--- /dev/null
+++ b/.github/workflows/testing.yml
@@ -0,0 +1,28 @@
+name: Testing
+
+on:
+  workflow_call:
+
+jobs:
+  python:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12.3"]
+        test-type: ["unit", "acceptance"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+      - name: Install dependencies
+        run: |
+          pip install --upgrade pip wheel
+          pip install .[dev]
+      - name: Perform ${{ matrix.test-type }} test
+        env:
+          PYTEST_ADDOPTS: "--color=yes"
+        run: |
+          pytest -vv tests/${{ matrix.test-type }}