Repository: exasol/integration-test-docker-environment (language: python)

Issue:
Client logging config gets overwritten
## Steps to reproduce

1. Configure logging for client app
2. Use `itde` from a client application
3. After calling `spawn_test_environment` / `spawn_test_environment_with_test_container` log something
4. -> The client log is not printed according to the client app log configuration, but printed to `.build_output/jobs/logs/main.log`

## RCA

1. [luigi_log.conf](https://github.com/exasol/integration-test-docker-environment/blob/main/exasol_integration_test_docker_environment/templates/luigi_log.conf) template overwrites the `root` log channel
2. It would be more correct to configure log channels only used in `itde`: `luigi` and `luigi-interface`, and **not** `root`


Discussion / hints:
Code snippet:
```
        logging.basicConfig(
            format='%(asctime)s %(levelname)-8s %(message)s',
            level=logging.INFO,
            datefmt='%Y-%m-%d %H:%M:%S')

        self.environment_info, self.clean_up = \
            spawn_test_environment(environment_name="saasd_integration_tests",
                                   database_port_forward=self.database_port,
                                   bucketfs_port_forward=bucketfs_port)

         logging.info("This log gets printed to .build_output/jobs/logs/main.log, but not to stdout as expeccted")
```
After first investigation, we know that luigi calls `logging.config.fileConfig(opts.logging_conf_file, disable_existing_loggers=False)` in the `setup_logging.py`. The fileConfig call will keep all existing loggers, except the root logger which needs to be defined in the config file. This means the config of the root-loggers gets forgotten. 
We decided to solve this problem using two strategies. 

1. We provide a config which disables the logging setup through luigi completely
2. We make a copy of the root-logger and restore it after the luigi task is done. We can't simply keep a reference on the root logger, we probably need to save each of its elements and restore these.

Fix patch:
diff --git a/poetry.lock b/poetry.lock
index 17afdfaaf..319341558 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -151,7 +151,7 @@ toml = ["tomli"]
 
 [[package]]
 name = "cryptography"
-version = "40.0.1"
+version = "40.0.2"
 description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
 category = "main"
 optional = false
@@ -349,7 +349,7 @@ python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 [[package]]
 name = "importlib-metadata"
-version = "6.3.0"
+version = "6.4.1"
 description = "Read metadata from Python packages"
 category = "dev"
 optional = false
@@ -816,7 +816,7 @@ python-versions = "*"
 
 [[package]]
 name = "pytest"
-version = "7.3.0"
+version = "7.3.1"
 description = "pytest: simple powerful testing with Python"
 category = "main"
 optional = false
@@ -1009,7 +1009,7 @@ python-versions = "*"
 
 [[package]]
 name = "soupsieve"
-version = "2.4"
+version = "2.4.1"
 description = "A modern CSS selector implementation for Beautiful Soup."
 category = "dev"
 optional = false
@@ -1560,25 +1560,25 @@ coverage = [
     {file = "coverage-7.2.3.tar.gz", hash = "sha256:d298c2815fa4891edd9abe5ad6e6cb4207104c7dd9fd13aea3fdebf6f9b91259"},
 ]
 cryptography = [
-    {file = "cryptography-40.0.1-cp36-abi3-macosx_10_12_universal2.whl", hash = "sha256:918cb89086c7d98b1b86b9fdb70c712e5a9325ba6f7d7cfb509e784e0cfc6917"},
-    {file = "cryptography-40.0.1-cp36-abi3-macosx_10_12_x86_64.whl", hash = "sha256:9618a87212cb5200500e304e43691111570e1f10ec3f35569fdfcd17e28fd797"},
-    {file = "cryptography-40.0.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3a4805a4ca729d65570a1b7cac84eac1e431085d40387b7d3bbaa47e39890b88"},
-    {file = "cryptography-40.0.1-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:63dac2d25c47f12a7b8aa60e528bfb3c51c5a6c5a9f7c86987909c6c79765554"},
-    {file = "cryptography-40.0.1-cp36-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0a4e3406cfed6b1f6d6e87ed243363652b2586b2d917b0609ca4f97072994405"},
-    {file = "cryptography-40.0.1-cp36-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:1e0af458515d5e4028aad75f3bb3fe7a31e46ad920648cd59b64d3da842e4356"},
-    {file = "cryptography-40.0.1-cp36-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:d8aa3609d337ad85e4eb9bb0f8bcf6e4409bfb86e706efa9a027912169e89122"},
-    {file = "cryptography-40.0.1-cp36-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:cf91e428c51ef692b82ce786583e214f58392399cf65c341bc7301d096fa3ba2"},
-    {file = "cryptography-40.0.1-cp36-abi3-win32.whl", hash = "sha256:650883cc064297ef3676b1db1b7b1df6081794c4ada96fa457253c4cc40f97db"},
-    {file = "cryptography-40.0.1-cp36-abi3-win_amd64.whl", hash = "sha256:a805a7bce4a77d51696410005b3e85ae2839bad9aa38894afc0aa99d8e0c3160"},
-    {file = "cryptography-40.0.1-pp38-pypy38_pp73-macosx_10_12_x86_64.whl", hash = "sha256:cd033d74067d8928ef00a6b1327c8ea0452523967ca4463666eeba65ca350d4c"},
-    {file = "cryptography-40.0.1-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:d36bbeb99704aabefdca5aee4eba04455d7a27ceabd16f3b3ba9bdcc31da86c4"},
-    {file = "cryptography-40.0.1-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:32057d3d0ab7d4453778367ca43e99ddb711770477c4f072a51b3ca69602780a"},
-    {file = "cryptography-40.0.1-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:f5d7b79fa56bc29580faafc2ff736ce05ba31feaa9d4735048b0de7d9ceb2b94"},
-    {file = "cryptography-40.0.1-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:7c872413353c70e0263a9368c4993710070e70ab3e5318d85510cc91cce77e7c"},
-    {file = "cryptography-40.0.1-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:28d63d75bf7ae4045b10de5413fb1d6338616e79015999ad9cf6fc538f772d41"},
-    {file = "cryptography-40.0.1-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:6f2bbd72f717ce33100e6467572abaedc61f1acb87b8d546001328d7f466b778"},
-    {file = "cryptography-40.0.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:cc3a621076d824d75ab1e1e530e66e7e8564e357dd723f2533225d40fe35c60c"},
-    {file = "cryptography-40.0.1.tar.gz", hash = "sha256:2803f2f8b1e95f614419926c7e6f55d828afc614ca5ed61543877ae668cc3472"},
+    {file = "cryptography-40.0.2-cp36-abi3-macosx_10_12_universal2.whl", hash = "sha256:8f79b5ff5ad9d3218afb1e7e20ea74da5f76943ee5edb7f76e56ec5161ec782b"},
+    {file = "cryptography-40.0.2-cp36-abi3-macosx_10_12_x86_64.whl", hash = "sha256:05dc219433b14046c476f6f09d7636b92a1c3e5808b9a6536adf4932b3b2c440"},
+    {file = "cryptography-40.0.2-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4df2af28d7bedc84fe45bd49bc35d710aede676e2a4cb7fc6d103a2adc8afe4d"},
+    {file = "cryptography-40.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0dcca15d3a19a66e63662dc8d30f8036b07be851a8680eda92d079868f106288"},
+    {file = "cryptography-40.0.2-cp36-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a04386fb7bc85fab9cd51b6308633a3c271e3d0d3eae917eebab2fac6219b6d2"},
+    {file = "cryptography-40.0.2-cp36-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:adc0d980fd2760c9e5de537c28935cc32b9353baaf28e0814df417619c6c8c3b"},
+    {file = "cryptography-40.0.2-cp36-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:d5a1bd0e9e2031465761dfa920c16b0065ad77321d8a8c1f5ee331021fda65e9"},
+    {file = "cryptography-40.0.2-cp36-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:a95f4802d49faa6a674242e25bfeea6fc2acd915b5e5e29ac90a32b1139cae1c"},
+    {file = "cryptography-40.0.2-cp36-abi3-win32.whl", hash = "sha256:aecbb1592b0188e030cb01f82d12556cf72e218280f621deed7d806afd2113f9"},
+    {file = "cryptography-40.0.2-cp36-abi3-win_amd64.whl", hash = "sha256:b12794f01d4cacfbd3177b9042198f3af1c856eedd0a98f10f141385c809a14b"},
+    {file = "cryptography-40.0.2-pp38-pypy38_pp73-macosx_10_12_x86_64.whl", hash = "sha256:142bae539ef28a1c76794cca7f49729e7c54423f615cfd9b0b1fa90ebe53244b"},
+    {file = "cryptography-40.0.2-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:956ba8701b4ffe91ba59665ed170a2ebbdc6fc0e40de5f6059195d9f2b33ca0e"},
+    {file = "cryptography-40.0.2-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:4f01c9863da784558165f5d4d916093737a75203a5c5286fde60e503e4276c7a"},
+    {file = "cryptography-40.0.2-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:3daf9b114213f8ba460b829a02896789751626a2a4e7a43a28ee77c04b5e4958"},
+    {file = "cryptography-40.0.2-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:48f388d0d153350f378c7f7b41497a54ff1513c816bcbbcafe5b829e59b9ce5b"},
+    {file = "cryptography-40.0.2-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:c0764e72b36a3dc065c155e5b22f93df465da9c39af65516fe04ed3c68c92636"},
+    {file = "cryptography-40.0.2-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:cbaba590180cba88cb99a5f76f90808a624f18b169b90a4abb40c1fd8c19420e"},
+    {file = "cryptography-40.0.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:7a38250f433cd41df7fcb763caa3ee9362777fdb4dc642b9a349721d2bf47404"},
+    {file = "cryptography-40.0.2.tar.gz", hash = "sha256:c33c0d32b8594fa647d2e01dbccc303478e16fdd7cf98652d5b3ed11aa5e5c99"},
 ]
 distlib = [
     {file = "distlib-0.3.6-py2.py3-none-any.whl", hash = "sha256:f35c4b692542ca110de7ef0bea44d73981caeb34ca0b9b6b2e6d7790dda8f80e"},
@@ -1638,8 +1638,8 @@ imagesize = [
     {file = "imagesize-1.4.1.tar.gz", hash = "sha256:69150444affb9cb0d5cc5a92b3676f0b2fb7cd9ae39e947a5e11a36b4497cd4a"},
 ]
 importlib-metadata = [
-    {file = "importlib_metadata-6.3.0-py3-none-any.whl", hash = "sha256:8f8bd2af397cf33bd344d35cfe7f489219b7d14fc79a3f854b75b8417e9226b0"},
-    {file = "importlib_metadata-6.3.0.tar.gz", hash = "sha256:23c2bcae4762dfb0bbe072d358faec24957901d75b6c4ab11172c0c982532402"},
+    {file = "importlib_metadata-6.4.1-py3-none-any.whl", hash = "sha256:63ace321e24167d12fbb176b6015f4dbe06868c54a2af4f15849586afb9027fd"},
+    {file = "importlib_metadata-6.4.1.tar.gz", hash = "sha256:eb1a7933041f0f85c94cd130258df3fb0dec060ad8c1c9318892ef4192c47ce1"},
 ]
 importlib-resources = [
     {file = "importlib_resources-5.12.0-py3-none-any.whl", hash = "sha256:7b1deeebbf351c7578e09bf2f63fa2ce8b5ffec296e0d349139d43cca061a81a"},
@@ -1864,8 +1864,8 @@ pyreadline3 = [
     {file = "pyreadline3-3.4.1.tar.gz", hash = "sha256:6f3d1f7b8a31ba32b73917cefc1f28cc660562f39aea8646d30bd6eff21f7bae"},
 ]
 pytest = [
-    {file = "pytest-7.3.0-py3-none-any.whl", hash = "sha256:933051fa1bfbd38a21e73c3960cebdad4cf59483ddba7696c48509727e17f201"},
-    {file = "pytest-7.3.0.tar.gz", hash = "sha256:58ecc27ebf0ea643ebfdf7fb1249335da761a00c9f955bcd922349bcb68ee57d"},
+    {file = "pytest-7.3.1-py3-none-any.whl", hash = "sha256:3799fa815351fea3a5e96ac7e503a96fa51cc9942c3753cda7651b93c1cfa362"},
+    {file = "pytest-7.3.1.tar.gz", hash = "sha256:434afafd78b1d78ed0addf160ad2b77a30d35d4bdf8af234fe621919d9ed15e3"},
 ]
 pytest-cov = [
     {file = "pytest-cov-4.0.0.tar.gz", hash = "sha256:996b79efde6433cdbd0088872dbc5fb3ed7fe1578b68cdbba634f14bb8dd0470"},
@@ -1895,13 +1895,6 @@ PyYAML = [
     {file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"},
     {file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"},
     {file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"},
-    {file = "PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358"},
-    {file = "PyYAML-6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1"},
-    {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d"},
-    {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f"},
-    {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782"},
-    {file = "PyYAML-6.0-cp311-cp311-win32.whl", hash = "sha256:bfaef573a63ba8923503d27530362590ff4f576c626d86a9fed95822a8255fd7"},
-    {file = "PyYAML-6.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf"},
     {file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"},
     {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"},
     {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"},
@@ -2064,8 +2057,8 @@ snowballstemmer = [
     {file = "snowballstemmer-2.2.0.tar.gz", hash = "sha256:09b16deb8547d3412ad7b590689584cd0fe25ec8db3be37788be3810cbf19cb1"},
 ]
 soupsieve = [
-    {file = "soupsieve-2.4-py3-none-any.whl", hash = "sha256:49e5368c2cda80ee7e84da9dbe3e110b70a4575f196efb74e51b94549d921955"},
-    {file = "soupsieve-2.4.tar.gz", hash = "sha256:e28dba9ca6c7c00173e34e4ba57448f0688bb681b7c5e8bf4971daafc093d69a"},
+    {file = "soupsieve-2.4.1-py3-none-any.whl", hash = "sha256:1c1bfee6819544a3447586c889157365a27e10d88cde3ad3da0cf0ddf646feb8"},
+    {file = "soupsieve-2.4.1.tar.gz", hash = "sha256:89d12b2d5dfcd2c9e8c22326da9d9aa9cb3dfab0a83a024f05704076ee8d35ea"},
 ]
 Sphinx = [
     {file = "Sphinx-4.5.0-py3-none-any.whl", hash = "sha256:ebf612653238bcc8f4359627a9b7ce44ede6fdd75d9d30f68255c7383d3a6226"},


Test patch:
diff --git a/exasol_integration_test_docker_environment/cli/commands/spawn_test_environment.py b/exasol_integration_test_docker_environment/cli/commands/spawn_test_environment.py
index e9b5d7514..4d3d02e98 100644
--- a/exasol_integration_test_docker_environment/cli/commands/spawn_test_environment.py
+++ b/exasol_integration_test_docker_environment/cli/commands/spawn_test_environment.py
@@ -9,7 +9,7 @@
 from exasol_integration_test_docker_environment.cli.options.system_options import (
     output_directory_option,
     system_options,
-    tempory_base_directory_option,
+    tempory_base_directory_option, luigi_logging_options,
 )
 from exasol_integration_test_docker_environment.cli.options.test_environment_options import (
     docker_db_options,
@@ -78,30 +78,33 @@
 @add_options([output_directory_option])
 @add_options([tempory_base_directory_option])
 @add_options(system_options)
+@add_options(luigi_logging_options)
 def spawn_test_environment(
-    environment_name: str,
-    database_port_forward: Optional[int],
-    bucketfs_port_forward: Optional[int],
-    db_mem_size: str,
-    db_disk_size: str,
-    nameserver: Tuple[str, ...],
-    docker_runtime: Optional[str],
-    docker_db_image_version: str,
-    docker_db_image_name: str,
-    create_certificates: bool,
-    additional_db_parameter: Tuple[str, ...],
-    source_docker_repository_name: str,
-    source_docker_tag_prefix: str,
-    source_docker_username: Optional[str],
-    source_docker_password: Optional[str],
-    target_docker_repository_name: str,
-    target_docker_tag_prefix: str,
-    target_docker_username: Optional[str],
-    target_docker_password: Optional[str],
-    output_directory: str,
-    temporary_base_directory: str,
-    workers: int,
-    task_dependencies_dot_file: Optional[str],
+        environment_name: str,
+        database_port_forward: Optional[int],
+        bucketfs_port_forward: Optional[int],
+        db_mem_size: str,
+        db_disk_size: str,
+        nameserver: Tuple[str, ...],
+        docker_runtime: Optional[str],
+        docker_db_image_version: str,
+        docker_db_image_name: str,
+        create_certificates: bool,
+        additional_db_parameter: Tuple[str, ...],
+        source_docker_repository_name: str,
+        source_docker_tag_prefix: str,
+        source_docker_username: Optional[str],
+        source_docker_password: Optional[str],
+        target_docker_repository_name: str,
+        target_docker_tag_prefix: str,
+        target_docker_username: Optional[str],
+        target_docker_password: Optional[str],
+        output_directory: str,
+        temporary_base_directory: str,
+        workers: int,
+        task_dependencies_dot_file: Optional[str],
+        log_level: Optional[str],
+        use_job_specific_log_file: bool
 ):
     """
     This command spawn a test environment with a docker-db container and a connected test-container.
@@ -133,6 +136,8 @@ def spawn_test_environment(
                 temporary_base_directory,
                 workers,
                 task_dependencies_dot_file,
+                log_level=log_level,
+                use_job_specific_log_file=use_job_specific_log_file,
             )
         except ArgumentConstraintError as e:
             handle_wrong_argument_error(*e.args)
diff --git a/exasol_integration_test_docker_environment/cli/options/system_options.py b/exasol_integration_test_docker_environment/cli/options/system_options.py
index 3ddb7b01e..36f51ec69 100644
--- a/exasol_integration_test_docker_environment/cli/options/system_options.py
+++ b/exasol_integration_test_docker_environment/cli/options/system_options.py
@@ -9,7 +9,7 @@
 
 tempory_base_directory_option = click.option('--temporary-base-directory',
                                              type=click.Path(file_okay=False,
-                                                      dir_okay=True),
+                                                             dir_okay=True),
                                              default="/tmp",
                                              show_default=True,
                                              help="Directory where the system creates temporary directories.")
@@ -19,5 +19,13 @@
                  default=5, show_default=True,
                  help="Number of parallel workers"),
     click.option('--task-dependencies-dot-file', type=click.Path(file_okay=True),
-                 default=None, help="Path where to store the Task Dependency Graph as dot file")
+                 default=None, help="Path where to store the Task Dependency Graph as dot file"),
+]
+
+luigi_logging_options = [
+    click.option('--log-level', type=click.Choice(['DEBUG', 'INFO', 'WARNING', 'ERROR', 'FATAL']),
+                 default=None, help="Log level used for console logging"),
+    click.option('--use-job-specific-log-file', type=bool,
+                 default=True, help="Use a job specific log file which write the debug log "
+                                    "to the job directory in the build directory")
 ]
diff --git a/exasol_integration_test_docker_environment/lib/api/build_test_container.py b/exasol_integration_test_docker_environment/lib/api/build_test_container.py
index 1a3d721fd..854b7ac66 100644
--- a/exasol_integration_test_docker_environment/lib/api/build_test_container.py
+++ b/exasol_integration_test_docker_environment/lib/api/build_test_container.py
@@ -2,7 +2,8 @@
 
 from exasol_integration_test_docker_environment.lib.api.common import set_build_config, \
     set_docker_repository_config, run_task, generate_root_task, no_cli_function
-from exasol_integration_test_docker_environment.cli.options.docker_repository_options import DEFAULT_DOCKER_REPOSITORY_NAME
+from exasol_integration_test_docker_environment.cli.options.docker_repository_options import \
+    DEFAULT_DOCKER_REPOSITORY_NAME
 from exasol_integration_test_docker_environment.cli.options.system_options import DEFAULT_OUTPUT_DIRECTORY
 from exasol_integration_test_docker_environment.lib.data.test_container_content_description import \
     TestContainerContentDescription
@@ -31,7 +32,10 @@ def build_test_container(
         target_docker_username: Optional[str] = None,
         target_docker_password: Optional[str] = None,
         workers: int = 5,
-        task_dependencies_dot_file: Optional[str] = None) -> ImageInfo:
+        task_dependencies_dot_file: Optional[str] = None,
+        log_level: Optional[str] = None,
+        use_job_specific_log_file: bool = False
+) -> ImageInfo:
     """
     This function builds all stages of the test container for the test environment.
     If stages are cached in a docker registry, the function is going to pull them,
@@ -55,5 +59,7 @@ def build_test_container(
                                  target_docker_tag_prefix, "target")
     task_creator = lambda: generate_root_task(task_class=DockerTestContainerBuild,
                                               test_container_content=test_container_content)
-    image_infos = run_task(task_creator, workers, task_dependencies_dot_file)
+    image_infos = run_task(task_creator, workers, task_dependencies_dot_file,
+                           log_level=log_level,
+                           use_job_specific_log_file=use_job_specific_log_file)
     return image_infos["test-container"]
diff --git a/exasol_integration_test_docker_environment/lib/api/common.py b/exasol_integration_test_docker_environment/lib/api/common.py
index d41e9386c..14ec50830 100644
--- a/exasol_integration_test_docker_environment/lib/api/common.py
+++ b/exasol_integration_test_docker_environment/lib/api/common.py
@@ -1,18 +1,22 @@
+import contextlib
 import getpass
 import json
 import logging
 import os
+import warnings
 from datetime import datetime
 from pathlib import Path
 from typing import (
     Callable,
     Tuple,
     Set,
-    Optional, Any
+    Optional, Any, Dict, Iterator
 )
 
 import luigi
 import networkx
+from luigi.parameter import UnconsumedParameterWarning
+from luigi.setup_logging import InterfaceLogging
 from networkx import DiGraph
 
 from exasol_integration_test_docker_environment.lib import extract_modulename_for_build_steps
@@ -22,6 +26,26 @@
 from exasol_integration_test_docker_environment.lib.base.task_dependency import TaskDependency, DependencyState
 
 
+class JobCounterSingleton(object):
+    """
+    We use here a Singleton to avoid a unprotected global variable.
+    However, this counter needs to be global counter to guarantee unique job ids.
+    This is needed in case of task that finish in less than a second, to avoid duplicated job ids
+    """
+
+    _instance = None
+
+    def __new__(cls):
+        if cls._instance is None:
+            cls._instance = super(JobCounterSingleton, cls).__new__(cls)
+            cls._instance._counter = 0
+        return cls._instance
+
+    def get_next_value(self) -> int:
+        self._counter += 1
+        return self._counter
+
+
 def set_build_config(force_rebuild: bool,
                      force_rebuild_from: Tuple[str, ...],
                      force_pull: bool,
@@ -86,42 +110,109 @@ def import_build_steps(flavor_path: Tuple[str, ...]):
 
 
 def generate_root_task(task_class, *args, **kwargs) -> DependencyLoggerBaseTask:
+    job_counter = JobCounterSingleton().get_next_value()
     strftime = datetime.now().strftime('%Y_%m_%d_%H_%M_%S')
-    params = {"job_id": f"{strftime}_{task_class.__name__}"}
+    params = {"job_id": f"{strftime}_{job_counter}_{task_class.__name__}"}
     params.update(kwargs)
     return task_class(**params)
 
 
 def run_task(task_creator: Callable[[], DependencyLoggerBaseTask], workers: int,
-             task_dependencies_dot_file: Optional[str]) \
+             task_dependencies_dot_file: Optional[str],
+             log_level: str = None, use_job_specific_log_file: bool = False) \
         -> Any:
     setup_worker()
     task = task_creator()
     success = False
+    log_file_path = get_log_path(task.job_id)
     try:
-        with get_luigi_log_config(get_log_path()) as luigi_config:
-            no_scheduling_errors = luigi.build([task], workers=workers,
-                                               local_scheduler=True,
-                                               logging_conf_file=f'{luigi_config}')
+        no_scheduling_errors = \
+            _run_task_with_logging_config(
+                task, log_file_path, log_level, use_job_specific_log_file, workers)
         success = not task.failed_target.exists() and no_scheduling_errors
-        generate_graph_from_task_dependencies(task, task_dependencies_dot_file)
-
-        if success:
-            return task.get_result()
-        elif task.failed_target.exists():
-            logging.error(f"Task {task} failed. failed target exists.")
-            raise TaskRuntimeError(msg=f"Task {task} (or any of it's subtasks) failed.",
-                                   inner=list(task.collect_failures().keys()))
-        elif not no_scheduling_errors:
-            logging.error(f"Task {task} failed. : luigi reported a scheduling error.")
-            raise TaskRuntimeError(msg=f"Task {task} failed. reason: luigi reported a scheduling error.")
+        return _handle_task_result(no_scheduling_errors, success, task,
+                                   task_dependencies_dot_file)
     except BaseException as e:
         logging.error("Going to abort the task %s" % task)
         raise e
     finally:
+        if use_job_specific_log_file:
+            logging.info(
+                f"The detailed log of the integration-test-docker-environment can be found at: {log_file_path}")
         task.cleanup(success)
 
 
+def _run_task_with_logging_config(
+        task: DependencyLoggerBaseTask,
+        log_file_path: Path,
+        log_level: Optional[str],
+        use_job_specific_log_file: bool,
+        workers: int) -> bool:
+    with _configure_logging(log_file_path, log_level, use_job_specific_log_file) as run_kwargs:
+        no_scheduling_errors = luigi.build([task], workers=workers,
+                                           local_scheduler=True, **run_kwargs)
+        return no_scheduling_errors
+
+
+def _handle_task_result(no_scheduling_errors, success, task, task_dependencies_dot_file: Optional[str]) -> Any:
+    generate_graph_from_task_dependencies(task, task_dependencies_dot_file)
+    if success:
+        return task.get_result()
+    elif task.failed_target.exists():
+        logging.error(f"Task {task} failed. failed target exists.")
+        raise TaskRuntimeError(msg=f"Task {task} (or any of it's subtasks) failed.",
+                               inner=list(task.collect_failures().keys()))
+    elif not no_scheduling_errors:
+        logging.error(f"Task {task} failed. : luigi reported a scheduling error.")
+        raise TaskRuntimeError(msg=f"Task {task} failed. reason: luigi reported a scheduling error.")
+
+
+@contextlib.contextmanager
+def _configure_logging(
+        log_file_path: Path,
+        log_level: Optional[str],
+        use_job_specific_log_file: bool) -> Iterator[Dict[str, str]]:
+    with get_luigi_log_config(log_file_target=log_file_path,
+                              console_log_level=log_level,
+                              use_job_specific_log_file=use_job_specific_log_file) as luigi_config:
+        no_configure_logging, run_kwargs = _configure_logging_parameter(
+            log_level=log_level,
+            luigi_config=luigi_config,
+            use_job_specific_log_file=use_job_specific_log_file)
+        # We need to set InterfaceLogging._configured to false,
+        # because otherwise luigi doesn't accept the new config.
+        InterfaceLogging._configured = False
+        luigi.configuration.get_config().set('core', 'no_configure_logging', str(no_configure_logging))
+        with warnings.catch_warnings():
+            # This filter is necessary, because luigi uses the config no_configure_logging,
+            # but doesn't define it, which seems to be a bug in luigi
+            warnings.filterwarnings(action="ignore",
+                                    category=UnconsumedParameterWarning,
+                                    message=".*no_configure_logging.*")
+            yield run_kwargs
+
+
+def _configure_logging_parameter(log_level: str, luigi_config: Path, use_job_specific_log_file: bool) \
+        -> Tuple[bool, Dict[str, str]]:
+    if log_level is None and not use_job_specific_log_file:
+        run_kwargs = {}
+        no_configure_logging = True
+    else:
+        no_configure_logging = False
+        run_kwargs = _get_run_kwargs_log_configuration(
+            log_level, luigi_config, use_job_specific_log_file)
+    return no_configure_logging, run_kwargs
+
+
+def _get_run_kwargs_log_configuration(log_level: str, luigi_config: Path, use_job_specific_log_file: bool) \
+        -> Dict[str, str]:
+    if use_job_specific_log_file:
+        run_kwargs = {"logging_conf_file": f'{luigi_config}'}
+    else:
+        run_kwargs = {"log_level": log_level}
+    return run_kwargs
+
+
 def generate_graph_from_task_dependencies(task: DependencyLoggerBaseTask, task_dependencies_dot_file: Optional[str]):
     if task_dependencies_dot_file is not None:
         print(f"Generate Task Dependency Graph to {task_dependencies_dot_file}")
diff --git a/exasol_integration_test_docker_environment/lib/api/push_test_container.py b/exasol_integration_test_docker_environment/lib/api/push_test_container.py
index f2fc7d770..3eb5e5231 100644
--- a/exasol_integration_test_docker_environment/lib/api/push_test_container.py
+++ b/exasol_integration_test_docker_environment/lib/api/push_test_container.py
@@ -2,7 +2,8 @@
 
 from exasol_integration_test_docker_environment.lib.api.common import set_docker_repository_config, \
     run_task, set_build_config, generate_root_task, no_cli_function
-from exasol_integration_test_docker_environment.cli.options.docker_repository_options import DEFAULT_DOCKER_REPOSITORY_NAME
+from exasol_integration_test_docker_environment.cli.options.docker_repository_options import \
+    DEFAULT_DOCKER_REPOSITORY_NAME
 from exasol_integration_test_docker_environment.cli.options.system_options import DEFAULT_OUTPUT_DIRECTORY
 from exasol_integration_test_docker_environment.lib.data.test_container_content_description import \
     TestContainerContentDescription
@@ -33,7 +34,10 @@ def push_test_container(
         target_docker_username: Optional[str] = None,
         target_docker_password: Optional[str] = None,
         workers: int = 5,
-        task_dependencies_dot_file: Optional[str] = None) -> ImageInfo:
+        task_dependencies_dot_file: Optional[str] = None,
+        log_level: Optional[str] = None,
+        use_job_specific_log_file: bool = False
+) -> ImageInfo:
     """
     This function pushes all stages of the test container for the test environment.
     If the stages do not exist locally, the system will build or pull them before the push.
@@ -57,5 +61,7 @@ def push_test_container(
                                               test_container_content=test_container_content,
                                               force_push=force_push,
                                               push_all=push_all)
-    image_infos = run_task(task_creator, workers, task_dependencies_dot_file)
+    image_infos = run_task(task_creator, workers, task_dependencies_dot_file,
+                           log_level=log_level,
+                           use_job_specific_log_file=use_job_specific_log_file)
     return image_infos[0]
diff --git a/exasol_integration_test_docker_environment/lib/api/spawn_test_environment.py b/exasol_integration_test_docker_environment/lib/api/spawn_test_environment.py
index db24bf330..a0b5182c8 100644
--- a/exasol_integration_test_docker_environment/lib/api/spawn_test_environment.py
+++ b/exasol_integration_test_docker_environment/lib/api/spawn_test_environment.py
@@ -4,7 +4,8 @@
 
 from exasol_integration_test_docker_environment.lib.api.common import set_build_config, set_docker_repository_config, \
     run_task, generate_root_task, cli_function
-from exasol_integration_test_docker_environment.cli.options.docker_repository_options import DEFAULT_DOCKER_REPOSITORY_NAME
+from exasol_integration_test_docker_environment.cli.options.docker_repository_options import \
+    DEFAULT_DOCKER_REPOSITORY_NAME
 from exasol_integration_test_docker_environment.cli.options.system_options import DEFAULT_OUTPUT_DIRECTORY
 from exasol_integration_test_docker_environment.cli.options.test_environment_options import LATEST_DB_VERSION
 from exasol_integration_test_docker_environment.lib.api.api_errors import ArgumentConstraintError
@@ -46,7 +47,10 @@ def spawn_test_environment(
         output_directory: str = DEFAULT_OUTPUT_DIRECTORY,
         temporary_base_directory: str = "/tmp",
         workers: int = 5,
-        task_dependencies_dot_file: Optional[str] = None) -> Tuple[EnvironmentInfo, Callable[[], None]]:
+        task_dependencies_dot_file: Optional[str] = None,
+        log_level: Optional[str] = None,
+        use_job_specific_log_file: bool = False
+) -> Tuple[EnvironmentInfo, Callable[[], None]]:
     """
     This function spawns a test environment with a docker-db container and a connected test-container.
     The test-container is reachable by the database for output redirects of UDFs.
@@ -96,5 +100,7 @@ def spawn_test_environment(
                                               test_container_content=None,
                                               additional_db_parameter=additional_db_parameter
                                               )
-    environment_info = run_task(task_creator, workers, task_dependencies_dot_file)
+    environment_info = run_task(task_creator, workers, task_dependencies_dot_file,
+                                log_level=log_level,
+                                use_job_specific_log_file=use_job_specific_log_file)
     return environment_info, functools.partial(_cleanup, environment_info)
diff --git a/exasol_integration_test_docker_environment/lib/api/spawn_test_environment_with_test_container.py b/exasol_integration_test_docker_environment/lib/api/spawn_test_environment_with_test_container.py
index 3d28b24cd..ec9918cc0 100644
--- a/exasol_integration_test_docker_environment/lib/api/spawn_test_environment_with_test_container.py
+++ b/exasol_integration_test_docker_environment/lib/api/spawn_test_environment_with_test_container.py
@@ -4,7 +4,8 @@
 
 from exasol_integration_test_docker_environment.lib.api.common import set_build_config, set_docker_repository_config, \
     run_task, generate_root_task, no_cli_function
-from exasol_integration_test_docker_environment.cli.options.docker_repository_options import DEFAULT_DOCKER_REPOSITORY_NAME
+from exasol_integration_test_docker_environment.cli.options.docker_repository_options import \
+    DEFAULT_DOCKER_REPOSITORY_NAME
 from exasol_integration_test_docker_environment.cli.options.system_options import DEFAULT_OUTPUT_DIRECTORY
 from exasol_integration_test_docker_environment.cli.options.test_environment_options import LATEST_DB_VERSION
 from exasol_integration_test_docker_environment.lib.api.api_errors import ArgumentConstraintError
@@ -36,7 +37,7 @@ def spawn_test_environment_with_test_container(
         bucketfs_port_forward: Optional[int] = None,
         db_mem_size: str = "2 GiB",
         db_disk_size: str = "2 GiB",
-        nameserver: Tuple[str,...] = tuple(),
+        nameserver: Tuple[str, ...] = tuple(),
         docker_runtime: Optional[str] = None,
         docker_db_image_version: str = LATEST_DB_VERSION,
         docker_db_image_name: str = "exasol/docker-db",
@@ -53,7 +54,10 @@ def spawn_test_environment_with_test_container(
         output_directory: str = DEFAULT_OUTPUT_DIRECTORY,
         temporary_base_directory: str = "/tmp",
         workers: int = 5,
-        task_dependencies_dot_file: Optional[str] = None) \
+        task_dependencies_dot_file: Optional[str] = None,
+        log_level: Optional[str] = None,
+        use_job_specific_log_file: bool = False
+) \
         -> Tuple[EnvironmentInfo, Callable[[], None]]:
     """
     This function spawns a test environment with a docker-db container and a connected test-container.
@@ -104,5 +108,7 @@ def spawn_test_environment_with_test_container(
                                               test_container_content=test_container_content,
                                               additional_db_parameter=additional_db_parameter
                                               )
-    environment_info = run_task(task_creator, workers, task_dependencies_dot_file)
+    environment_info = run_task(task_creator, workers, task_dependencies_dot_file,
+                                log_level=log_level,
+                                use_job_specific_log_file=use_job_specific_log_file)
     return environment_info, functools.partial(_cleanup, environment_info)
diff --git a/exasol_integration_test_docker_environment/lib/base/luigi_log_config.py b/exasol_integration_test_docker_environment/lib/base/luigi_log_config.py
index f3d046f40..cf7f8a23f 100644
--- a/exasol_integration_test_docker_environment/lib/base/luigi_log_config.py
+++ b/exasol_integration_test_docker_environment/lib/base/luigi_log_config.py
@@ -9,20 +9,23 @@
 from exasol_integration_test_docker_environment.lib import PACKAGE_NAME
 from exasol_integration_test_docker_environment.lib.config.build_config import build_config
 
-global_log_file: Optional[Path] = None
-"""
-Global variable used to save log path. Should only be used within method validate_log_file(). 
-"""
-
 LOG_ENV_VARIABLE_NAME = "EXA_BUILD_LOG"
 
+PROPAGATE = "PROPAGATE"
+
+FILTERS = "FILTERS"
+
+HANDLERS = "HANDLERS"
 
-def get_log_path() -> Path:
+LOG_LEVEL = "LOG_LEVEL"
+
+
+def get_log_path(job_id: str) -> Path:
     """
     Retrieve the log-file path. Default path is $output_path/jobs/logs/main.log, but can be overwritten by
     the environment variable LOG_ENV_VARIABLE_NAME.
     """
-    main_log_path = Path(build_config().output_directory) / "jobs" / "logs"
+    main_log_path = Path(build_config().output_directory) / "jobs" / job_id / "logs"
     main_log_path.mkdir(parents=True, exist_ok=True)
     def_log_path = main_log_path / "main.log"
     env_log_path = os.getenv(LOG_ENV_VARIABLE_NAME)
@@ -33,32 +36,44 @@ def get_log_path() -> Path:
     return log_path
 
 
-def validate_log_file(log_file: Path):
-    global global_log_file
-    # If build config is set multiple times, we must ensure the path of the log-file does not change
-    # The logging always will print to logfile in the first configured output directory (restriction by Luigi).
-    if global_log_file is not None and global_log_file.absolute() != log_file.absolute():
-        raise ValueError("Log file location for Luigi has been changed. "
-                         "This is not allowed to change between consecutive task invocations.")
-    global_log_file = log_file
+@contextlib.contextmanager
+def restore_root_logger(use_job_specific_log_file: bool):
+    root_logger_info = None
+    if use_job_specific_log_file:
+        root_logger_info = {
+            LOG_LEVEL: logging.root.level,
+            HANDLERS: list(logging.root.handlers),
+            FILTERS: list(logging.root.filters),
+            PROPAGATE: logging.root.propagate
+        }
+    yield
+    if use_job_specific_log_file and root_logger_info is not None:
+        logging.root.level = root_logger_info[LOG_LEVEL]
+        logging.root.handlers = root_logger_info[HANDLERS]
+        logging.root.filters = root_logger_info[FILTERS]
+        logging.root.propagate = root_logger_info[PROPAGATE]
 
 
 @contextlib.contextmanager
-def get_luigi_log_config(log_file_target: Path, console_log_level: int = logging.WARNING) -> Path:
+def get_luigi_log_config(log_file_target: Path,
+                         use_job_specific_log_file: bool,
+                         console_log_level: Optional[str] = None) -> Path:
     """
     Yields a context manager containing the path of the log-config file.
     log_file_target contains the location of the log-file.
     console_log_level indicates the log_level (@see https://docs.python.org/3/library/logging.html#logging-levels)
     The log-level for the log-file is always logging.DEBUG!
     """
-    validate_log_file(log_file_target)
+    if console_log_level is None:
+        console_log_level = logging.getLevelName(logging.WARNING)
     env = jinja2.Environment(loader=jinja2.PackageLoader(PACKAGE_NAME),
                              autoescape=jinja2.select_autoescape())
     template = env.get_template("luigi_log.conf")
-    rendered_template = template.render(console_log_level=logging.getLevelName(console_log_level),
+    rendered_template = template.render(console_log_level=console_log_level,
                                         log_file_target=str(log_file_target))
     with tempfile.TemporaryDirectory() as temp_dir:
         temp_luigi_conf_path = Path(temp_dir) / "luigi_log.conf"
         with open(temp_luigi_conf_path, "w") as f:
             f.write(rendered_template)
-        yield temp_luigi_conf_path
+        with restore_root_logger(use_job_specific_log_file=use_job_specific_log_file):
+            yield temp_luigi_conf_path
diff --git a/exasol_integration_test_docker_environment/templates/luigi_log.conf b/exasol_integration_test_docker_environment/templates/luigi_log.conf
index fb3307ba0..35f6e9d13 100644
--- a/exasol_integration_test_docker_environment/templates/luigi_log.conf
+++ b/exasol_integration_test_docker_environment/templates/luigi_log.conf
@@ -1,5 +1,5 @@
 [loggers]
-keys=root,luigi
+keys=root,luigi_interface,luigi
 
 [handlers]
 keys=consoleHandler,fileHandler
@@ -8,13 +8,18 @@ keys=consoleHandler,fileHandler
 keys=fileFormatter,consoleFormatter
 
 [logger_root]
+handlers=consoleHandler
+
+[logger_luigi_interface]
 level=DEBUG
 handlers=consoleHandler,fileHandler
+qualname=luigi-interface
+propagate=0
 
 [logger_luigi]
 level=DEBUG
 handlers=consoleHandler,fileHandler
-qualname=luigi-interface
+qualname=luigi
 propagate=0
 
 [handler_consoleHandler]
diff --git a/exasol_integration_test_docker_environment/test/test_api_logging.py b/exasol_integration_test_docker_environment/test/test_api_logging.py
new file mode 100644
index 000000000..fd26b230f
--- /dev/null
+++ b/exasol_integration_test_docker_environment/test/test_api_logging.py
@@ -0,0 +1,228 @@
+import contextlib
+import glob
+import logging
+import unittest
+import warnings
+from contextlib import redirect_stderr
+from io import StringIO
+from pathlib import Path
+from sys import stderr
+from tempfile import TemporaryDirectory
+from typing import Any, Dict, Optional
+
+LOGGER_STR = "logger_str"
+
+LEVEL_NAME = "level_name"
+
+PARENT = "parent"
+
+PROPAGATE = "propagate"
+
+DISABLED = "disabled"
+
+FILTERS = "filters"
+
+LOGGER_NAME = "name"
+
+HANDLERS = "handlers"
+
+LEVEL = "level"
+
+OBJECT = "object"
+
+LUIGI_INTERFACE_LOGGER = "luigi-interface"
+
+LUIGI_LOGGER = "luigi"
+
+API_CLIENT_LOGGING_TEST_LOGGER = "APIClientLoggingTest"
+
+ROOT_LOGGER = "root"
+
+from exasol_integration_test_docker_environment.lib.api.common import generate_root_task, run_task, \
+    set_build_config
+from exasol_integration_test_docker_environment.lib.base.dependency_logger_base_task import \
+    DependencyLoggerBaseTask
+
+
+@contextlib.contextmanager
+def ignore_resource_warning():
+    """
+    Ignore ResourceWarning to keep the captured output clean for the asserts
+    """
+    with warnings.catch_warnings():
+        warnings.filterwarnings(action="ignore",
+                                category=ResourceWarning,
+                                message=".*unclosed <socket.socket.*")
+        yield
+
+
+class DummyTask(DependencyLoggerBaseTask):
+
+    def run_task(self):
+        self.logger.info("DUMMY LOGGER")
+        self.return_object("DUMMY SUCCES")
+
+
+class APIClientLoggingTest(unittest.TestCase):
+
+    def setUp(self):
+        self.maxDiff = 30000
+        print(f"SetUp {self.__class__.__name__}", file=stderr)
+        self._build_output_temp_dir = TemporaryDirectory()
+        self._build_output_temp_dir.__enter__()
+        self.reset_logging()
+
+    def tearDown(self):
+        self._build_output_temp_dir.__exit__(None, None, None)
+
+    @ignore_resource_warning()
+    def dummy_api_command(self, log_level: Optional[str], use_job_specific_log_file: bool):
+        set_build_config(False,
+                         tuple(),
+                         False,
+                         False,
+                         self._build_output_temp_dir.name,
+                         "/tmp",
+                         "",
+                         "test")
+        task_creator = lambda: generate_root_task(task_class=DummyTask)
+        result = run_task(task_creator=task_creator,
+                          workers=5,
+                          task_dependencies_dot_file=None,
+                          log_level=log_level,
+                          use_job_specific_log_file=use_job_specific_log_file)
+        return result
+
+    def test_luigi_log_level_info_and_basic_logging_error(self):
+        with redirect_stderr(StringIO()) as f:
+            logging.basicConfig(
+                format='%(asctime)s %(levelname)-8s %(message)s',
+                level=logging.ERROR,
+                datefmt='%Y-%m-%d %H:%M:%S',
+                force=True)
+            logger_infos_before = self.create_logger_infos()
+            result = self.dummy_api_command(log_level="INFO", use_job_specific_log_file=False)
+            logger_infos_after = self.create_logger_infos()
+        self.assertEqual(logger_infos_before[ROOT_LOGGER], logger_infos_after[ROOT_LOGGER])
+        self.assertEqual(logger_infos_before[API_CLIENT_LOGGING_TEST_LOGGER],
+                         logger_infos_after[API_CLIENT_LOGGING_TEST_LOGGER])
+        self.assertEqual(logger_infos_before[LUIGI_LOGGER], logger_infos_after[LUIGI_LOGGER])
+        self.assertNotEqual(logger_infos_before[LUIGI_INTERFACE_LOGGER], logger_infos_after[LUIGI_INTERFACE_LOGGER])
+        self.assertEqual(logger_infos_after[LUIGI_INTERFACE_LOGGER][LEVEL_NAME], "INFO")
+        stderr = f.getvalue()
+        self.assertNotEqual(stderr, "")
+        self.assertRegex(stderr, ".*===== Luigi Execution Summary =====.*")
+        main_log_glob = list(Path(self._build_output_temp_dir.name).glob("**/main.log"))
+        self.assertEqual(main_log_glob, [])
+
+    def test_luigi_use_job_specific_log_file_and_basic_logging_error(self):
+        with redirect_stderr(StringIO()) as f:
+            logging.basicConfig(
+                format='%(asctime)s %(levelname)-8s %(message)s',
+                level=logging.ERROR,
+                datefmt='%Y-%m-%d %H:%M:%S',
+                force=True)
+            logger_infos_before = self.create_logger_infos()
+            result = self.dummy_api_command(log_level=None, use_job_specific_log_file=True)
+            logger_infos_after = self.create_logger_infos()
+        self.assertEqual(logger_infos_before[ROOT_LOGGER], logger_infos_after[ROOT_LOGGER])
+        self.assertEqual(logger_infos_before[API_CLIENT_LOGGING_TEST_LOGGER],
+                         logger_infos_after[API_CLIENT_LOGGING_TEST_LOGGER])
+        self.assertNotEqual(logger_infos_before[LUIGI_LOGGER], logger_infos_after[LUIGI_LOGGER])
+        self.assertNotEqual(logger_infos_before[LUIGI_INTERFACE_LOGGER], logger_infos_after[LUIGI_INTERFACE_LOGGER])
+        stderr = f.getvalue()
+        self.assertEqual(stderr, "")
+        main_log_glob = list(Path(self._build_output_temp_dir.name).glob("**/main.log"))
+        self.assertNotEqual(main_log_glob, [])
+
+    def test_luigi_log_level_error_and_basic_logging_info(self):
+        with redirect_stderr(StringIO()) as f:
+            logging.basicConfig(
+                format='%(asctime)s %(levelname)-8s %(message)s',
+                level=logging.INFO,
+                datefmt='%Y-%m-%d %H:%M:%S',
+                force=True)
+            logger_infos_before = self.create_logger_infos()
+            result = self.dummy_api_command(log_level="ERROR", use_job_specific_log_file=False)
+            logger_infos_after = self.create_logger_infos()
+        self.assertEqual(logger_infos_before[ROOT_LOGGER], logger_infos_after[ROOT_LOGGER])
+        self.assertEqual(logger_infos_before[API_CLIENT_LOGGING_TEST_LOGGER],
+                         logger_infos_after[API_CLIENT_LOGGING_TEST_LOGGER])
+        self.assertEqual(logger_infos_before[LUIGI_LOGGER], logger_infos_after[LUIGI_LOGGER])
+        self.assertNotEqual(logger_infos_before[LUIGI_INTERFACE_LOGGER], logger_infos_after[LUIGI_INTERFACE_LOGGER])
+        self.assertEqual(logger_infos_after[LUIGI_INTERFACE_LOGGER][LEVEL_NAME], "ERROR")
+        stderr = f.getvalue()
+        self.assertNotEqual(stderr, "")
+        self.assertRegex(stderr, ".*logging configured by default settings")
+        main_log_glob = list(Path(self._build_output_temp_dir.name).glob("**/main.log"))
+        self.assertEqual(main_log_glob, [])
+
+    def test_luigi_no_log_config_and_basic_logging_info(self):
+        with redirect_stderr(StringIO()) as f:
+            logging.basicConfig(
+                format='%(asctime)s %(levelname)-8s %(message)s',
+                level=logging.INFO,
+                datefmt='%Y-%m-%d %H:%M:%S',
+                force=True)
+            logger_infos_before = self.create_logger_infos()
+            result = self.dummy_api_command(log_level=None, use_job_specific_log_file=False)
+            logger_infos_after = self.create_logger_infos()
+        self.assertEqual(logger_infos_before, logger_infos_after)
+        stderr = f.getvalue()
+        self.assertRegex(stderr, ".*INFO     logging disabled in settings.*")
+        self.assertRegex(stderr, ".*===== Luigi Execution Summary =====.*")
+        main_log_glob = list(Path(self._build_output_temp_dir.name).glob("**/main.log"))
+        self.assertEqual(main_log_glob, [])
+
+    def test_luigi_no_log_config_and_basic_logging_error(self):
+        with redirect_stderr(StringIO()) as f:
+            logging.basicConfig(
+                format='%(asctime)s %(levelname)-8s %(message)s',
+                level=logging.ERROR,
+                datefmt='%Y-%m-%d %H:%M:%S',
+                force=True)
+            logger_infos_before = self.create_logger_infos()
+            result = self.dummy_api_command(log_level=None, use_job_specific_log_file=False)
+            logger_infos_after = self.create_logger_infos()
+        self.assertEqual(logger_infos_before, logger_infos_after)
+        stderr = f.getvalue()
+        self.assertEqual(stderr, "")
+        main_log_glob = list(Path(self._build_output_temp_dir.name).glob("**/main.log"))
+        self.assertEqual(main_log_glob, [])
+
+    def reset_logging(self):
+        logging.basicConfig(
+            format='%(asctime)s %(levelname)-8s %(message)s',
+            level=logging.ERROR,
+            datefmt='%Y-%m-%d %H:%M:%S',
+            force=True)
+        loggerDict = logging.root.manager.loggerDict
+        for key in list(loggerDict.keys()):
+            loggerDict[key].disabled = True
+            del loggerDict[key]
+
+    def create_logger_infos(self) -> Dict[str, Dict[str, Any]]:
+        logger_infos = {
+            ROOT_LOGGER: self.get_logger_info(logging.root),
+            API_CLIENT_LOGGING_TEST_LOGGER: self.get_logger_info(logging.getLogger("APIClientLoggingTest")),
+            LUIGI_LOGGER: self.get_logger_info(logging.getLogger("luigi")),
+            LUIGI_INTERFACE_LOGGER: self.get_logger_info(logging.getLogger("luigi-interface"))
+        }
+        return logger_infos
+
+    def get_logger_info(self, logger: logging.Logger) -> Dict[str, Any]:
+        logger_info = {}
+        logger_info[LOGGER_STR] = str(logger)
+        logger_info[LEVEL] = logger.level
+        logger_info[LEVEL_NAME] = logging.getLevelName(logger.level)
+        logger_info[HANDLERS] = list(logger.handlers)
+        logger_info[LOGGER_NAME] = logger.name
+        logger_info[FILTERS] = list(logger.filters)
+        logger_info[DISABLED] = logger.disabled
+        logger_info[PROPAGATE] = logger.propagate
+        logger_info[PARENT] = logger.parent
+        return logger_info
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/exasol_integration_test_docker_environment/test/test_click_api_consistency.py b/exasol_integration_test_docker_environment/test/test_click_api_consistency.py
index 4068aaea5..6ee38277c 100644
--- a/exasol_integration_test_docker_environment/test/test_click_api_consistency.py
+++ b/exasol_integration_test_docker_environment/test/test_click_api_consistency.py
@@ -46,7 +46,7 @@ def test_api_default_values(self):
             self.assertEqual(len(cli_defaults), len(api_spec_defaults))
             for api_default_value, cli_default in zip(api_spec_defaults, cli_defaults):
                 cli_param_name, cli_default_value = cli_default
-                if api_default_value != cli_default_value:
+                if api_default_value != cli_default_value and cli_param_name != "use_job_specific_log_file":
                     self.fail(f"Default value for parameter '{cli_param_name}' "
                               f"for method '{api_call.__name__}' does not match. "
                               f"API method has default value '{api_default_value}' "
diff --git a/exasol_integration_test_docker_environment/test/test_common_run_task.py b/exasol_integration_test_docker_environment/test/test_common_run_task.py
index 3d0c24c8d..d71d64026 100644
--- a/exasol_integration_test_docker_environment/test/test_common_run_task.py
+++ b/exasol_integration_test_docker_environment/test/test_common_run_task.py
@@ -28,8 +28,9 @@ def test_same_logging_file(self):
     def test_same_logging_file_custom_log_location(self):
         self._execute_in_new_process(target="run_test_same_logging_file_env_log_path")
 
-    def test_different_logging_file_raises_error(self):
-        self._execute_in_new_process(target="run_test_different_logging_file_raises_error")
+    def test_different_logging_file(self):
+        self._execute_in_new_process(target="run_test_different_logging_file")
+
 
 
 class TestTaskWithReturn(DependencyLoggerBaseTask):
diff --git a/exasol_integration_test_docker_environment/test/test_common_run_task_subprocess.py b/exasol_integration_test_docker_environment/test/test_common_run_task_subprocess.py
index 4780b20f9..bb909ba83 100644
--- a/exasol_integration_test_docker_environment/test/test_common_run_task_subprocess.py
+++ b/exasol_integration_test_docker_environment/test/test_common_run_task_subprocess.py
@@ -6,10 +6,8 @@
 import luigi
 
 from exasol_integration_test_docker_environment.lib.api.common import generate_root_task, run_task
-from exasol_integration_test_docker_environment.cli.options.system_options import DEFAULT_OUTPUT_DIRECTORY
 from exasol_integration_test_docker_environment.lib.base.dependency_logger_base_task import DependencyLoggerBaseTask
-
-from exasol_integration_test_docker_environment.lib.base.luigi_log_config import LOG_ENV_VARIABLE_NAME
+from exasol_integration_test_docker_environment.lib.base.luigi_log_config import LOG_ENV_VARIABLE_NAME, get_log_path
 
 
 class TestTask(DependencyLoggerBaseTask):
@@ -17,9 +15,10 @@ class TestTask(DependencyLoggerBaseTask):
 
     def run_task(self):
         self.logger.info(f"Logging: {self.x}")
+        self.return_object(self.job_id)
 
 
-def run_simple_tasks(log_path: Path) -> None:
+def run_simple_tasks() -> None:
     NUMBER_TASK = 5
     task_id_generator = (x for x in range(NUMBER_TASK))
 
@@ -27,12 +26,12 @@ def create_task():
         return generate_root_task(task_class=TestTask, x=f"{next(task_id_generator)}")
 
     for i in range(NUMBER_TASK):
-        run_task(create_task, workers=5, task_dependencies_dot_file=None)
+        jobid = run_task(create_task, workers=5, task_dependencies_dot_file=None, use_job_specific_log_file=True)
+        log_path = get_log_path(jobid)
+        assert log_path.exists()
 
-    assert log_path.exists()
-    with open(log_path, "r") as f:
-        log_content = f.read()
-        for i in range(NUMBER_TASK):
+        with open(log_path, "r") as f:
+            log_content = f.read()
             assert f"Logging: {i}" in log_content
 
 
@@ -43,8 +42,7 @@ def run_test_same_logging_file() -> None:
     """
     with tempfile.TemporaryDirectory() as temp_dir:
         os.chdir(temp_dir)
-        log_path = Path(temp_dir) / DEFAULT_OUTPUT_DIRECTORY / "jobs" / "logs" / "main.log"
-        run_simple_tasks(log_path=log_path)
+        run_simple_tasks()
 
 
 def run_test_same_logging_file_env_log_path() -> None:
@@ -56,34 +54,37 @@ def run_test_same_logging_file_env_log_path() -> None:
     with tempfile.TemporaryDirectory() as temp_dir:
         log_path = Path(temp_dir) / "main.log"
         os.environ[LOG_ENV_VARIABLE_NAME] = str(log_path)
-        run_simple_tasks(log_path=log_path)
+        run_simple_tasks()
 
 
-def run_test_different_logging_file_raises_error() -> None:
+def run_test_different_logging_file() -> None:
     """
-    Integration test which verifies that changing the log path from one invocation of run_task to the next will raise
-    an error.
+    Integration test which verifies that changing the log path from one invocation of run_task to the next will work
     """
 
     with tempfile.TemporaryDirectory() as temp_dir:
         task_creator = lambda: generate_root_task(task_class=TestTask, x="Test")
 
-        run_task(task_creator, workers=5, task_dependencies_dot_file=None)
-        log_path = Path(temp_dir) / "main.log"
-        os.environ[LOG_ENV_VARIABLE_NAME] = str(log_path)
-        exception_thrown = False
-        try:
-            run_task(task_creator, workers=5, task_dependencies_dot_file=None)
-        except ValueError as e:
-            exception_thrown = True
-        assert exception_thrown
+        use_specific_log_file(task_creator, temp_dir, "first")
+
+        use_specific_log_file(task_creator, temp_dir, "second")
+
+
+def use_specific_log_file(task_creator, temp_dir, test_name):
+    log_path = Path(temp_dir) / (test_name + ".log")
+    os.environ[LOG_ENV_VARIABLE_NAME] = str(log_path)
+    jobid = run_task(task_creator, workers=5, task_dependencies_dot_file=None, use_job_specific_log_file=True)
+    assert log_path.exists()
+    with open(log_path, "r") as f:
+        log_content = f.read()
+        assert f"Logging: Test" in log_content
 
 
 if __name__ == '__main__':
     test_type = sys.argv[1]
 
     dispatcher = {
-        "run_test_different_logging_file_raises_error": run_test_different_logging_file_raises_error,
+        "run_test_different_logging_file": run_test_different_logging_file,
         "run_test_same_logging_file_env_log_path": run_test_same_logging_file_env_log_path,
         "run_test_same_logging_file": run_test_same_logging_file,
     }
@@ -91,4 +92,3 @@ def run_test_different_logging_file_raises_error() -> None:
         dispatcher[test_type]()
     except KeyError as e:
         raise ValueError("Unknown Test argument") from e
-