Repository: RasaHQ/rasa
Pull request: Improve logging by adding cli option for custom formatting
**Proposed changes**:
- Fixes #2546 (Jira ticket [ATO-454](https://rasahq.atlassian.net/browse/ATO-454))
- Adds a new logging option parameter to the CLI labelled `--logging-config-file` which accepts a YAML file as value
- In this YAML file, the rasa developer should provide custom configuration of formatters and handlers (following closely the [dictConfig schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in the Python docs)

**Status (please check what you already did)**:
- [X] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/main/changelog) for instructions)
- [X] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


[ATO-454]: https://rasahq.atlassian.net/browse/ATO-454?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Problem statement:
Allow configuration of logging format
<!-- THIS INFORMATION IS MANDATORY - YOUR ISSUE WILL BE CLOSED IF IT IS MISSING. If you don't know your Rasa Core version, use `pip list | grep rasa_core`. Removing the below information is allowed for FEATURE REQUESTS. --> 

**Rasa Core version**: 0.11.3

**Python version**: 3.6.2

**Operating system** (windows, osx, ...): window

**Issue**:
This is for an analytics purpose. Current debug logs doesn't have the timestamp details like when the query is sent from the user. Please suggest

**Content of domain file** (if used & relevant):
```yaml

```

Hints from issue discussion:
Thanks for raising this issue, @tmbo  will get back to you about it soon.
You can configure the logger. It is a standard logger .e.g using
```
import logging
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
```
Where to configure this step? I am using below command to run the server

`python -m rasa_core.run --enable_api -d models/dialogue -u models/current/nlu --cors "*" --debug`
There is no option in the run script to change that - but that is actually a good point. I think it makes sense to integrate some loading of the configuration from a file, the python logging framework has that built in, we just need to use it https://docs.python.org/2/library/logging.config.html#logging-config-fileformat
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically closed due to inactivity. Please create a new issue if you need more help.

Any updates on that? 
We need logs in json format
Changing some configs in rasa command line required required a lot of surgery. I had an issue where we wanted a custom logger, and there was no way to configure it using the default commandline. Ended up creating my own start scripts and stubbing methods in `rasa.server` and `rasa.utils.common`.

Commit message: more cli test fixes

Patch to review:
diff --git a/changelog/2546.improvement.md b/changelog/2546.improvement.md
new file mode 100644
index 000000000000..ce19b00ffc9f
--- /dev/null
+++ b/changelog/2546.improvement.md
@@ -0,0 +1 @@
+Added CLI option `--logging-config-file` to enable configuration of custom logs formatting.
diff --git a/data/test_logging_config.yml b/data/test_logging_config.yml
new file mode 100644
index 000000000000..49590b545c98
--- /dev/null
+++ b/data/test_logging_config.yml
@@ -0,0 +1,19 @@
+version: 1
+disable_existing_loggers: false
+formatters:
+    normalFormatter:
+        format: "{\"time\": \"%(asctime)s\", \"name\": \"[%(name)s]\", \"levelname\": \"%(levelname)s\", \"message\": \"%(message)s\"}"
+handlers:
+    test_handler:
+        level: INFO
+        formatter: normalFormatter
+        class: logging.FileHandler
+        filename: "logging_test.log"
+loggers:
+    root:
+      handlers: [test_handler]
+      level: INFO
+    rasa:
+      handlers: [test_handler]
+      level: INFO
+      propagate: 0
diff --git a/rasa/__main__.py b/rasa/__main__.py
index 0918080f373f..507fad84cb4b 100644
--- a/rasa/__main__.py
+++ b/rasa/__main__.py
@@ -99,8 +99,9 @@ def main() -> None:
     cmdline_arguments = arg_parser.parse_args()
 
     log_level = getattr(cmdline_arguments, "loglevel", None)
+    logging_config_file = getattr(cmdline_arguments, "logging-config-file", None)
     configure_logging_and_warnings(
-        log_level, warn_only_once=True, filter_repeated_logs=True
+        log_level, logging_config_file, warn_only_once=True, filter_repeated_logs=True
     )
 
     tf_env.setup_tf_environment()
diff --git a/rasa/cli/arguments/default_arguments.py b/rasa/cli/arguments/default_arguments.py
index 804e1a5adeeb..1190b6eb8746 100644
--- a/rasa/cli/arguments/default_arguments.py
+++ b/rasa/cli/arguments/default_arguments.py
@@ -156,3 +156,10 @@ def add_logging_options(parser: argparse.ArgumentParser) -> None:
         dest="loglevel",
         const=logging.WARNING,
     )
+
+    logging_arguments.add_argument(
+        "--logging-config-file",
+        type=str,
+        help="If set, the name of the logging configuration file will be set "
+        "to the given name.",
+    )
diff --git a/rasa/utils/common.py b/rasa/utils/common.py
index 48fde9b254d2..0b5758211fb9 100644
--- a/rasa/utils/common.py
+++ b/rasa/utils/common.py
@@ -1,6 +1,7 @@
 import copy
 import inspect
 import logging
+import logging.config
 import logging.handlers
 import os
 import shutil
@@ -108,6 +109,7 @@ def read_global_config(path: Text) -> Dict[Text, Any]:
 
 def configure_logging_and_warnings(
     log_level: Optional[int] = None,
+    logging_config_file: Optional[Text] = None,
     warn_only_once: bool = True,
     filter_repeated_logs: bool = True,
 ) -> None:
@@ -117,11 +119,17 @@ def configure_logging_and_warnings(
         log_level: The log level to be used for the 'Rasa' logger. Pass `None` to use
             either the environment variable 'LOG_LEVEL' if it is specified, or the
             default log level otherwise.
+        logging_config_file: YAML file containing logging configuration to handle
+            custom formatting
         warn_only_once: determines whether user warnings should be filtered by the
             `warnings` module to appear only "once"
         filter_repeated_logs: determines whether `RepeatedLogFilter`s are added to
             the handlers of the root logger
     """
+    if logging_config_file is not None:
+        config_dict = rasa.shared.utils.io.read_yaml_file(logging_config_file)
+        logging.config.dictConfig(config_dict)
+
     if log_level is None:  # Log level NOTSET is 0 so we use `is None` here
         log_level_name = os.environ.get(ENV_LOG_LEVEL, DEFAULT_LOG_LEVEL)
         # Change log level from str to int (note that log_level in function parameter
diff --git a/tests/cli/test_rasa_data.py b/tests/cli/test_rasa_data.py
index 62181c0d7960..e4c7aa4017d2 100644
--- a/tests/cli/test_rasa_data.py
+++ b/tests/cli/test_rasa_data.py
@@ -114,8 +114,9 @@ def test_data_convert_nlu_yml(
 def test_data_split_help(run: Callable[..., RunResult]):
     output = run("data", "split", "nlu", "--help")
 
-    help_text = f"""usage: {RASA_EXE} data split nlu [-h] [-v] [-vv] [--quiet] [-u NLU]
-                           [--training-fraction TRAINING_FRACTION]
+    help_text = f"""usage: {RASA_EXE} data split nlu [-h] [-v] [-vv] [--quiet]\n
+                           [--logging-config-file LOGGING_CONFIG_FILE]\n
+                           [-u NLU] [--training-fraction TRAINING_FRACTION]\n
                            [--random-seed RANDOM_SEED] [--out OUT]"""
 
     lines = help_text.split("\n")
@@ -128,20 +129,22 @@ def test_data_split_help(run: Callable[..., RunResult]):
 def test_data_convert_help(run: Callable[..., RunResult]):
     output = run("data", "convert", "nlu", "--help")
 
-    help_text = (
-        f"""usage: {RASA_EXE} data convert nlu [-h] [-v] [-vv]"""
-        """ [--quiet] [-f {json,yaml}]"""
-    )
+    help_text = f"""usage: {RASA_EXE} data convert nlu [-h] [-v] [-vv] [--quiet]\n
+                           [--logging-config-file LOGGING_CONFIG_FILE]\n
+                           [-f {"{json,yaml}"}] [--data DATA [DATA ...]]"""
 
+    lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
-    assert help_text in printed_help
+    printed_help = {line.strip() for line in output.outlines}
+    for line in lines:
+        assert line.strip() in printed_help
 
 
 def test_data_validate_help(run: Callable[..., RunResult]):
     output = run("data", "validate", "--help")
 
     help_text = f"""usage: {RASA_EXE} data validate [-h] [-v] [-vv] [--quiet]
+                          [--logging-config-file LOGGING_CONFIG_FILE]
                           [--max-history MAX_HISTORY] [-c CONFIG]
                           [--fail-on-warnings] [-d DOMAIN]
                           [--data DATA [DATA ...]]
@@ -149,17 +152,23 @@ def test_data_validate_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_data_migrate_help(run: Callable[..., RunResult]):
     output = run("data", "migrate", "--help")
     printed_help = set(output.outlines)
 
-    help_text = f"""usage: {RASA_EXE} data migrate [-h] [-v] [-vv] [--quiet] [-d DOMAIN] [--out OUT]"""  # noqa: E501
-    assert help_text in printed_help
+    help_text = f"""usage: {RASA_EXE} data migrate [-h] [-v] [-vv] [--quiet]
+                          [--logging-config-file LOGGING_CONFIG_FILE]
+                          [-d DOMAIN] [--out OUT]"""
+    lines = help_text.split("\n")
+    # expected help text lines should appear somewhere in the output
+    printed_help = {line.strip() for line in output.outlines}
+    for line in lines:
+        assert line.strip() in printed_help
 
 
 def test_data_validate_stories_with_max_history_zero(monkeypatch: MonkeyPatch):
diff --git a/tests/cli/test_rasa_evaluate_markers.py b/tests/cli/test_rasa_evaluate_markers.py
index 59ebd717c276..721b620a6a74 100644
--- a/tests/cli/test_rasa_evaluate_markers.py
+++ b/tests/cli/test_rasa_evaluate_markers.py
@@ -50,6 +50,7 @@ def test_evaluate_markers_help(run: Callable[..., RunResult]):
     output = run("evaluate", "markers", "--help")
 
     help_text = f"""usage: {RASA_EXE} evaluate markers [-h] [-v] [-vv] [--quiet]
+    [--logging-config-file LOGGING_CONFIG_FILE]
     {{first_n,sample_n,all}} ..."""
 
     lines = [line.strip() for line in help_text.split("\n")]
@@ -64,6 +65,7 @@ def test_evaluate_markers_first_n_help(run: Callable[..., RunResult]):
     output = run("evaluate", "markers", "first_n", "--help")
 
     help_text = f"""usage: {RASA_EXE} evaluate markers first_n [-h] [-v] [-vv] [--quiet]
+    [--logging-config-file LOGGING_CONFIG_FILE]
     [--config CONFIG]
     [--no-stats | --stats-file-prefix [STATS_FILE_PREFIX]]
     [--endpoints ENDPOINTS] [-d DOMAIN]
@@ -80,8 +82,9 @@ def test_evaluate_markers_sample_n_help(run: Callable[..., RunResult]):
     # We need to specify an output_filename as that's the first positional parameter
     output = run("evaluate", "markers", "sample_n", "--help")
 
-    help_text = f"""usage: {RASA_EXE} evaluate markers sample_n [-h] [-v] [-vv] [--quiet] [--seed SEED]
-    [--config CONFIG]
+    help_text = f"""usage: {RASA_EXE} evaluate markers sample_n [-h] [-v] [-vv] [--quiet]
+    [--logging-config-file LOGGING_CONFIG_FILE]
+    [--seed SEED] [--config CONFIG]
     [--no-stats | --stats-file-prefix [STATS_FILE_PREFIX]]
     [--endpoints ENDPOINTS] [-d DOMAIN]
     count output_filename"""  # noqa: E501
@@ -97,7 +100,9 @@ def test_evaluate_markers_all_help(run: Callable[..., RunResult]):
     # We need to specify an output_filename as that's the first positional parameter
     output = run("evaluate", "markers", "all", "--help")
 
-    help_text = f"""usage: {RASA_EXE} evaluate markers all [-h] [-v] [-vv] [--quiet] [--config CONFIG]
+    help_text = f"""usage: {RASA_EXE} evaluate markers all [-h] [-v] [-vv] [--quiet]
+    [--logging-config-file LOGGING_CONFIG_FILE]
+    [--config CONFIG]
     [--no-stats | --stats-file-prefix [STATS_FILE_PREFIX]]
     [--endpoints ENDPOINTS] [-d DOMAIN]
     output_filename"""  # noqa: E501
diff --git a/tests/cli/test_rasa_export.py b/tests/cli/test_rasa_export.py
index a55f64cccf5f..45d231af8ece 100644
--- a/tests/cli/test_rasa_export.py
+++ b/tests/cli/test_rasa_export.py
@@ -27,7 +27,9 @@
 def test_export_help(run: Callable[..., RunResult]):
     output = run("export", "--help")
 
-    help_text = f"""usage: {RASA_EXE} export [-h] [-v] [-vv] [--quiet] [--endpoints ENDPOINTS]
+    help_text = f"""usage: {RASA_EXE} export [-h] [-v] [-vv] [--quiet]
+                   [--logging-config-file LOGGING_CONFIG_FILE]
+                   [--endpoints ENDPOINTS]
                    [--minimum-timestamp MINIMUM_TIMESTAMP]
                    [--maximum-timestamp MAXIMUM_TIMESTAMP]
                    [--offset-timestamps-by-seconds OFFSET_TIMESTAMPS_BY_SECONDS]
@@ -35,9 +37,9 @@ def test_export_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 @pytest.mark.parametrize(
diff --git a/tests/cli/test_rasa_init.py b/tests/cli/test_rasa_init.py
index 975c432c6e7e..b7f45cb856da 100644
--- a/tests/cli/test_rasa_init.py
+++ b/tests/cli/test_rasa_init.py
@@ -44,16 +44,15 @@ def test_not_found_init_path(run: Callable[..., RunResult]):
 def test_init_help(run: Callable[..., RunResult]):
     output = run("init", "--help")
 
-    help_text = (
-        f"""usage: {RASA_EXE} init [-h] [-v] [-vv] [--quiet]"""
-        """ [--no-prompt] [--init-dir INIT_DIR]"""
-    )
+    help_text = f"""usage: {RASA_EXE} init [-h] [-v] [-vv] [--quiet]
+        [--logging-config-file LOGGING_CONFIG_FILE] [--no-prompt]
+        [--init-dir INIT_DIR]"""
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_user_asked_to_train_model(run_with_stdin: Callable[..., RunResult]):
diff --git a/tests/cli/test_rasa_interactive.py b/tests/cli/test_rasa_interactive.py
index 8712ea47f453..31c7db8ae856 100644
--- a/tests/cli/test_rasa_interactive.py
+++ b/tests/cli/test_rasa_interactive.py
@@ -18,8 +18,10 @@
 def test_interactive_help(run: Callable[..., RunResult]):
     output = run("interactive", "--help")
 
-    help_text = f"""usage: {RASA_EXE} interactive [-h] [-v] [-vv] [--quiet] [--e2e] [-p PORT] [-m MODEL]
-                        [--data DATA [DATA ...]] [--skip-visualization]
+    help_text = f"""usage: {RASA_EXE} interactive [-h] [-v] [-vv] [--quiet]
+                        [--logging-config-file LOGGING_CONFIG_FILE] [--e2e]
+                        [-p PORT] [-m MODEL] [--data DATA [DATA ...]]
+                        [--skip-visualization]
                         [--conversation-id CONVERSATION_ID]
                         [--endpoints ENDPOINTS] [-c CONFIG] [-d DOMAIN]
                         [--out OUT] [--augmentation AUGMENTATION]
@@ -30,16 +32,17 @@ def test_interactive_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_interactive_core_help(run: Callable[..., RunResult]):
     output = run("interactive", "core", "--help")
 
-    help_text = f"""usage: {RASA_EXE} interactive core [-h] [-v] [-vv] [--quiet] [-m MODEL] [-s STORIES]
-                             [--skip-visualization]
+    help_text = f"""usage: {RASA_EXE} interactive core [-h] [-v] [-vv] [--quiet]
+                             [--logging-config-file LOGGING_CONFIG_FILE]
+                             [-m MODEL] [-s STORIES] [--skip-visualization]
                              [--conversation-id CONVERSATION_ID]
                              [--endpoints ENDPOINTS] [-c CONFIG] [-d DOMAIN]
                              [--out OUT] [--augmentation AUGMENTATION]
@@ -49,9 +52,9 @@ def test_interactive_core_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_pass_arguments_to_rasa_train(
diff --git a/tests/cli/test_rasa_run.py b/tests/cli/test_rasa_run.py
index cf8a9adcf805..2e21f1ef33aa 100644
--- a/tests/cli/test_rasa_run.py
+++ b/tests/cli/test_rasa_run.py
@@ -34,9 +34,10 @@ def test_run_help(
                 [--response-timeout RESPONSE_TIMEOUT]"""
 
     help_text = (
-        f"""usage: {RASA_EXE} run [-h] [-v] [-vv] [--quiet] [-m MODEL] [--log-file LOG_FILE]
-                [--use-syslog] [--syslog-address SYSLOG_ADDRESS]
-                [--syslog-port SYSLOG_PORT]
+        f"""usage: {RASA_EXE} run [-h] [-v] [-vv] [--quiet]
+                [--logging-config-file LOGGING_CONFIG_FILE] [-m MODEL]
+                [--log-file LOG_FILE] [--use-syslog]
+                [--syslog-address SYSLOG_ADDRESS] [--syslog-port SYSLOG_PORT]
                 [--syslog-protocol SYSLOG_PROTOCOL] [--endpoints ENDPOINTS]
                 """  # noqa: E501
         + version_dependent
@@ -52,9 +53,9 @@ def test_run_help(
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_run_action_help(
@@ -66,14 +67,16 @@ def test_run_action_help(
         # This is required because `argparse` behaves differently on
         # Python 3.9 and above. The difference is the changed formatting of help
         # output for CLI arguments with `nargs="*"
-        help_text = f"""usage: {RASA_EXE} run actions [-h] [-v] [-vv] [--quiet] [-p PORT]
+        help_text = f"""usage: {RASA_EXE} run actions [-h] [-v] [-vv] [--quiet]
+                        [--logging-config-file LOGGING_CONFIG_FILE] [-p PORT]
                         [--cors [CORS ...]] [--actions ACTIONS]"""  # noqa: E501
     else:
-        help_text = f"""usage: {RASA_EXE} run actions [-h] [-v] [-vv] [--quiet] [-p PORT]
+        help_text = f"""usage: {RASA_EXE} run actions [-h] [-v] [-vv] [--quiet]
+                        [--logging-config-file LOGGING_CONFIG_FILE] [-p PORT]
                         [--cors [CORS [CORS ...]]] [--actions ACTIONS]"""  # noqa: E501
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
diff --git a/tests/cli/test_rasa_shell.py b/tests/cli/test_rasa_shell.py
index d81fe5096693..143a055d545a 100644
--- a/tests/cli/test_rasa_shell.py
+++ b/tests/cli/test_rasa_shell.py
@@ -21,6 +21,7 @@ def test_shell_help(run: Callable[..., RunResult]):
 
     help_text = (
         f"""usage: {RASA_EXE} shell [-h] [-v] [-vv] [--quiet]
+                  [--logging-config-file LOGGING_CONFIG_FILE]
                   [--conversation-id CONVERSATION_ID] [-m MODEL]
                   [--log-file LOG_FILE] [--use-syslog]
                   [--syslog-address SYSLOG_ADDRESS]
@@ -40,19 +41,20 @@ def test_shell_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_shell_nlu_help(run: Callable[..., RunResult]):
     output = run("shell", "nlu", "--help")
 
-    help_text = f"""usage: {RASA_EXE} shell nlu [-h] [-v] [-vv] [--quiet] [-m MODEL]
+    help_text = f"""usage: {RASA_EXE} shell nlu [-h] [-v] [-vv] [--quiet]
+                      [--logging-config-file LOGGING_CONFIG_FILE] [-m MODEL]
                       [model-as-positional-argument]"""
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
diff --git a/tests/cli/test_rasa_test.py b/tests/cli/test_rasa_test.py
index 4173916ce8d7..d9726729a434 100644
--- a/tests/cli/test_rasa_test.py
+++ b/tests/cli/test_rasa_test.py
@@ -264,10 +264,11 @@ def test_test_core_comparison_after_train(
 def test_test_help(run: Callable[..., RunResult]):
     output = run("test", "--help")
 
-    help_text = f"""usage: {RASA_EXE} test [-h] [-v] [-vv] [--quiet] [-m MODEL] [-s STORIES]
-                 [--max-stories MAX_STORIES] [--endpoints ENDPOINTS]
-                 [--fail-on-prediction-errors] [--url URL]
-                 [--evaluate-model-directory] [-u NLU]
+    help_text = f"""usage: {RASA_EXE} test [-h] [-v] [-vv] [--quiet]
+                 [--logging-config-file LOGGING_CONFIG_FILE] [-m MODEL]
+                 [-s STORIES] [--max-stories MAX_STORIES]
+                 [--endpoints ENDPOINTS] [--fail-on-prediction-errors]
+                 [--url URL] [--evaluate-model-directory] [-u NLU]
                  [-c CONFIG [CONFIG ...]] [-d DOMAIN] [--cross-validation]
                  [-f FOLDS] [-r RUNS] [-p PERCENTAGES [PERCENTAGES ...]]
                  [--no-plot] [--successes] [--no-errors] [--no-warnings]
@@ -276,38 +277,41 @@ def test_test_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_test_nlu_help(run: Callable[..., RunResult]):
     output = run("test", "nlu", "--help")
 
-    help_text = f"""usage: {RASA_EXE} test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
-                     [-c CONFIG [CONFIG ...]] [-d DOMAIN] [--cross-validation]
-                     [-f FOLDS] [-r RUNS] [-p PERCENTAGES [PERCENTAGES ...]]
-                     [--no-plot] [--successes] [--no-errors] [--no-warnings]"""  # noqa: E501
+    help_text = f"""usage: {RASA_EXE} test nlu [-h] [-v] [-vv] [--quiet]
+                     [--logging-config-file LOGGING_CONFIG_FILE] [-m MODEL]
+                     [-u NLU] [--out OUT] [-c CONFIG [CONFIG ...]] [-d DOMAIN]
+                     [--cross-validation] [-f FOLDS] [-r RUNS]
+                     [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]
+                     [--successes] [--no-errors] [--no-warnings]"""  # noqa: E501
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_test_core_help(run: Callable[..., RunResult]):
     output = run("test", "core", "--help")
 
-    help_text = f"""usage: {RASA_EXE} test core [-h] [-v] [-vv] [--quiet] [-m MODEL [MODEL ...]]
-                      [-s STORIES] [--max-stories MAX_STORIES] [--out OUT]
-                      [--e2e] [--endpoints ENDPOINTS]
-                      [--fail-on-prediction-errors] [--url URL]
-                      [--evaluate-model-directory] [--no-plot] [--successes]
-                      [--no-errors] [--no-warnings]"""  # noqa: E501
+    help_text = f"""usage: {RASA_EXE} test core [-h] [-v] [-vv] [--quiet]
+                      [--logging-config-file LOGGING_CONFIG_FILE]
+                      [-m MODEL [MODEL ...]] [-s STORIES]
+                      [--max-stories MAX_STORIES] [--out OUT] [--e2e]
+                      [--endpoints ENDPOINTS] [--fail-on-prediction-errors]
+                      [--url URL] [--evaluate-model-directory] [--no-plot]
+                      [--successes] [--no-errors] [--no-warnings]"""
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
diff --git a/tests/cli/test_rasa_train.py b/tests/cli/test_rasa_train.py
index 0b30e3540adb..c2b55517da50 100644
--- a/tests/cli/test_rasa_train.py
+++ b/tests/cli/test_rasa_train.py
@@ -430,9 +430,10 @@ def test_train_nlu_persist_nlu_data(
 def test_train_help(run: Callable[..., RunResult]):
     output = run("train", "--help")
 
-    help_text = f"""usage: {RASA_EXE} train [-h] [-v] [-vv] [--quiet] [--data DATA [DATA ...]]
-                  [-c CONFIG] [-d DOMAIN] [--out OUT] [--dry-run]
-                  [--augmentation AUGMENTATION] [--debug-plots]
+    help_text = f"""usage: {RASA_EXE} train [-h] [-v] [-vv] [--quiet]
+                  [--logging-config-file LOGGING_CONFIG_FILE]
+                  [--data DATA [DATA ...]] [-c CONFIG] [-d DOMAIN] [--out OUT]
+                  [--dry-run] [--augmentation AUGMENTATION] [--debug-plots]
                   [--num-threads NUM_THREADS]
                   [--fixed-model-name FIXED_MODEL_NAME] [--persist-nlu-data]
                   [--force] [--finetune [FINETUNE]]
@@ -441,26 +442,27 @@ def test_train_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
-
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_train_nlu_help(run: Callable[..., RunResult]):
     output = run("train", "nlu", "--help")
 
-    help_text = f"""usage: {RASA_EXE} train nlu [-h] [-v] [-vv] [--quiet] [-c CONFIG] [-d DOMAIN]
-                      [--out OUT] [-u NLU] [--num-threads NUM_THREADS]
+    help_text = f"""usage: {RASA_EXE} train nlu [-h] [-v] [-vv] [--quiet]
+                      [--logging-config-file LOGGING_CONFIG_FILE] [-c CONFIG]
+                      [-d DOMAIN] [--out OUT] [-u NLU]
+                      [--num-threads NUM_THREADS]
                       [--fixed-model-name FIXED_MODEL_NAME]
                       [--persist-nlu-data] [--finetune [FINETUNE]]
                       [--epoch-fraction EPOCH_FRACTION]"""  # noqa: E501
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 def test_train_core_help(run: Callable[..., RunResult]):
@@ -470,17 +472,21 @@ def test_train_core_help(run: Callable[..., RunResult]):
         # This is required because `argparse` behaves differently on
         # Python 3.9 and above. The difference is the changed formatting of help
         # output for CLI arguments with `nargs="*"
-        help_text = f"""usage: {RASA_EXE} train core [-h] [-v] [-vv] [--quiet] [-s STORIES] [-d DOMAIN]
-                       [-c CONFIG [CONFIG ...]] [--out OUT]
-                       [--augmentation AUGMENTATION] [--debug-plots] [--force]
+        help_text = f"""usage: {RASA_EXE} train core [-h] [-v] [-vv] [--quiet]
+                       [--logging-config-file LOGGING_CONFIG_FILE]
+                       [-s STORIES] [-d DOMAIN] [-c CONFIG [CONFIG ...]]
+                       [--out OUT] [--augmentation AUGMENTATION]
+                       [--debug-plots] [--force]
                        [--fixed-model-name FIXED_MODEL_NAME]
                        [--percentages [PERCENTAGES ...]] [--runs RUNS]
                        [--finetune [FINETUNE]]
                        [--epoch-fraction EPOCH_FRACTION]"""  # noqa: E501
     else:
-        help_text = f"""usage: {RASA_EXE} train core [-h] [-v] [-vv] [--quiet] [-s STORIES] [-d DOMAIN]
-                       [-c CONFIG [CONFIG ...]] [--out OUT]
-                       [--augmentation AUGMENTATION] [--debug-plots] [--force]
+        help_text = f"""usage: {RASA_EXE} train core [-h] [-v] [-vv] [--quiet]
+                       [--logging-config-file LOGGING_CONFIG_FILE]
+                       [-s STORIES] [-d DOMAIN] [-c CONFIG [CONFIG ...]]
+                       [--out OUT] [--augmentation AUGMENTATION]
+                       [--debug-plots] [--force]
                        [--fixed-model-name FIXED_MODEL_NAME]
                        [--percentages [PERCENTAGES [PERCENTAGES ...]]]
                        [--runs RUNS] [--finetune [FINETUNE]]
@@ -488,9 +494,9 @@ def test_train_core_help(run: Callable[..., RunResult]):
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
 
 
 @pytest.mark.parametrize(
diff --git a/tests/cli/test_rasa_visualize.py b/tests/cli/test_rasa_visualize.py
index ba82533c22f0..b563cc962a78 100644
--- a/tests/cli/test_rasa_visualize.py
+++ b/tests/cli/test_rasa_visualize.py
@@ -7,11 +7,13 @@
 def test_visualize_help(run: Callable[..., RunResult]):
     output = run("visualize", "--help")
 
-    help_text = f"""usage: {RASA_EXE} visualize [-h] [-v] [-vv] [--quiet] [-d DOMAIN] [-s STORIES]
-                      [--out OUT] [--max-history MAX_HISTORY] [-u NLU]"""  # noqa: E501
+    help_text = f"""usage: {RASA_EXE} visualize [-h] [-v] [-vv] [--quiet]
+                      [--logging-config-file LOGGING_CONFIG_FILE] [-d DOMAIN]
+                      [-s STORIES] [--out OUT] [--max-history MAX_HISTORY]
+                      [-u NLU]"""  # noqa: E501
 
     lines = help_text.split("\n")
     # expected help text lines should appear somewhere in the output
-    printed_help = set(output.outlines)
+    printed_help = {line.strip() for line in output.outlines}
     for line in lines:
-        assert line in printed_help
+        assert line.strip() in printed_help
diff --git a/tests/utils/test_common.py b/tests/utils/test_common.py
index 8bc0d8a58bf8..e827bd102c9a 100644
--- a/tests/utils/test_common.py
+++ b/tests/utils/test_common.py
@@ -1,3 +1,4 @@
+import json
 import os
 import logging
 from pathlib import Path
@@ -159,20 +160,20 @@ def test_cli_missing_log_level_default_used():
     configure_logging_and_warnings()
     rasa_logger = logging.getLogger("rasa")
     # Default log level is currently INFO
-    rasa_logger.level == logging.INFO
+    assert rasa_logger.level == logging.INFO
     matplotlib_logger = logging.getLogger("matplotlib")
     # Default log level for libraries is currently ERROR
-    matplotlib_logger.level == logging.ERROR
+    assert matplotlib_logger.level == logging.ERROR
 
 
 def test_cli_log_level_debug_used():
     """Test CLI with log level uses for rasa logger whereas libraries stay default."""
     configure_logging_and_warnings(logging.DEBUG)
     rasa_logger = logging.getLogger("rasa")
-    rasa_logger.level == logging.DEBUG
+    assert rasa_logger.level == logging.DEBUG
     matplotlib_logger = logging.getLogger("matplotlib")
     # Default log level for libraries is currently ERROR
-    matplotlib_logger.level == logging.ERROR
+    assert matplotlib_logger.level == logging.ERROR
 
 
 @mock.patch.dict(os.environ, {"LOG_LEVEL": "WARNING"})
@@ -180,10 +181,10 @@ def test_cli_log_level_overrides_env_var_used():
     """Test CLI log level has precedence over env var."""
     configure_logging_and_warnings(logging.DEBUG)
     rasa_logger = logging.getLogger("rasa")
-    rasa_logger.level == logging.DEBUG
+    assert rasa_logger.level == logging.DEBUG
     matplotlib_logger = logging.getLogger("matplotlib")
     # Default log level for libraries is currently ERROR
-    matplotlib_logger.level == logging.ERROR
+    assert matplotlib_logger.level == logging.ERROR
 
 
 @mock.patch.dict(os.environ, {"LOG_LEVEL": "WARNING", "LOG_LEVEL_MATPLOTLIB": "INFO"})
@@ -191,6 +192,31 @@ def test_cli_missing_log_level_env_var_used():
     """Test CLI without log level uses env var for both rasa and libraries."""
     configure_logging_and_warnings()
     rasa_logger = logging.getLogger("rasa")
-    rasa_logger.level == logging.WARNING
+    assert rasa_logger.level == logging.WARNING
     matplotlib_logger = logging.getLogger("matplotlib")
-    matplotlib_logger.level == logging.INFO
+    assert matplotlib_logger.level == logging.INFO
+
+
+def test_cli_logging_configuration() -> None:
+    logging_config_file = "data/test_logging_config.yml"
+    configure_logging_and_warnings(logging_config_file=logging_config_file)
+    rasa_logger = logging.getLogger("rasa")
+
+    handlers = rasa_logger.handlers
+    assert len(handlers) == 1
+    assert isinstance(handlers[0], logging.FileHandler)
+    assert "test_handler" == rasa_logger.handlers[0].name
+
+    logging_message = "This is a test info log."
+    rasa_logger.info(logging_message)
+
+    handler_filename = handlers[0].baseFilename
+    assert Path(handler_filename).exists()
+
+    with open(handler_filename, "r") as logs:
+        data = logs.readlines()
+        logs_dict = json.loads(data[-1])
+        assert logs_dict.get("message") == logging_message
+
+        for key in ["time", "name", "levelname"]:
+            assert key in logs_dict.keys()
diff --git a/tests/utils/test_endpoints.py b/tests/utils/test_endpoints.py
index 847c940622b9..dadd8539485a 100644
--- a/tests/utils/test_endpoints.py
+++ b/tests/utils/test_endpoints.py
@@ -38,7 +38,7 @@ def test_concat_url(base, subpath, expected_result):
 def test_warning_for_base_paths_with_trailing_slash(caplog):
     test_path = "base/"
 
-    with caplog.at_level(logging.DEBUG, logger="rasa.utils.endpoints"):
+    with caplog.at_level(logging.DEBUG):
         assert endpoint_utils.concat_url(test_path, None) == test_path
 
     assert len(caplog.records) == 1


Code review comments:
diff_hunk: @@ -117,11 +119,17 @@ def configure_logging_and_warnings(
         log_level: The log level to be used for the 'Rasa' logger. Pass `None` to use
             either the environment variable 'LOG_LEVEL' if it is specified, or the
             default log level otherwise.
+        logging_config_file: YAML file containing logging configuration to handle
+            custom formatting
         warn_only_once: determines whether user warnings should be filtered by the
             `warnings` module to appear only "once"
         filter_repeated_logs: determines whether `RepeatedLogFilter`s are added to
             the handlers of the root logger
     """
+    if logging_config_file is not None:
+        config_dict = rasa.shared.utils.io.read_yaml_file(logging_config_file)
+        logging.config.dictConfig(config_dict)
path: rasa/utils/common.py
text: @user1:
nitpicking ⬇️ 
```suggestion
        logging_config_dict = rasa.shared.utils.io.read_yaml_file(logging_config_file)
        logging.config.dictConfig(logging_config_dict)
```