Repository: instructlab/instructlab

Issue:
Reconsider default logging format and logging format option
**Is your feature request related to a problem? Please describe.**
InstructLab has a hard-coded log format:

https://github.com/instructlab/instructlab/blob/49622efe32233b4da9fcd4978a8cac6a3a0d506f/src/instructlab/log.py#L7-L9

I see two problems with the approach:

- The log source information is not very useful and makes it hard to understand which library or part of the stack is emitting the log entry.
- The output has a hard-coded time stamp. For integration with systemd, journald, or rsyslog, we don't want to have a time stamp in log line. Journald and others already include time stamps.


The format elements `%(filename)s:%(lineno)d: %(funcName)s:` print the basename of the file (without directory), the line number, and the function name. The file name is not unique. Our dependencies have two files named `connectionpool.py` and three files named `_api.py`. It's impossible to say which messages are coming from `httpx`, `httpcore`, `urllib3`, or `filelock` package at first glance.

```console
$ ilab -vv model download
DEBUG 2024-07-15 16:01:20,501 _http.py:59: add_headers: Request de6cab31-27f2-4fd9-807a-c6a2910d178c: GET https://huggingface.co/api/models/instructlab/merlinite-7b-lab-GGUF/tree/main?recursive=True&expand=False (authenticated: False)
DEBUG 2024-07-15 16:01:20,502 connectionpool.py:1051: _new_conn: Starting new HTTPS connection (1): huggingface.co:443
DEBUG 2024-07-15 16:01:20,637 connectionpool.py:546: _make_request: https://huggingface.co:443 "GET /api/models/instructlab/merlinite-7b-lab-GGUF/tree/main?recursive=True&expand=False HTTP/11" 200 531
DEBUG 2024-07-15 16:01:20,638 _api.py:331: acquire: Attempting to acquire lock 140579829828624 on /home/heimes/.local/share/instructlab/models/.huggingface/.gitignore.lock
DEBUG 2024-07-15 16:01:20,638 _api.py:334: acquire: Lock 140579829828624 acquired on /home/heimes/.local/share/instructlab/models/.huggingface/.gitignore.lock
```

**Describe the solution you'd like**

- Add an option to `config.yaml` to set the log file format. That will allow users to customize logging, e.g. to drop the time stamp for journald.
- Default to `%(name):` instead of `%(filename)s:%(lineno)d: %(funcName)s:`. This logs the name of the logger, which is far more useful to understand log entries:

```
DEBUG 2024-07-15 16:15:22,154 huggingface_hub.utils._http: Request e6c766c4-0996-49d4-b094-0a5c7e7edebe: GET https://huggingface.co/api/models/instructlab/merlinite-7b-lab-GGUF/tree/main?recursive=True&expand=False (authenticated: False)
DEBUG 2024-07-15 16:15:22,154 urllib3.connectionpool: Starting new HTTPS connection (1): huggingface.co:443
DEBUG 2024-07-15 16:15:22,294 urllib3.connectionpool: https://huggingface.co:443 "GET /api/models/instructlab/merlinite-7b-lab-GGUF/tree/main?recursive=True&expand=False HTTP/11" 200 531
DEBUG 2024-07-15 16:15:22,298 filelock: Attempting to acquire lock 140648683333072 on /home/heimes/.local/share/instructlab/models/.huggingface/download/merlinite-7b-lab-Q4_K_M.gguf.lock
DEBUG 2024-07-15 16:15:22,299 filelock: Lock 140648683333072 acquired on /home/heimes/.local/share/instructlab/models/.huggingface/download/merlinite-7b-lab-Q4_K_M.gguf.lock
``` 



Discussion:
Almost fixed with https://github.com/instructlab/instructlab/pull/1741



Patch:
diff --git a/.spellcheck-en-custom.txt b/.spellcheck-en-custom.txt
index 1f002f3df3..41a661bb0f 100644
--- a/.spellcheck-en-custom.txt
+++ b/.spellcheck-en-custom.txt
@@ -133,6 +133,7 @@ Shaders
 Shivchander
 Srivastava
 sshkey
+subcommand
 subdirectory
 Sudalairaj
 Taj
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 797ef64049..93e47d6417 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,9 @@
 ## v0.19
 
+### Features
+
+* Add `log_format` to the `config.yaml` file to allow for customizing the log format.
+
 ### Breaking Changes
 
 * InstructLab now uses XDG-based directories on macOS, similar to Linux.
diff --git a/src/instructlab/configuration.py b/src/instructlab/configuration.py
index 75521d50ee..19db82a231 100644
--- a/src/instructlab/configuration.py
+++ b/src/instructlab/configuration.py
@@ -4,6 +4,7 @@
 from os import path
 from re import match
 from typing import Any, Optional, Union
+import logging
 import os
 import sys
 import textwrap
@@ -54,6 +55,11 @@ class _general(BaseModel):
     # additional fields with defaults
     log_level: StrictStr = Field(default="INFO", description="Log level for logging.")
     debug_level: int = Field(default=0, description="Debug level for logging.")
+    log_format: StrictStr = Field(
+        default="%(levelname)s %(asctime)s %(name)s:%(lineno)d: %(message)s",
+        description="Log format. https://docs.python.org/3/library/logging.html#logrecord-attributes",
+        validate_default=True,
+    )
 
     @field_validator("log_level")
     def validate_log_level(cls, v):
@@ -75,6 +81,19 @@ def validate_log_level(cls, v):
             )
         return v.upper()
 
+    @field_validator("log_format")
+    @classmethod
+    def validate_log_format(cls, log_format):
+        try:
+            logging.PercentStyle(log_format).validate()
+            return log_format
+        except ValueError as e:
+            raise ValueError(
+                f"\nFailed to configure log format: {e}\n"
+                "Have you specified a valid log format?\n"
+                "Consider reading: https://docs.python.org/3/library/logging.html#logrecord-attributes"
+            ) from e
+
     @model_validator(mode="after")
     def after_debug_level(self):
         # set debug level when log level is DEBUG
@@ -1063,7 +1082,9 @@ def init(
         log.configure_logging(
             log_level=config_obj.general.log_level.upper(),
             debug_level=config_obj.general.debug_level,
+            fmt=config_obj.general.log_format,
         )
+
         # subtly get the additional args per cfg section
         # if any are missing, add in sane defaults
         train_additional = ctx.default_map["train"]["additional_args"]
diff --git a/src/instructlab/log.py b/src/instructlab/log.py
index 6d43997721..5b5edbca06 100644
--- a/src/instructlab/log.py
+++ b/src/instructlab/log.py
@@ -3,10 +3,9 @@
 # Standard
 import logging
 import os
+import pathlib
 import sys
 
-FORMAT = "%(levelname)s %(asctime)s %(name)s:%(lineno)d: %(message)s"
-
 
 class CustomFormatter(logging.Formatter):
     def format(self, record):
@@ -37,15 +36,17 @@ def isatty(self):
         return False
 
 
-def stdout_stderr_to_logger(logger, log_file):
+def stdout_stderr_to_logger(
+    logger: logging.Logger, log_file: pathlib.Path | None, fmt: str
+):
     if log_file:
         # Use the existing log file if it exists and append to it
         mode = "a"
         # Create file handler
         file_handler = logging.FileHandler(log_file, mode, encoding="utf-8")
         logger.addHandler(file_handler)
-        # Create formatter and set it for both handlers
-        formatter = CustomFormatter(FORMAT)
+        # Create formatter and add it to the handler
+        formatter = CustomFormatter(fmt)
         file_handler.setFormatter(formatter)
 
     sys.stdout = LoggerWriter(logger.info)
@@ -56,7 +57,7 @@ def configure_logging(
     *,
     log_level: str,
     debug_level: int = 0,
-    fmt: str = FORMAT,
+    fmt: str,
 ) -> None:
     """Configure logging framework"""
     root = logging.getLogger()
diff --git a/src/instructlab/model/serve.py b/src/instructlab/model/serve.py
index 475d5dcbbd..a474d9718a 100644
--- a/src/instructlab/model/serve.py
+++ b/src/instructlab/model/serve.py
@@ -126,7 +126,9 @@ def serve(
         raise click.exceptions.Exit(1)
 
     # Redirect server stdout and stderr to the logger
-    log.stdout_stderr_to_logger(logger, log_file)
+    log.stdout_stderr_to_logger(
+        logger=logger, log_file=log_file, fmt=ctx.obj.config.general.log_format
+    )
 
     if chat_template is None:
         chat_template = ctx.obj.config.serve.chat_template


Test patch:
diff --git a/scripts/functional-tests.sh b/scripts/functional-tests.sh
index 8e9dbd2620..b6876efc92 100755
--- a/scripts/functional-tests.sh
+++ b/scripts/functional-tests.sh
@@ -19,6 +19,29 @@ set -ex
 #############
 # UTILITIES #
 #############
+init_config() {
+    # Generate initial configuration file
+    ilab config init --non-interactive
+
+    # It looks like GitHub action MacOS runner does not have graphics
+    # so we need to disable the GPU layers if we are running in GitHub actions
+    if [[ "$(uname)" == "Darwin" ]]; then
+        if command -v system_profiler; then
+            if system_profiler SPDisplaysDataType|grep "Metal Support"; then
+                echo "Metal GPU detected"
+            else
+                echo "No Metal GPU detected"
+                sed -i.bak -e 's/gpu_layers: -1/gpu_layers: 0/g;' "${ILAB_CONFIG_FILE}"
+            fi
+        else
+            echo "system_profiler not found, cannot determine GPU"
+        fi
+    fi
+
+    # Enable Debug in func tests with debug level 1
+    sed -i.bak -e 's/log_level:.*/log_level: DEBUG/g;' "${ILAB_CONFIG_FILE}"
+    sed -i.bak -e 's/debug_level:.*/debug_level: 1/g;' "${ILAB_CONFIG_FILE}"
+}
 
 cleanup() {
     set +e
@@ -41,16 +64,16 @@ cleanup() {
         "$TEST_CTX_SIZE_LAB_CHAT_LOG_FILE" \
         test_session_history
     rm -rf test_taxonomy
-    # revert port change from test_bind_port()
-    sed -i.bak 's/9999/8000/g' "${ILAB_CONFIG_FILE}"
 
     # revert model name change from test_model_print()
-    local original_model_name='merlinite-7b-lab-Q4_K_M'
-    local original_model_filename="${original_model_name}.gguf"
+    local original_model_filename="merlinite-7b-lab-Q4_K_M.gguf"
     local temporary_model_filename='foo.gguf'
-    sed -i.bak "s/baz/${original_model_name}/g" "${ILAB_CONFIG_FILE}"
     mv "${ILAB_CACHE_DIR}/models/${temporary_model_filename}" "${ILAB_CACHE_DIR}/models/${original_model_filename}" || true
-    rm -f "${ILAB_CONFIG_FILE}.bak" serve.log "${ILAB_CACHE_DIR}/models/${temporary_model_filename}"
+    rm -f serve.log "${ILAB_CACHE_DIR}/models/${temporary_model_filename}"
+
+    # reset config file and re-init defaults
+    init_config
+
     set -e
 }
 
@@ -132,6 +155,7 @@ wait_for_server(){
     fi
 }
 
+
 #########
 # TESTS #
 #########
@@ -405,6 +429,29 @@ test_server_welcome_message(){
     fi
 }
 
+test_log_format(){
+    # remove the log 'levelname' from the log format
+    sed -i.bak -e 's/log_format:.*/log_format: "%(name)s:%(lineno)d: %(message)s"/g' "${ILAB_CONFIG_FILE}"
+    ilab model serve &> serve.log &
+    PID_SERVE=$!
+
+    wait_for_server
+
+    if ! timeout 10 bash -c '
+        until test -s serve.log; do
+            echo "waiting for server log file to be created"
+            sleep 1
+        done
+        if grep INFO serve.log; then
+            echo "INFO log level found in server log - wrong log format"
+            exit 1
+        fi
+    '; then
+        echo "server log file was not created"
+        exit 1
+    fi
+}
+
 test_model_print(){
     local src_model="${ILAB_CACHE_DIR}/models/merlinite-7b-lab-Q4_K_M.gguf"
     local target_model="${ILAB_CACHE_DIR}/models/foo.gguf"
@@ -558,27 +605,8 @@ ilab --version
 # print system information
 ilab system info
 
-# pipe 3 carriage returns to ilab config init to get past the prompts
-echo -e "\n\n\n" | ilab config init
-
-# Enable Debug in func tests with debug level 1
-sed -i.bak -e 's/log_level:.*/log_level: DEBUG/g;' "${ILAB_CONFIG_FILE}"
-sed -i.bak -e 's/debug_level:.*/debug_level: 1/g;' "${ILAB_CONFIG_FILE}"
-
-# It looks like GitHub action MacOS runner does not have graphics
-# so we need to disable the GPU layers if we are running in GitHub actions
-if [[ "$(uname)" == "Darwin" ]]; then
-    if command -v system_profiler; then
-        if system_profiler SPDisplaysDataType|grep "Metal Support"; then
-            echo "Metal GPU detected"
-        else
-            echo "No Metal GPU detected"
-            sed -i.bak -e 's/gpu_layers: -1/gpu_layers: 0/g;' "${ILAB_CONFIG_FILE}"
-        fi
-    else
-        echo "system_profiler not found, cannot determine GPU"
-    fi
-fi
+# initialize the configuration file
+init_config
 
 # download the latest version of the ilab
 ilab model download
@@ -612,5 +640,7 @@ cleanup
 test_server_welcome_message
 cleanup
 test_model_print
+cleanup
+test_log_format
 
 exit 0
diff --git a/tests/test_config.py b/tests/test_config.py
index 7604eafbb5..330d954e3a 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -217,6 +217,11 @@ def test_expand_paths(self, tmp_path_home):
             "--qlora-adapter-name-or-path=$HOME/qlora-adapter-name-or-path"
         )
 
+    def test_validate_log_format_invalid(self):
+        cfg = config.get_default_config()
+        with pytest.raises(ValueError):
+            cfg.general.validate_log_format("INVALID")
+
     def test_get_model_family(self):
         good_cases = {
             # two known families
@@ -320,11 +325,12 @@ def test_config_modified_settings(self, tmp_path_home):
     ],
 )
 def test_logging(log_level, debug_level, root, instructlab, openai_httpx):
-    configure_logging(log_level=log_level, debug_level=debug_level)
+    configure_logging(log_level=log_level, debug_level=debug_level, fmt="%(message)s")
     assert logging.getLogger("root").getEffectiveLevel() == root
     assert logging.getLogger("instructlab").getEffectiveLevel() == instructlab
     assert logging.getLogger("openai").getEffectiveLevel() == openai_httpx
     assert logging.getLogger("httpx").getEffectiveLevel() == openai_httpx
+    assert logging.getLogger("root").handlers[0].formatter._fmt == "%(message)s"
 
 
 @patch.multiple(
diff --git a/tests/testdata/default_config.yaml b/tests/testdata/default_config.yaml
index 80e007271e..7d9d6dfd67 100644
--- a/tests/testdata/default_config.yaml
+++ b/tests/testdata/default_config.yaml
@@ -83,6 +83,9 @@ general:
   # Debug level for logging.
   # Default: 0
   debug_level: 0
+  # Log format. https://docs.python.org/3/library/logging.html#logrecord-attributes
+  # Default: %(levelname)s %(asctime)s %(name)s:%(lineno)d: %(message)s
+  log_format: '%(levelname)s %(asctime)s %(name)s:%(lineno)d: %(message)s'
   # Log level for logging.
   # Default: INFO
   log_level: INFO