Repository: osl-incubator/makim
Pull request #162: feat: Add an option for tasks to specify a log file, log format and log level
Author: login: ansh808s
Labels: name: released
Created: 2025-03-07T22:48:09Z | Merged: 2025-03-13T16:47:17Z

## Pull Request description

<!-- Describe the purpose of your PR and the changes you have made. -->
This PR aims to implement logging for tasks and also provides options to format logs
and specify level for the log (err,out,both)
<!-- Which issue this PR aims to resolve or fix? E.g.:
Solving issue #004
-->
resolves #38 
## How to test these changes

<!-- Example:

* run `$ abc -p 1234`
* open the web browser with url localhost:1234
* ...
-->

```yaml
groups:
  mygroup:
    tasks:
       tasks:
         log:
           path: /tmp/log.txt
           level: err
           format: "%(asctime)s - %(file)s - %(levelname)s - %(task)s - %(message)s"
```
or
```sh
makim smoke-tests.task-logger
```

<!-- Modify the options to suit your project. -->

## Pull Request checklists

This PR is a:

- [x] new feature

About this PR:

- [x] it includes tests.
- [x] the tests are executed on CI.
- [ ] the tests generate log file(s) (path).
- [x] pre-commit hooks were executed locally.
- [x] this PR requires a project documentation update.

Author's checklist:

- [x] I have reviewed the changes and it contains no misspelling.
- [x] The code is well commented, especially in the parts that contain more
      complexity.
- [x] New and old tests passed locally.

## Additional information
<img width="703" alt="Screenshot 2025-03-08 at 4 06 02 AM" src="https://github.com/user-attachments/assets/f46fb16f-c713-41e9-93f5-e5f230b77341" />
<img width="667" alt="Screenshot 2025-03-08 at 4 06 33 AM" src="https://github.com/user-attachments/assets/605b4543-ed17-46fa-ab40-4300842e4f70" />


<!-- Add any screenshot that helps to show the changes proposed -->

<!-- Add any other extra information that would help to understand the changes proposed by this PR -->

## Reviewer's checklist

Copy and paste this template for your review's note:

```
## Reviewer's Checklist

- [ ] I managed to reproduce the problem locally from the `main` branch
- [ ] I managed to test the new changes locally
- [ ] I confirm that the issues mentioned were fixed/resolved .
```


Linked issue #38: Add an option for targets to specify a log file
It would be nice to have an option to specify a log for for a target. also, it would be nice to define the level for the log (stdout, stderr, or both). For example:

```yaml

groups:
  mygroup:
    targets:
       mytarget:
         log:
           path: /tmp/log.txt
           level: err  # err, out, both
```
Linked issue comments: user: login: xmnlab
created_at: 2024-01-30T20:06:36Z
body: some explanation about output/error redirection with bash: https://linuxize.com/post/bash-redirect-stderr-stdout/
author_association: CONTRIBUTOR
user: login: ansh808s
created_at: 2025-03-05T04:31:21Z
body: @xmnlab I would like to work on this issue :)
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-04-02T00:48:57Z
body: :tada: This issue has been resolved in version 1.25.0 :tada:

The release is available on:
- `1.25.0`
- [GitHub release](https://github.com/osl-incubator/makim/releases/tag/1.25.0)

Your **[semantic-release](https://github.com/semantic-release/semantic-release)** bot :package::rocket:
author_association: NONE

Diff:
diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml
index 66f37a4..78af95e 100644
--- a/.github/workflows/main.yaml
+++ b/.github/workflows/main.yaml
@@ -130,6 +130,9 @@ jobs:
       - name: Run smoke test for scheduler
         run: makim smoke-tests.scheduler
 
+      - name: Run smoke test for task logger
+        run: makim smoke-tests.task-logger
+
       - name: Semantic Release PR Title Check
         uses: osl-incubator/semantic-release-pr-title-check@v1.4.1
         if: success() || failure()
diff --git a/.gitignore b/.gitignore
index b6e4761..fbd0fc4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -127,3 +127,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# log tests files
+tests/smoke/logs
diff --git a/.makim.yaml b/.makim.yaml
index 1c6e803..eb03e76 100644
--- a/.makim.yaml
+++ b/.makim.yaml
@@ -13,6 +13,7 @@ groups:
           rm -fr .pytest_cache
           rm -fr .ruff_cache
           rm -fr .mypy_cache
+          rm -fr ./tests/smoke/logs/
           find . -name '*.egg-info' -exec rm -fr {} +
           find . -name '*.egg' -exec rm -f {} +
           find . -name '*.pyc' -exec rm -f {} +
@@ -89,6 +90,7 @@ groups:
             - task: smoke-tests.shell-app
             - task: smoke-tests.ssh-remote-execution
             - task: smoke-tests.scheduler
+            - task: smoke-tests.task-logger
 
       ci:
         help: Run all tasks used on CI
@@ -462,6 +464,39 @@ groups:
           makim $VERBOSE_FLAG --file $MAKIM_FILE cron stop --all
           makim $VERBOSE_FLAG --file $MAKIM_FILE cron list
 
+      task-logger:
+        help: Test makim with file logs
+        args:
+          verbose-mode:
+            help: Run the all the tests in verbose mode
+            type: bool
+            action: store_true
+        env:
+          MAKIM_FILE: tests/smoke/.makim-file-log.yaml
+        backend: bash
+        run: |
+          export VERBOSE_FLAG='${{ "--verbose" if args.verbose_mode else "" }}'
+          makim $VERBOSE_FLAG --file $MAKIM_FILE --help
+          makim $VERBOSE_FLAG --file $MAKIM_FILE --version
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.out-no-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.err-no-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.both-no-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.out-file-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.err-file-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.both-file-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.out-task-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.err-task-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.both-task-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.out-level-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.err-level-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.both-level-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.out-message-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.err-message-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.both-message-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.out-time-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.err-time-format
+          makim $VERBOSE_FLAG --file $MAKIM_FILE test.both-time-format
+
   error:
     help: This group helps tests failure tasks
     tasks:
diff --git a/src/makim/core.py b/src/makim/core.py
index a690bbc..aee4b29 100644
--- a/src/makim/core.py
+++ b/src/makim/core.py
@@ -15,7 +15,7 @@
 from copy import deepcopy
 from itertools import product
 from pathlib import Path
-from typing import Any, Dict, List, Optional, TypedDict, Union, cast
+from typing import Any, Dict, List, Optional, TextIO, TypedDict, Union, cast
 
 import dotenv
 import paramiko
@@ -29,6 +29,7 @@
 from makim.console import get_terminal_size
 from makim.logs import MakimError, MakimLogs
 from makim.scheduler import MakimScheduler
+from makim.task_logging import FormattedLogStream, LogLevel, Tee
 
 
 def command_exists(command: str) -> bool:
@@ -184,7 +185,12 @@ def __getstate__(self) -> Dict[str, Any]:
             state['scheduler'] = None
         return state
 
-    def _call_shell_app(self, cmd: str) -> None:
+    def _call_shell_app(
+        self,
+        cmd: str,
+        out_stream: TextIO = sys.stdout,
+        err_stream: TextIO = sys.stderr,
+    ) -> None:
         self._load_shell_app()
 
         fd, filepath = tempfile.mkstemp(suffix=self.tmp_suffix, text=True)
@@ -196,8 +202,8 @@ def _call_shell_app(self, cmd: str) -> None:
             *self.shell_args,
             filepath,
             _in=sys.stdin,
-            _out=sys.stdout,
-            _err=sys.stderr,
+            _out=out_stream,
+            _err=err_stream,
             _bg=True,
             _bg_exc=False,
             _no_err=True,
@@ -867,7 +873,8 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
                         current_cmd, cast(dict[str, Any], host_config)
                     )
                 else:
-                    self._call_shell_app(current_cmd)
+                    out_stream, err_stream = self._get_output_stream()
+                    self._call_shell_app(current_cmd, out_stream, err_stream)
 
         # Run command for each matrix combination
         for matrix_vars in matrix_combinations or [{}]:
@@ -877,6 +884,55 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
         os.environ.clear()
         os.environ.update(self.env_scoped)
 
+    def _get_output_stream(self) -> tuple[TextIO, TextIO]:
+        """Set up logging streams based on task log configuration."""
+        log_config = self.task_data.get('log')
+
+        if not log_config:
+            return sys.stdout, sys.stderr
+
+        log_path = log_config.get('path')
+        log_level = LogLevel(log_config.get('level', 'both'))
+        log_format = log_config.get('format')
+
+        # Ensure log directory exists
+        log_dir = Path(log_path).parent
+        log_dir.mkdir(parents=True, exist_ok=True)
+
+        file_stream = open(log_path, 'a')
+
+        # Create default streams
+        stdout_stream = sys.stdout
+        stderr_stream = sys.stderr
+
+        def create_formatted_stream(level: LogLevel) -> FormattedLogStream:
+            return FormattedLogStream(
+                file_stream,
+                log_format,
+                level,
+                self.task_name,
+                self.file,
+            )
+
+        # Configure output stream based on log level
+        if log_level in (LogLevel.OUT, LogLevel.BOTH):
+            if log_format:
+                stdout_stream = create_formatted_stream(LogLevel.OUT)
+            else:
+                stdout_stream = file_stream
+
+        # Configure error stream based on log level
+        if log_level in (LogLevel.ERR, LogLevel.BOTH):
+            if log_format:
+                stderr_stream = create_formatted_stream(LogLevel.ERR)
+            else:
+                stderr_stream = file_stream
+
+        return (
+            cast(TextIO, Tee(sys.stdout, stdout_stream)),
+            cast(TextIO, Tee(sys.stderr, stderr_stream)),
+        )
+
     # public methods
     def load(
         self,
diff --git a/src/makim/schema.json b/src/makim/schema.json
index c484dfc..c8c3439 100644
--- a/src/makim/schema.json
+++ b/src/makim/schema.json
@@ -259,6 +259,26 @@
                           }
                         }
                       }
+                    },
+                    "log": {
+                      "type": "object",
+                      "description": "Logging configuration for the task.",
+                      "properties": {
+                        "path": {
+                          "type": "string",
+                          "description": "File path to store logs."
+                        },
+                        "level": {
+                          "type": "string",
+                          "description": "Specifies which stream to log. Options: 'err' for stderr, 'out' for stdout, 'both' for both.",
+                          "enum": ["err", "out", "both"]
+                        },
+                        "format": {
+                          "type": "string",
+                          "description": "Optional log format string (e.g., '%(levelname)s - %(asctime)s - %(message)s')."
+                        }
+                      },
+                      "additionalProperties": false
                     }
                   },
                   "additionalProperties": false
diff --git a/src/makim/task_logging.py b/src/makim/task_logging.py
new file mode 100644
index 0000000..1bd9bcb
--- /dev/null
+++ b/src/makim/task_logging.py
@@ -0,0 +1,129 @@
+"""Task Logging Module for Makimm."""
+
+import datetime
+
+from enum import Enum
+from typing import Any, TextIO
+
+
+class LogLevel(Enum):
+    """Enum of log levels."""
+
+    ERR = 'err'
+    OUT = 'out'
+    BOTH = 'both'
+
+
+class Tee:
+    """A stream wrapper that writes data to multiple streams simultaneously."""
+
+    def __init__(self, *streams: TextIO) -> None:
+        """
+        Initialize the Tee stream with output streams.
+
+        Parameters
+        ----------
+        streams : TextIO
+            Streams where data will be appended.
+        """
+        self.streams: tuple[TextIO, ...] = streams
+
+    def write(self, data: Any) -> None:
+        """
+        Write data to all given streams.
+
+        Parameters
+        ----------
+        data : str or bytes
+            The data to be written.
+        """
+        # Convert data bytes to utf-8
+        if isinstance(data, bytes):
+            data = data.decode('utf-8', errors='replace')
+
+        for stream in self.streams:
+            stream.write(data)
+
+    def flush(self) -> None:
+        """Flush all given streams."""
+        for stream in self.streams:
+            stream.flush()
+
+
+class FormattedLogStream:
+    """A stream wrapper that formats log messages based on format string."""
+
+    def __init__(
+        self,
+        stream: TextIO,
+        format: str,
+        level: LogLevel,
+        task: str,
+        file: str,
+    ) -> None:
+        """
+        Initialize the FormattedLogStream.
+
+        Parameters
+        ----------
+        stream : TextIO
+            The file stream to append logs.
+        format : str
+            The log format string (example, %(asctime)s - %(message)s").
+        level : LogLevel
+            The log level as specified in the task configuration.
+        task : str
+            The task from where logs are streamed.
+        file : str
+            The file where task is defined.
+        """
+        self.stream = stream
+        self.format = format
+        self.level = level.value.upper()
+        self.task = task
+        self.file = file
+        self._buffer = ''
+
+    def write(self, data: Any) -> None:
+        """
+        Write formatted log records.
+
+        Parameters
+        ----------
+        data : str or bytes
+            The data to write.
+        """
+        # Convert data bytes to utf-8
+        if isinstance(data, bytes):
+            data = data.decode('utf-8', errors='replace')
+        self._buffer += data
+        while '\n' in self._buffer:
+            text, self._buffer = self._buffer.split('\n', 1)
+            record = {
+                'asctime': datetime.datetime.now().strftime(
+                    '%Y-%m-%d %H:%M:%S'
+                ),
+                'task': self.task,
+                'file': self.file,
+                'levelname': self.level,
+                'message': text,
+            }
+            formatted_line = self.format % record
+            self.stream.write(formatted_line + '\n')
+
+    def flush(self) -> None:
+        """Send any pending text to the given stream."""
+        if not self._buffer:
+            self.stream.flush()
+            return
+
+        record = {
+            'asctime': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+            'task': self.task,
+            'file': self.file,
+            'levelname': self.level,
+            'message': self._buffer,
+        }
+        formatted_line = self.format % record
+        self.stream.write(formatted_line + '\n')
+        self._buffer = ''
diff --git a/tests/smoke/.makim-file-log.yaml b/tests/smoke/.makim-file-log.yaml
new file mode 100644
index 0000000..83aa96b
--- /dev/null
+++ b/tests/smoke/.makim-file-log.yaml
@@ -0,0 +1,395 @@
+groups:
+  setup:
+    tasks:
+      out-no-format:
+        help: Logging with level "out".
+        log:
+          path: ./tests/smoke/logs/out-no-format.txt
+          level: out
+        run: |
+          echo "out"
+
+      err-no-format:
+        help: Logging with level "err".
+        log:
+          path: ./tests/smoke/logs/err-no-format.txt
+          level: err
+        run: |
+          import sys
+          sys.stderr.write("err")
+
+      both-no-format:
+        help: Logging with level "both".
+        log:
+          path: ./tests/smoke/logs/both-no-format.txt
+          level: both
+        run: |
+          import sys
+          echo both
+          sys.stderr.write("both")
+
+      out-file-format:
+        help: Logging with level "out" with file formatting.
+        log:
+          path: ./tests/smoke/logs/out-file-format.txt
+          level: out
+          format: "%(file)s"
+        run: |
+          echo "out"
+
+      err-file-format:
+        help: Logging with level "err" with file formatting.
+        log:
+          path: ./tests/smoke/logs/err-file-format.txt
+          level: err
+          format: "%(file)s"
+        run: |
+          import sys
+          sys.stderr.write("err")
+
+      both-file-format:
+        help: Logging with level "both" with file formatting.
+        log:
+          path: ./tests/smoke/logs/both-file-format.txt
+          level: both
+          format: "%(file)s"
+        run: |
+          import sys
+          echo both
+          sys.stderr.write("both")
+
+      out-task-format:
+        help: Logging with level "out" with task formatting.
+        log:
+          path: ./tests/smoke/logs/out-task-format.txt
+          level: out
+          format: "%(task)s"
+        run: |
+          echo "out"
+
+      err-task-format:
+        help: Logging with level "err" with task formatting.
+        log:
+          path: ./tests/smoke/logs/err-task-format.txt
+          level: err
+          format: "%(task)s"
+        run: |
+          import sys
+          sys.stderr.write("err")
+
+      both-task-format:
+        help: Logging with level "both" with task formatting.
+        log:
+          path: ./tests/smoke/logs/both-task-format.txt
+          level: both
+          format: "%(task)s"
+        run: |
+          import sys
+          echo both
+          sys.stderr.write("both")
+
+      out-level-format:
+        help: Logging with level "out" with level formatting.
+        log:
+          path: ./tests/smoke/logs/out-level-format.txt
+          level: out
+          format: "%(levelname)s"
+        run: |
+          echo "out"
+
+      err-level-format:
+        help: Logging with level "err" with level formatting.
+        log:
+          path: ./tests/smoke/logs/err-level-format.txt
+          level: err
+          format: "%(levelname)s"
+        run: |
+          import sys
+          sys.stderr.write("err")
+
+      both-level-format:
+        help: Logging with level "both" with level formatting.
+        log:
+          path: ./tests/smoke/logs/both-level-format.txt
+          level: both
+          format: "%(levelname)s"
+        run: |
+          import sys
+          echo both
+          sys.stderr.write("both")
+
+      out-message-format:
+        help: Logging with level "out" with message formatting.
+        log:
+          path: ./tests/smoke/logs/out-message-format.txt
+          level: out
+          format: "%(message)s"
+        run: |
+          echo "out"
+
+      err-message-format:
+        help: Logging with level "err" with message formatting.
+        log:
+          path: ./tests/smoke/logs/err-message-format.txt
+          level: err
+          format: "%(message)s"
+        run: |
+          import sys
+          sys.stderr.write("err")
+
+      both-message-format:
+        help: Logging with level "both" with message formatting.
+        log:
+          path: ./tests/smoke/logs/both-message-format.txt
+          level: both
+          format: "%(message)s"
+        run: |
+          import sys
+          echo both
+          sys.stderr.write("both")
+
+      out-time-format:
+        help: Logging with level "out" with time formatting.
+        log:
+          path: ./tests/smoke/logs/out-time-format.txt
+          level: out
+          format: "%(asctime)s"
+        run: |
+          echo "out"
+
+      err-time-format:
+        help: Logging with level "err" with time formatting.
+        log:
+          path: ./tests/smoke/logs/err-time-format.txt
+          level: err
+          format: "%(asctime)s"
+        run: |
+          import sys
+          sys.stderr.write("err")
+
+      both-time-format:
+        help: Logging with level "both" with time formatting.
+        log:
+          path: ./tests/smoke/logs/both-time-format.txt
+          level: both
+          format: "%(asctime)s"
+        run: |
+          import sys
+          echo both
+          sys.stderr.write("both")
+
+  test:
+    tasks:
+      out-no-format:
+        help: Test logging with level "out".
+        hooks:
+          pre-run:
+            - task: setup.out-no-format
+        run: |
+          file_path = "./tests/smoke/logs/out-no-format.txt"
+          word_to_search = "out"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(word_to_search in line for line in file), "Log entry not found"
+
+      err-no-format:
+        help: Test logging with level "err".
+        hooks:
+          pre-run:
+            - task: setup.err-no-format
+        run: |
+          file_path = "./tests/smoke/logs/err-no-format.txt"
+          word_to_search = "err"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(word_to_search in line for line in file), "Log entry not found"
+
+      both-no-format:
+        help: Test logging with level "both".
+        hooks:
+          pre-run:
+            - task: setup.both-no-format
+        run: |
+          file_path = "./tests/smoke/logs/both-no-format.txt"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              contents = file.read()
+              assert "both" in contents, "Both logs not found"
+
+      out-file-format:
+        help: Test logging with file format.
+        hooks:
+          pre-run:
+            - task: setup.out-file-format
+        run: |
+          file_path = "./tests/smoke/logs/out-file-format.txt"
+          expected_format = "tests/smoke/.makim-file-log.yaml"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_format in line for line in file), "File format logging failed"
+
+      err-file-format:
+        help: Test logging with file format for error level.
+        hooks:
+          pre-run:
+            - task: setup.err-file-format
+        run: |
+          file_path = "./tests/smoke/logs/err-file-format.txt"
+          expected_format = "tests/smoke/.makim-file-log.yaml"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_format in line for line in file), "File format logging failed for error"
+
+      both-file-format:
+        help: Test logging with level "both" with file formatting.
+        hooks:
+          pre-run:
+            - task: setup.both-file-format
+        run: |
+          file_path = "./tests/smoke/logs/both-file-format.txt"
+          expected_format = "tests/smoke/.makim-file-log.yaml"
+          with open(file_path, 'r', encoding='utf-8') as file:
+            assert any(expected_format in line for line in file), "Expected file format log entry not found!"
+
+      out-task-format:
+        help: Test logging with task format.
+        hooks:
+          pre-run:
+            - task: setup.out-task-format
+        run: |
+          file_path = "./tests/smoke/logs/out-task-format.txt"
+          expected_task = "out-task-format"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              log_lines = [line.strip() for line in file]
+              assert any(expected_task in line for line in log_lines),"Task format logging failed."
+
+      err-task-format:
+        help: Test logging with task format for error level.
+        hooks:
+          pre-run:
+            - task: setup.err-task-format
+        run: |
+          file_path = "./tests/smoke/logs/err-task-format.txt"
+          expected_task = "err-task-format"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              log_lines = [line.strip() for line in file]
+              assert any(expected_task in line for line in log_lines), "Task format logging failed for error."
+
+      both-task-format:
+        help: Test logging with task format for both levels.
+        hooks:
+          pre-run:
+            - task: setup.both-task-format
+        run: |
+          file_path = "./tests/smoke/logs/both-task-format.txt"
+          expected_task = "both-task-format"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              log_lines = [line.strip() for line in file]
+              assert any(expected_task in line for line in log_lines), f"Task format logging failed for both. Log contents: {log_lines}"
+
+      out-level-format:
+        help: Test logging with level format.
+        hooks:
+          pre-run:
+            - task: setup.out-level-format
+        run: |
+          file_path = "./tests/smoke/logs/out-level-format.txt"
+          expected_level = "OUT"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_level in line for line in file), "Level format logging failed"
+
+      err-level-format:
+        help: Test logging with level format for error level.
+        hooks:
+          pre-run:
+            - task: setup.err-level-format
+        run: |
+          file_path = "./tests/smoke/logs/err-level-format.txt"
+          expected_level = "ERR"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_level in line for line in file), "Level format logging failed for error"
+
+      both-level-format:
+        help: Test logging with level format for both levels.
+        hooks:
+          pre-run:
+            - task: setup.both-level-format
+        run: |
+          file_path = "./tests/smoke/logs/both-level-format.txt"
+          expected_levels = ["OUT", "ERR"]
+          with open(file_path, 'r', encoding='utf-8') as file:
+              contents = file.read()
+              assert all(level in contents for level in expected_levels), "Level format logging failed for both"
+
+      out-message-format:
+        help: Test logging with message format.
+        hooks:
+          pre-run:
+            - task: setup.out-message-format
+        run: |
+          file_path = "./tests/smoke/logs/out-message-format.txt"
+          expected_message = "out"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_message in line for line in file), "Message format logging failed"
+
+      err-message-format:
+        help: Test logging with message format for error level.
+        hooks:
+          pre-run:
+            - task: setup.err-message-format
+        run: |
+          file_path = "./tests/smoke/logs/err-message-format.txt"
+          expected_message = "err"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_message in line for line in file), "Message format logging failed for error"
+
+      both-message-format:
+        help: Test logging with message format for both levels.
+        hooks:
+          pre-run:
+            - task: setup.both-message-format
+        run: |
+          file_path = "./tests/smoke/logs/both-message-format.txt"
+          expected_message = "both"
+          with open(file_path, 'r', encoding='utf-8') as file:
+              assert any(expected_message in line for line in file), "Message format logging failed for both"
+
+      out-time-format:
+        help: Test logging with timestamp format.
+        hooks:
+          pre-run:
+            - task: setup.out-time-format
+        run: |
+          import datetime
+          import re
+          file_path = "./tests/smoke/logs/out-time-format.txt"
+
+          with open(file_path, 'r', encoding='utf-8') as file:
+              log_lines = [line.strip() for line in file if line.strip()]
+          timestamp_line = next((line for line in log_lines if re.match(r"^\d{4}-\d{2}-\d{2}", line)), None)
+          assert timestamp_line, "No timestamp found in the log file."
+
+      err-time-format:
+        help: Test logging with timestamp format for error level.
+        hooks:
+          pre-run:
+            - task: setup.err-time-format
+        run: |
+          import datetime
+          import re
+          file_path = "./tests/smoke/logs/err-time-format.txt"
+
+          with open(file_path, 'r', encoding='utf-8') as file:
+              log_lines = [line.strip() for line in file if line.strip()]
+          timestamp_line = next((line for line in log_lines if re.match(r"^\d{4}-\d{2}-\d{2}", line)), None)
+          assert timestamp_line, "No timestamp found in the log file."
+
+      both-time-format:
+        help: Test logging with timestamp format for both levels.
+        hooks:
+          pre-run:
+            - task: setup.both-time-format
+        run: |
+          import datetime
+          import re
+          file_path = "./tests/smoke/logs/both-time-format.txt"
+
+          with open(file_path, 'r', encoding='utf-8') as file:
+              log_lines = [line.strip() for line in file if line.strip()]
+          timestamp_line = next((line for line in log_lines if re.match(r"^\d{4}-\d{2}-\d{2}", line)), None)
+          assert timestamp_line, "No timestamp found in the log file."


Reviews:
user: login: xmnlab
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T17:02:39Z
user: login: xmnlab
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T17:07:51Z
user: login: xmnlab
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T17:10:01Z
user: login: xmnlab
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T17:14:09Z
user: login: ansh808s
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-09T04:54:42Z
user: login: ansh808s
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T12:31:48Z
user: login: ansh808s
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T12:36:33Z
user: login: xmnlab
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T19:03:12Z
user: login: xmnlab
body: thanks  @ansh808s !

I just added a very small comment here.

I will test it locally as well .. and maybe I will add a few more comments soon
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T19:04:25Z
user: login: xmnlab
body: hi @ansh808s , thanks for working on that.

in the .makim-file-log.yaml, there are tests for err, out, and both. 
probably, the tests for `both` should have an output and an error as well. 
currently they have an echo.

btw I tested a "both" test, but the output wasn't present in the file, it just had the time printed in the file, for example:

```bash
20:08 $ makim --file tests/smoke/.makim-file-log.yaml test.both-time-format
 DEFAULT SHELL: xonsh
Makim file: tests/smoke/.makim-file-log.yaml
both
(makim) ✘-INT ~/dev/osl-projects/osl-incubator/makim [feat/task-log L|…1] 
20:11 $ cat tests/smoke/makim_log.txt 
2025-03-12 20:09:01
```

maybe you should add a separated test to check if the content of the log file is correct.


additionally, the log path is set to `./tests/smoke/makim_log.txt` probably it should be something like `./tests/smoke/logs/<the name of the task>.txt` and add `tests/smoke/logs` to the .gitignore
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T20:20:07Z
user: login: ansh808s
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T03:24:56Z
user: login: xmnlab
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T16:16:43Z
user: login: xmnlab
body: LGTM! thanks @ansh808s for working on that.
great work.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T16:18:45Z

Review comments:
diff_hunk: @@ -13,6 +13,7 @@ groups:
           rm -fr .pytest_cache
           rm -fr .ruff_cache
           rm -fr .mypy_cache
+          rm ./tests/smoke/makim_log.txt
path: .makim.yaml
user: login: xmnlab
body: ```suggestion
          rm -f ./tests/smoke/makim_log.txt
```
`-f` prevents to fail if the file doesn't exist
created_at: 2025-03-08T17:02:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -866,7 +872,7 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
                         current_cmd, cast(dict[str, Any], host_config)
                     )
                 else:
-                    self._call_shell_app(current_cmd)
+                    self._execute_task_with_log(current_cmd)
path: src/makim/core.py
user: login: xmnlab
body: maybe instead of creating this wrapper `_execute_task_with_log` .. we could keep using just `_call_shell_app`.
and inside `_call_shell_app`, it could call a function `_get_output_stream` that return both out and err streams.

what do you think?
created_at: 2025-03-08T17:07:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -866,7 +872,7 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
                         current_cmd, cast(dict[str, Any], host_config)
                     )
                 else:
-                    self._call_shell_app(current_cmd)
+                    self._execute_task_with_log(current_cmd)
path: src/makim/core.py
user: login: xmnlab
body: so `_execute_task_with_log` will be renamed to `_get_output_stream` .. and it will return the err and out streams for files or pointing to the sys objects.
created_at: 2025-03-08T17:10:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -876,6 +882,95 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
         os.environ.clear()
         os.environ.update(self.env_scoped)
 
+    def _execute_task_with_log(self, current_cmd: Any) -> None:
+        """Execute task using the custom logging configuration."""
+        log_config = self.task_data.get('log')
+        if log_config:
path: src/makim/core.py
user: login: xmnlab
body: please avoid unnecessary nesting blocks
take a look here fore more details: https://www.youtube.com/watch?v=CFRhGnuXG-4
created_at: 2025-03-08T17:14:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -13,6 +13,7 @@ groups:
           rm -fr .pytest_cache
           rm -fr .ruff_cache
           rm -fr .mypy_cache
+          rm ./tests/smoke/makim_log.txt
path: .makim.yaml
user: login: ansh808s
body: > `-f` prevents to fail if the file doesn't exist


that makes sense, I've made the change, thanks for the suggestion!
created_at: 2025-03-09T04:54:42Z
author_association: MEMBER
diff_hunk: @@ -866,7 +872,7 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
                         current_cmd, cast(dict[str, Any], host_config)
                     )
                 else:
-                    self._call_shell_app(current_cmd)
+                    self._execute_task_with_log(current_cmd)
path: src/makim/core.py
user: login: ansh808s
body: > maybe instead of creating this wrapper `_execute_task_with_log` .. we could keep using just `_call_shell_app`. and inside `_call_shell_app`, it could call a function `_get_output_stream` that return both out and err streams.
> 
> what do you think?

yeah this code flow looks more readable and better i have refactored it accordingly, thanks for pointing it out
created_at: 2025-03-10T12:31:48Z
author_association: MEMBER
diff_hunk: @@ -876,6 +882,95 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
         os.environ.clear()
         os.environ.update(self.env_scoped)
 
+    def _execute_task_with_log(self, current_cmd: Any) -> None:
+        """Execute task using the custom logging configuration."""
+        log_config = self.task_data.get('log')
+        if log_config:
path: src/makim/core.py
user: login: ansh808s
body: thanks for the video guidance appreciate it!, I tried and and reduced the nesting to two block levels please let me know if it still requires some refactoring or structure :)
created_at: 2025-03-10T12:36:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,129 @@
+"""Task Logging Module for Makimm."""
+
+import datetime
+
+from enum import Enum
+from typing import Any, TextIO
+
+
+class LogLevel(Enum):
+    """Enum of log levels."""
+
+    ERR = 'err'
+    OUT = 'out'
+    BOTH = 'both'
+
+
+class Tee:
+    """A stream wrapper that writes data to multiple streams simultaneously."""
+
+    def __init__(self, *streams: TextIO) -> None:
+        """
+        Initialize the Tee stream with output streams.
+
+        Parameters
+        ----------
+        streams : TextIO
+            Streams where data will be appended.
+        """
+        self.streams: tuple[TextIO, ...] = streams
+
+    def write(self, data: Any) -> None:
+        """
+        Write data to all given streams.
+
+        Parameters
+        ----------
+        data : str or bytes
+            The data to be written.
+        """
+        # Convert data bytes to utf-8
+        if isinstance(data, bytes):
+            data = data.decode('utf-8', errors='replace')
+
+        for stream in self.streams:
+            stream.write(data)
+
+    def flush(self) -> None:
+        """Flush all given streams."""
+        for stream in self.streams:
+            stream.flush()
+
+
+class FormattedLogStream:
+    """A stream wrapper that formats log messages based on format string."""
+
+    def __init__(
+        self,
+        stream: TextIO,
+        format: str,
+        level: LogLevel,
+        task: str,
+        file: str,
+    ) -> None:
+        """
+        Initialize the FormattedLogStream.
+
+        Parameters
+        ----------
+        stream : TextIO
+            The file stream to append logs.
+        format : str
+            The log format string (example, %(asctime)s - %(message)s").
+        level : LogLevel
+            The log level as specified in the task configuration.
+        task : str
+            The task from where logs are streamed.
+        file : str
+            The file where task is defined
+        """
+        self.stream = stream
+        self.format = format
+        self.level = level.value.upper()
+        self.task = task
+        self.file = file
+        self._buffer = ''
+
+    def write(self, data: Any) -> None:
+        """
+        Write formatted log records.
+
+        Parameters
+        ----------
+        data : str or bytes
+            The data to write.
+        """
+        # Convert data bytes to utf-8
+        if isinstance(data, bytes):
+            data = data.decode('utf-8', errors='replace')
+        self._buffer += data
+        while '\n' in self._buffer:
+            text, self._buffer = self._buffer.split('\n', 1)
+            record = {
+                'asctime': datetime.datetime.now().strftime(
+                    '%Y-%m-%d %H:%M:%S'
+                ),
+                'task': self.task,
+                'file': self.file,
+                'levelname': self.level,
+                'message': text,
+            }
+            formatted_line = self.format % record
+            self.stream.write(formatted_line + '\n')
+
+    def flush(self) -> None:
+        """Send any pending text to the given stream."""
+        if self._buffer:
path: src/makim/task_logging.py
user: login: xmnlab
body: ```suggestion
        if not self._buffer:
            self.stream.flush()
            return
```

sorry, I am a never nester XD
created_at: 2025-03-12T19:03:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,129 @@
+"""Task Logging Module for Makimm."""
+
+import datetime
+
+from enum import Enum
+from typing import Any, TextIO
+
+
+class LogLevel(Enum):
+    """Enum of log levels."""
+
+    ERR = 'err'
+    OUT = 'out'
+    BOTH = 'both'
+
+
+class Tee:
+    """A stream wrapper that writes data to multiple streams simultaneously."""
+
+    def __init__(self, *streams: TextIO) -> None:
+        """
+        Initialize the Tee stream with output streams.
+
+        Parameters
+        ----------
+        streams : TextIO
+            Streams where data will be appended.
+        """
+        self.streams: tuple[TextIO, ...] = streams
+
+    def write(self, data: Any) -> None:
+        """
+        Write data to all given streams.
+
+        Parameters
+        ----------
+        data : str or bytes
+            The data to be written.
+        """
+        # Convert data bytes to utf-8
+        if isinstance(data, bytes):
+            data = data.decode('utf-8', errors='replace')
+
+        for stream in self.streams:
+            stream.write(data)
+
+    def flush(self) -> None:
+        """Flush all given streams."""
+        for stream in self.streams:
+            stream.flush()
+
+
+class FormattedLogStream:
+    """A stream wrapper that formats log messages based on format string."""
+
+    def __init__(
+        self,
+        stream: TextIO,
+        format: str,
+        level: LogLevel,
+        task: str,
+        file: str,
+    ) -> None:
+        """
+        Initialize the FormattedLogStream.
+
+        Parameters
+        ----------
+        stream : TextIO
+            The file stream to append logs.
+        format : str
+            The log format string (example, %(asctime)s - %(message)s").
+        level : LogLevel
+            The log level as specified in the task configuration.
+        task : str
+            The task from where logs are streamed.
+        file : str
+            The file where task is defined
+        """
+        self.stream = stream
+        self.format = format
+        self.level = level.value.upper()
+        self.task = task
+        self.file = file
+        self._buffer = ''
+
+    def write(self, data: Any) -> None:
+        """
+        Write formatted log records.
+
+        Parameters
+        ----------
+        data : str or bytes
+            The data to write.
+        """
+        # Convert data bytes to utf-8
+        if isinstance(data, bytes):
+            data = data.decode('utf-8', errors='replace')
+        self._buffer += data
+        while '\n' in self._buffer:
+            text, self._buffer = self._buffer.split('\n', 1)
+            record = {
+                'asctime': datetime.datetime.now().strftime(
+                    '%Y-%m-%d %H:%M:%S'
+                ),
+                'task': self.task,
+                'file': self.file,
+                'levelname': self.level,
+                'message': text,
+            }
+            formatted_line = self.format % record
+            self.stream.write(formatted_line + '\n')
+
+    def flush(self) -> None:
+        """Send any pending text to the given stream."""
+        if self._buffer:
path: src/makim/task_logging.py
user: login: ansh808s
body: I will try to become one soon XD...I will update it
created_at: 2025-03-13T03:24:56Z
author_association: MEMBER
diff_hunk: @@ -877,6 +884,56 @@ def process_matrix_combination(matrix_vars: dict[str, Any]) -> None:
         os.environ.clear()
         os.environ.update(self.env_scoped)
 
+    def _get_output_stream(self) -> tuple[TextIO, TextIO]:
+        """Set up logging streams based on task log configuration."""
+        log_config = self.task_data.get('log')
+
+        if not log_config:
+            return sys.stdout, sys.stderr
+
+        log_path = log_config.get('path')
+        log_level = LogLevel(log_config.get('level', 'both'))
+        log_format = log_config.get('format')
+
+        # Ensure log directory exists
+        log_dir = Path(log_path).parent
+        log_dir.mkdir(parents=True, exist_ok=True)
+
+        file_stream = open(log_path, 'a')
+
+        # Create default streams
+        stdout_stream = sys.stdout
+        stderr_stream = sys.stderr
+
+        def create_formatted_stream(level: LogLevel) -> FormattedLogStream:
+            return FormattedLogStream(
+                file_stream,
+                log_format,
+                level,
+                self.task_name,
+                self.file,
+            )
+
+        # Configure output stream based on log level
+        if log_level in (LogLevel.OUT, LogLevel.BOTH):
+            if log_format:
+                stdout_stream = create_formatted_stream(LogLevel.OUT)
+            else:
+                stdout_stream = file_stream
+
+        # Configure error stream based on log level
+        if log_level in (LogLevel.ERR, LogLevel.BOTH):
+            if log_format:
+                stderr_stream = create_formatted_stream(LogLevel.ERR)
+            else:
+                stderr_stream = file_stream
+
+            # Wrap original streams with configured streams
path: src/makim/core.py
user: login: xmnlab
body: ```suggestion
```
created_at: 2025-03-13T16:16:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: ansh808s
created_at: 2025-03-13T03:51:30Z
body: > hi @ansh808s , thanks for working on that.





> btw I tested a "both" test, but the output wasn't present in the file, it just had the time printed in the file, for example:
> 
> ```shell
> 20:08 $ makim --file tests/smoke/.makim-file-log.yaml test.both-time-format
>  DEFAULT SHELL: xonsh
> Makim file: tests/smoke/.makim-file-log.yaml
> both
> (makim) ✘-INT ~/dev/osl-projects/osl-incubator/makim [feat/task-log L|…1] 
> 20:11 $ cat tests/smoke/makim_log.txt 
> 2025-03-12 20:09:01
> ```
Hello @xmnlab  as you can see in the test.both-time-format
```yaml
groups:
  setup:
    tasks:
      both-time-format:
        help: Logging with level "both" with time formatting.
        log:
          path: ./tests/smoke/makim_log.txt
          level: both
          format: "%(asctime)s"
        run: |
          echo "both"
   tests:
     tasks:
       both-time-format:
            help: Test logging with timestamp format for both levels.
            hooks:
              pre-run:
                - task: setup.both-time-format
            run: |
              import datetime
              import re
              file_path = "./tests/smoke/makim_log.txt"
              current_year = str(datetime.datetime.now().year)
    
              with open(file_path, 'r', encoding='utf-8') as file:
                  log_lines = [line.strip() for line in file if line.strip()]
              timestamp_line = next((line for line in log_lines if re.match(r"^\d{4}-\d{2}-\d{2}", line)), None)
              assert timestamp_line, "No timestamp found in the log file."
   
```
If we see the `format` only `asctime` is present, `message` isn't there so the output wouldn't be there only the items present in `format` would be there in the log file, if this sounds counter-intuitive and the output log should always be there even if it isn't mentioned in format, please let me know I will add it



> 
> maybe you should add a separated test to check if the content of the log file is correct.
> 
also there are separate tests to check the log file content are correct or not this is the structure of tests:

```yaml
groups:
  setup:
      tasks:
        out-no-format:
          help: Logging with level "out".
          log:
            path: ./tests/smoke/makim_log.txt
            level: out
          run: |
            echo "out"
          
  test:
      tasks:
        out-no-format:
          help: Test logging with level "out".
          hooks:
            pre-run:
              - task: setup.out-no-format
          run: |
            file_path = "./tests/smoke/makim_log.txt"
            word_to_search = "out"
            with open(file_path, 'r', encoding='utf-8') as file:
                assert any(word_to_search in line for line in file), "Log entry not found"
```
Is it good or you are suggesting something else please let me know

> in the .makim-file-log.yaml, there are tests for err, out, and both. probably, the tests for `both` should have an output and an error as well. currently they have an echo.

okay I will surely add `stderr` output as well, thanks


> additionally, the log path is set to `./tests/smoke/makim_log.txt` probably it should be something like `./tests/smoke/logs/<the name of the task>.txt` and add `tests/smoke/logs` to the .gitignore

Sure I will add these changes

Thanks for the review! 😄 (I accidentally requested for re-review I will make the changes and let you know :> )

author_association: MEMBER
user: login: review-notebook-app[bot]
created_at: 2025-03-13T06:42:45Z
body: Check out this pull request on&nbsp; <a href="https://app.reviewnb.com/osl-incubator/makim/pull/162"><img align="absmiddle"  alt="ReviewNB" height="28" class="BotMessageButtonImage" src="https://raw.githubusercontent.com/ReviewNB/support/master/images/button_reviewnb.png"/></a> 

 See visual diffs & provide feedback on Jupyter Notebooks. 

---

 <i>Powered by <a href='https://www.reviewnb.com/?utm_source=gh'>ReviewNB</a></i>
author_association: NONE
user: login: xmnlab
created_at: 2025-03-13T16:04:29Z
body: @ansh808s , got it, thanks! I misunderstood the tests. 
I will take a look into that again!
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-02T00:48:36Z
body: :tada: This PR is included in version 1.25.0 :tada:

The release is available on:
- `1.25.0`
- [GitHub release](https://github.com/osl-incubator/makim/releases/tag/1.25.0)

Your **[semantic-release](https://github.com/semantic-release/semantic-release)** bot :package::rocket:
author_association: NONE