Repository: aimhubio/aim

Issue:
Extend `aim.ext.tensorboard_tracker.run.Run` to also allow system stats, parameters, and stdout capture.
## 🚀 Feature

Allow capturing of system parameters and terminal logs by the `aim.ext.tensorboard_tracker.run.Run`, as this is great feature shouldn't be only available to the default `Run`.

### Motivation

The new feature of allowing continuous syncing from `tensorboard` files to `aim` is really nice, but because `aim.ext.tensorboard_tracker.run.Run` inherits from `BasicRun` rather than `Run`, it misses out on the ability to log the standard out, system stats and system parameters. Since `aim.ext.tensorboard_tracker.run.Run` should be a possible replacement for `Run`, I don't see a reason why this behaviour shouldn't be allowed.

It has been highlighted in Discord by @mihran113:

> The reason behind inheriting from basic run is exactly to avoid terminal log tracking and system param tracking actually, cause we don’t want to add anything else rather than what’s tracked via tensorboard. Cause there can be times when live tracking is done from a different process, and catching that process’s terminal logs and system params won’t make any sense I guess. If you’re interested you can open a PR to address those points, cause adding the possibility to enable those won’t make any harm as well.

so I believe the *default* arguments should *not* do this extra logging, but still optionally allow this behaviour. 

### Pitch

Have `aim.ext.tensorboard_tracker.run.Run` inherit from `aim.sdk.run.Run` instead of `aim.sdk.run.BasicRun`, so that it can utilise it's extra capabilities.

### Alternatives

Instead of inheritance we could change the system resource tracking be a mixin? 

Extend `aim.ext.tensorboard_tracker.run.Run` to also allow system stats, parameters, and stdout capture.
## 🚀 Feature

Allow capturing of system parameters and terminal logs by the `aim.ext.tensorboard_tracker.run.Run`, as this is great feature shouldn't be only available to the default `Run`.

### Motivation

The new feature of allowing continuous syncing from `tensorboard` files to `aim` is really nice, but because `aim.ext.tensorboard_tracker.run.Run` inherits from `BasicRun` rather than `Run`, it misses out on the ability to log the standard out, system stats and system parameters. Since `aim.ext.tensorboard_tracker.run.Run` should be a possible replacement for `Run`, I don't see a reason why this behaviour shouldn't be allowed.

It has been highlighted in Discord by @mihran113:

> The reason behind inheriting from basic run is exactly to avoid terminal log tracking and system param tracking actually, cause we don’t want to add anything else rather than what’s tracked via tensorboard. Cause there can be times when live tracking is done from a different process, and catching that process’s terminal logs and system params won’t make any sense I guess. If you’re interested you can open a PR to address those points, cause adding the possibility to enable those won’t make any harm as well.

so I believe the *default* arguments should *not* do this extra logging, but still optionally allow this behaviour. 

### Pitch

Have `aim.ext.tensorboard_tracker.run.Run` inherit from `aim.sdk.run.Run` instead of `aim.sdk.run.BasicRun`, so that it can utilise it's extra capabilities.

### Alternatives

Instead of inheritance we could change the system resource tracking be a mixin? 



Discussion:

Resolving patch:
diff --git a/aim/ext/tensorboard_tracker/run.py b/aim/ext/tensorboard_tracker/run.py
--- a/aim/ext/tensorboard_tracker/run.py
+++ b/aim/ext/tensorboard_tracker/run.py
@@ -1,6 +1,6 @@
 from typing import Optional, Union
 
-from aim.sdk.run import BasicRun
+from aim.sdk.run import Run as SdkRun
 from aim.ext.tensorboard_tracker.tracker import TensorboardTracker
 
 from typing import TYPE_CHECKING
@@ -9,14 +9,23 @@
     from aim.sdk.repo import Repo
 
 
-class Run(BasicRun):
-    def __init__(self, run_hash: Optional[str] = None, *,
-                 sync_tensorboard_log_dir: str,
-                 repo: Optional[Union[str, 'Repo']] = None,
-                 experiment: Optional[str] = None,
-                 force_resume: Optional[bool] = False,
-                 ):
-        super().__init__(run_hash, repo=repo, read_only=False, experiment=experiment, force_resume=force_resume)
+class Run(SdkRun):
+    def __init__(
+            self, run_hash: Optional[str] = None, *,
+            sync_tensorboard_log_dir: str,
+            repo: Optional[Union[str, 'Repo']] = None,
+            experiment: Optional[str] = None,
+            force_resume: Optional[bool] = False,
+            system_tracking_interval: Optional[Union[int, float]] = None,
+            log_system_params: Optional[bool] = False,
+            capture_terminal_logs: Optional[bool] = False,
+    ):
+        super().__init__(
+            run_hash, repo=repo, read_only=False, experiment=experiment, force_resume=force_resume,
+            system_tracking_interval=system_tracking_interval, log_system_params=log_system_params,
+            capture_terminal_logs=capture_terminal_logs
+        )
+
         self['tb_log_directory'] = sync_tensorboard_log_dir
         self._tensorboard_tracker = TensorboardTracker(self._tracker, sync_tensorboard_log_dir)
         self._tensorboard_tracker.start()


Test patch:
diff --git a/tests/ext/test_tensorboard_run.py b/tests/ext/test_tensorboard_run.py
new file mode 100644
--- /dev/null
+++ b/tests/ext/test_tensorboard_run.py
@@ -0,0 +1,86 @@
+import time
+
+from aim.ext.resource.log import LogLine
+from aim.ext.tensorboard_tracker.run import Run as TensorboardRun
+
+from tests.base import TestBase
+from tests.utils import full_class_name
+
+
+class TestTensorboardRun(TestBase):
+
+    def test_tensorboard_tracker_run__default_no_capture(self):
+        # Given
+        run = TensorboardRun(sync_tensorboard_log_dir="dummy", repo=self.repo)
+        run['testcase'] = full_class_name(TensorboardRun)
+        run_hash = run.hash
+        console_statement = 'no console capture is being done'
+
+        # When
+        print(console_statement)
+        time.sleep(3)  # allow tracker to add resource usage metrics
+        del run
+
+        # Then
+        tracked_run = self.repo.get_run(run_hash)
+        self.assertIsNone(tracked_run.metrics().dataframe())
+
+    def test_tensorboard_tracker_run__system_stats_captured(self):
+        # Given
+        run = TensorboardRun(
+            sync_tensorboard_log_dir="dummy", repo=self.repo, system_tracking_interval=1
+        )
+        run['testcase'] = full_class_name(TensorboardRun)
+        run_hash = run.hash
+
+        # When
+        time.sleep(3)  # allow tracker to add resource usage metrics
+        del run
+
+        # Then
+        tracked_run = self.repo.get_run(run_hash)
+        metrics_recorded = set(tracked_run.metrics().dataframe()['metric.name'].unique())
+        self.assertTrue("__system__cpu" in metrics_recorded)
+
+    def test_tensorboard_tracker_run__terminal_capture(self):
+        # Given
+        run = TensorboardRun(
+            sync_tensorboard_log_dir="dummy", repo=self.repo, capture_terminal_logs=True,
+        )
+        run['testcase'] = full_class_name(TensorboardRun)
+        run_hash = run.hash
+        console_statement = 'no console capture has worked'
+
+        # When
+        print(console_statement)
+        time.sleep(3)  # allow tracker to add terminal logs
+        del run
+
+        # Then
+        tracked_run = self.repo.get_run(run_hash)
+        terminal_logs = tracked_run.get_terminal_logs()
+        log_found = False
+        for log_item in terminal_logs.data.values():
+            log_line = log_item[0][0]
+            if isinstance(log_line, LogLine):
+                if console_statement in str(log_line.data):
+                    log_found = True
+
+        self.assertTrue(log_found)
+
+    def test_tensorboard_tracker_run__system_params_captured(self):
+        # Given
+        run = TensorboardRun(
+            sync_tensorboard_log_dir="dummy", repo=self.repo, log_system_params=True
+        )
+        run['testcase'] = full_class_name(TensorboardRun)
+        run_hash = run.hash
+
+        # When
+        time.sleep(3)  # allow tracker to add system params
+        del run
+
+        # Then
+        tracked_run = self.repo.get_run(run_hash)
+        system_params = tracked_run.get('__system_params')
+        self.assertIsNotNone(system_params)