Repository: aws-deadline/deadline-cloud-test-fixtures
Pull request #183: feat: Add support to fetch and match worker logs
Author: login: leongdl
Created: 2025-04-03T20:57:17Z | Merged: 2025-04-04T18:32:57Z

### What was the problem/requirement? (What/Why)
- We are adding some new features related to worker features. Currently the test fixtures only have features related to Job logs. I am extending the test fixtures to worker logs.

### What was the solution? (How)
- Add new fixtures and helpers to get worker log group, log stream.
- Small refactor to log class to re-use existing log search logic for test verification.
- Allows tests to use `assert_pattern_in_log` and `assert_pattern_not_in_log`.

### What is the impact of this change?
- We will have great test coverage!

### How was this change tested?
1. `hatch build`
2. `hatch run fmt`
3. `hatch run all:lint`
4. Used the new worker log functions in a new E2E test case for worker agent. 

### Was this change documented?
Not Applicable.

### Is this a breaking change?
Not Applicable.

----

*By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.*


Diff:
diff --git a/src/deadline_test_fixtures/deadline/__init__.py b/src/deadline_test_fixtures/deadline/__init__.py
index e191e6e..e1ea767 100644
--- a/src/deadline_test_fixtures/deadline/__init__.py
+++ b/src/deadline_test_fixtures/deadline/__init__.py
@@ -23,6 +23,7 @@
     Task,
     TaskParameterValue,
     TaskStatus,
+    WorkerLog,
 )
 from .worker import (
     CommandResult,
@@ -70,4 +71,5 @@
     "TaskStatus",
     "WindowsInstanceBuildWorker",
     "WindowsInstanceWorkerBase",
+    "WorkerLog",
 ]
diff --git a/src/deadline_test_fixtures/deadline/resources.py b/src/deadline_test_fixtures/deadline/resources.py
index f61c4d6..7e39d81 100644
--- a/src/deadline_test_fixtures/deadline/resources.py
+++ b/src/deadline_test_fixtures/deadline/resources.py
@@ -1288,8 +1288,7 @@ def assert_log_does_not_contain(
 
 
 @dataclass
-class SessionLog:
-    session_id: str
+class CloudWatchLogs:
     logs: list[CloudWatchLogEvent]
 
     def assert_pattern_in_log(
@@ -1365,3 +1364,13 @@ def from_api_response(response: dict) -> CloudWatchLogEvent:
             message=response["message"],
             timestamp=response["timestamp"],
         )
+
+
+@dataclass
+class SessionLog(CloudWatchLogs):
+    session_id: str
+
+
+@dataclass
+class WorkerLog(CloudWatchLogs):
+    worker_id: str
diff --git a/src/deadline_test_fixtures/deadline/worker.py b/src/deadline_test_fixtures/deadline/worker.py
index 7434095..d221537 100644
--- a/src/deadline_test_fixtures/deadline/worker.py
+++ b/src/deadline_test_fixtures/deadline/worker.py
@@ -16,15 +16,18 @@
 import tempfile
 import time
 from dataclasses import dataclass, field, InitVar, replace
-from typing import Any, ClassVar, Optional, cast
+from typing import TYPE_CHECKING, Any, ClassVar, Optional, cast
 
 from ..models import (
     PipInstall,
     PosixSessionUser,
 )
-from .resources import Fleet
+from .resources import CloudWatchLogEvent, Fleet, WorkerLog
 from ..util import call_api, wait_for
 
+if TYPE_CHECKING:
+    from botocore.paginate import PageIterator, Paginator
+
 LOG = logging.getLogger(__name__)
 
 DOCKER_CONTEXT_DIR = os.path.join(os.path.dirname(__file__), "..", "containers", "worker")
@@ -48,6 +51,15 @@ def get_worker_id(self) -> str:
         pass
 
 
+@dataclass(frozen=True)
+class WorkerLogConfig:
+    cloudwatch_log_group: str
+    """The name of the CloudWatch Log Group that the Agent log should be streamed to"""
+
+    cloudwatch_log_stream: str
+    """The name of the CloudWatch Log Stream that the Agent log should be streamed to"""
+
+
 @dataclass(frozen=True)
 class CommandResult:  # pragma: no cover
     exit_code: int
@@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response.get("log"):
+            LOG.info(f"Log Config structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log config yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config: Optional[WorkerLogConfig] = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
+        filter_log_events_pages: PageIterator = call_api(
+            description=f"Fetching log events for worker {self.worker_id} in log group {log_config.cloudwatch_log_group}",
+            fn=lambda: filter_log_events_paginator.paginate(
+                logGroupName=log_config.cloudwatch_log_group,
+                logStreamNames=[log_config.cloudwatch_log_stream],
+            ),
+        )
+        log_events = filter_log_events_pages.build_full_result()
+        log_events = [CloudWatchLogEvent.from_api_response(e) for e in log_events["events"]]
+        # For debugging test cases.
+        # LOG.info(log_events)
+
+        return WorkerLog(worker_id=self.worker_id, logs=log_events)  # type: ignore[arg-type]
+
     def send_command(self, command: str) -> CommandResult:
         """Send a command via SSM to a shell on a launched EC2 instance. Once the command has fully
         finished the result of the invocation is returned.


Reviews:
user: login: ttblanchard
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:33:57Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:57:31Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:57:40Z
user: login: ttblanchard
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T15:45:58Z
user: login: jairaws
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-04T16:19:12Z
user: login: godobyte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:30:09Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:48:26Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:49:17Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:49:44Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:49:54Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:50:01Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:50:08Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:50:30Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:50:41Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:50:55Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:52:12Z
user: login: leongdl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T16:52:48Z
user: login: jairaws
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-04T17:00:02Z
user: login: godobyte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T17:09:20Z
user: login: godobyte
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T17:19:59Z
user: login: godobyte
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:20:57Z
user: login: ttblanchard
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:24:36Z

Review comments:
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def get_worker_logs(self) -> Optional[WorkerLogConfig]:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: ttblanchard
body: To make this a little more clear, should we change this method to `_get_worker_log_config`? 
created_at: 2025-04-03T21:27:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_all_worker_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: ttblanchard
body: Perhaps just `get_logs`? Would be more consistent with our [Job implementation](https://github.com/aws-deadline/deadline-cloud-test-fixtures/blob/13d60e1d504c72d6df6629b5e2948f0bfc10408f/src/deadline_test_fixtures/deadline/resources.py#L532)
created_at: 2025-04-03T21:33:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def get_worker_logs(self) -> Optional[WorkerLogConfig]:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: Sure, we can do that. 

I was thinking if other tests may want to get the WorkerLogConfig. Both works.
created_at: 2025-04-03T21:57:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_all_worker_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: Sure, easy
created_at: 2025-04-03T21:57:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: nit: type
created_at: 2025-04-04T16:12:05Z
author_association: NONE
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: Q: do we have a data class for GetWorker response?
created_at: 2025-04-04T16:14:46Z
author_association: NONE
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: If `log` isn't a key in `response`, wouldn't this line throw an exception?
created_at: 2025-04-04T16:16:10Z
author_association: NONE
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: nit: type
created_at: 2025-04-04T16:16:41Z
author_association: NONE
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
+        filter_log_events_pages: PageIterator = call_api(
+            description=f"Fetching log events for worker {self.worker_id} in log group {log_config.cloudwatch_log_group}",
+            fn=lambda: filter_log_events_paginator.paginate(
+                logGroupName=log_config.cloudwatch_log_group,
+                logStreamNames=[log_config.cloudwatch_log_stream],
+            ),
+        )
+        log_events = filter_log_events_pages.build_full_result()
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: nit: type
created_at: 2025-04-04T16:18:13Z
author_association: NONE
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: godobyte
body: nit - what about `get_worker_log_config`?
created_at: 2025-04-04T16:18:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
+        filter_log_events_pages: PageIterator = call_api(
+            description=f"Fetching log events for worker {self.worker_id} in log group {log_config.cloudwatch_log_group}",
+            fn=lambda: filter_log_events_paginator.paginate(
+                logGroupName=log_config.cloudwatch_log_group,
+                logStreamNames=[log_config.cloudwatch_log_stream],
+            ),
+        )
+        log_events = filter_log_events_pages.build_full_result()
+        log_events = [CloudWatchLogEvent.from_api_response(e) for e in log_events["events"]]
+        # For debugging test cases.
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: Could we remove this comment? Doesn't seem to be adding any value.
created_at: 2025-04-04T16:19:00Z
author_association: NONE
diff_hunk: @@ -16,15 +16,18 @@
 import tempfile
 import time
 from dataclasses import dataclass, field, InitVar, replace
-from typing import Any, ClassVar, Optional, cast
+from typing import TYPE_CHECKING, Any, ClassVar, Optional, cast
 
 from ..models import (
     PipInstall,
     PosixSessionUser,
 )
-from .resources import Fleet
+from .resources import CloudWatchLogEvent, Fleet, WorkerLog
 from ..util import call_api, wait_for
 
+if TYPE_CHECKING:
+    from botocore.paginate import PageIterator, Paginator
path: src/deadline_test_fixtures/deadline/worker.py
user: login: godobyte
body: curious - why only import those during type checking?
created_at: 2025-04-04T16:22:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
path: src/deadline_test_fixtures/deadline/worker.py
user: login: godobyte
body: nit - you meant LogConfig?
created_at: 2025-04-04T16:24:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
path: src/deadline_test_fixtures/deadline/worker.py
user: login: godobyte
body: nit - you meant log config? log group should always exist as it's at fleet level.
created_at: 2025-04-04T16:25:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
path: src/deadline_test_fixtures/deadline/worker.py
user: login: godobyte
body: Please make sure caller has the `logs:FilterLogEvents` permission to perform this operation. Local test and automated tests use different infrastructure setup.
created_at: 2025-04-04T16:29:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -16,15 +16,18 @@
 import tempfile
 import time
 from dataclasses import dataclass, field, InitVar, replace
-from typing import Any, ClassVar, Optional, cast
+from typing import TYPE_CHECKING, Any, ClassVar, Optional, cast
 
 from ..models import (
     PipInstall,
     PosixSessionUser,
 )
-from .resources import Fleet
+from .resources import CloudWatchLogEvent, Fleet, WorkerLog
 from ..util import call_api, wait_for
 
+if TYPE_CHECKING:
+    from botocore.paginate import PageIterator, Paginator
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: This was done in the rest of the library too, there's some differences in versions of python I think.
created_at: 2025-04-04T16:48:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: We don't have typing in this library for boto classes :( 
created_at: 2025-04-04T16:49:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: good call, I'll change it to .get()
created_at: 2025-04-04T16:49:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: Sure
created_at: 2025-04-04T16:49:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: sure
created_at: 2025-04-04T16:50:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: Sure
created_at: 2025-04-04T16:50:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: This is the same in the rest of the library and we don't check 
created_at: 2025-04-04T16:50:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
+        filter_log_events_pages: PageIterator = call_api(
+            description=f"Fetching log events for worker {self.worker_id} in log group {log_config.cloudwatch_log_group}",
+            fn=lambda: filter_log_events_paginator.paginate(
+                logGroupName=log_config.cloudwatch_log_group,
+                logStreamNames=[log_config.cloudwatch_log_stream],
+            ),
+        )
+        log_events = filter_log_events_pages.build_full_result()
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: there's no typing for boto in this package.
created_at: 2025-04-04T16:50:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
+        filter_log_events_pages: PageIterator = call_api(
+            description=f"Fetching log events for worker {self.worker_id} in log group {log_config.cloudwatch_log_group}",
+            fn=lambda: filter_log_events_paginator.paginate(
+                logGroupName=log_config.cloudwatch_log_group,
+                logStreamNames=[log_config.cloudwatch_log_stream],
+            ),
+        )
+        log_events = filter_log_events_pages.build_full_result()
+        log_events = [CloudWatchLogEvent.from_api_response(e) for e in log_events["events"]]
+        # For debugging test cases.
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: We do, otherwise everyone wants to remove LOG.info :)
created_at: 2025-04-04T16:50:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: Arrrg, I had it as the a non private name before but went for `_` so no :) Not changing it

created_at: 2025-04-04T16:52:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
path: src/deadline_test_fixtures/deadline/worker.py
user: login: leongdl
body: The data class comes from the design and structure of this library. There's no use of boto3 generated types
created_at: 2025-04-04T16:52:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
path: src/deadline_test_fixtures/deadline/worker.py
user: login: jairaws
body: Could we type it as `Dict[str, Any]`?
created_at: 2025-04-04T17:00:02Z
author_association: NONE
diff_hunk: @@ -252,6 +264,46 @@ def set_stopped_status(self):
             LOG.exception(f"Failed to update worker status: {error}")
             raise
 
+    def _get_worker_logs(self) -> Optional[WorkerLogConfig]:
+        """Get the log group and log stream for the worker. Retain the API structure"""
+        response = self.deadline_client.get_worker(
+            farmId=self.configuration.farm_id,
+            fleetId=self.configuration.fleet.id,
+            workerId=self.worker_id,
+        )
+        if log_config := response["log"]:
+            LOG.info(f"LogGroup structure {log_config}")
+            if log_config_options := log_config.get("options"):
+                log_group_name = log_config_options.get("logGroupName")
+                log_stream_name = log_config_options.get("logStreamName")
+                if log_group_name and log_stream_name:
+                    return WorkerLogConfig(
+                        cloudwatch_log_group=log_group_name, cloudwatch_log_stream=log_stream_name
+                    )
+        # Default, no log group yet.
+        return None
+
+    def get_logs(self, *, logs_client: botocore.client.BaseClient) -> WorkerLog:
+        # Get the worker log group and stream from the service.
+        log_config = self._get_worker_logs()
+        if not log_config:
+            return WorkerLog(worker_id=self.worker_id, logs=[])  # type: ignore[arg-type]
+
+        filter_log_events_paginator: Paginator = logs_client.get_paginator("filter_log_events")
path: src/deadline_test_fixtures/deadline/worker.py
user: login: godobyte
body: This introduce FilterLogEvents call to fleet logs while the prior code filter on session logs no? The permission for CWL is configured at log group level, that's why I suggested checking. 

Took a quick look seems we have have this permission on all deadline log groups `log-group:/aws/deadline/*` , should be good then.
created_at: 2025-04-04T17:09:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-04-04T18:28:49Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=aws-deadline_deadline-cloud-test-fixtures&pullRequest=183)


author_association: NONE