Repository: apache/airflow

Issue:
TaskInstances do not succeed when using enable_logging=True option in DockerSwarmOperator
### Apache Airflow Provider(s)

docker

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.1.0
apache-airflow-providers-docker==3.3.0

### Apache Airflow version

2.5.0

### Operating System

centos 7

### Deployment

Other Docker-based deployment

### Deployment details

Running an a docker-swarm cluster deployed locally. 

### What happened

Same issue as https://github.com/apache/airflow/issues/13675

With logging_enabled=True the DAG never completes and stays in running. 

When using DockerSwarmOperator together with the default enable_logging=True option, tasks do not succeed and stay in state running. When checking the docker service logs I can clearly see that the container ran and ended successfully. Airflow however does not recognize that the container finished and keeps the tasks in state running.

### What you think should happen instead

DAG should complete.

### How to reproduce

Docker-compose deployment:
```console
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.5.0/docker-compose.yaml'
docker compose up airflow-init
docker compose up -d
```

DAG code:

```python
from airflow import DAG
from docker.types import Mount, SecretReference
from airflow.providers.docker.operators.docker_swarm import DockerSwarmOperator
from datetime import timedelta
from airflow.utils.dates import days_ago
from airflow.models import Variable


# Setup default args for the job
default_args = {
	'owner': 'airflow',
	'start_date': days_ago(2),
	'retries': 0
}
 
# Create the DAG
dag = DAG(
    'test_dag',         # DAG ID
    default_args=default_args,
    schedule_interval='0 0 * * *', 
    catchup=False
)

# # Create the DAG object
with dag as dag:
	  docker_swarm_task = DockerSwarmOperator(
			task_id="job_run",
			image="<any image>",
			execution_timeout=timedelta(minutes=5),
			command="<specific code>",
			api_version='auto',
			tty=True,
			enable_logging=True
		)
```

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

https://github.com/apache/airflow/issues/13675

enable_logging=**False**,
@maxim317, yes disabling logging is a workaround. Ideally we would like the logs though in airflow.
Marked it as good first issue, hopefully someone wil take a look at this one.
can I work on this?
Feel free, but there is a linked PR for that so better ask there.
yeah I see the pull request recently updated and active. my bad!

Resolving patch:
diff --git a/airflow/providers/docker/operators/docker_swarm.py b/airflow/providers/docker/operators/docker_swarm.py
--- a/airflow/providers/docker/operators/docker_swarm.py
+++ b/airflow/providers/docker/operators/docker_swarm.py
@@ -17,6 +17,9 @@
 """Run ephemeral Docker Swarm services."""
 from __future__ import annotations
 
+import re
+from datetime import datetime
+from time import sleep
 from typing import TYPE_CHECKING
 
 from docker import types
@@ -179,23 +182,34 @@ def _has_service_terminated(self) -> bool:
     def _stream_logs_to_output(self) -> None:
         if not self.service:
             raise Exception("The 'service' should be initialized before!")
-        logs = self.cli.service_logs(
-            self.service["ID"], follow=True, stdout=True, stderr=True, is_tty=self.tty
-        )
-        line = ""
-        for log in logs:
-            try:
-                log = log.decode()
-            except UnicodeDecodeError:
-                continue
-            if log == "\n":
-                self.log.info(line)
-                line = ""
-            else:
-                line += log
-        # flush any remaining log stream
-        if line:
-            self.log.info(line)
+        last_line_logged, last_timestamp = "", 0
+
+        def stream_new_logs(last_line_logged, since=0):
+            logs = self.cli.service_logs(
+                self.service["ID"],
+                follow=False,
+                stdout=True,
+                stderr=True,
+                is_tty=self.tty,
+                since=since,
+                timestamps=True,
+            )
+            logs = b"".join(logs).decode().splitlines()
+            if last_line_logged in logs:
+                logs = logs[logs.index(last_line_logged) + 1 :]
+            for line in logs:
+                match = re.match(r"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6,}Z) (.*)", line)
+                timestamp, message = match.groups()
+                self.log.info(message)
+            # Floor nanoseconds to microseconds
+            last_timestamp = re.sub(r"(\.\d{6})\d+Z", r"\1Z", timestamp)
+            last_timestamp = datetime.strptime(last_timestamp, "%Y-%m-%dT%H:%M:%S.%fZ")
+            last_timestamp = last_timestamp.timestamp()
+            return last_line_logged, last_timestamp
+
+        while not self._has_service_terminated():
+            sleep(2)
+            last_line_logged, last_timestamp = stream_new_logs(last_line_logged, since=last_timestamp)
 
     def on_kill(self) -> None:
         if self.hook.client_created and self.service is not None:


Test patch:
diff --git a/tests/providers/docker/operators/test_docker_swarm.py b/tests/providers/docker/operators/test_docker_swarm.py
--- a/tests/providers/docker/operators/test_docker_swarm.py
+++ b/tests/providers/docker/operators/test_docker_swarm.py
@@ -40,7 +40,7 @@ def _client_tasks_side_effect():
                 yield [{"Status": {"State": "complete"}}]
 
         def _client_service_logs_effect():
-            yield b"Testing is awesome."
+            yield b"2023-12-05T00:00:00.000000000Z Testing is awesome."
 
         client_mock = mock.Mock(spec=APIClient)
         client_mock.create_service.return_value = {"ID": "some_id"}
@@ -98,8 +98,8 @@ def _client_service_logs_effect():
             base_url="unix://var/run/docker.sock", tls=False, version="1.19", timeout=DEFAULT_TIMEOUT_SECONDS
         )
 
-        client_mock.service_logs.assert_called_once_with(
-            "some_id", follow=True, stdout=True, stderr=True, is_tty=True
+        client_mock.service_logs.assert_called_with(
+            "some_id", follow=False, stdout=True, stderr=True, is_tty=True, since=0, timestamps=True
         )
 
         csargs, cskwargs = client_mock.create_service.call_args_list[0]
@@ -108,7 +108,7 @@ def _client_service_logs_effect():
         assert cskwargs["labels"] == {"name": "airflow__adhoc_airflow__unittest"}
         assert cskwargs["name"].startswith("airflow-")
         assert cskwargs["mode"] == types.ServiceMode(mode="replicated", replicas=3)
-        assert client_mock.tasks.call_count == 5
+        assert client_mock.tasks.call_count == 6
         client_mock.remove_service.assert_called_once_with("some_id")
 
     @mock.patch("airflow.providers.docker.operators.docker_swarm.types")