Repository: apache/airflow

Issue:
docker container still running while dag run failed
### Apache Airflow version

2.1.4

### What happened

 I have operator run with docker . 

When dag run failed , docker.py try to remove container but remove failed and got the following error:

`2022-04-20 00:03:50,381] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 301, in _run_image_with_mounts
    for line in lines:
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/types/daemon.py", line 32, in __next__
    return next(self._stream)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 412, in <genexpr>
    gen = (data for (_, data) in gen)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 92, in frames_iter_no_tty
    (stream, n) = next_frame_header(socket)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 64, in next_frame_header
    data = read_exactly(socket, 8)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 49, in read_exactly
    next_data = read(socket, n - len(data))
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 29, in read
    select.select([socket], [], [])
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1238, in signal_handler
    raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.35/containers/de4cd812f8b0dcc448d591d1bd28fa736b1712237c8c8848919be512938bd515?v=False&link=False&force=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/airflow/dags/operators/byx_base_operator.py", line 611, in execute
    raise e
  File "/usr/local/airflow/dags/operators/byx_base_operator.py", line 591, in execute
    self.execute_job(context)
  File "/usr/local/airflow/dags/operators/byx_datax_operator.py", line 93, in execute_job
    result = call_datax.execute(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 343, in execute
    return self._run_image()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 265, in _run_image
    return self._run_image_with_mounts(self.mounts, add_tmp_variable=False)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 317, in _run_image_with_mounts
    self.cli.remove_container(self.container['Id'])
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 1010, in remove_container
    self._raise_for_status(res)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.35/containers/de4cd812f8b0dcc448d591d1bd28fa736b1712237c8c8848919be512938bd515?v=False&link=False&force=False: Conflict ("You cannot remove a running container de4cd812f8b0dcc448d591d1bd28fa736b1712237c8c8848919be512938bd515. Stop the container before attempting removal or force remove")
`

### What you think should happen instead

the container should removed successful when dag run failed

### How to reproduce

step 1: create a dag with execute DockerOperator operation

step 2: trigger dag 

step 3: mark dag run to failed simulate dag run failed, and the remove container failed error will appear and the docker container still running.



### Operating System

NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2"

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!


Resolving patch:
diff --git a/airflow/providers/docker/operators/docker.py b/airflow/providers/docker/operators/docker.py
--- a/airflow/providers/docker/operators/docker.py
+++ b/airflow/providers/docker/operators/docker.py
@@ -20,6 +20,7 @@
 import io
 import pickle
 import tarfile
+import warnings
 from tempfile import TemporaryDirectory
 from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Sequence, Union
 
@@ -125,7 +126,7 @@ class DockerOperator(BaseOperator):
     :param dns_search: Docker custom DNS search domain
     :param auto_remove: Auto-removal of the container on daemon side when the
         container's process exits.
-        The default is False.
+        The default is never.
     :param shm_size: Size of ``/dev/shm`` in bytes. The size must be
         greater than 0. If omitted uses system default.
     :param tty: Allocate pseudo-TTY to the container
@@ -175,7 +176,7 @@ def __init__(
         docker_conn_id: Optional[str] = None,
         dns: Optional[List[str]] = None,
         dns_search: Optional[List[str]] = None,
-        auto_remove: bool = False,
+        auto_remove: str = "never",
         shm_size: Optional[int] = None,
         tty: bool = False,
         privileged: bool = False,
@@ -189,7 +190,20 @@ def __init__(
     ) -> None:
         super().__init__(**kwargs)
         self.api_version = api_version
-        self.auto_remove = auto_remove
+        if type(auto_remove) == bool:
+            warnings.warn(
+                "bool value for auto_remove is deprecated, please use 'never', 'success', or 'force' instead",
+                DeprecationWarning,
+                stacklevel=2,
+            )
+        if str(auto_remove) == "False":
+            self.auto_remove = "never"
+        elif str(auto_remove) == "True":
+            self.auto_remove = "success"
+        elif str(auto_remove) in ("never", "success", "force"):
+            self.auto_remove = auto_remove
+        else:
+            raise ValueError("unsupported auto_remove option, use 'never', 'success', or 'force' instead")
         self.command = command
         self.container_name = container_name
         self.cpus = cpus
@@ -334,8 +348,10 @@ def _run_image_with_mounts(
                     return None
             return None
         finally:
-            if self.auto_remove:
+            if self.auto_remove == "success":
                 self.cli.remove_container(self.container['Id'])
+            elif self.auto_remove == "force":
+                self.cli.remove_container(self.container['Id'], force=True)
 
     def _attempt_to_retrieve_result(self):
         """
diff --git a/airflow/providers/docker/operators/docker_swarm.py b/airflow/providers/docker/operators/docker_swarm.py
--- a/airflow/providers/docker/operators/docker_swarm.py
+++ b/airflow/providers/docker/operators/docker_swarm.py
@@ -160,11 +160,12 @@ def _run_service(self) -> None:
                 self.log.info('Service status before exiting: %s', self._service_status())
                 break
 
+        self.log.info("auto_removeauto_removeauto_removeauto_removeauto_remove : %s", str(self.auto_remove))
         if self.service and self._service_status() != 'complete':
-            if self.auto_remove:
+            if self.auto_remove == "success":
                 self.cli.remove_service(self.service['ID'])
             raise AirflowException('Service did not complete: ' + repr(self.service))
-        elif self.auto_remove:
+        elif self.auto_remove == "success":
             if not self.service:
                 raise Exception("The 'service' should be initialized before!")
             self.cli.remove_service(self.service['ID'])


Test patch: