Repository: apache/airflow

Issue:
DockerOperator fails to pull an image
**Apache Airflow version**: 2.0

**Environment**:

- **OS** (from /etc/os-release): Debian GNU/Linux 10 (buster)
- **Kernel** (`uname -a`): Linux 37365fa0b59b 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 GNU/Linux
- **Others**: running inside a docker container, forked puckel/docker-airflow

**What happened**:

`DockerOperator` does not attempt to pull an image unless force_pull is set to True, instead displaying a misleading 404 error.

**What you expected to happen**:

`DockerOperator` should attempt to pull an image when it is not present locally.

**How to reproduce it**:

Make sure you don't have an image tagged `debian:buster-slim` present locally.
```
DockerOperator(
        task_id=f'try_to_pull_debian',
        image='debian:buster-slim',
        command=f'''echo hello''',
        force_pull=False
    )
```
prints: `{taskinstance.py:1396} ERROR - 404 Client Error: Not Found ("No such image: ubuntu:latest")`
This, on the other hand:
```
DockerOperator(
        task_id=f'try_to_pull_debian',
        image='debian:buster-slim',
        command=f'''echo hello''',
        force_pull=True
    )
```
pulls the image and prints `{docker.py:263} INFO - hello`

**Anything else we need to know**:

I overrode `DockerOperator` to track down what I was doing wrong and found the following:

When trying to run an image that's not present locally, `self.cli.images(name=self.image)` in the line:
https://github.com/apache/airflow/blob/8723b1feb82339d7a4ba5b40a6c4d4bbb995a4f9/airflow/providers/docker/operators/docker.py#L286
returns a non-empty array even when the image has been deleted from the local machine. 

In fact, `self.cli.images` appears to return non-empty arrays even when supplied with nonsense image names.

<details><summary>force_pull_false.log</summary> 
[2021-01-27 06:15:28,987] {__init__.py:124} DEBUG - Preparing lineage inlets and outlets
[2021-01-27 06:15:28,987] {__init__.py:168} DEBUG - inlets: [], outlets: []
[2021-01-27 06:15:28,987] {config.py:21} DEBUG - Trying paths: ['/usr/local/airflow/.docker/config.json', '/usr/local/airflow/.dockercfg']
[2021-01-27 06:15:28,987] {config.py:25} DEBUG - Found file at path: /usr/local/airflow/.docker/config.json
[2021-01-27 06:15:28,987] {auth.py:182} DEBUG - Found 'auths' section
[2021-01-27 06:15:28,988] {auth.py:142} DEBUG - Found entry (registry='https://index.docker.io/v1/', username='xxxxxxx')
[2021-01-27 06:15:29,015] {connectionpool.py:433} DEBUG - http://localhost:None "GET /version HTTP/1.1" 200 851
[2021-01-27 06:15:29,060] {connectionpool.py:433} DEBUG - http://localhost:None "GET /v1.41/images/json?filter=debian%3Abuster-slim&only_ids=0&all=0 HTTP/1.1" 200 None
[2021-01-27 06:15:29,060] {docker.py:224} INFO - Starting docker container from image debian:buster-slim
[2021-01-27 06:15:29,063] {connectionpool.py:433} DEBUG - http://localhost:None "POST /v1.41/containers/create HTTP/1.1" 404 48
[2021-01-27 06:15:29,063] {taskinstance.py:1396} ERROR - 404 Client Error: Not Found ("No such image: debian:buster-slim")
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 305, in execute
    return self._run_image()
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 231, in _run_image
    self.container = self.cli.create_container(
  File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 427, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 438, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 267, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: debian:buster-slim")
</details>

<details><summary>force_pull_true.log</summary>
[2021-01-27 06:17:01,811] {__init__.py:124} DEBUG - Preparing lineage inlets and outlets
[2021-01-27 06:17:01,811] {__init__.py:168} DEBUG - inlets: [], outlets: []
[2021-01-27 06:17:01,811] {config.py:21} DEBUG - Trying paths: ['/usr/local/airflow/.docker/config.json', '/usr/local/airflow/.dockercfg']
[2021-01-27 06:17:01,811] {config.py:25} DEBUG - Found file at path: /usr/local/airflow/.docker/config.json
[2021-01-27 06:17:01,811] {auth.py:182} DEBUG - Found 'auths' section
[2021-01-27 06:17:01,812] {auth.py:142} DEBUG - Found entry (registry='https://index.docker.io/v1/', username='xxxxxxxxx')
[2021-01-27 06:17:01,825] {connectionpool.py:433} DEBUG - http://localhost:None "GET /version HTTP/1.1" 200 851
[2021-01-27 06:17:01,826] {docker.py:287} INFO - Pulling docker image debian:buster-slim
[2021-01-27 06:17:01,826] {auth.py:41} DEBUG - Looking for auth config
[2021-01-27 06:17:01,826] {auth.py:242} DEBUG - Looking for auth entry for 'docker.io'
[2021-01-27 06:17:01,826] {auth.py:250} DEBUG - Found 'https://index.docker.io/v1/'
[2021-01-27 06:17:01,826] {auth.py:54} DEBUG - Found auth config
[2021-01-27 06:17:04,399] {connectionpool.py:433} DEBUG - http://localhost:None "POST /v1.41/images/create?tag=buster-slim&fromImage=debian HTTP/1.1" 200 None
[2021-01-27 06:17:04,400] {docker.py:301} INFO - buster-slim: Pulling from library/debian
[2021-01-27 06:17:04,982] {docker.py:301} INFO - a076a628af6f: Pulling fs layer
[2021-01-27 06:17:05,884] {docker.py:301} INFO - a076a628af6f: Downloading
[2021-01-27 06:17:11,429] {docker.py:301} INFO - a076a628af6f: Verifying Checksum
[2021-01-27 06:17:11,429] {docker.py:301} INFO - a076a628af6f: Download complete
[2021-01-27 06:17:11,480] {docker.py:301} INFO - a076a628af6f: Extracting
</details>


Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

> Thanks for opening your first issue here! Be sure to follow the issue template!

Relatedly, I looked for an issue template in the [contributing guidelines](https://github.com/apache/airflow/blob/df1503ea0aad34f6e1ed005444dacae5f78f7fde/CONTRIBUTING.rst#report-bugs) and couldn't find one.
> > Thanks for opening your first issue here! Be sure to follow the issue template!
> 
> Relatedly, I looked for an issue template in the [contributing guidelines](https://github.com/apache/airflow/blob/df1503ea0aad34f6e1ed005444dacae5f78f7fde/CONTRIBUTING.rst#report-bugs) and couldn't find one.

The issue template is just all the questions asked when you create this Github issue like the Airflow version, how to reproduce, etc (https://github.com/apache/airflow/issues/new?assignees=&labels=kind%3Abug&template=bug_report.md&title=)
> > > Thanks for opening your first issue here! Be sure to follow the issue template!
> > 
> > 
> > Relatedly, I looked for an issue template in the [contributing guidelines](https://github.com/apache/airflow/blob/df1503ea0aad34f6e1ed005444dacae5f78f7fde/CONTRIBUTING.rst#report-bugs) and couldn't find one.
> 
> The issue template is just all the questions asked when you create this Github issue like the Airflow version, how to reproduce, etc (https://github.com/apache/airflow/issues/new?assignees=&labels=kind%3Abug&template=bug_report.md&title=)

Thank you! I'll update my issue using that template. 
I have updated the original issue using the bug report template.
I am facing the same issue. This seems to only happen when running Airflow inside a Docker container. I had a non-dockerized deployment for which it worked fine without the `force_pull=True` option. Since going to a docker-compose deployment, I've been having the same 404 error.
```
[2021-02-18 09:24:44,360] {docker.py:224} INFO - Starting docker container from image busybox:stable
[2021-02-18 09:24:44,394] {taskinstance.py:1396} ERROR - 404 Client Error: Not Found ("No such image: busybox:stable")
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/home/airflow/.local/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create
```
Looks like the problem here is the filter argument being removed in docker API version 1.41: https://github.com/docker/cli/blob/v20.10.0-rc1/docs/deprecated.md#filter-param-for-imagesjson-endpoint.  This fix (https://github.com/docker/docker-py/commit/1757c974fa3a05b0e9b783af85242b18df09d05d) is available in `docker>=4.4.1`

I confirmed this by installing 4.4.4 and running a sample docker dag that was failing earlier due to the image not being present.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -263,7 +263,7 @@ def get_sphinx_theme_version() -> str:
     'sphinxcontrib-spelling==5.2.1',
 ]
 docker = [
-    'docker~=3.0',
+    'docker',
 ]
 druid = [
     'pydruid>=0.4.1',


Test patch: