Repository: pantsbuild/pants

Issue:
New default of `pull=True` on `docker_image` target breaks use of transitive base images
**Describe the bug**

(Sorry for the wordy title, couldn't think of a better way to put it)

In our repo, we have a "base" `Dockerfile` containing common build steps. It's a lightweight image, so we don't push it anywhere - instead, all images that use it build off of it using:
```
ARG BASE=src/infra/docker/color-base-python:color-base-python
FROM $BASE
```

On Pants 2.14, if I `./pants package some/top-level/Dockerfile` then Pants will see the `FROM` dependency and build `src/infra/docker/color-base-python` first. When trying out Pants 2.15.0a0 this no longer happens, so the top-level `package` fails with:

```
#3 [internal] load metadata for docker.io/color/color-base-python:latest
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for docker.io/color/color-base-python:latest:
------
ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
```

`./pants dependencies` of the top-level image still shows `src/infra/docker/color-base-python:color-base-python` so dependency inference still works, but something in the sequencing of `docker build` commands has broken.

**Pants version**

v2.15.0a0

**OS**

Both

**Additional info**

https://app.toolchain.com/organizations/color/repos/color/builds/pants_run_2022_11_03_12_17_40_671_7f0821ce4406484b85bbbd6bf52264e0/type/goal/?subTab=package



Discussion:
Interesting - when I `docker image ls` after hitting the error, I _do_ see `color/color-base-python` in the list. So the base image is getting built, but not in time for the other build?
I thought I fixed this in 2.14.x. WTH?
https://github.com/pantsbuild/pants/pull/16419
Interesting x2 - the error occurs even if I leave `color/color-base-python` in place locally 🤔 are we doing some sort of build isolation between docker images as part of the environments work?
Looking at a `__run.sh` from a failure, I see `--build-arg $'BASE=color/color-base-python:latest'` in the `docker build` args. Is that `$` supposed to be there? EDIT: I also see `$`s in front of `--label`/`--tag` values and other args so maybe it's a red herring
Oh! `$'--pull=True'` is in the `docker build` args 👀 👀 👀 
So I think https://github.com/pantsbuild/pants/pull/16607 is the culprit. Defaulting to `True` for the new option is a breaking change.
I can make a quick PR to flip the default.
Interesting, can you explain why `--pull=true` would cause this?
`--pull=true` tells the daemon to (effectively) run a `docker pull` of the base image before building, which is guaranteed to fail if there is no remote repository for your base
Ah OK, so this isn't a Pants logic error a la https://github.com/pantsbuild/pants/pull/16419? phew

Resolving patch:
diff --git a/src/python/pants/backend/docker/target_types.py b/src/python/pants/backend/docker/target_types.py
--- a/src/python/pants/backend/docker/target_types.py
+++ b/src/python/pants/backend/docker/target_types.py
@@ -348,12 +348,13 @@ def options(self) -> Iterator[str]:
 
 class DockerImageBuildPullOptionField(DockerBuildOptionFieldValueMixin, BoolField):
     alias = "pull"
-    default = True
+    default = False
     help = softwrap(
         """
         If true, then docker will always attempt to pull a newer version of the image.
 
-        Useful to disable it when building images from other intermediate goals.
+        NOTE: This option cannot be used on images that build off of "transient" base images
+        referenced by address (i.e. `FROM path/to/your/base/Dockerfile`).
         """
     )
     docker_build_option = "--pull"


Test patch:
diff --git a/src/python/pants/backend/docker/goals/package_image_integration_test.py b/src/python/pants/backend/docker/goals/package_image_integration_test.py
--- a/src/python/pants/backend/docker/goals/package_image_integration_test.py
+++ b/src/python/pants/backend/docker/goals/package_image_integration_test.py
@@ -3,6 +3,8 @@
 
 from __future__ import annotations
 
+from textwrap import dedent
+
 import pytest
 
 from pants.backend.docker.goals.package_image import DockerFieldSet
@@ -46,7 +48,7 @@ def run_docker(
     return result
 
 
-def test_docker_build(rule_runner) -> None:
+def test_docker_build(rule_runner: RuleRunner) -> None:
     """This test requires a running docker daemon."""
     rule_runner.write_files(
         {
@@ -61,3 +63,30 @@ def test_docker_build(rule_runner) -> None:
     assert "Built docker image: test-image:1.0" == result.artifacts[0].extra_log_lines[0]
     assert "Docker image ID:" in result.artifacts[0].extra_log_lines[1]
     assert "<unknown>" not in result.artifacts[0].extra_log_lines[1]
+
+
+def test_docker_build_multi_layer(rule_runner: RuleRunner) -> None:
+    rule_runner.write_files(
+        {
+            "src/BUILD": dedent(
+                """\
+                docker_image(name='test-image', image_tags=['1.0'])
+                docker_image(name='base', source='Dockerfile.base')
+                """
+            ),
+            "src/Dockerfile": dedent(
+                """\
+                ARG BASE=src:base
+                FROM $BASE
+                """
+            ),
+            "src/Dockerfile.base": "FROM python:3.8",
+        }
+    )
+    target = rule_runner.get_target(Address("src", target_name="test-image"))
+    result = run_docker(rule_runner, target)
+    assert len(result.artifacts) == 1
+    assert len(result.artifacts[0].extra_log_lines) == 2
+    assert "Built docker image: test-image:1.0" == result.artifacts[0].extra_log_lines[0]
+    assert "Docker image ID:" in result.artifacts[0].extra_log_lines[1]
+    assert "<unknown>" not in result.artifacts[0].extra_log_lines[1]
diff --git a/src/python/pants/backend/docker/goals/package_image_test.py b/src/python/pants/backend/docker/goals/package_image_test.py
--- a/src/python/pants/backend/docker/goals/package_image_test.py
+++ b/src/python/pants/backend/docker/goals/package_image_test.py
@@ -450,7 +450,7 @@ def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "env1:1.2.3",
             "--file",
@@ -473,13 +473,13 @@ def check_docker_proc(process: Process):
 
 
 def test_docker_build_pull(rule_runner: RuleRunner) -> None:
-    rule_runner.write_files({"docker/test/BUILD": 'docker_image(name="args1", pull=False)'})
+    rule_runner.write_files({"docker/test/BUILD": 'docker_image(name="args1", pull=True)'})
 
     def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=False",
+            "--pull=True",
             "--tag",
             "args1:latest",
             "--file",
@@ -510,7 +510,7 @@ def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--squash",
             "--tag",
             "args1:latest",
@@ -523,7 +523,7 @@ def check_docker_proc_no_squash(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "args2:latest",
             "--file",
@@ -559,7 +559,7 @@ def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "args1:1.2.3",
             "--build-arg",
@@ -654,7 +654,7 @@ def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "img1:latest",
             "--build-arg",
@@ -712,7 +712,7 @@ def check_docker_proc(process: Process):
             f"id=project-secret,src={rule_runner.build_root}/secrets/mysecret",
             "--secret",
             f"id=target-secret,src={rule_runner.build_root}/docker/test/mysecret",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "img1:latest",
             "--file",
@@ -747,7 +747,7 @@ def check_docker_proc(process: Process):
             "build",
             "--ssh",
             "default",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "img1:latest",
             "--file",
@@ -791,7 +791,7 @@ def check_docker_proc(process: Process):
             "build.host=tbs06",
             "--label",
             "build.job=13934",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "img1:latest",
             "--build-arg",
@@ -932,7 +932,7 @@ def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--target",
             expected_target,
             "--tag",
@@ -1151,7 +1151,7 @@ def check_docker_proc(process: Process):
         assert process.argv == (
             "/dummy/docker",
             "build",
-            "--pull=True",
+            "--pull=False",
             "--tag",
             "plugin:latest",
             "--tag",