Repository: apache/airflow

Issue:
CI: Reuse Breeze CI image building for CI
We already have a basic Build image function in `Breeze` - we should be able to use it to build images in CI.

There are actions that build images in `./build-imaages.yml` and they should simply (similarely to free-space) use Python commands develped in ./dev/breeze.


Discussion:
@potiuk I changed the file ci_build_ci_image_on_ci.sh for build_image.py, it is the unique part where we call to build the image. Please let me know if I am in the right way.

https://github.com/edithturn/airflow/blob/a02e8562a01b251d214f06f69bf1f15c91af536c/.github/workflows/build-images.yml#L231-L235

Another thing I was thinking is If I finished building the image, it means that it is ready to push and It is made for ci_push_ci_images.sh, and it makes a login in docker registry to push the image. Will, it is possible to test it in my docker registry with my own credentials? How will I prove that the image is well built and ready to upload?

Note: I am testing it in my own remote since these are changes made in the build-image.yaml

> @potiuk I changed the file ci_build_ci_image_on_ci.sh for build_image.py, it is the unique part where we call to build the image. Please let me know if I am in the right way.
> 
> https://github.com/edithturn/airflow/blob/a02e8562a01b251d214f06f69bf1f15c91af536c/.github/workflows/build-images.yml#L231-L235

Yeah. But we also should make it an entrypoint similar to "airflow-freespace", (`airlfow-build-image`) this way when we run it in CI, we shoudl be able to - similarly as `airflow-freespace` run it as `run: airflow-build-image`. 

> Another thing I was thinking is If I finished building the image, it means that it is ready to push and It is made for ci_push_ci_images.sh, and it makes a login in docker registry to push the image. Will, it is possible to test it in my docker registry with my own credentials? How will I prove that the image is well built and ready to upload?

Yeah. It should work in your repo. That should be no problem. If everything works fine with the build image and all the variables are properly used, the image should be built as:

```
ghcr.io/edithurn/airflow/main/ci/python3.7
```

GIHUB_REPOSITORY variable should be used to determine that.

See for example here: https://github.com/potiuk/airflow/runs/5177272408?check_suite_focus=true

```
Build CI image ghcr.io/potiuk/airflow/main/ci/python3.7
```

Of course there might be some issues with the python build_image.py to use that variable (we did not try it before) - but then this is something that should be fixed if it will continue using "apache/airflow" :).

> Note: I am testing it in my own remote since these are changes made in the build-image.yaml

Yep. The only way to test it :)

Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`
![Screenshot from 2022-02-20 10-46-33](https://user-images.githubusercontent.com/58795858/154858493-a17dae89-1c74-43b9-9859-bde5231f6ede.png)
 
 I saw that in breeze.py the build_image is being used: 
 https://github.com/apache/airflow/blob/9ad4de835cbbd296b9dbd1ff0ea88c1cd0050263/dev/breeze/src/airflow_breeze/breeze.py#L162-L164


 Could it be a  good example to see if the  buid_image is working effectible?

Another doubt I have. What should be the parameter for build_image, since in the build_image this recipe: **kwargs as an argument, it means many arguments, like the python version, directories, etc. How I could know what will be these parameters?
> Understoonn @potiuk, I was testing and the image is not being built appropriately. I just print a `docker images`

Actually I think now it will be better to just use "Breeze2 build-ci-image" command directly - no need for a separate entrypoint. I think you should be able to run the command even now - as "Breeze2" entrypoint is already installed at this point.

> Another doubt I have. What should be the parameter for build_image, since in the build_image this recipe: **kwargs as an argument, it means many arguments, like the python version, directories, etc. How I could know what will be these parameters?

Those parameters should be (as they are now) passed through environment variables. When you run it as `./Breeze2 build-ci-image` you should just make sure that al the necessary variables that are currently set in the `build-images.yml` are properly configured as proper click options env variables fallbacks https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables

For example `PYTHON_MAJOR_MINOR_VERSION` variables (which is set from matrix in the CI) should be used as fallback in the `--python` flag in `build-ci-image`
@potiuk I have a doubt about the construction of the image. 
The image we are building for build_image is Dockerfile.ci, and this creates the URL based in ARG PYTHON_BASE_IMAGE="python:3.7-slim-buster" but seeing your, the name of the image is different, without slim-buster (see picture).

Is this the URL of the basic image that we take to build that Dockerfil?
I am getting a: denied, is this because of the malformation of the URL?

![Screenshot from 2022-02-22 13-33-19](https://user-images.githubusercontent.com/58795858/155196627-0e9f4447-f347-4cbb-89a1-0f098a1511f0.png)
 
Few comments: 

* use `Breeze2` instead of `./Breeze2` - similarly as `airflow-freespace`, Breeze2 is already installed by the earlier `pip install` step
* You will need to modify the options of build-ci-image command to use ENV_VARIABLES. In the original Breeze, we read the variables during intialization, but in Python command we just need to make sure the options can be read from the env variable. See this chapter in click docs: https://click.palletsprojects.com/en/8.0.x/options/#values-from-environment-variables

You will need to modify this code: https://github.com/apache/airflow/blob/e8afff9cb52c8c2bc8bd6587656c67703fd87956/dev/breeze/src/airflow_breeze/breeze.py#L93

For example:

```
@click.option('-p', '--python', help='Choose your python version')
```

You will have to change to:

```
@click.option('-p', '--python', help='Choose your python version', envvar='PYTHON_MAJOR_MINOR_VERSION')
```

If you look at the execution of the build step in CI you will see that there are already many variables set there by ci,yaml and selective checks. 

```
env:
    MOUNT_SELECTED_LOCAL_SOURCES: false
    FORCE_ANSWER_TO_QUESTIONS: yes
    CHECK_IMAGE_FOR_REBUILD: true
    SKIP_CHECK_REMOTE_IMAGE: true
    DB_RESET: true
    VERBOSE: true
    GITHUB_REPOSITORY: edithturn/airflow
    GITHUB_USERNAME: edithturn
    CONSTRAINTS_GITHUB_REPOSITORY: apache/airflow
    GITHUB_REGISTRY_PULL_IMAGE_TAG: latest
    INSTALL_PROVIDERS_FROM_SOURCES: true
    AIRFLOW_LOGIN_TO_GITHUB_REGISTRY: true
    BACKEND: postgres
    PYTHON_MAJOR_MINOR_VERSION: 3.7
    UPGRADE_TO_NEWER_DEPENDENCIES: false
    DOCKER_CACHE: disable:
```

For example the case you mentioned is handled by `GITHUB_REPOSITORY` variable. 
You will find that there is this option:
```
@click.option('--github-repository', help='Choose repository to push/pull image.')
```

And again adding `envvar='GITHUB_REPOSITORY'`) should help. 

You will find the variables used for each option in https://github.com/apache/airflow/blob/main/breeze#L799  - there is "parse-flags" and each flag has appropriate env variable assigned.

@potiuk I was struggling with a problem to know why the `.build` folder was not created when executing build_image. It is that `.build ` is only created in the breeze main. Really sorry for the time it took me to realize it.

https://github.com/apache/airflow/blob/5d89dea56843d7b76d5e308e373ba16ecbcffa77/dev/breeze/src/airflow_breeze/breeze.py#L358-L360

So what I did was copy the line to build_image:
https://github.com/edithturn/airflow/blob/e282d37962393d76802ec8170b6c0d4cef290f20/dev/breeze/src/airflow_breeze/breeze.py#L166-L169
With this, I am not having the problem "file not found", so I could start to add the other variable environments that I need to build the image.

As far I know, we don't run breeze in the Ci GitHub Actions, so these breeze main where that directory is created never will execute it never will create that directory which is needed to build the image. 

@potiuk another thing I found is with the env variable **UPGRADE_TO_NEWER_DEPENDENCIES**, it is setting a random number.
![Screenshot from 2022-03-01 00-07-40](https://user-images.githubusercontent.com/58795858/156110340-5d885ed4-4a2e-46d3-9991-de0d0a6c9fb2.png)  [JOB LOG](https://github.com/edithturn/airflow/runs/5370490394?check_suite_focus=true)

It is because I am **pushing** the code and, It falls into this validation.
https://github.com/apache/airflow/blob/82e1abf7144f5386cb9dbf9775e9c78da51ec673/scripts/ci/selective_ci_checks.sh#L45-L54

It should not fail when it is random number (I saw it in one of your executions: https://github.com/potiuk/airflow/runs/5344689690?check_suite_focus=true), so I have to find a way to continue the process if this value is random, right? I must be setting the variable to boolean yes or yes somewhere, and that is why it fails.

Could you please confirm that?
> As far I know, we don't run breeze in the Ci GitHub Actions, so these breeze main where that directory is created never will execute it never will create that directory which is needed to build the image.

Hmm. I think this will be fixed when #21145 is merged - for now this is good to keep it but once we merge the change from @bowrna we should remove it.
> It should not fail when it is random number (I saw it in one of your executions: https://github.com/potiuk/airflow/runs/5344689690?check_suite_focus=true), so I have to find a way to continue the process if this value is random, right? I must be setting the variable to boolean yes or yes somewhere, and that is why it fails.

AAARGH. The "Upgrade to newer depencies" is rather special. It can be "true", "false" OR ANYTHING_ELSE :). Simply change the type of the parameter to "string". We might want to improve it in the future, but for now it should be fine to just change it to "string" and possibly add some description for that one in "Help" 
Thanks Jarek, It worked setting it as a string.
woho, it passed the build-ci-image, now I am checking the problem with Push
![Screenshot from 2022-03-03 10-01-12](https://user-images.githubusercontent.com/58795858/156591118-107e0c7d-3d7a-4abf-b0b9-76f1d49eb541.png)


But I think it is weird because I am not taking much than one env variable for this.
I found that the generated image is wrong, nothing inside, 0B.
REPOSITORY                                                                            TAG                                        IMAGE ID       CREATED         SIZE
ghcr.io/edithturn/airflow/main/ci/python3.9   eda6a85304bb2b598674a94e1b877a920e1451db   71de1148337f   N/A             0B

@potiuk could you please assign me this task? 🚀 
![Screenshot from 2022-03-03 11-09-03](https://user-images.githubusercontent.com/58795858/156604155-3b4b17a3-4a13-4d4c-832f-04c4f27d07d9.png)



Resolving patch:
diff --git a/dev/breeze/src/airflow_breeze/breeze.py b/dev/breeze/src/airflow_breeze/breeze.py
--- a/dev/breeze/src/airflow_breeze/breeze.py
+++ b/dev/breeze/src/airflow_breeze/breeze.py
@@ -69,10 +69,7 @@ def main():
 
 
 option_verbose = click.option(
-    "-v",
-    "--verbose",
-    is_flag=True,
-    help="Print verbose information about performed steps",
+    "-v", "--verbose", is_flag=True, help="Print verbose information about performed steps", envvar='VERBOSE'
 )
 
 option_python_version = click.option(
@@ -80,6 +77,7 @@ def main():
     '--python',
     type=click.Choice(ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS),
     help='Choose your python version',
+    envvar='PYTHON_MAJOR_MINOR_VERSION',
 )
 
 option_backend = click.option(
@@ -187,54 +185,79 @@ def shell(
 @click.option(
     '--additional-extras',
     help='This installs additional extra package while installing airflow in the image.',
+    envvar='ADDITIONAL_AIRFLOW_EXTRAS',
 )
 @option_python_version
 @click.option(
-    '--additional-dev-apt-deps', help='Additional apt dev dependencies to use when building the images.'
+    '--additional-dev-apt-deps',
+    help='Additional apt dev dependencies to use when building the images.',
+    envvar='ADDITIONAL_DEV_APT_DEPS',
 )
 @click.option(
     '--additional-runtime-apt-deps',
     help='Additional apt runtime dependencies to use when building the images.',
+    envvar='ADDITIONAL_RUNTIME_APT_DEPS',
 )
 @click.option(
-    '--additional-python-deps', help='Additional python dependencies to use when building the images.'
+    '--additional-python-deps',
+    help='Additional python dependencies to use when building the images.',
+    envvar='ADDITIONAL_PYTHON_DEPS',
 )
 @click.option(
-    '--additional_dev_apt_command', help='Additional command executed before dev apt deps are installed.'
+    '--additional-dev-apt-command',
+    help='Additional command executed before dev apt deps are installed.',
+    envvar='ADDITIONAL_DEV_APT_COMMAND',
 )
 @click.option(
-    '--additional_runtime_apt_command',
+    '--additional-runtime-apt-command',
     help='Additional command executed before runtime apt deps are installed.',
+    envvar='ADDITIONAL_RUNTIME_APT_COMMAND',
 )
 @click.option(
-    '--additional_dev_apt_env', help='Additional environment variables set when adding dev dependencies.'
+    '--additional-dev-apt-env',
+    help='Additional environment variables set when adding dev dependencies.',
+    envvar='ADDITIONAL_DEV_APT_ENV',
 )
 @click.option(
-    '--additional_runtime_apt_env',
+    '--additional-runtime-apt-env',
     help='Additional environment variables set when adding runtime dependencies.',
+    envvar='ADDITIONAL_RUNTIME_APT_ENV',
+)
+@click.option(
+    '--dev-apt-command',
+    help='The basic command executed before dev apt deps are installed.',
+    envvar='DEV_APT_COMMAND',
 )
-@click.option('--dev-apt-command', help='The basic command executed before dev apt deps are installed.')
 @click.option(
     '--dev-apt-deps',
     help='The basic apt dev dependencies to use when building the images.',
+    envvar='DEV_APT_DEPS',
 )
 @click.option(
-    '--runtime-apt-command', help='The basic command executed before runtime apt deps are installed.'
+    '--runtime-apt-command',
+    help='The basic command executed before runtime apt deps are installed.',
+    envvar='RUNTIME_APT_COMMAND',
 )
 @click.option(
     '--runtime-apt-deps',
     help='The basic apt runtime dependencies to use when building the images.',
+    envvar='RUNTIME_APT_DEPS',
 )
-@click.option('--github-repository', help='Choose repository to push/pull image.')
+@click.option('--github-repository', help='Choose repository to push/pull image.', envvar='GITHUB_REPOSITORY')
 @click.option('--build-cache', help='Cache option')
-@click.option('--platform', help='Builds image for the platform specified.')
+@click.option('--platform', help='Builds image for the platform specified.', envvar='PLATFORM')
 @click.option(
     '-d',
     '--debian-version',
     help='Debian version used for the image.',
     type=click.Choice(ALLOWED_DEBIAN_VERSIONS),
+    envvar='DEBIAN_VERSION',
+)
+@click.option(
+    '--upgrade-to-newer-dependencies',
+    help='Upgrades PIP packages to latest versions available without looking at the constraints.',
+    envvar='UPGRADE_TO_NEWER_DEPENDENCIES',
 )
-@click.option('--upgrade-to-newer-dependencies', is_flag=True)
 def build_ci_image(
     verbose: bool,
     additional_extras: Optional[str],
@@ -254,7 +277,7 @@ def build_ci_image(
     build_cache: Optional[str],
     platform: Optional[str],
     debian_version: Optional[str],
-    upgrade_to_newer_dependencies: bool,
+    upgrade_to_newer_dependencies: str,
 ):
     """Builds docker CI image without entering the container."""
 
@@ -263,6 +286,7 @@ def build_ci_image(
             f"\n[blue]Building image of airflow from {__AIRFLOW_SOURCES_ROOT} "
             f"python version: {python}[/]\n"
         )
+    create_directories()
     build_image(
         verbose,
         additional_extras=additional_extras,
@@ -282,7 +306,7 @@ def build_ci_image(
         docker_cache=build_cache,
         platform=platform,
         debian_version=debian_version,
-        upgrade_to_newer_dependencies=str(upgrade_to_newer_dependencies).lower(),
+        upgrade_to_newer_dependencies=upgrade_to_newer_dependencies,
     )
 
 
diff --git a/dev/breeze/src/airflow_breeze/utils/path_utils.py b/dev/breeze/src/airflow_breeze/utils/path_utils.py
--- a/dev/breeze/src/airflow_breeze/utils/path_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/path_utils.py
@@ -75,5 +75,5 @@ def create_directories():
     DIST_DIR.mkdir(parents=True, exist_ok=True)
     CACHE_TMP_FILE_DIR = tempfile.TemporaryDirectory()
     # add trap to this cache_tmp_file_dir
-    OUTPUT_LOG = Path(CACHE_TMP_FILE_DIR, 'out.log')
+    OUTPUT_LOG = Path(CACHE_TMP_FILE_DIR.name, 'out.log')
     OUTPUT_LOG.mkdir(parents=True, exist_ok=True)


Test patch: