Repository: apache/airflow

Issue:
CI: Rewrite `free_space` script in Python
In our build/ci.yml (maybe other?) files we are using a "free_space" script that performs cleanup of the machine before running the tasks. This allows us to reclaim memory and disk space for our tasks.

Example: https://github.com/apache/airflow/blob/eaa8ac72fc901de163b912a94dbe675045d2a009/.github/workflows/ci.yml#L334

This should be written in Python and our ci.yml and build.yml should be updated. We shoudl also be able to remove free_space script from the repo.



Discussion:
I see airflow/scripts/ci/tools is where we have all the sh files.
This script free_space.sh contains another script: 
`$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh` and this another one.

Should I focus just on free_space.sh, right?

I would like to work on this issue :)
The "init" script contains some common setup which are needed for Bash and defines common functions used by both Breeze and CI, but they are largely unnecessary for Python and if we find that we need some common code we will extract it to some common packages that will be used by both CI and Breeze.
So yeah. Just port what the `free_space" does - the common parts we will extract later if /when needed.

Resolving patch:
diff --git a/dev/breeze/src/airflow_breeze/ci/__init__.py b/dev/breeze/src/airflow_ci/__init__.py
similarity index 100%
rename from dev/breeze/src/airflow_breeze/ci/__init__.py
rename to dev/breeze/src/airflow_ci/__init__.py
diff --git a/dev/breeze/src/airflow_ci/freespace.py b/dev/breeze/src/airflow_ci/freespace.py
new file mode 100755
--- /dev/null
+++ b/dev/breeze/src/airflow_ci/freespace.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""freespace.py for clean environment before start CI"""
+
+import shlex
+import subprocess
+from typing import List
+
+import click
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+option_verbose = click.option(
+    "--verbose",
+    envvar='VERBOSE',
+    is_flag=True,
+    help="Print verbose information about free space steps",
+)
+
+
+@click.group()
+def main():
+    pass
+
+
+@option_verbose
+def run_command(cmd: List[str], verbose, *, check: bool = True, **kwargs):
+    if verbose:
+        console.print(f"\n[green]$ {' '.join(shlex.quote(c) for c in cmd)}[/]\n")
+    try:
+        subprocess.run(cmd, check=check, **kwargs)
+    except subprocess.CalledProcessError as ex:
+        print("========================= OUTPUT start ============================")
+        print(ex.stderr)
+        print(ex.stdout)
+        print("========================= OUTPUT end ============================")
+
+
+@main.command()
+@option_verbose
+def free_space(verbose):
+    run_command(["sudo", "swapoff", "-a"], verbose)
+    run_command(["sudo", "rm", "-f", "/swapfile"], verbose)
+    run_command(["sudo", "apt", "clean", "||", "true"], verbose)
+    run_command(["docker", "system", "prune", "--all", "--force", "volumes"], verbose)
+    run_command(["df", "-h"], verbose)
+    run_command(["docker", "logout", "ghcr.io"], verbose)
+
+
+if __name__ == '__main__':
+    main()


Test patch: