Repository: E3SM-Project/mache
Pull request #229: Add a utility that shows updates to `config_machines.xml`
Author: login: xylar
Labels: name: enhancement
description: New feature or request
Created: 2025-02-04T21:22:57Z | Merged: 2025-03-31T19:35:48Z

<!--
Thank you for your pull request.
Please add a description of what is accomplished in the PR here at the top:
-->

<!--
Below are a few things we ask you or your reviewers to kindly check. 
***Remove checks that are not relevant by deleting the line(s) below.***
-->
Checklist
* [ ] `Testing` comment in the PR documents testing used to verify the changes

<!--
Please note any issues this fixes using closing keywords: https://help.github.com/articles/closing-issues-using-keywords
-->




Diff:
diff --git a/.gitignore b/.gitignore
index 611a79e9..290554a1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -139,3 +139,6 @@ cython_debug/
 
 # pycharm
 .idea/
+
+# xml files in the root directory
+/*.xml
diff --git a/mache/cime_machine_config/__init__.py b/mache/cime_machine_config/__init__.py
index e69de29b..a7c95dfe 100644
--- a/mache/cime_machine_config/__init__.py
+++ b/mache/cime_machine_config/__init__.py
@@ -0,0 +1,79 @@
+import difflib
+
+from lxml import etree
+from termcolor import colored
+
+
+def extract_supported_machines(input_xml, output_xml, supported_machines):
+    """
+    Extract supported machines from an XML file and write to a new XML file.
+
+    Parameters
+    ----------
+    input_xml : str
+        The path to the input XML file.
+    output_xml : str
+        The path to the output XML file.
+    supported_machines : list of str
+        A list of supported machine names.
+    """
+    tree = etree.parse(input_xml)
+    root = tree.getroot()
+
+    for machine in root.findall('machine'):
+        mach_name = machine.get('MACH')
+        if mach_name not in supported_machines:
+            root.remove(machine)
+
+    tree.write(output_xml, pretty_print=True)
+
+
+def compare_machine_configs(old_xml, new_xml, machine_name):
+    """
+    Compare the XML within the `machine` tag for a given machine between the
+    old and new XML files and print the differences.
+
+    Parameters
+    ----------
+    old_xml : str
+        The path to the old XML file.
+    new_xml : str
+        The path to the new XML file.
+    machine_name : str
+        The name of the machine to compare.
+    """
+    old_tree = etree.parse(old_xml)
+    new_tree = etree.parse(new_xml)
+
+    old_machine = old_tree.find(f".//machine[@MACH='{machine_name}']")
+    new_machine = new_tree.find(f".//machine[@MACH='{machine_name}']")
+
+    if old_machine is None:
+        print(f'Machine {machine_name} not found in {old_xml}.')
+        return
+
+    if new_machine is None:
+        print(f'Machine {machine_name} not found in {new_xml}.')
+        return
+
+    old_machine_str = etree.tostring(old_machine, pretty_print=True).decode()
+    new_machine_str = etree.tostring(new_machine, pretty_print=True).decode()
+
+    diff = difflib.unified_diff(
+        old_machine_str.splitlines(),
+        new_machine_str.splitlines(),
+        fromfile='old',
+        tofile='new',
+        lineterm='',
+    )
+
+    print()
+    print(colored(f'Comparing machine: {machine_name}', 'blue'))
+    for line in diff:
+        if line.startswith('-'):
+            print(colored(line, 'red'))
+        elif line.startswith('+'):
+            print(colored(line, 'green'))
+        else:
+            print(line)
+    print()
diff --git a/mache/io.py b/mache/io.py
new file mode 100644
index 00000000..57c930b7
--- /dev/null
+++ b/mache/io.py
@@ -0,0 +1,19 @@
+import requests
+
+
+def download_file(url, local_filename):
+    """
+    Download a file from a URL.
+
+    Parameters
+    ----------
+    url : str
+        The URL of the file to download.
+    local_filename : str
+        The local path where the file will be saved.
+    """
+    with requests.get(url, stream=True) as r:
+        r.raise_for_status()
+        with open(local_filename, 'wb') as f:
+            for chunk in r.iter_content(chunk_size=8192):
+                f.write(chunk)
diff --git a/mache/machines/__init__.py b/mache/machines/__init__.py
index e69de29b..1cb2f7b7 100644
--- a/mache/machines/__init__.py
+++ b/mache/machines/__init__.py
@@ -0,0 +1,20 @@
+from importlib import resources as importlib_resources
+from pathlib import Path
+
+
+def get_supported_machines():
+    """
+    Get a list of supported machines by reading .cfg files in the
+    mache.machines directory.
+
+    Returns
+    -------
+    list of str
+        A sorted list of supported machine names.
+    """
+    machines = []
+    for file in importlib_resources.files('mache.machines').iterdir():
+        file_path = Path(str(file))
+        if file_path.suffix == '.cfg' and file_path.name != 'default.cfg':
+            machines.append(file_path.stem)
+    return sorted(machines)
diff --git a/spec-file.txt b/spec-file.txt
index 7051004e..76906a27 100644
--- a/spec-file.txt
+++ b/spec-file.txt
@@ -4,7 +4,9 @@ jinja2
 lxml
 pyyaml
 progressbar2
+requests
 rsync
+termcolor
 
 # Building
 setuptools >=60
diff --git a/utils/README.md b/utils/README.md
new file mode 100644
index 00000000..503ef069
--- /dev/null
+++ b/utils/README.md
@@ -0,0 +1,16 @@
+# Utilities for Mache Package
+
+This directory contains utilities that are helpful for making updates to the
+mache package.
+
+## update CIME machine config
+
+The `update_cime_machine_config.py` script can be used to download the latest
+version of the `config_machines.xml` file from E3SM's `master` branch, then
+compare it from the previous version stored in `mache` and show changes related
+to supported machines.
+
+A developer can then copy `new_config_machines.xml` into
+`mache/cime_machine_config/config_machines.xml` as part of a PR that makes
+relevant updates. They should also make the changes associated with the
+differences that this utility displays in the appropriate `mache/spack` files.
diff --git a/utils/update_cime_machine_config.py b/utils/update_cime_machine_config.py
new file mode 100755
index 00000000..e865360a
--- /dev/null
+++ b/utils/update_cime_machine_config.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+
+import argparse
+
+import mache.version
+from mache.cime_machine_config import (
+    compare_machine_configs,
+    extract_supported_machines,
+)
+from mache.io import download_file
+from mache.machines import get_supported_machines
+
+
+def main():
+    """
+    Main function to download the XML file, get supported machines, and extract
+    them, then compare the machine configurations between the old and new XML.
+    """
+    parser = argparse.ArgumentParser(
+        description='Get and display the updates made to '
+        'config_supported_machines.xml',
+    )
+
+    parser.add_argument(
+        '-v',
+        '--version',
+        action='version',
+        version=mache.version.__version__,
+        help='Show version number and exit',
+    )
+
+    parser.parse_args()
+
+    url = (
+        'https://raw.githubusercontent.com/E3SM-Project/E3SM/refs/heads/'
+        'master/cime_config/machines/config_machines.xml'
+    )
+    new_filename = 'new_config_machines.xml'
+    download_file(url, new_filename)
+    machines = get_supported_machines()
+    extract_supported_machines(
+        new_filename, 'new_config_supported_machines.xml', machines
+    )
+
+    old_filename = 'mache/cime_machine_config/config_machines.xml'
+    extract_supported_machines(
+        old_filename, 'old_config_supported_machines.xml', machines
+    )
+
+    for machine in machines:
+        compare_machine_configs(
+            'old_config_supported_machines.xml',
+            'new_config_supported_machines.xml',
+            machine,
+        )
+
+
+if __name__ == '__main__':
+    main()


Reviews:
user: login: xylar
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-28T01:06:22Z
user: login: altheaden
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-28T15:47:58Z
user: login: altheaden
body: This looks good to me after completing my testing which I documented above.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-31T18:54:22Z

Review comments:
diff_hunk: @@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+
+import argparse
+
+from mache.cime_machine_config import (
+    compare_machine_configs,
+    extract_supported_machines,
+)
+from mache.io import download_file
+from mache.machines import get_supported_machines
+
+
+def main():
+    """
+    Main function to download the XML file, get supported machines, and extract
+    them, then compare the machine configurations between the old and new XML.
+    """
+    parser = argparse.ArgumentParser(
+        description='Get and display the updates made to '
+        'config_supported_machines.xml',
+    )
+
+    parser.add_argument(
+        '-v',
+        '--version',
+        action='version',
+        version='',
path: utils/update_cime_machine_config.py
user: login: xylar
body: @altheaden, maybe it would make sense to show the `mache` version here?
```suggestion
        version=mache.version.__version__,
```
created_at: 2025-03-28T01:06:22Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+
+import argparse
+
+from mache.cime_machine_config import (
+    compare_machine_configs,
+    extract_supported_machines,
+)
+from mache.io import download_file
+from mache.machines import get_supported_machines
+
+
+def main():
+    """
+    Main function to download the XML file, get supported machines, and extract
+    them, then compare the machine configurations between the old and new XML.
+    """
+    parser = argparse.ArgumentParser(
+        description='Get and display the updates made to '
+        'config_supported_machines.xml',
+    )
+
+    parser.add_argument(
+        '-v',
+        '--version',
+        action='version',
+        version='',
path: utils/update_cime_machine_config.py
user: login: altheaden
body: Oops, I forgot to mention that I had left it blank as a placeholder since I wasn't sure what to put. I'll go ahead and make it the mache version!
created_at: 2025-03-28T15:47:58Z
author_association: COLLABORATOR

PR comments:
user: login: xylar
created_at: 2025-02-04T21:26:26Z
body: Also need to update `conda/meta.yaml` with the new dependencies.
author_association: COLLABORATOR
user: login: altheaden
created_at: 2025-03-27T16:28:09Z
body: @xylar I just pushed the argument parsing. Let me know if there's anything else you'd like me to add or change!
author_association: COLLABORATOR
user: login: altheaden
created_at: 2025-03-28T16:01:56Z
body: @xylar Things should be all sorted with the version now. 
author_association: COLLABORATOR
user: login: altheaden
created_at: 2025-03-31T18:53:50Z
body: ## Testing
I ran the utility with no arguments and saw that it did provide the updates as expected. I ran it with `--help` and `--version` as well and got the expected result for both.
author_association: COLLABORATOR
user: login: xylar
created_at: 2025-03-31T19:35:45Z
body: Thanks so much, @altheaden!
author_association: COLLABORATOR