Repository: mantidproject/mantid
Pull request #38976: 38882 gsas2scriptable path search revision
Author: login: JackEAllen
Labels: name: Diffraction
description: Issues and pull requests related to diffraction
name: Powder
description: Issues and pull requests related to powder diffraction
name: ISIS: Diffraction
description: Issue and pull requests relating to Diffraction at ISIS
Created: 2025-02-25T13:14:35Z | Merged: 2025-03-27T15:30:27Z

### Description of work

#### Summary of work
<!-- Please provide a short, high level description of the work that was done.
-->
Dynamically search for `GSASIIscriptable.py` from outer most parent library of [GSAS-II](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/) using a recursive directory search with a depth limit of 3 set to avoid searching too deeply into a given directory structure.

Improve error message return descriptions where:
* `GSASIIscriptable.py` is not found.
* `GSASIIscriptable.py` has been found, but an import error occurred.
* Python binaries could not been found in `/gsas2_parent_dir/GSAS-II/bin/python`


<!-- Why has this work been done? If there is no linked issue please provide appropriate context for this work.
#### Purpose of work
This can be removed if a github issue is referenced below
-->

The location of `GSASIIscriptable.py` is not consistent across different versions of [GSAS-II](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/). For newer version of GSAS-II ([5758](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/releases/tag/5758) or [later](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/releases/tag/latest)), `GSASIIscriptable.py` is not correctly located

### Fixes #38882  <!-- and fix #xxxx or close #xxxx xor resolves #xxxx. One line per issue fixed. -->
<!-- alternative
*There is no associated issue.*
-->

<!-- If the original issue was raised by a user they should be named here. Do not leak email addresses
**Report to:** [user name]
-->

#### Further detail of work
<!-- Please provide a more detailed description of the work that has been undertaken.
-->
[Mantid anaconda label](https://anaconda.org/mantid/mantidworkbench): `new_gsas_path_search_test` - **PLEASE REMOVE LABEL FROM ANACONDA ONCE PR IS APPROVED.**

### To test:

<!-- Instructions for testing.
There should be sufficient instructions for someone unfamiliar with the application to test - unless a specific
reviewer is requested.
If instructions for replicating the fault are contained in the linked issue then it is OK to refer back to these.
-->

**Testing with newer versions of GSAS-II:**

1. Install a newer version of GSAS-II such as version [5758](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/releases/tag/5758) or [later](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/releases/tag/latest).
2. Checkout the `main` branch: `git switch main && git fetch && git pull`
3. Launch workbench from CLI `workbench` and follow the [Engineering Diffraction Test 11](https://developer.mantidproject.org/Testing/EngineeringDiffraction/EngineeringDiffractionTestGuide.html#test-11)
4. Notice that if the path is set to the outer most directory of GSAS-II, you will receive an error similar to:
```python
GSAS-II call failed with error: Traceback (most recent call last):
  File "/mantidproject/mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py", line 195, in main
    import GSASIIscriptable as G2sc
ModuleNotFoundError: No module named 'GSASIIscriptable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mantidproject/mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py", line 232, in <module>
    main()
  File "/mantidproject/mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py", line 197, in main
    raise ImportError(f"GSAS-II was not found at {import_path}")
ImportError: GSAS-II was not found at /gsas2/GSASII
```
5. If the path is intuitively set closer to `GSASIIscriptable.py`, maybe `gsas2/GSAS-II`, you will receive an similar error to:
```python
GSAS-II call failed with error: [Errno 2] No such file or directory: '/gsas2/GSAS-II/bin/python'. Please check the Path to GSASII in the Engineering Diffraction Settings is valid
```
6. Switch to the the PR branch: `git switch 38882_gsas2scriptable_path_search_revision`
7. Repeat steps `3-5`, and notice that the error for step `4` GSAS should now load correctly. For step `5`, notice that the same error will be returned with the additional statement: `Path must be outer most directory of GSAS-II installation.` This message should better aid a user to the directory needed for GSAS-II to load correctly.
8. Move the `GSASIIscriptable.py` to another directory within the GSAS library such as directly under the parent folder. the rough location of `GSASIIscriptable.py` from the parent directory of the library will be: `/gsas2/GSAS-II/GSASII/GSASIIscriptable.py` for newer versions of GSAS-II
9. On the GSAS II tab in the Engineering Diffraction interface, click refine after following step `3`. You should receive an import error that specified that `GSASIIscriptable.py`has been found, but an import error occurred similar to:
 ```python
  File "/mantidproject/mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py", line 200, in find_gsasii
    raise ImportError(
ImportError: GSASIIscriptable (GSASIIscriptable.py) module found in /gsas2 but could not be imported
```
10. Move the `GSASIIscriptable.py` back to it's original location: ~`/gsas2/GSAS-II/GSASII/GSASIIscriptable.py`

**Testing with older versions of GSAS-II:**
1. Login to [IDAaaS](https://isis.analysis.stfc.ac.uk/) and create a new EnginX workspace.
2. Check the version of GSAS in the workspace is `4579`. This can be checked before launching the workspace by clicking on the setting cog at the top right of the workspace under the "Included Software" subheading.
3. Launch the EnginX workspace.
4. Open a terminal `Applications >> System >> Terminal`
5. Execute the following command: 
```bash
mamba create -n mantid-test -c mantid/label/new_gsas_path_search_test -c mantid/label/nightly mantidworkbench
``` 
`-c mantid/label/nightly` is required due to a an mslice version being pinned which will prevent you from creating a valid environment otherwise.
6. Activate your environment `mamba activate mantid-test` and open workbench `workbench` from the CLI.
7. Repeat steps `3-5`, from "Testing with newer versions of GSAS-II" and notice the update error messages, mentioned in step `7`.

<!-- Ensure the base of this PR is correct (e.g. release-next or main)
Finally, don't forget to add the appropriate labels, milestones, etc.!  -->

---

### Reviewer

Please comment on the points listed below ([full description](http://developer.mantidproject.org/ReviewingAPullRequest.html)).
**Your comments will be used as part of the gatekeeper process, so please comment clearly on what you have checked during your review.** If changes are made to the PR during the review process then your final comment will be the most important for gatekeepers. In this comment you should make it clear why any earlier review is still valid, or confirm that all requested changes have been addressed.

#### Code Review

- Is the code of an acceptable quality?
- Does the code conform to the [coding standards](http://developer.mantidproject.org/Standards/)?
- Are the unit tests small and test the class in isolation?
- If there is GUI work does it follow the [GUI standards](http://developer.mantidproject.org/Standards/GUIStandards.html)?
- If there are changes in the release notes then do they describe the changes appropriately?
- Do the release notes conform to the [release notes guide](https://developer.mantidproject.org/Standards/ReleaseNotesGuide.html)?

#### Functional Tests

- Do changes function as described? Add comments below that describe the tests performed?
- Do the changes handle unexpected situations, e.g. bad input?
- Has the relevant (user and developer) documentation been added/updated?

Does everything look good? Mark the review as **Approve**. A member of `@mantidproject/gatekeepers` will take care of it.

### Gatekeeper

If you need to request changes to a PR then please add a comment and set the review status to "Request changes". This will stop the PR from showing up in the list for other gatekeepers.


Linked issue #38882: GSASIIscriptable Path is Not Correctly Located For Newer Versions of GSAS-II
<!-- Issues which impact security should have a Vulnerability label added to them -->

<!-- If this issue is raised on behalf of a user they should be named here.
**Original reporter:** [username facility]/[nobody]
-->

**Describe the bug**
<!-- A clear and concise description of what the bug is. -->

The location of `GSASIIscriptable.py` is not consistent across different versions of GSAS-II. For newer version of GSAS-II (`5758` or later), `GSASIIscriptable.py` is not correctly located by `mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py`

It is no longer possible to download older version of [GSAS](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/), previously accessible from the GSAS SVN server for each OS. From further investigation, only versions `5758` and later are available to download from the [GSAS repository](https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/releases)

To resolve this, instead of looking for a hard coded path using `import_path = os.path.join(path_to_gsas2, "GSASII")`, a recursive search should be taken to search all subdirectories within the parent GSAS-II installation directory for `GSASIIscriptable.py`

**To Reproduce**

1. Install a version of GSAS-II that is `>= 5758` from https://advancedphotonsource.github.io/GSAS-II-tutorials/install.html for your specific OS.
2. Go to `Interfaces>>Diffraction>>Engineering Diffraction`
3. Go to the `Calibration` tab, select `Create New Calibration` and un-tick the `Crop Calibration` option.
4. For the `Calibration Sample #` enter `305738` and click the Calibrate button.
5. On the `Focus` tab, enter `Sample Run #` `305761` and `Vanadium #` `307521` and click the Focus button.
6. Change to the `GSASII` tab. Enter a project name in the `Project Name` text input. The Instrument Group path should be pre-filled to a `.prm` file output by the calibration and the Focused Data path should be pre-filled to the `.gss` file output from the `Focus` tab.
7. Enter a path to a valid phase file which will end in `.cif` such as `mantid/scripts/Engineering/ENGINX/phase_info/FE_GAMMA.cif` into the `Phase` text input.
8. Enter the Engineering Diffraction settings menu by clicking the cog wheel in the bottom left.
9. Click `Browse` and select the path to the outer most parent directory of your GSAS-II installation which may be called `gsas2` or `g2full` and is likely installed in your home directory in the `Path to GSASII` input area. Click `Apply` and then `OK`.
10. On the `GSAS II` tab, click `Refine`
11. This will cause an error which can be seen in the `Messages` panel on the right of the Mantid Workbench window, stating that the path to GSAS-II is incorrect similar to:
```bash
GSAS-II call failed with error: [Errno 2] No such file or directory: '/Users/username/gsas2/GSAS-II/bin/python'. Please check the Path to GSASII in the Engineering Diffraction Settings is valid. 
``` 
(For older versions of GSAS-II such as `4579`, which a linux version is available on IDAaaS for EnginX that can be copied, no error will be returned and the GSAS II Plot will be plotted as expected)

Platform/Version (please complete the following information):**
 - OS: Windows, RHEL 7, Ubuntu, macOS
 - GSAS-II Version: 15758` and later
 - Mantid Version 6.11.0 (Likely affect previous version also)

**Additional context**
<!--Add any other context about the problem here.-->


Diff:
diff --git a/docs/source/release/v6.13.0/Diffraction/Engineering/Bugfixes/38882.rst b/docs/source/release/v6.13.0/Diffraction/Engineering/Bugfixes/38882.rst
new file mode 100644
index 000000000000..13c4fb3eaf9c
--- /dev/null
+++ b/docs/source/release/v6.13.0/Diffraction/Engineering/Bugfixes/38882.rst
@@ -0,0 +1 @@
+- `#38882 <https://github.com/mantidproject/mantid/issues/38882>`_ : Fix issue with ``GSAS-II GSASIIscriptable.py`` hard-coded path which is invalid for newer version of GSAS-II (versions 5758 and later).
diff --git a/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py b/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
index 8740aa95a10b..33a3171545f5 100644
--- a/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
+++ b/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
@@ -8,6 +8,10 @@
 import os
 import sys
 import json
+from pathlib import Path
+import fnmatch
+from types import ModuleType
+from typing import Generator
 
 
 def print_histogram_R_factors(project):
@@ -164,6 +168,42 @@ def export_lattice_parameters(temp_save_directory, name_of_project, project):
             file.write(parameters_json)
 
 
+def limited_rglob(directory: Path, pattern: str, max_depth: int) -> Generator[Path, None, None]:
+    """
+    Recursively search for files matching the pattern in the directory up to a specified depth.
+    """
+    if not directory.is_dir():
+        raise FileNotFoundError(f"The provided directory path '{directory}' is not a valid directory.")
+    for root, dirs, files in os.walk(directory):
+        current_depth = len(Path(root).parts) - len(directory.parts)
+        if current_depth > max_depth:
+            dirs[:] = []
+        if current_depth <= max_depth:
+            for file in files:
+                if fnmatch.fnmatch(file, pattern):
+                    yield Path(root) / file
+
+
+def find_gsasii(directory_path: Path, file_name: str) -> ModuleType:
+    """
+    Recursively search for the file_name in the directory_path - the search checks all subdirectories
+    :Raises ImportError: If the file_name is not found in the directory_path
+    """
+    for file_path in limited_rglob(directory_path, file_name, max_depth=3):
+        if file_path.is_file():
+            if str(file_path.parent) not in sys.path:
+                sys.path.append(str(file_path.parent))
+            try:
+                import GSASIIscriptable as G2sc
+
+                return G2sc
+            except ImportError as exc:
+                raise ImportError(
+                    f"GSASIIscriptable ({file_name}) module found in {str(file_path.parent)} but could not be imported"
+                ) from exc
+    raise ImportError(f"GSASIIscriptable module '{file_name}' could not be found using the provided path: {directory_path}")
+
+
 def main():
     # Parse Inputs from Mantid
     inputs_dict = json.loads(sys.argv[1])
@@ -188,13 +228,7 @@ def main():
     number_of_regions = inputs_dict["number_of_regions"]
 
     # Call GSASIIscriptable
-    import_path = None
-    try:
-        import_path = os.path.join(path_to_gsas2, "GSASII")
-        sys.path.insert(0, import_path)
-        import GSASIIscriptable as G2sc
-    except ModuleNotFoundError:
-        raise ImportError(f"GSAS-II was not found at {import_path}")
+    G2sc = find_gsasii(Path(path_to_gsas2), "GSASIIscriptable.py")
 
     project_path = os.path.join(temporary_save_directory, project_name + ".gpx")
     gsas_project = G2sc.G2Project(filename=project_path)
diff --git a/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/model.py b/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/model.py
index 842f63e8324a..16ae4a85002f 100644
--- a/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/model.py
+++ b/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/model.py
@@ -286,7 +286,8 @@ def call_subprocess(self, command_string_list):
         except Exception as exc:
             logger.error(
                 f"GSAS-II call failed with error: {str(exc)}. "
-                "Please check the Path to GSASII in the Engineering Diffraction Settings is valid"
+                "Please check the Path to GSASII in the Engineering Diffraction Settings is valid. "
+                "Path must be outer most directory of GSAS-II installation."
             )
             return None
 
diff --git a/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/test/test_call_G2sc.py b/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/test/test_call_G2sc.py
index 531dcba8c7da..ddd8d7e7f7d2 100644
--- a/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/test/test_call_G2sc.py
+++ b/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/test/test_call_G2sc.py
@@ -25,9 +25,13 @@
     export_reflections,
     export_refined_instrument_parameters,
     export_lattice_parameters,
+    limited_rglob,
+    find_gsasii,
 )
 
 import numpy as np
+from pathlib import Path
+from types import ModuleType
 
 
 class GSAS2ViewTest(unittest.TestCase):
@@ -268,3 +272,105 @@ def test_export_lattice_parameters(self):
         with open(os.path.join(self.temp_save_directory, "project_cell_parameters_phase_2.txt"), "rt", encoding="utf-8") as file:
             str_2 = file.read()
             self.assertEqual(str_2, """{"Microstrain":10}""")
+
+
+class TestLimitedRglob(unittest.TestCase):
+    def setUp(self):
+        self.test_dir = tempfile.mkdtemp()
+        self.addCleanup(shutil.rmtree, self.test_dir)
+
+    def create_test_files(self, files):
+        for file in files:
+            file_path = os.path.join(self.test_dir, file)
+            os.makedirs(os.path.dirname(file_path), exist_ok=True)
+            with open(file_path, "w") as f:
+                f.write("test")
+
+    def test_limited_rglob_match_at_parent_level(self):
+        files = ["file1.txt", "file2.log"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=0))
+        expected = [Path(self.test_dir) / "file1.txt"]
+        self.assertEqual(result, expected)
+
+    def test_limited_rglob_single_level(self):
+        files = ["dir1/file1.txt", "dir1/file2.txt"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=1))
+        expected = [Path(self.test_dir) / file for file in files]
+        self.assertCountEqual(result, expected)
+
+    def test_limited_rglob_exceeds_max_depth(self):
+        files = ["dir1/dir2/file1.txt", "dir1/dir2/dir3/file2.txt"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=1))
+        self.assertCountEqual(result, [])
+
+    def test_limited_rglob_no_match(self):
+        files = ["dir1/file1.txt", "dir1/file2.log"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.csv", max_depth=1))
+        self.assertEqual(result, [])
+
+    def test_limited_rglob_multiple_levels(self):
+        files = ["dir1/dir2/dir3/file1.txt", "dir1/dir2/dir3/dir4/file2.txt", "dir1/dir2/dir3/dir4/dir5/file3.txt"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=4))
+        expected = [
+            Path(self.test_dir) / "dir1/dir2/dir3/file1.txt",
+            Path(self.test_dir) / "dir1/dir2/dir3/dir4/file2.txt",
+        ]
+        self.assertCountEqual(result, expected)
+
+    def test_limited_rglob_exact_max_depth(self):
+        files = ["dir1/dir2/file1.txt", "dir1/dir2/dir3/file2.txt"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=2))
+        expected = [Path(self.test_dir) / "dir1/dir2/file1.txt"]
+        self.assertCountEqual(result, expected)
+
+    def test_limited_rglob_different_patterns(self):
+        files = ["dir1/file1.txt", "dir1/file2.log", "dir1/file3.csv"]
+        self.create_test_files(files)
+        result_txt = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=1))
+        result_log = list(limited_rglob(Path(self.test_dir), "*.log", max_depth=1))
+        result_csv = list(limited_rglob(Path(self.test_dir), "*.csv", max_depth=1))
+        self.assertCountEqual(result_txt, [Path(self.test_dir) / "dir1/file1.txt"])
+        self.assertCountEqual(result_log, [Path(self.test_dir) / "dir1/file2.log"])
+        self.assertCountEqual(result_csv, [Path(self.test_dir) / "dir1/file3.csv"])
+
+    def test_limited_rglob_empty_directory(self):
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=1))
+        self.assertEqual(result, [])
+
+    def test_limited_rglob_similar_directory_names(self):
+        files = ["dir1/file1.txt", "dir1_2/file2.txt"]
+        self.create_test_files(files)
+        result = list(limited_rglob(Path(self.test_dir), "*.txt", max_depth=1))
+        expected = [
+            Path(self.test_dir) / "dir1/file1.txt",
+            Path(self.test_dir) / "dir1_2/file2.txt",
+        ]
+        self.assertCountEqual(result, expected)
+
+    def test_limited_rglob_invalid_directory(self):
+        with self.assertRaises(FileNotFoundError):
+            list(limited_rglob(Path("/invalid/directory"), "*.txt", max_depth=1))
+
+    def test_find_gsasii_success(self):
+        files = ["GSASIIscriptable.py"]
+        self.create_test_files(files)
+        with mock.patch("builtins.__import__", return_value=mock.Mock(spec=ModuleType)):
+            result = find_gsasii(Path(self.test_dir), "GSASIIscriptable.py")
+            self.assertIsInstance(result, ModuleType)
+
+    def test_find_gsasii_not_found(self):
+        with self.assertRaises(ImportError):
+            find_gsasii(Path(self.test_dir), "GSASIIscriptable.py")
+
+    def test_find_gsasii_import_error(self):
+        files = ["GSASIIscriptable.py"]
+        self.create_test_files(files)
+        with mock.patch("builtins.__import__", side_effect=ImportError):
+            with self.assertRaises(ImportError):
+                find_gsasii(Path(self.test_dir), "GSASIIscriptable.py")


Reviews:
user: login: RichardWaiteSTFC
body: This is a big improvement thanks for doing this! I have tested on windows with a newer version and everything works great and the error messages are a lot more helpful.

I have yet to test on IDAaaS as they seem to have network issues, in the meantime I have  a minor suggestion for the release note. Also what do you think about adding a unit test?
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T13:24:43Z
user: login: JackEAllen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:12:33Z
user: login: JackEAllen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:15:39Z
user: login: JackEAllen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:20:38Z
user: login: RichardWaiteSTFC
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:32:03Z
user: login: RichardWaiteSTFC
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:33:02Z
user: login: JackEAllen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:38:01Z
user: login: RichardWaiteSTFC
body: Test instructions work on IDAaaS (old GSAS version) and on my windows with newer GSAS version. Thanks for adding the test - it all looks great to me!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T13:25:58Z

Review comments:
diff_hunk: @@ -0,0 +1 @@
+- `#38882 <https://github.com/mantidproject/mantid/issues/38882>`_ : Fix issue with ``GSAS-II GSASIIscriptable.py`` hard-coded path which is invalid for newer version of GSAS-II.
path: docs/source/release/v6.13.0/Diffraction/Engineering/Bugfixes/38882.rst
user: login: RichardWaiteSTFC
body: Probably worth adding a version number which we know doesn't work on main? Just so people get a feel for what `newer` versions of GSAS-II means?
created_at: 2025-03-07T13:14:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,6 +168,41 @@ def export_lattice_parameters(temp_save_directory, name_of_project, project):
             file.write(parameters_json)
 
 
+def limited_rglob(directory: Path, pattern: str, max_depth: int) -> Generator[Path, None, None]:
path: qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
user: login: RichardWaiteSTFC
body: Technically this could have a unit test associate with it in `test_call_G2sc.py` - perhaps setting up a temp directory that reflects the old and new GSAS version setup? But that might be overkill - what do you think?

Don't worry about making any changes as this seems to work well. However, it occurred to me the logic in this function is quite complicated for what it is doing (but maybe I'm not grasping all the subtleties)!

For example on L177
```
if depth >= max_depth:
            dirs[:] = []  # Max depth reached, don't recurse further
```
Is there any reason one couldn't break here and return the appropriate null path (anything that isn't a file I think)?
Could depth instead just be a counter and do `depth += 1` instead of `depth = root[len(str(directory)) + len(os.path.sep) :].count(os.path.sep) + 1` etc.?
created_at: 2025-03-07T13:22:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,6 +168,41 @@ def export_lattice_parameters(temp_save_directory, name_of_project, project):
             file.write(parameters_json)
 
 
+def limited_rglob(directory: Path, pattern: str, max_depth: int) -> Generator[Path, None, None]:
path: qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
user: login: JackEAllen
body: Using break would exit the loop entirely, which might  mean we don't find the file we are searching for if there are other directories at the same level to process. 

By using `dirs[:] = []`, we ensure that the traversal stops at the current directory level without affecting the parent directory's traversal if there are other directories at at the same level. This approach basically future proofs against the path ever changes again. 

Using `depth = root[len(str(directory)) + len(os.path.sep) :].count(os.path.sep) + 1` will have the downside of being more complicated compared to using a counter due to the usage of string slicing and counting separators, but will be more accurate. 
The depth is recalculated based on the actual path, ensuring it accurately reflects the directory structure which will be resilient against the directory structure possibly changing during execution (though this is very unlikely). The main benefit is that this will also handle edge cases where directories might be skipped or have varying depths. 

I agree that this method is likely too complicated for just the case of searching for `GSASIIscriptable.py`.
My plan is to potentially move this method out of `engineering_diffraction/tabs/gsas2/call_G2sc.py` and into `engineering_diffraction/tabs/gsas2/model.py` as part of a larger refactor and use the same method to also search for the python binaries which are searched for within `mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/model.py.call_subprocess` to protect against possible being moved elsewhere within the GSAS-II package. 
created_at: 2025-03-07T15:12:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,6 +168,41 @@ def export_lattice_parameters(temp_save_directory, name_of_project, project):
             file.write(parameters_json)
 
 
+def limited_rglob(directory: Path, pattern: str, max_depth: int) -> Generator[Path, None, None]:
path: qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
user: login: JackEAllen
body: Though adding a test could be considered overkill as path searching errors will be caught elsewhere when searching for `GSASIIscriptable.py` I believe you're right to suggest adding a test. Thank you for catching this!

I'll work on adding a test to check a variety of different directory structures as this will also be a useful test to have for when this method is possibly moved into `mantid/qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/model.py`
created_at: 2025-03-07T15:15:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1 @@
+- `#38882 <https://github.com/mantidproject/mantid/issues/38882>`_ : Fix issue with ``GSAS-II GSASIIscriptable.py`` hard-coded path which is invalid for newer version of GSAS-II.
path: docs/source/release/v6.13.0/Diffraction/Engineering/Bugfixes/38882.rst
user: login: JackEAllen
body: Good point! There may be other older versions of GSAS-II between the one installed on IDAaaS (`4579`) and the most recent version I have tested with (`5758`) that may also cause a path issue prior to this PR that I'm not aware of, however we should specify at least one known version that breaks on main currently. I will update the release notes to include the newest version of GSAS-II `5758` that this PR aims to resolve. 
created_at: 2025-03-07T15:20:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,6 +168,41 @@ def export_lattice_parameters(temp_save_directory, name_of_project, project):
             file.write(parameters_json)
 
 
+def limited_rglob(directory: Path, pattern: str, max_depth: int) -> Generator[Path, None, None]:
path: qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
user: login: RichardWaiteSTFC
body: Thanks for adding a test!
RE logic/counters etc. - I didn't mean drag and drop the counter etc.  into the current logic. I know it would require different ifs and loops etc. To be clear I'm not suggesting you change what you have! Also I hadn't thought of the directory changing mid-execution - a good point!
created_at: 2025-03-07T15:32:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1 @@
+- `#38882 <https://github.com/mantidproject/mantid/issues/38882>`_ : Fix issue with ``GSAS-II GSASIIscriptable.py`` hard-coded path which is invalid for newer version of GSAS-II.
path: docs/source/release/v6.13.0/Diffraction/Engineering/Bugfixes/38882.rst
user: login: RichardWaiteSTFC
body: Yep I think there's no point going through GSAS github trying to find out when the directory changes, but agreed, if you could add that it will now work for version `5758` as you suggest - that would be great thanks!
created_at: 2025-03-07T15:33:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,6 +168,41 @@ def export_lattice_parameters(temp_save_directory, name_of_project, project):
             file.write(parameters_json)
 
 
+def limited_rglob(directory: Path, pattern: str, max_depth: int) -> Generator[Path, None, None]:
path: qt/python/mantidqtinterfaces/mantidqtinterfaces/Engineering/gui/engineering_diffraction/tabs/gsas2/call_G2sc.py
user: login: JackEAllen
body: I'll refresh the review once I've added a test 👍 
created_at: 2025-03-07T15:38:01Z
author_association: CONTRIBUTOR

PR comments:
user: login: RichardWaiteSTFC
created_at: 2025-03-10T13:42:07Z
body: Works on IDAaaS with older version of GSAS so looks good - just waiting on the new test (no rush!)
author_association: CONTRIBUTOR
user: login: JackEAllen
created_at: 2025-03-27T13:46:00Z
body: The directory and file testing could possibly be improved if [pyfakefs](https://anaconda.org/conda-forge/pyfakefs) was added as a dependency to Mantid
author_association: CONTRIBUTOR