Repository: Chia-Network/chia-blockchain
Pull request: CHIA-1697: Add new flag to support recursively scanning and following links
Fixes #16268

Adds new harvester option `follow_links` that only works when used with `recursive_scan` and uses `glob.glob` instead of `Path.rglob` in order to follow links when scanning recursively for plots.



Problem statement:
[Bug] Recursive plot scan ignores hard links
### What happened?

I have a main plotting directory that has a few mount points and a few hard links.

When I switched to using recursive searching I noticed that plots from mount points were picked fine, but no plots from hard-linked directories were farmed.

Here's the sample directory, I have to specify plot01-plot07 manually as an additional plot paths:

> -rwxrwxrwx+ 1 guest users 108826485488 Oct 10  2021 plot-k32-2021-10-10-05-01-a7279687971312438a432eb673fc13363d52f87aee459445e527701137c50197.plot
-rwxrwxrwx+ 1 guest users 108802486755 Oct 10  2021 plot-k32-2021-10-10-06-02-927a8d973e0f0fc18e67a0f3c9ffea66382c9e17d667f51eef172b993126ed78.plot
-rwxrwxrwx+ 1 guest users 108817698217 Oct 10  2021 plot-k32-2021-10-10-07-40-ff279245c837e85ff014feded1b7a6452d3601f62d4a27043ba5bc53c8d50d5a.plot
-rwxrwxrwx+ 1 guest users 108810943123 Oct 11  2021 plot-k32-2021-10-11-18-20-2da2729d47eae307c9f3a4ef93a0e8a8e95a8176b640d9d475fc37597edea022.plot
-rwxrwxrwx+ 1 guest users  88878366720 Sep  4 12:43 plot-k32-c04-2023-09-04-12-38-442cf42234781fd850739075976926ded7bd8ae0f1ed63fa538ebe507ee38500.plot
-rwxrwxrwx+ 1 guest users  88929333248 Sep  4 13:14 plot-k32-c04-2023-09-04-13-08-da133f47058df6bec61f323d8d779e6399668b33e0a8c37cb3cf04c452be0584.plot
-rwxrwxrwx+ 1 guest users  88847151104 Sep  4 13:48 plot-k32-c04-2023-09-04-13-42-719ff8172974391ca876362785ab006d226cc1d1fa275567d6b3da9161b0344d.plot
-rwxrwxrwx+ 1 guest users  88949571584 Sep  4 14:19 plot-k32-c04-2023-09-04-14-13-9c912efe43b2971edcca1d18aebb985fc9c0031748f45828efa04bda63ac53cc.plot
lrwxrwxrwx+ 1 root  root            16 Sep 29  2021 plots01 -> /volume2/plots01
lrwxrwxrwx+ 1 root  root            16 Sep 29  2021 plots02 -> /volume3/plots02
lrwxrwxrwx+ 1 root  root            16 Sep 29  2021 plots03 -> /volume4/plots03
lrwxrwxrwx+ 1 root  root            16 Sep 29  2021 plots04 -> /volume5/plots04
lrwxrwxrwx+ 1 root  root            16 Oct 16  2021 plots05 -> /volume6/plots05
lrwxrwxrwx+ 1 root  root            16 Sep 29  2021 plots06 -> /volume7/plots06
lrwxrwxrwx+ 1 root  root            16 Oct 14  2021 plots07 -> /volume8/plots07
drwxrwxrwx  4 root  root         16384 Apr  1  2022 plots08
drwxrwxrwx  4 root  root         12288 Apr  1  2022 plots09
drwxrwxrwx  4 root  root         24576 Feb 12  2022 plots10
drwxrwxrwx+ 2 guest users         4096 Sep 30  2021 plots11
drwxrwxrwx+ 2 guest users         4096 Oct  6  2021 plots12
drwxrwxrwx  4 root  root         24576 Apr  1  2022 plots13
drwxrwxrwx  4 root  root         20480 Apr  1  2022 plots14
drwxrwxrwx  4 root  root         16384 Apr  1  2022 plots15
drwxrwxrwx  4 root  root         24576 Apr  1  2022 plots16
drwxrwxrwx+ 2 guest users         4096 Sep  4 12:02 plots17
drwxrwxrwx  4 root  root         20480 Apr  1  2022 plots18
drwxrwxrwx  4 root  root         16384 Apr  1  2022 plots19
drwxrwxrwx  4 root  root         24576 Feb 12  2022 plots20
drwxrwxrwx  1 root  root       2097152 Jul 22 09:58 plots21
drwxrwxrwx  1 root  root       2097152 Jul 23 10:14 plots22
drwxrwxrwx+ 2 guest users         4096 Sep  4 12:02 plots23
drwxrwxrwx+ 2 guest users         4096 Sep  4 12:02 plots24
drwxrwxrwx+ 2 guest users         4096 Sep  4 12:02 plots25
drwxrwxrwx+ 2 guest users         4096 Sep  4 12:02 plots26
drwxrwxrwx+ 2 guest users         4096 Sep  4 12:02 plots27


### Version

2.0.0

### What platform are you using?

Linux

### What ui mode are you using?

GUI

### Relevant log output

_No response_

Hints from issue discussion:
We are using pathlib.rglob(*.plot) for resolving plots. Maybe there is an issue in that for hard links.

Commit message: Remove unneeded hardlink checks as hardlinks to directories aren't a thing

Patch to review:
diff --git a/chia/_tests/plotting/test_plot_manager.py b/chia/_tests/plotting/test_plot_manager.py
index 6989a77b8d33..07dd7e159063 100644
--- a/chia/_tests/plotting/test_plot_manager.py
+++ b/chia/_tests/plotting/test_plot_manager.py
@@ -737,3 +737,46 @@ async def test_recursive_plot_scan(environment: Environment) -> None:
     add_plot_directory(env.root_path, str(sub_dir_1_0_1.path))
     expected_result.loaded = []
     await env.refresh_tester.run(expected_result)
+
+
+@pytest.mark.limit_consensus_modes(reason="does not depend on consensus rules")
+@pytest.mark.parametrize("follow_links", [True, False])
+@pytest.mark.anyio
+async def test_recursive_plot_scan_symlinks(environment: Environment, follow_links: bool) -> None:
+    env: Environment = environment
+    # Create a directory tree with some subdirectories containing plots, others not.
+    root_plot_dir = env.root_path / "root"
+    sub_dir_0: Directory = Directory(root_plot_dir / "0", env.dir_1.plots[0:2])
+    sub_dir_0_1: Directory = Directory(sub_dir_0.path / "1", env.dir_1.plots[2:3])
+
+    # Create a plot directory outside of the root plot directory
+    other_plot_dir: Directory = Directory(env.root_path / "other", env.dir_1.plots[3:7])
+
+    # Create a symlink to the other plot directory from inside the root plot directory
+    symlink_path = Path(root_plot_dir / "other")
+
+    try:
+        symlink_path.symlink_to(env.root_path / "other")
+    except NotImplementedError:
+        pytest.skip("Symlinking is not supported on this platform")
+
+    # Adding the root without `recursive_plot_scan` and running a test should not load any plots (match an empty result)
+    expected_result = PlotRefreshResult()
+    add_plot_directory(env.root_path, str(root_plot_dir))
+    await env.refresh_tester.run(expected_result)
+
+    if follow_links:
+        expected_plot_list = sub_dir_0.plot_info_list() + sub_dir_0_1.plot_info_list() + other_plot_dir.plot_info_list()
+    else:
+        expected_plot_list = sub_dir_0.plot_info_list() + sub_dir_0_1.plot_info_list()
+
+    # Set the recursive scan flag in the config and symlink flag
+    with lock_and_load_config(env.root_path, "config.yaml") as config:
+        config["harvester"]["recursive_plot_scan"] = True
+        config["harvester"]["recursive_follow_links"] = follow_links
+        save_config(env.root_path, "config.yaml", config)
+
+    # With the flag enabled it should load all expected plots
+    expected_result.loaded = expected_plot_list  # type: ignore[assignment]
+    expected_result.processed = len(expected_plot_list)
+    await env.refresh_tester.run(expected_result)
diff --git a/chia/plotting/util.py b/chia/plotting/util.py
index b0b1642e46d9..a04a21876f54 100644
--- a/chia/plotting/util.py
+++ b/chia/plotting/util.py
@@ -110,13 +110,14 @@ def get_plot_filenames(root_path: Path) -> dict[Path, list[Path]]:
     all_files: dict[Path, list[Path]] = {}
     config = load_config(root_path, "config.yaml")
     recursive_scan: bool = config["harvester"].get("recursive_plot_scan", DEFAULT_RECURSIVE_PLOT_SCAN)
+    recursive_follow_links: bool = config["harvester"].get("recursive_follow_links", False)
     for directory_name in get_plot_directories(root_path, config):
         try:
             directory = Path(directory_name).resolve()
         except (OSError, RuntimeError):
             log.exception(f"Failed to resolve {directory_name}")
             continue
-        all_files[directory] = get_filenames(directory, recursive_scan)
+        all_files[directory] = get_filenames(directory, recursive_scan, recursive_follow_links)
     return all_files
 
 
@@ -219,7 +220,7 @@ def update_harvester_config(
         save_config(root_path, "config.yaml", config)
 
 
-def get_filenames(directory: Path, recursive: bool) -> list[Path]:
+def get_filenames(directory: Path, recursive: bool, follow_links: bool) -> list[Path]:
     try:
         if not directory.exists():
             log.warning(f"Directory: {directory} does not exist.")
@@ -229,8 +230,19 @@ def get_filenames(directory: Path, recursive: bool) -> list[Path]:
         return []
     all_files: list[Path] = []
     try:
-        glob_function = directory.rglob if recursive else directory.glob
-        all_files = [child for child in glob_function("*.plot") if child.is_file() and not child.name.startswith("._")]
+        if follow_links and recursive:
+            import glob
+
+            files = glob.glob(str(directory / "**" / "*.plot"), recursive=True)
+            for file in files:
+                filepath = Path(file).resolve()
+                if filepath.is_file() and not filepath.name.startswith("._"):
+                    all_files.append(filepath)
+        else:
+            glob_function = directory.rglob if recursive else directory.glob
+            all_files = [
+                child for child in glob_function("*.plot") if child.is_file() and not child.name.startswith("._")
+            ]
         log.debug(f"get_filenames: {len(all_files)} files found in {directory}, recursive: {recursive}")
     except Exception as e:
         log.warning(f"Error reading directory {directory} {e}")


Code review comments:
diff_hunk: @@ -737,3 +737,46 @@ async def test_recursive_plot_scan(environment: Environment) -> None:
     add_plot_directory(env.root_path, str(sub_dir_1_0_1.path))
     expected_result.loaded = []
     await env.refresh_tester.run(expected_result)
+
+
+@pytest.mark.limit_consensus_modes(reason="does not depend on consensus rules")
path: chia/_tests/plotting/test_plot_manager.py
text: @user1:
Should this be applied as a file-level thing at least?

https://docs.pytest.org/en/stable/reference/reference.html#globalvar-pytestmark

```python
pytestmark = [pytest.mark.limit_consensus_modes(reason="does not depend on consensus rules")]
```

@author:
oh yea, that would be nice

diff_hunk: @@ -737,3 +737,46 @@ async def test_recursive_plot_scan(environment: Environment) -> None:
     add_plot_directory(env.root_path, str(sub_dir_1_0_1.path))
     expected_result.loaded = []
     await env.refresh_tester.run(expected_result)
+
+
+@pytest.mark.limit_consensus_modes(reason="does not depend on consensus rules")
+@pytest.mark.parametrize("follow_links", [True, False])
path: chia/_tests/plotting/test_plot_manager.py
text: @user2:
```suggestion
@user1(name="follow_links", false="no-follow", true="follow")
```

diff_hunk: @@ -737,3 +737,46 @@ async def test_recursive_plot_scan(environment: Environment) -> None:
     add_plot_directory(env.root_path, str(sub_dir_1_0_1.path))
     expected_result.loaded = []
     await env.refresh_tester.run(expected_result)
+
+
+@pytest.mark.limit_consensus_modes(reason="does not depend on consensus rules")
+@pytest.mark.parametrize("follow_links", [True, False])
+@pytest.mark.anyio
+async def test_recursive_plot_scan_symlinks(environment: Environment, follow_links: bool) -> None:
+    env: Environment = environment
+    # Create a directory tree with some subdirectories containing plots, others not.
+    root_plot_dir = env.root_path / "root"
+    sub_dir_0: Directory = Directory(root_plot_dir / "0", env.dir_1.plots[0:2])
+    sub_dir_0_1: Directory = Directory(sub_dir_0.path / "1", env.dir_1.plots[2:3])
+
+    # Create a plot directory outside of the root plot directory
+    other_plot_dir: Directory = Directory(env.root_path / "other", env.dir_1.plots[3:7])
+
+    # Create a symlink to the other plot directory from inside the root plot directory
+    symlink_path = Path(root_plot_dir / "other")
+
+    try:
+        symlink_path.symlink_to(env.root_path / "other")
+    except NotImplementedError:
path: chia/_tests/plotting/test_plot_manager.py
text: @user1:
Or just remove it maybe?
```suggestion
    except NotImplementedError:  #pragma: no cover
```

If keeping it, might want to also handle the new 3.13 behavior. https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.symlink_to  Would need to import `UnsupportedOperation` from `pathlib` as well.  Or could get fancier than this and make the expected exception version specific...

```suggestion
    except (NotImplementedError, UnsupportedOperation):  #pragma: no cover
```


@author:
Yea, I was thinking of just removing it - everything does symlinks afaik