Repository: Chia-Network/chia-blockchain

Issue:
[BUG] Harvester plots_refresh_parameter not keeping track of plots
The harvester is losing track of my plots during replotting, and my plot count is going down.

I have 5156 plots. All of my drives are full, so I am currently replotting.
I have a script that, upon completion of a new NFT plot, it identifies an OG plot, deletes it, and moves the new NFT plot in its place.
I am currently creating 64 plots every 24h.
Earlier today, my harvester did indeed show that it was farming `5156` plots. I check my status a few moments ago, and it shows that I'm only farming `5129` plots.

![2021-08-29_01-35](https://user-images.githubusercontent.com/8990544/131241137-9b0aa144-d510-49d8-8e6c-fd3b0346de39.png)
(Then some time passes for the farmer to rediscover all my plots)
![2021-08-29_01-36](https://user-images.githubusercontent.com/8990544/131241230-535680eb-f101-464b-bd3b-cbe2bca1bf7f.png)

Ok, so back to `5156` after restarting the farmer. It lost track of 27 plots.

When I first discovered this bug this morning, there was a Δ of 85 plots that the harvester wasn't tracking.

> 2021-08-28T13:54:20.544 harvester chia.harvester.harvester: INFO  ...  Total 5071 plots

`chia stop all -d`
`chia start farmer`

> 2021-08-28T14:49:27.451 harvester chia.harvester.harvester: INFO  ...  Total 5156 plots


What I suspect:
My config.yaml had been with me since before launch, and I decided to compare it against https://github.com/Chia-Network/chia-blockchain/blob/main/chia/util/initial-config.yaml to see if I needed to refresh it any. There were a few things that I was missing.
I had `plot_loading_frequency_seconds: 120`, which isn't anywhere in the current initial-config.yaml, so I swapped that out with: https://github.com/Chia-Network/chia-blockchain/blob/main/chia/util/initial-config.yaml#L128-L131
I think that this may not be discovering new plots for some reason. I'm going to attempt to remove those lines, and go back to `plot_loading_frequency_seconds: 120`. I'll check back in the morning to see if the count is off.


Chia: 1.2.5
Distro: Arch Linux [x86_64]
Kernel: Linux 5.13.12-arch1-1
Processor: 24 x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
Processor: 24 x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
RAM: 64362 MiB


Discussion:
We are having issues as well with the harvesters not counting all the plots since upgrading to 1.2.5.  We will try to use a fresh config file and see if we can get those missing plots back.  Our configs are very old as well.
Changing the config back to `plot_loading_frequency_seconds: 120` did not seem to do anything.

This may be due to how the new manager bans badbit/failbit plots.
My final plot management scripts for plotting, and replotting, move plots into active harvester directories with the `.plot` extension. If the file is still being copied when the manager finds it, this causes the manager to throw:

> harvester chia.plotting.manager   : ERROR    Failed to open file /<path_to_plot>/plot-k32-<plotId>.plot. badbit or failbit after reading size 4 at position 107547188232 Traceback (most recent call last):                                                                            File "/<path_to>/chia-blockchain/chia/plotting/manager.py", line 191, in process_file        prover = DiskProver(str(file_path))                                                       RuntimeError: badbit or failbit after reading size 4 at position 107547188232

See here:
https://github.com/Chia-Network/chia-blockchain/blob/53149f2e57743e203aba1d25a89786edcf1f78ab/chia/plotting/manager.py#L198-#L199

I don't know if this outright bans it, or puts it in a loop to try to read it again in 20m. The behavior I'm seeing makes me think it's the former.

For now, I've simply modified my re-plot script (https://github.com/wallentx/farm-and-ranch-supply-depot/blob/main/bin/final-plot-management/replot/replot#L83) to move the plot into the active harvester directory with "-TMP" appended, and removing that when it's finished being copied.

I'll follow up if this solves it
Yep, that solved it. It seems like the ban on badbit/failbit lasts until a restart of the farmer?
Yes, agreed.  I am replotting as well (from a different machine) and then copying the files over from external drives.  Previously, while the copy was in progress the plots may be flagged as bad, but it would eventually farm them a little bit after they finished copying.  Now, once they get flagged as bad it does not appear to ever attempt to farm them again.  Closing and re-opening caused it to refresh all the plots and the ones that had been flagged as bad disappeared and started farming as normal.  

Quite annoying when copying file all day (and all night) to have to now restart the client every once in a while to make sure they're being farmed.  Wish I had stuck with the older version...
> Yep, that solved it. It seems like the ban on badbit/failbit lasts until a restart of the farmer?

restarting the harvester alone appears to be sufficient.

`chia start harvester -r`
On Linux, use rsync to copy plots if you don't want to script changing file names. rsync creates a temporary file while transferring and renames it to the final file name after transfer is complete. 

Resolving patch:
diff --git a/chia/plotting/manager.py b/chia/plotting/manager.py
--- a/chia/plotting/manager.py
+++ b/chia/plotting/manager.py
@@ -162,9 +162,10 @@ def process_file(file_path: Path) -> Dict:
             if file_path.exists():
                 if (
                     file_path in self.failed_to_open_filenames
-                    and (time.time() - self.failed_to_open_filenames[file_path]) > 1200
+                    and (time.time() - self.failed_to_open_filenames[file_path])
+                    < self.refresh_parameter.retry_invalid_seconds
                 ):
-                    # Try once every 20 minutes to open the file
+                    # Try once every `refresh_parameter.retry_invalid_seconds` seconds to open the file
                     return new_provers
                 if file_path in self.plots:
                     try:
@@ -268,6 +269,9 @@ def process_file(file_path: Path) -> Dict:
                         result.loaded_plots += 1
                         result.loaded_size += stat_info.st_size
 
+                    if file_path in self.failed_to_open_filenames:
+                        del self.failed_to_open_filenames[file_path]
+
                 except Exception as e:
                     tb = traceback.format_exc()
                     log.error(f"Failed to open file {file_path}. {e} {tb}")
diff --git a/chia/plotting/util.py b/chia/plotting/util.py
--- a/chia/plotting/util.py
+++ b/chia/plotting/util.py
@@ -16,6 +16,7 @@
 @dataclass
 class PlotsRefreshParameter:
     interval_seconds: int = 120
+    retry_invalid_seconds: int = 1200
     batch_size: int = 30
     batch_sleep_milliseconds: int = 10
 


Test patch:
diff --git a/tests/core/test_farmer_harvester_rpc.py b/tests/core/test_farmer_harvester_rpc.py
--- a/tests/core/test_farmer_harvester_rpc.py
+++ b/tests/core/test_farmer_harvester_rpc.py
@@ -1,6 +1,8 @@
 # flake8: noqa: E501
 import logging
+from pathlib import Path
 from secrets import token_bytes
+from shutil import copy, move
 
 import pytest
 from blspy import AugSchemeMPL
@@ -198,6 +200,7 @@ async def test_case(
                 await time_out_assert(5, harvester.plot_manager.needs_refresh, value=False)
                 result = await client_2.get_plots()
                 assert len(result["plots"]) == expect_total_plots
+                assert len(harvester.plot_manager.failed_to_open_filenames) == 0
 
             # Add plot_dir with two new plots
             await test_case(
@@ -291,6 +294,39 @@ async def test_case(
                 expect_total_plots=0,
             )
 
+            # Test re-trying if processing a plot failed
+            # First save the plot
+            retry_test_plot = Path(plot_dir_sub / filename_2).resolve()
+            retry_test_plot_save = Path(plot_dir_sub / "save").resolve()
+            copy(retry_test_plot, retry_test_plot_save)
+            # Invalidate the plot
+            with open(plot_dir_sub / filename_2, "r+b") as file:
+                file.write(bytes(100))
+            # Add it and validate it fails to load
+            await harvester.add_plot_directory(str(plot_dir_sub))
+            expected_result.loaded_plots = 0
+            expected_result.removed_plots = 0
+            expected_result.processed_files = 1
+            expected_result.remaining_files = 0
+            harvester.plot_manager.start_refreshing()
+            await time_out_assert(5, harvester.plot_manager.needs_refresh, value=False)
+            assert retry_test_plot in harvester.plot_manager.failed_to_open_filenames
+            # Make sure the file stays in `failed_to_open_filenames` and doesn't get loaded or processed in the next
+            # update round
+            expected_result.loaded_plots = 0
+            expected_result.processed_files = 0
+            harvester.plot_manager.trigger_refresh()
+            await time_out_assert(5, harvester.plot_manager.needs_refresh, value=False)
+            assert retry_test_plot in harvester.plot_manager.failed_to_open_filenames
+            # Now decrease the re-try timeout, restore the valid plot file and make sure it properly loads now
+            harvester.plot_manager.refresh_parameter.retry_invalid_seconds = 0
+            move(retry_test_plot_save, retry_test_plot)
+            expected_result.loaded_plots = 1
+            expected_result.processed_files = 1
+            harvester.plot_manager.trigger_refresh()
+            await time_out_assert(5, harvester.plot_manager.needs_refresh, value=False)
+            assert retry_test_plot not in harvester.plot_manager.failed_to_open_filenames
+
             targets_1 = await client.get_reward_targets(False)
             assert "have_pool_sk" not in targets_1
             assert "have_farmer_sk" not in targets_1