Repository: Chia-Network/chia-blockchain

Issue:
[Bug] Harvester delays on plots refresh with many plots
### What happened?

1.3.0

Got 3 reports of big harvesters seeing a delay between the triggering of `interval_timer:` and the first batch processing:

Harvester with 7800~ plots, the harvester is starting `_plot_refresh_callback: event started` at `05.369` seconds
the following first event of `refresh_batch: 300 files in directories...` does not start until `11.181` seconds, and first batch processed at `11.292` seconds
ending up with a `5.9` seconds of delay
in the below log output, it receives a challenge from the farmer at `06.083` seconds, which it does not process until the first batch is processed. That in turns delays the proof lookups of the challenge by roughly that amount of time that it took to process the first batch refresh `5.2` seconds
This in turns results in `WARNING  Looking up qualities on - took: 5.17`

Further monitoring on 1.2.11, showing lookup times:
![paste (14)](https://user-images.githubusercontent.com/32275244/157868222-8cd6541b-8da3-45c1-a1b1-977143f9c1c3.png)
And again on 1.3.0
![paste (13)](https://user-images.githubusercontent.com/32275244/157868282-c54007dd-a5b3-4173-aa91-09ace87a3e9b.png)
Yellow is over 1 second, and red is above 5 seconds.
This is run on Linux


Harvester with 12000~ plots are seeing lookup times around 15~ seconds.
Unknown OS


Harvester with 32000~ plots are seeing lookup times around 250~ seconds.
Windows 10 Enterprise


DEBUG logs:
[7800-debug.log](https://github.com/Chia-Network/chia-blockchain/files/8231953/7800-debug.log)
[32k-debug.log](https://github.com/Chia-Network/chia-blockchain/files/8231954/32k-debug.log)

Thanks to @bhorvitz @reythia and @mrkkwar1 for help gathering data, and offering to test.

### Version

 1.3.0

### What platform are you using?

Linux & Windows

### What ui mode are you using?

CLI

### Relevant log output

```shell
Snippet from 7800 plots on harvester
2022-03-09T16:46:30.143 harvester harvester_server        : DEBUG    <- new_signage_point_harvester from peer 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17 192.168.187.41
2022-03-09T16:46:30.353 harvester chia.harvester.harvester: INFO     19 plots were eligible for farming e39fa1c3a6... Found 0 proofs. Time: 0.20918 s. Total 7872 plots
2022-03-09T16:46:30.353 harvester harvester_server        : DEBUG    -> farming_info to peer 192.168.187.41 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17
2022-03-09T16:46:30.353 harvester harvester_server        : DEBUG    Time taken to process new_signage_point_harvester from 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17 is 0.20984625816345215 seconds
2022-03-09T16:46:42.125 harvester harvester_server        : DEBUG    <- new_signage_point_harvester from peer 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17 192.168.187.41
2022-03-09T16:46:57.372 harvester harvester_server        : DEBUG    -> farming_info to peer 192.168.187.41 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17
2022-03-09T16:46:57.372 harvester harvester_server        : DEBUG    Time taken to process new_signage_point_harvester from 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17 is 0.35970258712768555 seconds
2022-03-09T16:47:05.369 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event started, loaded 0, removed 0, processed 0, remaining 7872, duration: 0.00 seconds, total plots: 7872
2022-03-09T16:47:06.083 harvester harvester_server        : DEBUG    <- new_signage_point_harvester from peer 46d4ca65bf9cca95f806fed742b252bfd4c7a1ffe0c3e4cc591c1847a23dae17 192.168.187.41
2022-03-09T16:47:11.181 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.292 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.11 seconds
2022-03-09T16:47:11.292 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 7572, duration: 0.11 seconds, total plots: 7872
2022-03-09T16:47:11.292 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.294 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.300 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-01/plot-k32-2021-08-25-17-13-6a5f228f7ee339fbb173938c33f7bb7c2621f77a6324a54de791b4e3f9b3e67b.plot took: 5.176668643951416. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.301 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.301 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 7272, duration: 0.01 seconds, total plots: 7872
2022-03-09T16:47:11.301 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.302 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.308 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-15/plot-k32-2021-09-12-23-07-939de4ce360030b7882ec1d552a83792850479d2c798d605d5c65f44a1e65c27.plot took: 5.184684991836548. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.309 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.309 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 6972, duration: 0.01 seconds, total plots: 7872
2022-03-09T16:47:11.309 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-11/plot-k32-2021-09-10-19-12-b65cd47246c03b86dcfc5d0c902a92286b8151774d01899721bd151bfdf73f8e.plot took: 5.185422658920288. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.309 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.310 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-10/plot-k32-2021-09-24-03-24-d71100f98269a9c0f73edf77b0e6e7b6656a8a6822422622db084076a837fe45.plot took: 5.185967206954956. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.311 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.316 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-19/plot-k32-2021-08-01-16-16-c2966e4ee8644df4402a48b8cd3b35cab7967e21fc7a92bb35d565aa71a26867.plot took: 5.192744970321655. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.317 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.317 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 6672, duration: 0.01 seconds, total plots: 7872
2022-03-09T16:47:11.317 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.318 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.324 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.325 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 6372, duration: 0.01 seconds, total plots: 7872
2022-03-09T16:47:11.325 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.326 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.332 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.332 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 6072, duration: 0.01 seconds, total plots: 7872
2022-03-09T16:47:11.332 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.333 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.361 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-04/plot-k32-2021-09-22-12-06-60aa944e89a547d72b446e9abe4b8c007860f333c939cce42ed69cc3372636cc.plot took: 5.237456798553467. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.363 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-06/plot-k32-2021-08-28-20-30-af29b589022f530267118c94b84275d21738d93911c8b4863d6ae4438a1fd248.plot took: 5.239156246185303. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.363 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-01/plot-k32-2021-09-04-18-09-adfaa0541b67aec04a0ac3d87867e7f823e1bd4f3115938d111666143e763023.plot took: 5.239412784576416. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.363 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-20/plot-k32-2021-09-07-02-01-c5b40edf057c3081d4752cfea7cf17f690980be642ff98a95fb39bf517466024.plot took: 5.23961067199707. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.364 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-14/plot-k32-2021-10-03-04-29-122a0812f71d9e267a5e681d7f437005db274433f7589ec591b8616cff56c42f.plot took: 5.2397847175598145. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.364 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.03 seconds
2022-03-09T16:47:11.364 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-16/plot-k32-2021-09-28-05-35-3a34d5ad7ce969ce0973ccba730ba7aa485d6bdcedbb126123561f2559fa0b8b.plot took: 5.240145683288574. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.364 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 5772, duration: 0.03 seconds, total plots: 7872
2022-03-09T16:47:11.364 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-00/plot-k32-2021-09-07-07-55-0223cbe6bd23bc85f806e53ab94ebf3fe0a29e1c289eae2b16eda2baffb632c1.plot took: 5.240473508834839. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.364 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.366 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.370 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/00-13/plot-k32-2021-09-19-15-16-a8c40a52eb714587e3dc98aefed166cff6b3b0d53c24df2c4e07e89637eeccaa.plot took: 5.2467474937438965. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.372 harvester chia.harvester.harvester: WARNING  Looking up qualities on /plots/01-06/plot-k32-2021-09-18-17-24-58ad3af937b8f448cec111417f38a54be7fd604e56c78ec91734aec2a41e140d.plot took: 5.248072385787964. This should be below 5 seconds to minimize risk of losing rewards.
2022-03-09T16:47:11.372 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.372 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 5472, duration: 0.01 seconds, total plots: 7872
2022-03-09T16:47:11.372 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2022-03-09T16:47:11.374 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {PosixPath('/plots/00-21'), PosixPath('/plots/01-22'), PosixPath('/plots/01-18'), PosixPath('/plots/01-15'), PosixPath('/plots/01-09'), PosixPath('/plots/00-14'), PosixPath('/plots/01-12'), PosixPath('/plots/00-04'), PosixPath('/plots/00-12'), PosixPath('/plots/00-18'), PosixPath('/plots/01-10'), PosixPath('/plots/01-06'), PosixPath('/plots/00-00'), PosixPath('/plots/00-07'), PosixPath('/plots/00-13'), PosixPath('/plots/00-06'), PosixPath('/plots/01-19'), PosixPath('/plots/01-21'), PosixPath('/plots/00-17'), PosixPath('/plots/01-01'), PosixPath('/plots/01-05'), PosixPath('/plots/01-13'), PosixPath('/plots/00-02'), PosixPath('/plots/01-04'), PosixPath('/plots/01-02'), PosixPath('/plots/01-00'), PosixPath('/plots/00-10'), PosixPath('/plots/01-11'), PosixPath('/plots/01-17'), PosixPath('/plots/00-16'), PosixPath('/plots/01-20'), PosixPath('/plots/01-03'), PosixPath('/plots/01-07'), PosixPath('/plots/00-09'), PosixPath('/plots/00-20'), PosixPath('/plots/00-08'), PosixPath('/plots/01-14'), PosixPath('/plots/00-22'), PosixPath('/plots/00-03'), PosixPath('/plots/01-16'), PosixPath('/plots/00-05'), PosixPath('/plots/00-11'), PosixPath('/plots/00-01'), PosixPath('/plots/00-19'), PosixPath('/plots/00-23'), PosixPath('/plots/01-23'), PosixPath('/plots/00-15'), PosixPath('/plots/01-08')}
2022-03-09T16:47:11.380 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.01 seconds
2022-03-09T16:47:11.380 harvester chia.harvester.harvester: DEBUG    _plot_refresh_callback: event batch_processed, loaded 0, removed 0, processed 300, remaining 5172, duration: 0.01 seconds, total plots: 7872
```



Discussion:

Resolving patch:
diff --git a/chia/plotting/manager.py b/chia/plotting/manager.py
--- a/chia/plotting/manager.py
+++ b/chia/plotting/manager.py
@@ -244,30 +244,30 @@ def _refresh_task(self):
                     if path not in plot_paths:
                         self.no_key_filenames.remove(path)
 
-                with self:
-                    filenames_to_remove: List[str] = []
-                    for plot_filename, paths_entry in self.plot_filename_paths.items():
-                        loaded_path, duplicated_paths = paths_entry
-                        loaded_plot = Path(loaded_path) / Path(plot_filename)
-                        if loaded_plot not in plot_paths:
-                            filenames_to_remove.append(plot_filename)
+                filenames_to_remove: List[str] = []
+                for plot_filename, paths_entry in self.plot_filename_paths.items():
+                    loaded_path, duplicated_paths = paths_entry
+                    loaded_plot = Path(loaded_path) / Path(plot_filename)
+                    if loaded_plot not in plot_paths:
+                        filenames_to_remove.append(plot_filename)
+                        with self:
                             if loaded_plot in self.plots:
                                 del self.plots[loaded_plot]
+                        total_result.removed.append(loaded_plot)
+                        # No need to check the duplicates here since we drop the whole entry
+                        continue
+
+                    paths_to_remove: List[str] = []
+                    for path in duplicated_paths:
+                        loaded_plot = Path(path) / Path(plot_filename)
+                        if loaded_plot not in plot_paths:
+                            paths_to_remove.append(path)
                             total_result.removed.append(loaded_plot)
-                            # No need to check the duplicates here since we drop the whole entry
-                            continue
-
-                        paths_to_remove: List[str] = []
-                        for path in duplicated_paths:
-                            loaded_plot = Path(path) / Path(plot_filename)
-                            if loaded_plot not in plot_paths:
-                                paths_to_remove.append(path)
-                                total_result.removed.append(loaded_plot)
-                        for path in paths_to_remove:
-                            duplicated_paths.remove(path)
-
-                    for filename in filenames_to_remove:
-                        del self.plot_filename_paths[filename]
+                    for path in paths_to_remove:
+                        duplicated_paths.remove(path)
+
+                for filename in filenames_to_remove:
+                    del self.plot_filename_paths[filename]
 
                 for remaining, batch in list_to_batches(plot_paths, self.refresh_parameter.batch_size):
                     batch_result: PlotRefreshResult = self.refresh_batch(batch, plot_directories)


Test patch: