Repository: Chia-Network/chia-blockchain

Issue:
[Bug] ERROR    Exception: dictionary changed size during iteration 
### What happened?

Errors in some remote harvesters' logs and corresponding farmer log.  A remote harvester showing this ERROR is either plotting (using bladebit) or receiving (NFT) plots over rsync (using plotman).  From time to time remote harvesters would lose connection to the farmer. 

### Version

1.2.11

### What platform are you using?

Linux

### What ui mode are you using?

CLI

### Relevant log output

```shell
ON REMOTE HARVESTER:  xxx.xxx.xxx.xxx is farmer LAN IP.
2021-12-31T23:23:46.652 harvester harvester_server        : ERROR    Exception: dictionary changed size during iteration <class 'RuntimeError'>, closing connection {'host': '192.168.1.174', 'port': 8447}. Traceback (most recent call last):
  File "/home/jh/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
    return fut.result()
  File "/home/jh/chia-blockchain/chia/server/server.py", line 573, in wrapped_coroutine
    raise e
  File "/home/jh/chia-blockchain/chia/server/server.py", line 566, in wrapped_coroutine
    result = await coroutine
  File "/home/jh/chia-blockchain/chia/harvester/harvester_api.py", line 185, in new_signage_point_harvester
    for try_plot_filename, try_plot_info in self.harvester.plot_manager.plots.items():
RuntimeError: dictionary changed size during iteration

2021-12-31T23:23:46.656 harvester harvester_server        : WARNING  Banning xxx.xxx.xxx.xxx for 10 seconds
2021-12-31T23:23:49.436 harvester harvester_server        : WARNING  Peer xxx.xxx.xxx.xxx is still banned, not connecting to it
2021-12-31T23:23:52.438 harvester harvester_server        : WARNING  Peer xxx.xxx.xxx.xxx is still banned, not connecting to it
2021-12-31T23:23:55.443 harvester harvester_server        : WARNING  Peer xxx.xxx.xxx.xxx is still banned, not connecting to it

ON FARMER: yyy.yyy.yyy.yyy is remote harvester IP
2021-12-31T21:23:18.934 farmer chia.farmer.farmer         : ERROR    Harvester 'yyy.yyy.yyy.yyy/zzzzzzzzzzzzzzzzzzzzzzz' did not respond: (version mismatch or time out 90s)
```



Discussion:

Resolving patch:
diff --git a/chia/plotting/manager.py b/chia/plotting/manager.py
--- a/chia/plotting/manager.py
+++ b/chia/plotting/manager.py
@@ -166,11 +166,12 @@ def __exit__(self, exc_type, exc_value, exc_traceback):
         self._lock.release()
 
     def reset(self):
-        self.last_refresh_time = time.time()
-        self.plots.clear()
-        self.plot_filename_paths.clear()
-        self.failed_to_open_filenames.clear()
-        self.no_key_filenames.clear()
+        with self:
+            self.last_refresh_time = time.time()
+            self.plots.clear()
+            self.plot_filename_paths.clear()
+            self.failed_to_open_filenames.clear()
+            self.no_key_filenames.clear()
 
     def set_refresh_callback(self, callback: Callable):
         self._refresh_callback = callback  # type: ignore
@@ -243,29 +244,30 @@ def _refresh_task(self):
                     if path not in plot_paths:
                         self.no_key_filenames.remove(path)
 
-                filenames_to_remove: List[str] = []
-                for plot_filename, paths_entry in self.plot_filename_paths.items():
-                    loaded_path, duplicated_paths = paths_entry
-                    loaded_plot = Path(loaded_path) / Path(plot_filename)
-                    if loaded_plot not in plot_paths:
-                        filenames_to_remove.append(plot_filename)
-                        if loaded_plot in self.plots:
-                            del self.plots[loaded_plot]
-                        total_result.removed.append(loaded_plot)
-                        # No need to check the duplicates here since we drop the whole entry
-                        continue
-
-                    paths_to_remove: List[str] = []
-                    for path in duplicated_paths:
-                        loaded_plot = Path(path) / Path(plot_filename)
+                with self:
+                    filenames_to_remove: List[str] = []
+                    for plot_filename, paths_entry in self.plot_filename_paths.items():
+                        loaded_path, duplicated_paths = paths_entry
+                        loaded_plot = Path(loaded_path) / Path(plot_filename)
                         if loaded_plot not in plot_paths:
-                            paths_to_remove.append(path)
+                            filenames_to_remove.append(plot_filename)
+                            if loaded_plot in self.plots:
+                                del self.plots[loaded_plot]
                             total_result.removed.append(loaded_plot)
-                    for path in paths_to_remove:
-                        duplicated_paths.remove(path)
-
-                for filename in filenames_to_remove:
-                    del self.plot_filename_paths[filename]
+                            # No need to check the duplicates here since we drop the whole entry
+                            continue
+
+                        paths_to_remove: List[str] = []
+                        for path in duplicated_paths:
+                            loaded_plot = Path(path) / Path(plot_filename)
+                            if loaded_plot not in plot_paths:
+                                paths_to_remove.append(path)
+                                total_result.removed.append(loaded_plot)
+                        for path in paths_to_remove:
+                            duplicated_paths.remove(path)
+
+                    for filename in filenames_to_remove:
+                        del self.plot_filename_paths[filename]
 
                 for remaining, batch in list_to_batches(plot_paths, self.refresh_parameter.batch_size):
                     batch_result: PlotRefreshResult = self.refresh_batch(batch, plot_directories)


Test patch: