Repository: Chia-Network/chia-blockchain
Pull request: List unopenable plots at end of plots check
Closes https://github.com/Chia-Network/chia-blockchain/issues/2825

Problem statement:
[BUG-MINOR] plots marked as "Invalid plot header magic" are not included in summary by `chia plots check`
The fact that chia seems to be randomly taking some plots with "Invalid plot header magic" aside, it is also not reporting these plots as invalid in the `chia plots check`  summary:

```
(venv) speedster@cohen:~$ chia plots check -n 100 -g /farm/acre26/plots/chachacha-plot-k32-2021-04-24-02-28-00211f7b93d4e342a7af6bb4dd3f2bacce5c4e161c90c2ab38a69704b8e69e1b.plot
2021-04-26T10:07:48.324  chia.plotting.check_plots        : INFO     Loading plots in config.yaml using plot_tools loading code

2021-04-26T10:07:52.439  chia.plotting.plot_tools         : INFO     Searching directories ['/farm/acre12/plots', '/farm/acre14/plots', '/farm/acre15/plots', '/farm/acre16/plots', '/farm/acre17/plots', '/farm/acre18/plots', '/farm/acre19/plots', '/farm/acre20/plots', '/farm/acre21/plots', '/farm/acre22/plots', '/farm/acre23/plots', '/farm/acre24/plots', '/farm/acre5/plots', '/farm/acre6/plots', '/farm/acre7/plots', '/farm/acre8/plots', '/farm/acre2/plots', '/farm/acre4/plots', '/farm/acre3/plots', '/farm/acre25/plots', '/farm/acre28/plots', '/mnt/transfer/plots', '/farm/acre26/plots']
2021-04-26T10:07:52.508  chia.plotting.plot_tools         : INFO     Only loading plots that contain "/farm/acre26/plots/chachacha-plot-k32-2021-04-24-02-28-00211f7b93d4e342a7af6bb4dd3f2bacce5c4e161c90c2ab38a69704b8e69e1b.plot" in the file or directory name
2021-04-26T10:07:52.699  chia.plotting.plot_tools         : ERROR    Failed to open file /farm/acre26/plots/chachacha-plot-k32-2021-04-24-02-28-00211f7b93d4e342a7af6bb4dd3f2bacce5c4e161c90c2ab38a69704b8e69e1b.plot. Invalid plot header magic Traceback (most recent call last):
  File "/home/speedster/dev/chia-blockchain/chia/plotting/plot_tools.py", line 183, in process_file
    prover = DiskProver(str(filename))
ValueError: Invalid plot header magic

2021-04-26T10:07:52.700  chia.plotting.plot_tools         : INFO     Loaded a total of 0 plots of size 0.0 TiB, in 0.3909459114074707 seconds
2021-04-26T10:07:52.701  chia.plotting.check_plots        : INFO     
2021-04-26T10:07:52.702  chia.plotting.check_plots        : INFO     
2021-04-26T10:07:52.702  chia.plotting.check_plots        : INFO     Summary
2021-04-26T10:07:52.702  chia.plotting.check_plots        : INFO     Found 0 valid plots, total size 0.00000 TiB
2021-04-26T10:07:52.702  chia.plotting.check_plots        : WARNING  1 invalid plots found:
(venv) speedster@cohen:~$ 
```

1) All "invalid" plots should be reported in the `chia plots check` summary
2) I am crossing my fingers that when looking for this bug you uncover whatever bug which has started to randomly tag plots that passed muster previously as "Invalid plot header magic"

Hints from issue discussion:
Same here 
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
chia dev team would need to update if they fixed
> chia dev team would need to update if they fixed

Hi @fiveangle, have you plotted on servers like HP ProLiant? Can I ask a couple of questions in a personal conversation - dmitry.nesterovith@gmail.com
This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.
closing as, like chia devs, i no longer care

Commit message: drop .keys() in len() check

Patch to review:
diff --git a/chia/plotting/check_plots.py b/chia/plotting/check_plots.py
index b1d8291e53bd..f2b4d8f3017c 100644
--- a/chia/plotting/check_plots.py
+++ b/chia/plotting/check_plots.py
@@ -1,5 +1,6 @@
 import logging
 from collections import Counter
+import os
 from pathlib import Path
 from time import time, sleep
 from typing import List
@@ -94,7 +95,6 @@ def check_plots(root_path, num, challenge_start, grep_string, list_duplicates, d
         log.info("")
         log.info(f"Starting to test each plot with {num} challenges each\n")
     total_good_plots: Counter = Counter()
-    total_bad_plots = 0
     total_size = 0
     bad_plots_list: List[Path] = []
 
@@ -165,7 +165,6 @@ def check_plots(root_path, num, challenge_start, grep_string, list_duplicates, d
                 total_good_plots[pr.get_size()] += 1
                 total_size += plot_path.stat().st_size
             else:
-                total_bad_plots += 1
                 log.error(f"\tProofs {total_proofs} / {challenges}, {round(total_proofs/float(challenges), 4)}")
                 bad_plots_list.append(plot_path)
     log.info("")
@@ -175,11 +174,17 @@ def check_plots(root_path, num, challenge_start, grep_string, list_duplicates, d
     log.info(f"Found {total_plots} valid plots, total size {total_size / (1024 * 1024 * 1024 * 1024):.5f} TiB")
     for (k, count) in sorted(dict(total_good_plots).items()):
         log.info(f"{count} plots of size {k}")
-    grand_total_bad = total_bad_plots + len(plot_manager.failed_to_open_filenames)
+    grand_total_bad = len(bad_plots_list) + len(plot_manager.failed_to_open_filenames)
     if grand_total_bad > 0:
         log.warning(f"{grand_total_bad} invalid plots found:")
-        for bad_plot_path in bad_plots_list:
-            log.warning(f"{bad_plot_path}")
+        if len(bad_plots_list) > 0:
+            log.warning(f"    {len(bad_plots_list)} bad plots:")
+            for bad_plot_path in bad_plots_list:
+                log.warning(os.fspath(bad_plot_path))
+        if len(plot_manager.failed_to_open_filenames) > 0:
+            log.warning(f"    {len(plot_manager.failed_to_open_filenames)} unopenable plots:")
+            for unopenable_plot_path in plot_manager.failed_to_open_filenames.keys():
+                log.warning(os.fspath(unopenable_plot_path))
     if len(plot_manager.no_key_filenames) > 0:
         log.warning(
             f"There are {len(plot_manager.no_key_filenames)} plots with a farmer or pool public key that "


Code review comments:
diff_hunk: @@ -175,11 +174,17 @@ def check_plots(root_path, num, challenge_start, grep_string, list_duplicates, d
     log.info(f"Found {total_plots} valid plots, total size {total_size / (1024 * 1024 * 1024 * 1024):.5f} TiB")
     for (k, count) in sorted(dict(total_good_plots).items()):
         log.info(f"{count} plots of size {k}")
-    grand_total_bad = total_bad_plots + len(plot_manager.failed_to_open_filenames)
+    grand_total_bad = len(bad_plots_list) + len(plot_manager.failed_to_open_filenames)
     if grand_total_bad > 0:
         log.warning(f"{grand_total_bad} invalid plots found:")
-        for bad_plot_path in bad_plots_list:
-            log.warning(f"{bad_plot_path}")
+        if len(bad_plots_list) > 0:
+            log.warning(f"    {len(bad_plots_list)} bad plots:")
+            for bad_plot_path in bad_plots_list:
+                log.warning(os.fspath(bad_plot_path))
path: chia/plotting/check_plots.py
text: @user1:
```suggestion
                log.warning(str(bad_plot_path))
```

diff_hunk: @@ -175,11 +174,17 @@ def check_plots(root_path, num, challenge_start, grep_string, list_duplicates, d
     log.info(f"Found {total_plots} valid plots, total size {total_size / (1024 * 1024 * 1024 * 1024):.5f} TiB")
     for (k, count) in sorted(dict(total_good_plots).items()):
         log.info(f"{count} plots of size {k}")
-    grand_total_bad = total_bad_plots + len(plot_manager.failed_to_open_filenames)
+    grand_total_bad = len(bad_plots_list) + len(plot_manager.failed_to_open_filenames)
     if grand_total_bad > 0:
         log.warning(f"{grand_total_bad} invalid plots found:")
-        for bad_plot_path in bad_plots_list:
-            log.warning(f"{bad_plot_path}")
+        if len(bad_plots_list) > 0:
+            log.warning(f"    {len(bad_plots_list)} bad plots:")
+            for bad_plot_path in bad_plots_list:
+                log.warning(os.fspath(bad_plot_path))
+        if len(plot_manager.failed_to_open_filenames) > 0:
+            log.warning(f"    {len(plot_manager.failed_to_open_filenames)} unopenable plots:")
+            for unopenable_plot_path in plot_manager.failed_to_open_filenames.keys():
+                log.warning(os.fspath(unopenable_plot_path))
path: chia/plotting/check_plots.py
text: @user1:
```suggestion
                log.warning(str(unopenable_plot_path))
```

diff_hunk: @@ -1,5 +1,6 @@
 import logging
 from collections import Counter
+import os
path: chia/plotting/check_plots.py
text: @user1:
```suggestion
```