Repository: wimglenn/advent-of-code-data

Issue:
aoc runner sends requests to advent of code even if solution file does not exist
What I think is happening is even if there is a non-answer provided, the library is still trying to fetch the answer from AOC. However, if the problem isn't solved yet then the runner has nothing to cache, so this behavior repeats. Additionally, if several solutions are missing, this fires off the rate limiter, which quickly maxes out. I believe this behavior shouldn't be the default. Maybe missing solutions, which are already well-handled, should trigger a `skipped` or `error` state, and continue instead of trying to fetch a solution. Below is my entry point and console logs:

```python
def mysolve(year: str, day: str, data: str) -> tuple[str, str]:
    ans_1, ans_2 = "", ""
    try:
        mod_name = f"aoc_i13e.aoc{year}.q{day:02d}"
        mod = importlib.import_module(name=mod_name)

        part_1 = getattr(mod, "part_1")
        ans_1 = str(part_1(data))

        part_2 = getattr(mod, "part_2")
        ans_2 = str(part_2(data))
    except (ModuleNotFoundError, AttributeError):
        pass

    return ans_1, ans_2
```

```
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
   0.20s   2024/11 - Plutonian Pebbles                          i13e/default   ✔ part a: 194557                             ? part b:  (correct answer unknown)
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=0.16s)
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=0.32s)
   0.10s   2024/12 - Garden Groups                              i13e/default   ? part a:  (correct answer unknown)          ? part b:  (correct answer unknown)
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=0.64s)
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=1.28s)
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=2.56s)
   0.10s   2024/13 - Claw Contraption                           i13e/default   ? part a:  (correct answer unknown)          ? part b:  (correct answer unknown)
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=5.12s)
   0.10s   2024/14 - Restroom Redoubt                           i13e/default   ? part a:  (correct answer unknown)          ? part b:  (correct answer unknown)
```

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/wimglenn/advent-of-code-data/pull/157?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn) Report
Attention: Patch coverage is `96.00000%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 91.15%. Comparing base [(`143461e`)](https://app.codecov.io/gh/wimglenn/advent-of-code-data/commit/143461e989f8d5355ec07c928cc846bf315a6c09?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn) to head [(`de205ca`)](https://app.codecov.io/gh/wimglenn/advent-of-code-data/commit/de205ca8f9f3e67fd6182c94ae7954769f4e3967?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn).

| [Files with missing lines](https://app.codecov.io/gh/wimglenn/advent-of-code-data/pull/157?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn) | Patch % | Lines |
|---|---|---|
| [tests/test\_runner.py](https://app.codecov.io/gh/wimglenn/advent-of-code-data/pull/157?src=pr&el=tree&filepath=tests%2Ftest_runner.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn#diff-dGVzdHMvdGVzdF9ydW5uZXIucHk=) | 94.73% | [1 Missing :warning: ](https://app.codecov.io/gh/wimglenn/advent-of-code-data/pull/157?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #157      +/-   ##
==========================================
+ Coverage   91.11%   91.15%   +0.04%     
==========================================
  Files          25       25              
  Lines        2781     2805      +24     
  Branches      252      253       +1     
==========================================
+ Hits         2534     2557      +23     
- Misses        179      180       +1     
  Partials       68       68              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/wimglenn/advent-of-code-data/pull/157?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Wim+Jeantine-Glenn).


Patch:
diff --git a/aocd/models.py b/aocd/models.py
index d4b1b64..ffcf06d 100644
--- a/aocd/models.py
+++ b/aocd/models.py
@@ -50,6 +50,10 @@ AOCD_DATA_DIR = AOCD_DATA_DIR.expanduser()
 AOCD_CONFIG_DIR = Path(os.environ.get("AOCD_CONFIG_DIR", AOCD_DATA_DIR)).expanduser()
 URL = "https://adventofcode.com/{year}/day/{day}"
 
+# aocd will refuse to submit these as answers, and if a user entry point happens to
+# return them the runner will consider it to mean the solution hasn't been implemented
+NON_ANSWER = "", b"", None, b"None", "None"
+
 
 class User:
     _token2id = None
@@ -461,7 +465,7 @@ class Puzzle:
         # actual submit logic. not meant to be invoked directly - users are expected
         # to use aocd.post.submit function, puzzle answer setters, or the aoc.runner
         # which autosubmits answers by default.
-        if value in {"", b"", None, b"None", "None"}:
+        if value in NON_ANSWER:
             raise AocdError(f"cowardly refusing to submit non-answer: {value!r}")
         if not isinstance(value, str):
             value = self._coerce_val(value)
diff --git a/aocd/runner.py b/aocd/runner.py
index ee4618b..4919810 100644
--- a/aocd/runner.py
+++ b/aocd/runner.py
@@ -20,6 +20,7 @@ import pebble.concurrent
 from .exceptions import AocdError
 from .models import _load_users
 from .models import AOCD_CONFIG_DIR
+from .models import NON_ANSWER
 from .models import Puzzle
 from .utils import _cli_guess
 from .utils import AOC_TZ
@@ -370,6 +371,9 @@ def run_for(
             os.environ.pop(f"AOCD_EXTRA", None)
             runtime = format_time(walltime, timeout)
             line = "   ".join([runtime, progress])
+            if a in NON_ANSWER and b in NON_ANSWER and not error:
+                a = b = ""
+                error = f"Skipping {year}/{day:<2d} (entry-point returned non-answers)"
             if error:
                 assert a == b == ""
                 icon = colored("✖", "red")


Test patch:
diff --git a/tests/test_runner.py b/tests/test_runner.py
index 78b7d7f..e4dd81b 100644
--- a/tests/test_runner.py
+++ b/tests/test_runner.py
@@ -86,6 +86,10 @@ def bugged_entry_point(year, day, data):
     raise Exception(123, 456)
 
 
+def non_answer_entry_point(year, day, data):
+    return "", ""
+
+
 def test_results(mocker, capsys):
     ep = mocker.Mock()
     ep.name = "testuser"
@@ -202,6 +206,30 @@ def test_run_error(aocd_data_dir, mocker, capsys):
     assert "part b" not in out  # because it's 25 dec, no part b puzzle
 
 
+def test_run_non_answers(aocd_data_dir, mocker, capsys):
+    prose_dir = aocd_data_dir / "prose"
+    prose_dir.mkdir()
+    puzzle_file = prose_dir / "2018_25_prose.0.html"
+    puzzle_file.write_text("<h2>--- Day 25: The Puzzle Title ---</h2>")
+    input_path = aocd_data_dir / "testauth.testuser.000" / "2018_25_input.txt"
+    input_path.write_text("someinput")
+    answer_path = aocd_data_dir / "testauth.testuser.000" / "2018_25a_answer.txt"
+    answer_path.write_text("answ")
+    ep = mocker.Mock()
+    ep.name = "testuser"
+    ep.load.return_value = non_answer_entry_point
+    mocker.patch("aocd.runner.get_plugins", return_value=[ep])
+    run_for(
+        plugs=["testuser"],
+        years=[2018],
+        days=[25],
+        datasets={"default": "thetesttoken"},
+    )
+    out, err = capsys.readouterr()
+    txt = "Skipping 2018/25 (entry-point returned non-answers)"
+    assert txt in out
+
+
 def test_run_and_autosubmit(aocd_data_dir, mocker, capsys, pook):
     prose_dir = aocd_data_dir / "prose"
     prose_dir.mkdir()