Repository: pantsbuild/pants

Issue:
Improve error message for unrecognized symbols
When setting up a new repo, I forgot to activate the Python backend, so Pants failed to recognize `python_library`. The error message didn't offer any suggestion for a remedy:

```
MappingError: Failed to parse build-support/bin/BUILD:
Name 'python_library' is not defined.

All registered symbols: ['archive', 'files', 'relocated_files', 'resources', 'target']
```

It's not clear what to do. Ideally, every error message will give some guidance on how you might fix it. When we're programming, it's crucial to remember that we are writing things for humans. Check out https://elm-lang.org/news/compiler-errors-for-humans for some thoughts on error messages.

To test this out, change a BUILD file by introducing a typo, then run `./pants --no-print-stacktrace list path/to/dir::`.

 We should ask something like

> If you expect to see more symbols registered, have you added the relevant backend to `backend_packages` in `pants.toml`? See .. for a list of all backends.

It would be great to also add a "Did you mean?" check.

Possible wording:

```
 MappingError: Failed to parse build-support/bin/BUILD:
Name 'pex2_binary' is not defined.

Did you mean pex_binary? If you expect to see more symbols activated in the below list, refer to https://www.pantsbuild.org/docs/enabling-backends for all available backends to activate.

All registered symbols: ['_python_requirements_file', 'archive', 'files', 'pants_requirement', 'pex_binary', 'pipenv_requirements', 'protobuf_library', 'python_artifact', 'python_awslambda', 'python_distribution', 'python_integration_tests', 'python_library', 'python_requirement_library', 'python_requirements', 'python_tests', 'relocated_files', 'resources', 'setup_py', 'target']
```

If there isn't anything similar, leave that part off.

--

This error message happens here:

https://github.com/pantsbuild/pants/blob/c57d3ca2b8cb3bce31417b53ae0343ecd7c11b1b/src/python/pants/engine/internals/parser.py#L99-L104

You can start with adding the Backends message before "did you mean", as it's easier to add. When we refer to docs, we use this little helper + f-strings to ensure we use the right version of docs:

https://github.com/pantsbuild/pants/blob/c57d3ca2b8cb3bce31417b53ae0343ecd7c11b1b/src/python/pants/util/docutil.py#L7-L9

Ensure that change is working properly and that the URL is clickable. You can also try running `./pants test src/python/pants/engine/internals/parser_test.py`. This test should fail and will need updating:

https://github.com/pantsbuild/pants/blob/c57d3ca2b8cb3bce31417b53ae0343ecd7c11b1b/src/python/pants/engine/internals/parser_test.py#L20-L34

(Normally, this test would be seen as really brittle because it's using exact string matching. But for error messages, we want this; we want to ensure that our error messages are worded exactly how we want and that things like formatting are working properly. The brittleness is a virtue here.)

--

The more interesting part is now how to do the "Did you mean?" calculation. This is how we do it for command line args, thanks to Python's `difflib` module:

https://github.com/pantsbuild/pants/blob/c57d3ca2b8cb3bce31417b53ae0343ecd7c11b1b/src/python/pants/help/flag_error_help_printer.py#L30-L57

Note that it's possible for us to have multiple possible matches. The wording of the error message will need to sound natural for both a single possibility, and multiple.

Finally, you'll want to add a test for the case where there is a "did you mean?", but also one where we don't have it. In that test linked above, you have tight control over the `Parser` object and what symbols get set up, so you'll want to experiment with different configurations to get the test setup you're looking for. You'll likely want to use an inner helper function to deduplicate things, but you can start by copying and pasting all the code between both edge cases until they're working, then deduplicate.



Discussion:

Resolving patch:
diff --git a/src/python/pants/engine/internals/parser.py b/src/python/pants/engine/internals/parser.py
--- a/src/python/pants/engine/internals/parser.py
+++ b/src/python/pants/engine/internals/parser.py
@@ -4,6 +4,7 @@
 import os.path
 import tokenize
 from dataclasses import dataclass
+from difflib import get_close_matches
 from io import StringIO
 from typing import Any, Dict, Iterable, List, Tuple, cast
 
@@ -90,6 +91,7 @@ def parse(
         # those extra symbols will not see our target aliases etc. This also means that if multiple
         # prelude files are present, they probably cannot see each others' symbols. We may choose
         # to change this at some point.
+
         global_symbols = dict(self._symbols)
         for k, v in extra_symbols.symbols.items():
             if hasattr(v, "__globals__"):
@@ -101,7 +103,24 @@ def parse(
         except NameError as e:
             valid_symbols = sorted(s for s in global_symbols.keys() if s != "__builtins__")
             original = e.args[0].capitalize()
-            raise ParseError(f"{original}.\n\nAll registered symbols: {valid_symbols}")
+            help_str = (
+                "If you expect to see more symbols activated in the below list,"
+                f" refer to {docs_url('enabling-backends')} for all available"
+                " backends to activate."
+            )
+
+            candidates = get_close_matches(build_file_content, valid_symbols)
+            if candidates:
+                if len(candidates) == 1:
+                    formatted_candidates = candidates[0]
+                elif len(candidates) == 2:
+                    formatted_candidates = " or ".join(candidates)
+                else:
+                    formatted_candidates = f"{', '.join(candidates[:-1])}, or {candidates[-1]}"
+                help_str = f"Did you mean {formatted_candidates}?\n\n" + help_str
+            raise ParseError(
+                f"{original}.\n\n{help_str}\n\nAll registered symbols: {valid_symbols}"
+            )
 
         error_on_imports(build_file_content, filepath)
 


Test patch:
diff --git a/src/python/pants/engine/internals/parser_test.py b/src/python/pants/engine/internals/parser_test.py
--- a/src/python/pants/engine/internals/parser_test.py
+++ b/src/python/pants/engine/internals/parser_test.py
@@ -1,10 +1,13 @@
 # Copyright 2020 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
+from __future__ import annotations
+
 import pytest
 
 from pants.build_graph.build_file_aliases import BuildFileAliases
 from pants.engine.internals.parser import BuildFilePreludeSymbols, ParseError, Parser
+from pants.util.docutil import docs_url
 from pants.util.frozendict import FrozenDict
 
 
@@ -17,18 +20,34 @@ def test_imports_banned() -> None:
     assert "Import used in dir/BUILD at line 4" in str(exc.value)
 
 
-def test_unrecognized_symbol() -> None:
-    parser = Parser(
-        target_type_aliases=["tgt"],
-        object_aliases=BuildFileAliases(
-            objects={"obj": 0},
-            context_aware_object_factories={"caof": lambda parse_context: lambda _: None},
-        ),
-    )
-    prelude_symbols = BuildFilePreludeSymbols(FrozenDict({"prelude": 0}))
-    with pytest.raises(ParseError) as exc:
-        parser.parse("dir/BUILD", "fake", prelude_symbols)
-    assert (
-        str(exc.value)
-        == "Name 'fake' is not defined.\n\nAll registered symbols: ['caof', 'obj', 'prelude', 'tgt']"
-    )
+def test_unrecogonized_symbol() -> None:
+    def perform_test(extra_targets: list[str], dym: str) -> None:
+
+        parser = Parser(
+            target_type_aliases=["tgt", *extra_targets],
+            object_aliases=BuildFileAliases(
+                objects={"obj": 0},
+                context_aware_object_factories={"caof": lambda parse_context: lambda _: None},
+            ),
+        )
+        prelude_symbols = BuildFilePreludeSymbols(FrozenDict({"prelude": 0}))
+        fmt_extra_sym = str(extra_targets)[1:-1] + (", ") if len(extra_targets) != 0 else ""
+        with pytest.raises(ParseError) as exc:
+            parser.parse("dir/BUILD", "fake", prelude_symbols)
+        assert str(exc.value) == (
+            f"Name 'fake' is not defined.\n\n{dym}"
+            "If you expect to see more symbols activated in the below list,"
+            f" refer to {docs_url('enabling-backends')} for all available"
+            " backends to activate.\n\n"
+            f"All registered symbols: ['caof', {fmt_extra_sym}'obj', 'prelude', 'tgt']"
+        )
+
+    test_targs = ["fake1", "fake2", "fake3", "fake4", "fake5"]
+
+    perform_test([], "")
+    dym_one = "Did you mean fake1?\n\n"
+    perform_test(test_targs[:1], dym_one)
+    dym_two = "Did you mean fake2 or fake1?\n\n"
+    perform_test(test_targs[:2], dym_two)
+    dym_many = "Did you mean fake5, fake4, or fake3?\n\n"
+    perform_test(test_targs, dym_many)