Repository: pantsbuild/pants

Issue:
Improve module_mapping
Feature request for improving the `module_mapping` configuration.

As this is duplicated information, and placed in the `BUILD` file for efficiency reasons, this could be improved upon for usability.

One possibility is to have a pre-defined list of module_mappings for a set of well known commonly used packages.

Another is to make use of the lock file to work around ambiguities.

Excerpt from conversation on slack:

> [@kaos] ... the mapping we input here, is already available information from the packages themselves from the top_level.txt file once installed. Can't we use this information from the source, instead of having to manually provide it?

> [@jsirois] We can. The issue is efficiency. So, the context where the module_mapping is used is in dependency inference. Pants is looking at a Python source file, picking out import statements, and determining from those what other files and distributions the Python source file being examined depends on. Right now, with the module_mapping, this can all be done by looking at local files. To infer the module_mapping itself, Pants would need to resolve all distributions declared anywhere in the repo up front, then use that resolve to build a module_mapping.

> [..] The one case where we could do this in a fairly unambiguously performant way is when you have a "lockfile" set up as outlined here: https://www.pantsbuild.org/docs/python-third-party-dependencies#using-a-lockfile-strongly-recommended

> [..] Without that though, we need to scan every BUILD file for potential python_requirements or python_requirement_library targets (https://www.pantsbuild.org/docs/reference-python_requirement_library) and resolve those. Performance aside, there are ambiguities this introduces. It may be that you have conflicting requirements that are not meant to be resolved together, To be safe, we'd then need to resolve each requirement seperately or ... there is really no valid way to pick resolve sets, and picking a resolve set requires inferring the dependencies of your python source files. Here we hit a loop.

> [..] So, short answer - we could do this for lockfiles I think. Is this something you'd be interested in chipping in support for? We're happy to shepherd new contributors.
Either way, it at least seems worthy of tracking with a feature request issue if you don't mind filing an issue 

> [@benjyw] I think we could provide a default module_mapping for some well-known common cases, to reduce this friction even further. E.g., pkg_resources->setuptools.

https://pantsbuild.slack.com/archives/C046T6T9U/p1614762271091800



Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/dependency_inference/default_module_mapping.py b/src/python/pants/backend/python/dependency_inference/default_module_mapping.py
new file mode 100644
--- /dev/null
+++ b/src/python/pants/backend/python/dependency_inference/default_module_mapping.py
@@ -0,0 +1,28 @@
+# Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+# notice: using sets here to ensure the mapping is hashable
+DEFAULT_MODULE_MAPPING = {
+    "ansicolors": ("colors",),
+    "attrs": ("attr",),
+    "beautifulsoup4": ("bs4",),
+    "djangorestframework": ("rest_framework",),
+    "enum34": ("enum",),
+    "paho_mqtt": ("paho",),
+    "protobuf": ("google.protobuf",),
+    "pycrypto": ("Crypto",),
+    "pyopenssl": ("OpenSSL",),
+    "python-dateutil": ("dateutil",),
+    "python-jose": ("jose",),
+    "PyYAML": ("yaml",),
+    "pymongo": (
+        "bson",
+        "gridfs",
+    ),
+    "pytest_runner": ("ptr",),
+    "python_dateutil": ("dateutil",),
+    "setuptools": (
+        "easy_install",
+        "pkg_resources",
+    ),
+}
diff --git a/src/python/pants/backend/python/dependency_inference/module_mapper.py b/src/python/pants/backend/python/dependency_inference/module_mapper.py
--- a/src/python/pants/backend/python/dependency_inference/module_mapper.py
+++ b/src/python/pants/backend/python/dependency_inference/module_mapper.py
@@ -182,7 +182,7 @@ async def map_third_party_modules_to_addresses() -> ThirdPartyPythonModuleMappin
     for tgt in all_targets:
         if not tgt.has_field(PythonRequirementsField):
             continue
-        module_map = tgt.get(ModuleMappingField).value or {}  # type: ignore[var-annotated]
+        module_map = tgt.get(ModuleMappingField).value
         for python_req in tgt[PythonRequirementsField].value:
             modules = module_map.get(
                 python_req.project_name,
diff --git a/src/python/pants/backend/python/target_types.py b/src/python/pants/backend/python/target_types.py
--- a/src/python/pants/backend/python/target_types.py
+++ b/src/python/pants/backend/python/target_types.py
@@ -10,10 +10,11 @@
 from dataclasses import dataclass
 from enum import Enum
 from textwrap import dedent
-from typing import Iterable, Iterator, Optional, Tuple, Union, cast
+from typing import Dict, Iterable, Iterator, Optional, Tuple, Union, cast
 
 from pkg_resources import Requirement
 
+from pants.backend.python.dependency_inference.default_module_mapping import DEFAULT_MODULE_MAPPING
 from pants.backend.python.macros.python_artifact import PythonArtifact
 from pants.backend.python.subsystems.pytest import PyTest
 from pants.core.goals.package import OutputPathField
@@ -41,6 +42,7 @@
 from pants.python.python_setup import PythonSetup
 from pants.source.filespec import Filespec
 from pants.util.docutil import docs_url
+from pants.util.frozendict import FrozenDict
 
 logger = logging.getLogger(__name__)
 
@@ -591,6 +593,14 @@ class ModuleMappingField(DictStringToStringSequenceField):
         'name as the default module, e.g. "Django" will default to `["django"]`.\n\nThis is '
         "used for Pants to be able to infer dependencies in BUILD files."
     )
+    value: FrozenDict[str, Tuple[str, ...]]
+
+    @classmethod
+    def compute_value(
+        cls, raw_value: Optional[Dict[str, Iterable[str]]], *, address: Address
+    ) -> FrozenDict[str, Tuple[str, ...]]:
+        provided_mapping = super().compute_value(raw_value, address=address)
+        return FrozenDict({**DEFAULT_MODULE_MAPPING, **(provided_mapping or {})})
 
 
 class PythonRequirementLibrary(Target):


Test patch:
diff --git a/src/python/pants/backend/python/macros/pants_requirement_test.py b/src/python/pants/backend/python/macros/pants_requirement_test.py
--- a/src/python/pants/backend/python/macros/pants_requirement_test.py
+++ b/src/python/pants/backend/python/macros/pants_requirement_test.py
@@ -39,7 +39,9 @@ def assert_pants_requirement(
     assert target[PythonRequirementsField].value == (
         Requirement.parse(f"{expected_dist}=={pants_version()}"),
     )
-    assert target[ModuleMappingField].value == FrozenDict({expected_dist: (expected_module,)})
+    actual_value = target[ModuleMappingField].value
+    assert isinstance(actual_value, FrozenDict)
+    assert actual_value.get(expected_dist) == (expected_module,)
 
 
 def test_target_name(rule_runner: RuleRunner) -> None:
diff --git a/src/python/pants/backend/python/target_types_test.py b/src/python/pants/backend/python/target_types_test.py
--- a/src/python/pants/backend/python/target_types_test.py
+++ b/src/python/pants/backend/python/target_types_test.py
@@ -3,17 +3,19 @@
 
 import logging
 from textwrap import dedent
-from typing import List, Optional
+from typing import Dict, Iterable, List, Optional, Tuple
 
 import pytest
 from _pytest.logging import LogCaptureFixture
 from pkg_resources import Requirement
 
+from pants.backend.python.dependency_inference.default_module_mapping import DEFAULT_MODULE_MAPPING
 from pants.backend.python.dependency_inference.rules import import_rules
 from pants.backend.python.macros.python_artifact import PythonArtifact
 from pants.backend.python.subsystems.pytest import PyTest
 from pants.backend.python.target_types import (
     EntryPoint,
+    ModuleMappingField,
     PexBinary,
     PexBinaryDependencies,
     PexEntryPointField,
@@ -43,6 +45,7 @@
 )
 from pants.testutil.option_util import create_subsystem
 from pants.testutil.rule_runner import QueryRule, RuleRunner
+from pants.util.frozendict import FrozenDict
 
 
 def test_timeout_validation() -> None:
@@ -315,3 +318,20 @@ def test_python_distribution_dependency_injection() -> None:
         [InjectPythonDistributionDependencies(tgt[PythonDistributionDependencies])],
     )
     assert injected == InjectedDependencies([Address("project", target_name="my_binary")])
+
+
+@pytest.mark.parametrize(
+    ["raw_value", "expected"],
+    (
+        (None, {}),
+        ({"new_dist": ["new_module"]}, {"new_dist": ("new_module",)}),
+        ({"PyYAML": ["custom_yaml"]}, {"PyYAML": ("custom_yaml",)}),
+    ),
+)
+def test_module_mapping_field(
+    raw_value: Optional[Dict[str, Iterable[str]]], expected: Dict[str, Tuple[str]]
+) -> None:
+    merged = dict(DEFAULT_MODULE_MAPPING)
+    merged.update(expected)
+    actual_value = ModuleMappingField(raw_value, address=Address("", target_name="tests")).value
+    assert actual_value == FrozenDict(merged)