Repository: pantsbuild/pants

Issue:
Django `manage.py` files should infer dependencies on migrations from `INSTALLED_APPS`
Currently, running any Django management command that depends on migrations requires running `manage.py` outside the sandbox, or (apparently) explicitly declaring migrations packages as dependencies of `manage.py`. `manage.py` knows which migrations to run by way of the apps declared in the `INSTALLED_APPS` value in a Django settings file.

With the experimental Django plugin, it appears that Django migrations are able to infer dependencies on other migrations by way of understanding the `AppConfig.name` value in the `apps.py` file for those other migrations. It appears that the leaf migrations still need to be declared automatically to be picked up as a dependency.

With that in mind, it looks like `INSTALLED_APPS` values can be inspected -- they either declare a package spec, or an `AppConfig` class, with a name that consists of a package spec. That should be enough information to infer dependencies on those migrations.

Pants should do that :)

**UPDATE**: It looks like management commands are in the same boat, and should also get picked up.


Discussion:
The trouble with `INSTALLED_APPS` is that it may not consist only of hardcoded values in a single place, so static analysis may work for simple cases.. perhaps good enough?
@kaos Agreed, it's not going to be complete. It'll be about as good as our ability to infer string imports, which exists for other strings in `settings.py` files, so probably good enough?
String imports should already get us an inferred dep from settings.py to the package containing an app. But that just turns into a dep on that package's `__init__.py`, which is typically empty.

So the question in my mind is: do we want to 1) infer a dep from settings.py on the migrations inside that package, or 2) infer a dep from an app package's `__init__.py` onto various things inside its package (in this case, migrations).

The latter seems "correct" in the sense that this is probably what you mean when you list an app in settings.py. You're not saying "depend on this app's migrations", you're saying "depend on ~everything in this app". Doing fine-grained deps on slivers of code inside apps may be incompatible with Django's whole thing.

I'm not at all sure what the right answer is. 
@benjyw Well, for starters, Python packages don't need `__init__.py ` files any more, and running `PANTS_PYTHON_INFER_STRING_IMPORTS_MIN_DOTS=1 pants dependencies src/python/DjangoProjectName/config/settings/base.py` with an `INSTALLED_APP` that contains no `__init__.py` doesn't work.

The good news is that it's also valid to point `INSTALLED_APPS` to an `AppConfig` class.

How would we feel about hanging the inferred dependencies (the `migrations` and `management/commands` packages) off `apps.py`?
Yeah, that makes sense. In practice - unrelated to Pants - Django chokes without `__init__.py`s, as it relies on them for its own inferr-y things, but you're right that app.py is the right place for this. 
This seems easy enough to prototype. I'll have a go over the next day or
three.

On Wed, 29 Nov 2023, 15:00 Benjy Weinberger, ***@***.***>
wrote:

> String imports should already get us an inferred dep from settings.py to
> the package containing an app. But that just turns into a dep on that
> package's __init__.py, which is typically empty.
>
> So the question in my mind is: do we want to 1) infer a dep from
> settings.py on the migrations inside that package, or 2) infer a dep from
> an app package's __init__.py onto various things inside its package (in
> this case, migrations).
>
> The latter seems "correct" in the sense that this is probably what you
> mean when you list an app in settings.py. You're not saying "depend on this
> app's migrations", you're saying "depend on ~everything in this app". Doing
> fine-grained deps on slivers of code inside apps may be incompatible with
> Django's whole thing.
>
> I'm not at all sure what the right answer is.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pantsbuild/pants/issues/20246#issuecomment-1832835952>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAEX5XCMUJWJJQBJNULBXHDYG65BTAVCNFSM6AAAAAA775KRV2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSHAZTKOJVGI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Another idea could be to depend on the owning primary target of `__init__.py` instead.

In our django project at $-work, we don't have `app.py` we have `apps.py`. Also, I think it makes sense to depend on all top-level modules in the app, as have been pointed out already, Django won't work well with a subset of those any way.

What I mean with "primary owning target" is that if we look up `__init__.py` and find it is owned by a `python_source` target that has been generated from a `python_sources` target, the target generator would be the primary target in this case. If we depend on that, we'll depend on all files that is picked up by the sources field of the `python_sources` target and don't need worry about what files are present and not nor what they're called.
And if some project needs/wants some file(s) to be excluded from this inferred dependency you can split them up into multiple targets, and only have the ones you do want together with the `__init__.py` one. Does this make sense to you?
> In our django project at $-work, we don't have `app.py` we have `apps.py`

yup, `apps.py` is the standard. The trick is that at the moment, there's no transitive dependency to the `migrations` or `management.commands` subpackages, and I'm not sure how depending on the `__init__.py` at the same level as `apps.py` would help with that.

My current approach, which I'm going to try out in an internal plugin, is going to be to use the existing `app_detector.py` script to output the package names that the `apps.py` file depends on, and then see if we can resolve known subpackages of each of those package names arising from `AppConfig`. I think that'll work OK in my case, and it should be a matter of expanding from there.
yea, incremental progress is likely the only viable way here..
I have an internal plugin that will find all migrations in the same directory as an `__init__.py` file in a `migrations` (also a `management.commands`) subpackage of a package declared in an `AppConfig` class. I'll tidy it up for external use later today.

Resolving patch:
diff --git a/src/python/pants/backend/python/framework/django/dependency_inference.py b/src/python/pants/backend/python/framework/django/dependency_inference.py
--- a/src/python/pants/backend/python/framework/django/dependency_inference.py
+++ b/src/python/pants/backend/python/framework/django/dependency_inference.py
@@ -23,6 +23,7 @@
 from pants.backend.python.util_rules import pex
 from pants.backend.python.util_rules.interpreter_constraints import InterpreterConstraints
 from pants.backend.python.util_rules.pex import PexRequest, VenvPex, VenvPexProcess
+from pants.base.specs import FileGlobSpec, RawSpecs
 from pants.core.util_rules.source_files import SourceFilesRequest
 from pants.core.util_rules.stripped_source_files import StrippedSourceFiles
 from pants.engine.fs import CreateDigest, FileContent
@@ -30,7 +31,7 @@
 from pants.engine.internals.selectors import Get
 from pants.engine.process import ProcessResult
 from pants.engine.rules import collect_rules, rule
-from pants.engine.target import InferDependenciesRequest, InferredDependencies
+from pants.engine.target import InferDependenciesRequest, InferredDependencies, Targets
 from pants.engine.unions import UnionRule
 from pants.util.logging import LogLevel
 from pants.util.resources import read_resource
@@ -41,19 +42,17 @@ class InferDjangoDependencies(InferDependenciesRequest):
 
 
 _visitor_resource = "scripts/dependency_visitor.py"
+_implicit_dependency_packages = (
+    "management.commands",
+    "migrations",
+)
 
 
-@rule
-async def django_parser_script(
+async def _django_migration_dependencies(
     request: InferDjangoDependencies,
     python_setup: PythonSetup,
     django_apps: DjangoApps,
 ) -> InferredDependencies:
-    source_field = request.field_set.source
-    # NB: This doesn't consider https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-MIGRATION_MODULES
-    if not PurePath(source_field.file_path).match("migrations/*.py"):
-        return InferredDependencies([])
-
     stripped_sources = await Get(
         StrippedSourceFiles, SourceFilesRequest([request.field_set.source])
     )
@@ -117,6 +116,77 @@ async def django_parser_script(
     )
 
 
+async def _django_app_implicit_dependencies(
+    request: InferDjangoDependencies,
+    python_setup: PythonSetup,
+    django_apps: DjangoApps,
+) -> InferredDependencies:
+    file_path = request.field_set.source.file_path
+    apps = [
+        django_app for django_app in django_apps.values() if django_app.config_file == file_path
+    ]
+    if not apps:
+        return InferredDependencies([])
+
+    app_package = apps[0].name
+
+    implicit_dependency_packages = [
+        f"{app_package}.{subpackage}" for subpackage in _implicit_dependency_packages
+    ]
+
+    resolve = request.field_set.resolve.normalized_value(python_setup)
+
+    resolved_dependencies = await Get(
+        ResolvedParsedPythonDependencies,
+        ResolvedParsedPythonDependenciesRequest(
+            request.field_set,
+            ParsedPythonDependencies(
+                ParsedPythonImports(
+                    (package, ParsedPythonImportInfo(0, False))
+                    for package in implicit_dependency_packages
+                ),
+                ParsedPythonAssetPaths(),
+            ),
+            resolve,
+        ),
+    )
+
+    spec_paths = [
+        address.spec_path
+        for result in resolved_dependencies.resolve_results.values()
+        if result.status in (ImportOwnerStatus.unambiguous, ImportOwnerStatus.disambiguated)
+        for address in result.address
+    ]
+
+    targets = await Get(
+        Targets,
+        RawSpecs,
+        RawSpecs.create(
+            specs=[FileGlobSpec(f"{spec_path}/*.py") for spec_path in spec_paths],
+            description_of_origin="Django implicit dependency detection",
+        ),
+    )
+
+    return InferredDependencies(sorted(target.address for target in targets))
+
+
+@rule
+async def infer_django_dependencies(
+    request: InferDjangoDependencies,
+    python_setup: PythonSetup,
+    django_apps: DjangoApps,
+) -> InferredDependencies:
+    source_field = request.field_set.source
+    # NB: This doesn't consider https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-MIGRATION_MODULES
+    path = PurePath(source_field.file_path)
+    if path.match("migrations/*.py"):
+        return await _django_migration_dependencies(request, python_setup, django_apps)
+    elif path.match("apps.py"):
+        return await _django_app_implicit_dependencies(request, python_setup, django_apps)
+    else:
+        return InferredDependencies([])
+
+
 def rules():
     return [
         *collect_rules(),
diff --git a/src/python/pants/backend/python/framework/django/detect_apps.py b/src/python/pants/backend/python/framework/django/detect_apps.py
--- a/src/python/pants/backend/python/framework/django/detect_apps.py
+++ b/src/python/pants/backend/python/framework/django/detect_apps.py
@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import os
 from collections import defaultdict
 from dataclasses import dataclass
 
@@ -27,12 +28,30 @@
 
 
 @dataclass(frozen=True)
-class DjangoApps:
-    label_to_name: FrozenDict[str, str]
+class DjangoApp:
+    name: str
+    config_file: str
 
-    def add_from_json(self, json_bytes: bytes) -> "DjangoApps":
-        apps = dict(self.label_to_name, **json.loads(json_bytes.decode()))
-        return DjangoApps(FrozenDict(sorted(apps.items())))
+
+class DjangoApps(FrozenDict[str, DjangoApp]):
+    @property
+    def label_to_name(self) -> FrozenDict[str, str]:
+        return FrozenDict((label, app.name) for label, app in self.items())
+
+    @property
+    def label_to_file(self) -> FrozenDict[str, str]:
+        return FrozenDict((label, app.config_file) for label, app in self.items())
+
+    def add_from_json(self, json_bytes: bytes, strip_prefix="") -> "DjangoApps":
+        json_dict: dict[str, dict[str, str]] = json.loads(json_bytes.decode())
+        apps = {
+            label: DjangoApp(
+                val["app_name"], val["config_file"].partition(f"{strip_prefix}{os.sep}")[2]
+            )
+            for label, val in json_dict.items()
+        }
+        combined = dict(self, **apps)
+        return DjangoApps(sorted(combined.items()))
 
 
 _script_resource = "scripts/app_detector.py"
@@ -119,7 +138,10 @@ async def detect_django_apps(python_setup: PythonSetup) -> DjangoApps:
                 description="Detect Django apps",
             ),
         )
-        django_apps = django_apps.add_from_json(process_result.stdout or b"{}")
+        django_apps = django_apps.add_from_json(
+            process_result.stdout or b"{}", strip_prefix=apps_sandbox_prefix
+        )
+
     return django_apps
 
 
diff --git a/src/python/pants/backend/python/framework/django/scripts/app_detector.py b/src/python/pants/backend/python/framework/django/scripts/app_detector.py
--- a/src/python/pants/backend/python/framework/django/scripts/app_detector.py
+++ b/src/python/pants/backend/python/framework/django/scripts/app_detector.py
@@ -77,7 +77,7 @@ def handle_file(apps, path):
     visitor.visit(tree)
 
     if visitor.app_name:
-        apps[visitor.app_label] = visitor.app_name
+        apps[visitor.app_label] = {"app_name": visitor.app_name, "config_file": path}
 
 
 def main(paths):


Test patch:
diff --git a/src/python/pants/backend/python/framework/django/dependency_inference_test.py b/src/python/pants/backend/python/framework/django/dependency_inference_test.py
--- a/src/python/pants/backend/python/framework/django/dependency_inference_test.py
+++ b/src/python/pants/backend/python/framework/django/dependency_inference_test.py
@@ -46,12 +46,12 @@ def rule_runner() -> RuleRunner:
     return rule_runner
 
 
-def do_test_migration_dependencies(rule_runner: RuleRunner, constraints: str) -> None:
-    rule_runner.write_files(
-        {
-            "BUILD": "python_source(name='t', source='path/to/app0/migrations/0001_initial.py')",
-            "path/to/app0/migrations/0001_initial.py": dedent(
-                """\
+def files(constraints: str) -> dict[str, str]:
+    return {
+        "BUILD": "python_source(name='t', source='path/to/app0/migrations/0001_initial.py')",
+        "path/to/app0/migrations/__init__.py": "",
+        "path/to/app0/migrations/0001_initial.py": dedent(
+            """\
             class Migration(migrations.Migration):
                 dependencies = [
                 ("app1", "0012_some_migration"),
@@ -60,43 +60,82 @@ class Migration(migrations.Migration):
 
                 operations = []
             """
-            ),
-            "path/to/app1/BUILD": dedent(
-                f"""\
+        ),
+        "path/to/app1/BUILD": dedent(
+            f"""\
                 python_source(
+                  name="app1",
                   source="apps.py",
                   interpreter_constraints=['{constraints}'],
                 )
+                python_source(
+                  name="initpy",
+                  source="__init__.py",
+                )
                 """
-            ),
-            "path/to/app1/apps.py": dedent(
-                """\
+        ),
+        "path/to/app1/__init__.py": "",
+        "path/to/app1/apps.py": dedent(
+            """\
                 class App1AppConfig(AppConfig):
                     name = "path.to.app1"
                     label = "app1"
                 """
-            ),
-            "path/to/app1/migrations/BUILD": "python_source(source='0012_some_migration.py')",
-            "path/to/app1/migrations/0012_some_migration.py": "",
-            "another/path/app2/BUILD": dedent(
-                f"""\
+        ),
+        "path/to/app1/management/commands/BUILD": dedent(
+            """\
+                python_source(source='do_a_thing.py')
+                python_source(
+                    name="initpy",
+                    source="__init__.py",
+                )
+                """
+        ),
+        "path/to/app1/management/commands/__init__.py": "",
+        "path/to/app1/management/commands/do_a_thing.py": "",
+        "path/to/app1/migrations/BUILD": dedent(
+            """\
+                python_source(source='0012_some_migration.py')
+                python_source(
+                    name="initpy",
+                    source="__init__.py",
+                )
+                """
+        ),
+        "path/to/app1/migrations/__init__.py": "",
+        "path/to/app1/migrations/0012_some_migration.py": "",
+        "another/path/app2/BUILD": dedent(
+            f"""\
                 python_source(
                   source="apps.py",
                   interpreter_constraints=['{constraints}'],
                 )
                 """
-            ),
-            "another/path/app2/apps.py": dedent(
-                """\
+        ),
+        "another/path/app2/__init__.py": "",
+        "another/path/app2/apps.py": dedent(
+            """\
                 class App2AppConfig(AppConfig):
                     name = "another.path.app2"
                     label = "app2_label"
                 """
-            ),
-            "another/path/app2/migrations/BUILD": "python_source(source='0042_another_migration.py')",
-            "another/path/app2/migrations/0042_another_migration.py": "",
-        }
-    )
+        ),
+        "another/path/app2/migrations/BUILD": dedent(
+            """\
+                python_source(source='0042_another_migration.py')
+                python_source(
+                    name="initpy",
+                    source="__init__.py",
+                )
+                """
+        ),
+        "another/path/app2/migrations/__init__.py": "",
+        "another/path/app2/migrations/0042_another_migration.py": "",
+    }
+
+
+def do_test_migration_dependencies(rule_runner: RuleRunner, constraints: str) -> None:
+    rule_runner.write_files(files(constraints))
     tgt = rule_runner.get_target(Address("", target_name="t"))
     result = rule_runner.request(
         InferredDependencies,
@@ -112,21 +151,45 @@ class App2AppConfig(AppConfig):
     }
 
 
+def do_test_implicit_app_dependencies(rule_runner: RuleRunner, constraints: str) -> None:
+    rule_runner.write_files(files(constraints))
+    tgt = rule_runner.get_target(Address("path/to/app1", target_name="app1"))
+    print(tgt)
+    result = rule_runner.request(
+        InferredDependencies,
+        [
+            dependency_inference.InferDjangoDependencies(
+                PythonImportDependenciesInferenceFieldSet.create(tgt)
+            )
+        ],
+    )
+    assert set(result.include) == {
+        Address("path/to/app1/migrations", target_name="migrations"),
+        Address("path/to/app1/migrations", target_name="initpy"),
+        Address("path/to/app1/management/commands", target_name="commands"),
+        Address("path/to/app1/management/commands", target_name="initpy"),
+    }
+
+
 @skip_unless_python27_present
 def test_works_with_python2(rule_runner: RuleRunner) -> None:
     do_test_migration_dependencies(rule_runner, constraints="CPython==2.7.*")
+    do_test_implicit_app_dependencies(rule_runner, constraints="CPython==2.7.*")
 
 
 @skip_unless_python37_present
 def test_works_with_python37(rule_runner: RuleRunner) -> None:
     do_test_migration_dependencies(rule_runner, constraints="CPython==3.7.*")
+    do_test_implicit_app_dependencies(rule_runner, constraints="CPython==3.7.*")
 
 
 @skip_unless_python38_present
 def test_works_with_python38(rule_runner: RuleRunner) -> None:
     do_test_migration_dependencies(rule_runner, constraints="CPython==3.8.*")
+    do_test_implicit_app_dependencies(rule_runner, constraints="CPython==3.8.*")
 
 
 @skip_unless_python39_present
 def test_works_with_python39(rule_runner: RuleRunner) -> None:
     do_test_migration_dependencies(rule_runner, constraints="CPython==3.9.*")
+    do_test_implicit_app_dependencies(rule_runner, constraints="CPython==3.9.*")
diff --git a/src/python/pants/backend/python/framework/django/detect_apps_test.py b/src/python/pants/backend/python/framework/django/detect_apps_test.py
--- a/src/python/pants/backend/python/framework/django/detect_apps_test.py
+++ b/src/python/pants/backend/python/framework/django/detect_apps_test.py
@@ -7,7 +7,7 @@
 
 from pants.backend.python.dependency_inference import parse_python_dependencies
 from pants.backend.python.framework.django import detect_apps
-from pants.backend.python.framework.django.detect_apps import DjangoApps
+from pants.backend.python.framework.django.detect_apps import DjangoApp, DjangoApps
 from pants.backend.python.target_types import PythonSourceTarget
 from pants.backend.python.util_rules import pex
 from pants.core.util_rules import stripped_source_files
@@ -114,7 +114,13 @@ class App3AppConfig(AppConfig):
         [],
     )
     assert result == DjangoApps(
-        FrozenDict({"app1": "path.to.app1", "app2_label": "another.path.app2", "app3": "some.app3"})
+        FrozenDict(
+            {
+                "app1": DjangoApp("path.to.app1", "path/to/app1/apps.py"),
+                "app2_label": DjangoApp("another.path.app2", "another/path/app2/apps.py"),
+                "app3": DjangoApp("some.app3", "some/app3/apps.py"),
+            }
+        )
     )