Repository: pantsbuild/pants

Issue:
Not able to load resources when using pants vs sbt
**Describe the bug**
When using sbt we are able to call `Thread.currentThread().getContextClassLoader().getResources` and get a list of URLs. When using pants the list is empty. 

This at the moment limits us from using Flyway with pants.

**Pants version**
2.13.0a1 and main.

**OS**
MacOS

**Additional info**
Example repo to reproduce the issue:
https://github.com/somdoron/test-pants-resources

I think the issue is, that pants only compress files in the resources zip file and not the directories.


Discussion:
when running `unzip -v` on the sbt jar I get:
```
Archive:  root_2.13-0.1.0-SNAPSHOT.jar
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
     275  Defl:N      128  54% 01-01-2010 00:00 ce03dda4  META-INF/MANIFEST.MF
       0  Stored        0   0% 01-01-2010 00:00 00000000  db/
       0  Stored        0   0% 01-01-2010 00:00 00000000  db/migration/
       0  Stored        0   0% 01-01-2010 00:00 00000000  example/
      38  Defl:N       31  18% 01-01-2010 00:00 de34bf7a  BUILD
       0  Defl:N        2   0% 01-01-2010 00:00 00000000  db/migration/V1.sql
       0  Defl:N        2   0% 01-01-2010 00:00 00000000  db/migration/V2.sql
    1643  Defl:N      836  49% 01-01-2010 00:00 7fe7b92b  example/Hello$.class
     699  Defl:N      544  22% 01-01-2010 00:00 97b6a14c  example/Hello.class
```

So the folder is an item in the zip file when running the same on pants fatjar:

```
Archive:  dist/src.main.scala.example/app.jar
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      38  Defl:N       31  18% 07-20-2022 00:26 de34bf7a  BUILD
       0  Stored        0   0% 07-20-2022 00:26 00000000  db/migration/V1.sql
       0  Stored        0   0% 07-20-2022 00:26 00000000  db/migration/V2.sql
      84  Defl:N       74  12% 07-20-2022 00:24 f5bae605  META-INF/MANIFEST.MF
```

So it seems the directory is missing from the zip file produced by pants.

> Both sbt and pants able to call `getResource("db/migration/V1.sql")` and get back a URL, however only sbt receives list of URLs when calling `.getResources("db/migration")`.

Yea, a JAR needs to contain directory entries in order for a `getResources(directory)` call to work... but I would have thought that we were doing that, since the `zip` CLI will create them.

@chrisjrn : are you able to take a look at this one? As mentioned, this is usually controlled by the `-D` flag to `zip`:
```
-D
--no-dir-entries
    Do not create entries in the zip archive for directories. Directory entries are created by default so that their attributes can be saved in the zip archive. The environment variable ZIPOPT can be used to change the default options. For example under Unix with sh: 
```
It seems because files are provided specifically directories are skipped if I'm applying the patch below the directories are picked.

```
diff --git a/src/python/pants/jvm/resources.py b/src/python/pants/jvm/resources.py
index 5b565b82a..e97b54e28 100644
--- a/src/python/pants/jvm/resources.py
+++ b/src/python/pants/jvm/resources.py
@@ -82,8 +82,9 @@ async def assemble_resources_jar(
         Process(
             argv=[
                 zip.path,
+                "-r",
                 output_filename,
-                *source_files.snapshot.files,
+                "."
             ],
             description="Build resources JAR for {request.component}",
             input_digest=resources_jar_input_digest,
```
One option would be to put the entire input `Snapshot` in a subdirectory, and then invoke `zip` inside that subdirectory (via [Process.working_directory](https://github.com/pantsbuild/pants/blob/9f7b5161207b0804bcae38ef11bb99d4291a75a0/src/rust/engine/process_execution/src/lib.rs#L466-L470)), while creating the output file in the parent directory.

Something like:
```python
subdir = "subdir"
input_digest = await Get(Digest, AddPrefix(source_files.snapshot.digest, subdir))
Process(
    argv=[
        zip.path,
        "-r",
        f"../{output_filename}",
        ".",
    ],
    working_directory=subdir,
    input_digest=input_digest,
    ..
)
```
I tried that, but apparently pants is looking for the output at the work directory.
taking a look at this. I'll let you know what I find.
I think I've got the example repo working; just need to re-work how our resources unit tests work before submitting the PR.
Per @somdoron's investigation, directory names need to be provided explicitly as input files. #16264 walks the parents of each filename and adds those to the set of input files for the zip file. The repro repo works as expected now.

Resolving patch:
diff --git a/src/python/pants/jvm/resources.py b/src/python/pants/jvm/resources.py
--- a/src/python/pants/jvm/resources.py
+++ b/src/python/pants/jvm/resources.py
@@ -3,6 +3,7 @@
 import itertools
 import logging
 from itertools import chain
+from pathlib import Path
 
 from pants.core.target_types import ResourcesFieldSet, ResourcesGeneratorFieldSet
 from pants.core.util_rules import stripped_source_files
@@ -71,6 +72,13 @@ async def assemble_resources_jar(
     output_filename = f"{request.component.representative.address.path_safe_spec}.resources.jar"
     output_files = [output_filename]
 
+    # #16231: Valid JAR files need the directories of each resource file as well as the files
+    # themselves.
+
+    paths = {Path(filename) for filename in source_files.snapshot.files}
+    directories = {parent for path in paths for parent in path.parents}
+    input_files = {str(path) for path in chain(paths, directories)}
+
     resources_jar_input_digest = source_files.snapshot.digest
     resources_jar_result = await Get(
         ProcessResult,
@@ -78,7 +86,7 @@ async def assemble_resources_jar(
             argv=[
                 zip.path,
                 output_filename,
-                *source_files.snapshot.files,
+                *sorted(input_files),
             ],
             description="Build partial JAR containing resources files",
             input_digest=resources_jar_input_digest,


Test patch:
diff --git a/src/python/pants/jvm/resources_test.py b/src/python/pants/jvm/resources_test.py
--- a/src/python/pants/jvm/resources_test.py
+++ b/src/python/pants/jvm/resources_test.py
@@ -3,13 +3,20 @@
 
 from __future__ import annotations
 
+from io import BytesIO
+from zipfile import ZipFile
+
 import pytest
 
 from pants.build_graph.address import Address
 from pants.core.target_types import ResourcesGeneratorTarget, ResourceTarget
 from pants.core.target_types import rules as core_target_types_rules
 from pants.engine.addresses import Addresses
-from pants.jvm import classpath, resources, testutil
+from pants.engine.fs import DigestContents, FileContent
+from pants.engine.internals.native_engine import Digest
+from pants.jvm import resources, testutil
+from pants.jvm.classpath import Classpath
+from pants.jvm.classpath import rules as classpath_rules
 from pants.jvm.goals import lockfile
 from pants.jvm.resolve.coursier_fetch import CoursierResolvedLockfile
 from pants.jvm.resolve.coursier_fetch import rules as coursier_fetch_rules
@@ -29,10 +36,12 @@ def rule_runner() -> RuleRunner:
             *coursier_fetch_rules(),
             *lockfile.rules(),
             *resources.rules(),
-            *classpath.rules(),
+            *classpath_rules(),
             *util_rules(),
             *testutil.rules(),
+            QueryRule(Classpath, (Addresses,)),
             QueryRule(RenderedClasspath, (Addresses,)),
+            QueryRule(DigestContents, (Digest,)),
         ],
         target_types=[
             ResourcesGeneratorTarget,
@@ -43,36 +52,53 @@ def rule_runner() -> RuleRunner:
     return rule_runner
 
 
+def filenames_from_zip(file_content: FileContent) -> list[str]:
+    z = ZipFile(BytesIO(file_content.content))
+    files = z.filelist
+    return [file_.filename for file_ in files]
+
+
 @maybe_skip_jdk_test
 def test_resources(rule_runner: RuleRunner) -> None:
     rule_runner.write_files(
         {
-            "BUILD": "resources(name='root', sources=['*.txt'])",
+            "BUILD": "resources(name='root', sources=['**/*.txt'])",
             "one.txt": "",
             "two.txt": "",
+            "three/four.txt": "",
+            "three/five.txt": "",
+            "three/six/seven/eight.txt": "",
             "3rdparty/jvm/default.lock": EMPTY_LOCKFILE,
         }
     )
 
     # Building the generator target should exclude the individual files and result in a single jar
     # for the generator.
-    rendered_classpath = rule_runner.request(
-        RenderedClasspath, [Addresses([Address(spec_path="", target_name="root")])]
+    classpath = rule_runner.request(
+        Classpath, [Addresses([Address(spec_path="", target_name="root")])]
     )
-    assert rendered_classpath.content == {
-        ".root.resources.jar": {
-            "one.txt",
-            "two.txt",
-        }
+
+    contents = rule_runner.request(DigestContents, list(classpath.digests()))
+    assert contents[0].path == ".root.resources.jar"
+    resources_filenames = set(filenames_from_zip(contents[0]))
+    expected = {
+        "one.txt",
+        "two.txt",
+        "three/",
+        "three/four.txt",
+        "three/five.txt",
+        "three/six/",
+        "three/six/seven/",
+        "three/six/seven/eight.txt",
     }
 
+    assert resources_filenames == expected
+
     # But requesting a single file should individually package it.
-    rendered_classpath = rule_runner.request(
-        RenderedClasspath,
+    classpath = rule_runner.request(
+        Classpath,
         [Addresses([Address(spec_path="", target_name="root", relative_file_path="one.txt")])],
     )
-    assert rendered_classpath.content == {
-        ".one.txt.root.resources.jar": {
-            "one.txt",
-        }
-    }
+    contents = rule_runner.request(DigestContents, list(classpath.digests()))
+    assert contents[0].path == ".one.txt.root.resources.jar"
+    assert filenames_from_zip(contents[0]) == ["one.txt"]