Repository: conda/conda-build

Issue:
Support for (collected) license files via directories, globs, list files
### Actual Behavior

Currently, we can specify either single files or a list of files for the `about: license_file:` field.


In some language ecosystems like Rust or Go it is common to vendor dependencies.
The license files of these dependencies are then included in the source distributions but often/usually there is no readily composed list of them available. Hence, it would make sense to use helper scripts during the build to gather a list of those license files and use that list as an input for the `about: license_file:` field.

### Expected Behavior

Allow one or more of the following:
1. Allow entries in `license_file` to point to directories and recursive include all files in them as "license files".
    (Appending a suffix `/` could be made a requirement to make this behavior opt-in.)
2. Allow (recursive) globs in `license_file` entries to be given instead of only static file names.
3. Allow some additional syntax to recognize entries in `license_file` not as "license files" directly, but just as list files.
   (e.g., have a `gather_licenses` script create a `${SRC_DIR}/thirdparty-licenses` file in which every line references the actual license files and have `license_file: <some new syntax>thirdparty-licenses` expand to the file list.)


All of the options above bare the possibility to accidentally include additional non-license files. But I believe it can be rather subjective to judge if one or the other is more prone to it. Hence, I think all of them more or less equally good and I would probably go with the one that is easiest to use and document\*, i.e., option 1.

(\* usage instruction could be simple as "move thirdparty license files to a `$SRC_DIR/additional-licenses` folder and add to `license_file` an entry `additional-licenses/` (ending with `/` to denote  a directory for which all files are included recursively)".)


Discussion:
This feature request is only for the `license_file` field and not the `license` field. One could argue that it would be nice to parse all these additional licenses and compose a SPDX license expression (or a similar summary) from them. That, however, would be an elaborate undertaking and should be a separate feature request -- and, frankly, composing such expressions/summary should be done in a separate tool, IMO.

Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -768,12 +768,19 @@ def copy_license(m):
         if license_file == "":
             continue
         src_file = join(m.config.work_dir, license_file)
-        if not os.path.isfile(src_file):
+        if not os.path.isfile(src_file) and not os.path.isdir(src_file):
             src_file = os.path.join(m.path, license_file)
-        if os.path.isfile(src_file):
+        if os.path.isdir(src_file) and not src_file.endswith("/"):
+            raise ValueError(
+                "License entry in about/license_file ({}) points to a directory but does not "
+                "end with a '/'. Make sure the directory only contains license files and "
+                "append a '/' to include the folder and all of its content as license "
+                "information.".format(src_file)
+            )
+        if os.path.isfile(src_file) or os.path.isdir(src_file):
             # Rename absolute file paths or relative file paths starting with .. or .
             if os.path.isabs(license_file) or license_file.startswith("."):
-                filename = "LICENSE{}.txt".format(count)
+                filename = "LICENSE{}{}".format(count, ".txt" if os.path.isfile(src_file) else "")
                 count += 1
             else:
                 filename = license_file


Test patch:
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/license-dir-from-recipe/first-license.txt b/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/license-dir-from-recipe/first-license.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/license-dir-from-recipe/first-license.txt
@@ -0,0 +1 @@
+first license from recipe
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/license-dir-from-recipe/second-license.txt b/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/license-dir-from-recipe/second-license.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/license-dir-from-recipe/second-license.txt
@@ -0,0 +1 @@
+second license from recipe
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/meta.yaml b/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/dir-no-slash-suffix/meta.yaml
@@ -0,0 +1,11 @@
+package:
+  name: test_about_license_file_dir-no-slash-suffix
+  version: 1.0
+
+source:
+  path: ../../source
+
+about:
+  license_file:
+    - license-dir-from-source/
+    - license-dir-from-recipe
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/dir/license-dir-from-recipe/first-license.txt b/tests/test-recipes/metadata/_about_license_file/recipes/dir/license-dir-from-recipe/first-license.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/dir/license-dir-from-recipe/first-license.txt
@@ -0,0 +1 @@
+first license from recipe
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/dir/license-dir-from-recipe/second-license.txt b/tests/test-recipes/metadata/_about_license_file/recipes/dir/license-dir-from-recipe/second-license.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/dir/license-dir-from-recipe/second-license.txt
@@ -0,0 +1 @@
+second license from recipe
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/dir/meta.yaml b/tests/test-recipes/metadata/_about_license_file/recipes/dir/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/dir/meta.yaml
@@ -0,0 +1,11 @@
+package:
+  name: test_about_license_file_dir
+  version: 1.0
+
+source:
+  path: ../../source
+
+about:
+  license_file:
+    - license-dir-from-source/
+    - license-dir-from-recipe/
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/list/license-from-recipe.txt b/tests/test-recipes/metadata/_about_license_file/recipes/list/license-from-recipe.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/list/license-from-recipe.txt
@@ -0,0 +1 @@
+license from recipe
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/list/meta.yaml b/tests/test-recipes/metadata/_about_license_file/recipes/list/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/list/meta.yaml
@@ -0,0 +1,11 @@
+package:
+  name: test_about_license_file_list
+  version: 1.0
+
+source:
+  path: ../../source
+
+about:
+  license_file:
+    - license-from-source.txt
+    - license-from-recipe.txt
diff --git a/tests/test-recipes/metadata/_about_license_file/recipes/single/meta.yaml b/tests/test-recipes/metadata/_about_license_file/recipes/single/meta.yaml
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/recipes/single/meta.yaml
@@ -0,0 +1,9 @@
+package:
+  name: test_about_license_file_single
+  version: 1.0
+
+source:
+  path: ../../source
+
+about:
+  license_file: license-from-source.txt
diff --git a/tests/test-recipes/metadata/_about_license_file/source/license-dir-from-source/first-license.txt b/tests/test-recipes/metadata/_about_license_file/source/license-dir-from-source/first-license.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/source/license-dir-from-source/first-license.txt
@@ -0,0 +1 @@
+first license from source
diff --git a/tests/test-recipes/metadata/_about_license_file/source/license-dir-from-source/second-license.txt b/tests/test-recipes/metadata/_about_license_file/source/license-dir-from-source/second-license.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/source/license-dir-from-source/second-license.txt
@@ -0,0 +1 @@
+second license from source
diff --git a/tests/test-recipes/metadata/_about_license_file/source/license-from-source.txt b/tests/test-recipes/metadata/_about_license_file/source/license-from-source.txt
new file mode 100644
--- /dev/null
+++ b/tests/test-recipes/metadata/_about_license_file/source/license-from-source.txt
@@ -0,0 +1 @@
+license from source
diff --git a/tests/test_api_build.py b/tests/test_api_build.py
--- a/tests/test_api_build.py
+++ b/tests/test_api_build.py
@@ -728,6 +728,31 @@ def test_about_json_content(testing_metadata):
     assert 'root_pkgs' in about and about['root_pkgs']
 
 
+def test_about_license_file(testing_workdir, testing_config):
+    base_dir = os.path.join(metadata_dir, "_about_license_file/recipes")
+
+    recipe = os.path.join(base_dir, "single")
+    outputs = api.build(recipe, config=testing_config)
+    assert package_has_file(outputs[0], "info/licenses/license-from-source.txt")
+
+    recipe = os.path.join(base_dir, "list")
+    outputs = api.build(recipe, config=testing_config)
+    assert package_has_file(outputs[0], "info/licenses/license-from-source.txt")
+    assert package_has_file(outputs[0], "info/licenses/license-from-recipe.txt")
+
+    recipe = os.path.join(base_dir, "dir")
+    outputs = api.build(recipe, config=testing_config)
+    assert package_has_file(outputs[0], "info/licenses/license-dir-from-source/first-license.txt")
+    assert package_has_file(outputs[0], "info/licenses/license-dir-from-source/second-license.txt")
+    assert package_has_file(outputs[0], "info/licenses/license-dir-from-recipe/first-license.txt")
+    assert package_has_file(outputs[0], "info/licenses/license-dir-from-recipe/second-license.txt")
+
+    recipe = os.path.join(base_dir, "dir-no-slash-suffix")
+    assert os.path.isdir(recipe)
+    with pytest.raises(ValueError, match=r"license_file.*license-dir-from-recipe.*directory"):
+        api.build(recipe, config=testing_config)
+
+
 @pytest.mark.slow
 @pytest.mark.xfail(parse_version(conda.__version__) < parse_version("4.3.14"),
                    reason="new noarch supported starting with conda 4.3.14")