Repository: jameslamb/pydistcheck (language: python)

Issue:
[feature request] support checking conda packages
### What change would you like to see?

This project currently only supports two formats:

* Python sdists (`*.tar.gz`)
* Python wheels (`*.whl`)

It should also support conda packages. Ideally all conda packages, but at least Python code packaged for conda.

### How would implementing this improve `pydistcheck`?

All the same benefits it brings for sdists and wheels. Would allow `pydistcheck` to be used to enforce all the desirable properties of packages built with e.g. `conda-build`, and to identify areas for improvement in portability of such packages.

### Notes

Related to #199.

See https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/packages.html for details, especially on the `.conda` format added in `conda` 4.7 ([May 2019](https://github.com/conda/conda/releases/tag/4.7.0)).

I don't think this project should take on support for pre-`conda 4.7` format `.bz2` conda packages. Or at least, that's not as high a priority.
add support for conda packages
Closes #200.

Adds support for checking `conda` packages with `pydistcheck` 🥳 

It supports both formats:

* `.tar.bz2` = the format `conda-build` has supported for most of its existence
* `.conda` = the format introduced in conda 4.7 (4ish years ago) that's gaining broader acceptance across different `conda` tools and channels

### An example

What's in the `pytorch` conda packages?

```shell
# install pydistcheck
pip install .

# download some pytorch conda packages
mkdir -p ./downloads

python ./bin/get-conda-release-files.py \
        'pytorch' \
        'pytorch' \
        ./downloads

# check them
pydistcheck --inspect ./downloads/*
```

Lots of interesting stuff to see!

For example, for `linux-64-pytorch-2.2.0-py3.10_cpu_0.tar.bz2`:

* it's 79.4MB compressed and 0.4GB uncompressed
* it contains 12,000+ files
* looks like it's bundling a copy of the cpython shared library (maybe?) and that that was compiled with debug symbols included

```text
==================== running pydistcheck ====================

checking './downloads/linux-64-pytorch-2.2.0-py3.10_cpu_0.tar.bz2'
----- package inspection summary -----
file size
  * compressed size: 79.4M
  * uncompressed size: 0.4G
  * compression space saving: 79.3%
contents
  * directories: 0
  * files: 12076 (11 compiled)
size by extension
  * .so - 298893.6K (76.1%)
  * .py - 25532.7K (6.5%)
  * .h - 25511.6K (6.5%)
  * .pyc - 19139.5K (4.9%)
  * .txt - 6748.1K (1.7%)
  * no-extension - 6236.5K (1.6%)
  * .0 - 5213.9K (1.3%)
  * .json - 3005.1K (0.8%)
  * .yaml - 742.5K (0.2%)
  * .pyi - 626.9K (0.2%)
  * .cuh - 600.7K (0.2%)
  * .cmake - 254.5K (0.1%)
  * .hpp - 241.5K (0.1%)
  * .cpp - 101.8K (0.0%)
  * .js - 18.8K (0.0%)
  * .mjs - 11.0K (0.0%)
  * .sh - 5.9K (0.0%)
  * .bat - 3.9K (0.0%)
  * .template - 2.9K (0.0%)
  * .cu - 0.5K (0.0%)
  * .ini - 0.5K (0.0%)
  * .html - 0.4K (0.0%)
  * .bzl - 0.3K (0.0%)
  * .in - 0.2K (0.0%)
  * .md - 0.1K (0.0%)
  * .bazel - 0.1K (0.0%)
  * .typed - 0.0K (0.0%)
largest files
  * (0.3G) lib/python3.10/site-packages/torch/lib/libtorch_cpu.so
  * (22.1M) lib/python3.10/site-packages/torch/lib/libtorch_python.so
  * (6.6M) lib/python3.10/site-packages/torch-2.2.0-py3.10.egg-info/SOURCES.txt
  * (5.1M) lib/python3.10/site-packages/torch/bin/protoc-3.13.0.0
  * (5.1M) lib/python3.10/site-packages/torch/bin/protoc
------------ check results -----------
1. [compiled-objects-have-debug-symbols] Found compiled object containing debug symbols. For details, extract the distribution contents and run 'objdump --all-headers "lib/python3.10/site-packages/torch/_C.cpython-310-x86_64-linux-gnu.so"'.
2. [distro-too-large-compressed] Compressed size 79.4M is larger than the allowed size (50.0M).
3. [distro-too-large-uncompressed] Uncompressed size 0.4G is larger than the allowed size (75.0M).
4. [too-many-files] Found 12076 files. Only 2000 allowed.
```

Hopefully this addition to `pydistcheck` will be helpful for project maintainers to detect unexpected package changes, and identify areas for improvement.

### References

conda:

* https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/packages.html#conda-file-format
* https://docs.conda.io/projects/conda/en/latest/release-notes.html
* https://docs.conda.io/projects/conda-build/en/stable/resources/package-spec.html
* https://github.com/conda/conda-docs/issues/796

working with zstd-compressed stuff in Python:

* https://stackoverflow.com/a/55260983/3986677
* https://github.com/indygreg/python-zstandard


Discussion / hints:
One thing that'll have to change..... right now the code that checks for debug symbols implicitly assumes it'll only be called on members of a zip file (because right now this project only supports sdists and wheels and skips debug-symbol checks for sdists).

https://github.com/jameslamb/pydistcheck/blob/779258c833dec41c30cedf3cb1eb010f72340abe/src/pydistcheck/shared_lib_utils.py#L72-L75


Fix patch:
diff --git a/.github/workflows/manual-run.yml b/.github/workflows/manual-run.yml
index c777a1c..8fa1729 100644
--- a/.github/workflows/manual-run.yml
+++ b/.github/workflows/manual-run.yml
@@ -30,7 +30,7 @@ jobs:
               pipx
           make install
           mkdir -p ./tmp-dir
-          python bin/get-release-files.py\
+          python bin/get-pypi-files.py\
             ${{ inputs.package_name }} \
             ./tmp-dir
           pydistcheck ./tmp-dir/*
diff --git a/bin/get-release-files.py b/bin/get-pypi-files.py
similarity index 100%
rename from bin/get-release-files.py
rename to bin/get-pypi-files.py


Test patch:
diff --git a/bin/run-smoke-tests.sh b/bin/run-smoke-tests.sh
index fd8a6cc..4fc5fd9 100755
--- a/bin/run-smoke-tests.sh
+++ b/bin/run-smoke-tests.sh
@@ -9,7 +9,7 @@ get-files() {
     rm -rf ./smoke-tests
     mkdir ./smoke-tests
     echo ""
-    python bin/get-release-files.py \
+    python bin/get-pypi-files.py \
         "${pkg_name}" \
         ./smoke-tests
 }
diff --git a/tests/test_cli.py b/tests/test_cli.py
index c40678c..15fcb34 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -7,11 +7,14 @@
 
 from pydistcheck.cli import check
 
-BASE_PACKAGES = ["base-package-0.1.0.tar.gz", "base-package-0.1.0.zip"]
+BASE_PACKAGES = [
+    "base-package-0.1.0.tar.gz",
+    "base-package-0.1.0.zip",
+]
 PROBLEMATIC_PACKAGES = ["problematic-package-0.1.0.tar.gz", "problematic-package-0.1.0.zip"]
 MACOS_SUFFIX = "macosx_10_15_x86_64.macosx_11_6_x86_64.macosx_12_5_x86_64.whl"
 MANYLINUX_SUFFIX = "manylinux_2_28_x86_64.manylinux_2_5_x86_64.manylinux1_x86_64.whl"
-BASE_WHEELS = [
+BASEBALL_PACKAGES = [
     f"baseballmetrics-0.1.0-py3-none-{MACOS_SUFFIX}",
     f"baseballmetrics-0.1.0-py3-none-{MANYLINUX_SUFFIX}",
 ]
@@ -39,7 +42,7 @@ def _assert_log_matches_pattern(result: Result, pattern: str, num_times: int = 1
     assert num_matches_found == num_times, msg
 
 
-@pytest.mark.parametrize("distro_file", BASE_PACKAGES + BASE_WHEELS)
+@pytest.mark.parametrize("distro_file", BASE_PACKAGES + BASEBALL_PACKAGES)
 def test_check_runs_without_error(distro_file):
     runner = CliRunner()
     result = runner.invoke(check, [os.path.join(TEST_DATA_DIR, distro_file)])