Repository: conan-io/conan

Issue:
[question] Keying packages on libstdc++ version? Autodetecting it?
### What is your question?

Hi! I'm trying to push packages to a conan artifactory instance and reuse them in compatible environments.
To that end I added keying for the packages through `settings_user.yml`.
Specifically, I added keying on (distro, distro_version). However I found that's not enough since we have easybuild-based environments which provide libstdc++ versions that are newer than the distro's, and then packages produced in that environment cannot be used on the vanilla distro itself because they demand newer symbols from libstdc++.

I thought I might further key the package on a `compiler.libcxx.version` setting field I would add, and was wondering if that makes sense and what's the best practice here. 

I was also wondering if I could somehow extend the APIs available at profile templating time in order to be able to call some code to autodetect this property, instead of doing it outside and passing it in via an environment variable. WDYT?

Lastly, I was wondering if more things (like the `distro` module and maybe also `subprocess`) could be made available during template formatting to make this keying easier?

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
Hi @nextsilicon-itay-bookstein  - thanks for reaching out.

Typically when the compiler is `gcc`, `compiler.version` should effectively "track" the version of libstdc++ (as it goes hand in hand with a particular major version of GCC, if I'm not mistaken).  Is this not the case?   

of course, it's a different story for `glibc`, or if the compiler is `clang` whilst using `libstdc++`
I ended up relying on these two articles:
* https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html
* https://maskray.me/blog/2023-06-25-c++-standard-library-abi-compatibility

Because the property I'm trying to capture is "abi compat on dynamic linking", I did the following:
* Added a property `abiversion` under each libcxx variant I support (in `settings_user.yml`)
* For libstdc++ I extracted the full suffix of `libstdc++.so` (e.g. "6.0.30")
* For libc++ I extracted `_LIBCPP_ABI_VERSION` + `_LIBCPP_VERSION` (e.g. "1.17.0.6")
> https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

Looking at this - 
```
GCC 10.1.0: libstdc++.so.6.0.28

GCC 11.1.0: libstdc++.so.6.0.29

GCC 12.1.0: libstdc++.so.6.0.30

GCC 13.1.0: libstdc++.so.6.0.31

GCC 13.2.0: libstdc++.so.6.0.32

GCC 14.1.0: libstdc++.so.6.0.33
```

It seems that versions of libstdc++ go hand in hand with the compiler version, up to the minor component of the version.

Hence I'm surprised - I would have imagined that settings like the following:

```
compiler=gcc
compiler.version=13.2
```

would already be enough to capture the binary compatibility around libstdc++ -  especially considering that I'm unaware of  getting  g++ to use a different version of libstdc++ than the one that corresponds to the same version ! so there will always be a 1:1 mapping 

We're using clang with libstdc++ 🙂
Also I haven't checked what happens when you use a newer gcc on a system whose systemwide libstdc++ version is older. Maybe it always picks its own libstdc++ and then you're right for gcc. But as a general rule using a compiler that isn't the system compiler opens these cans of worms.
As some extra hint, note that Conan 2 has the ``profile.py`` plugin, that can be used to check, complete (we use it for completing ``runtime_type`` based on ``build_type`` for MSVC), so it might also be used for injecting some functionality. I agree that it seems that profile templates would be a slightly better place, though it is true that too heavy system detection in profiles like using ``subprocess`` sounds too much, and might be an indicator that pre-defined profiles (maybe with ``profile.py`` checks) could be easier than auto-detecting profiles.
Hi @nextsilicon-itay-bookstein 

Following up on this after a while, sorry for the delays.

It seems that a possible approach to deal with different libstdc++ versions that are not implicitly modeled by any other settings, would be:

- Define the custom sub-setting ``compiler.libcxx.version`` in the ``settings_user.yml``
- In the profile, use the ``detect_api`` to get the ``gcc`` version:
  ```
  {% set compiler, version, _ = detect_api.detect_gcc_compiler() %}
  ```
- In the profile, use that ``version`` to map to the respective libcxx version, something like (not tested):
  ```
  {% set libcxx_version = {"10.1": "6.0.28", "11.1": "6.0.29", "12.1": "6.0.30", ...}.get(str(version)) %}
  ```
- Assign it to the subsetting:
  ```
  compiler.libcxx.version = {{ libcxx_version }}
  ```

Please let us know if this helps, and if there is any further question or comment. Thanks for your feedback!
We ended up doing something along these lines, but a bit more flexible. We run an external script (through subprocess.check_output) which autodetects the version of either libstdc++ or libc++ by invoking the compiler. It can then also encode abi properties of libc++. This prevents sharing of incompatible binaries between distros / compilers - better under-share than over-share.
I leave it to you to consider whether this should be an area of upstream improvement rather than downstream user customization, but for now the downstream customization addresses our needs 🙂

Also we found ability to call external functionality when rendering the profile templates very useful, even though the method of retrieving subprocess.check_output is very decidedly a hack.

We use it both to detect the stdlib version and to locate address sanitizer's shared library, which is needed for shared-libasan on our Conan deps.
> We ended up doing something along these lines, but a bit more flexible. We run an external script (through subprocess.check_output) which autodetects the version of either libstdc++ or libc++ by invoking the compiler. It can then also encode abi properties of libc++. 

Thanks for the feedback, sounds good

> This prevents sharing of incompatible binaries between distros / compilers - better under-share than over-share.

It depends on many things, yes, for environments fully controlled, this might have advantages, for environments such sharing in public, like ConanCenter, this is extremely unpractical

> I leave it to you to consider whether this should be an area of upstream improvement rather than downstream user customization, but for now the downstream customization addresses our needs 

You might have now more experience than us, regarding this. Do you think that is it possible a general and robust ``detect_api.detect_libcxx_version`` or something in that line that would be useful for other users, without requiring high maintenance (like requiring different implementations in different compiler in different platforms, etc)? Would you like to contribute something to be built-in in Conan?

> Also we found ability to call external functionality when rendering the profile templates very useful, even though the method of retrieving subprocess.check_output is very decidedly a hack.

Why a hack? This is mostly what the ``detect_api`` does under the hood.
>
> Why a hack? This is mostly what the detect_api does under the hood.

The biggest hack right now is in the method by which we retrieve `subprocess.check_output`:
```
{% set check_output = os.sys.modules["builtins"].__import__("subprocess").check_output %}
```
So it might be useful to expose something cleaner from the namespace dict passed to the jinja2 engine.

>
> It depends on many things, yes, for environments fully controlled, this might have advantages, for environments such sharing in public, like ConanCenter, this is extremely unpractical

I also wanted to add that in some environments we use a Foreign Package Manager called Lmod (which is based on easybuild), and it creates situations wherein you have an alternative (newer) libstdc++ on some distro. That's why, for us, keying merely on distro version (which usually also implies some system-provided version of libstdc++) wasn't enough.

>
> You might have now more experience than us, regarding this. Do you think that is it possible a general and robust `detect_api.detect_libcxx_version` or something in that line that would be useful for other users, without requiring high maintenance (like requiring different implementations in different compiler in different platforms, etc)? Would you like to contribute something to be built-in in Conan?

So, I think I might as well first show how our customizations look like, and maybe then we can discuss what parts of this make sense to upstream. This is a snippet of our `settings_user.yml`, used for keying on distro version and on libcxx ABI:
```
os:
    Linux:
        # Values determined according to values in distro os-release files. See:
        # https://docs.python.org/3/library/platform.html#platform.freedesktop_os_release
        distro:
            debian:
                version: ["10", "11", "12"]
            rocky:
                version: ["8.0", "8.1", "8.2", "8.3", "8.4", "8.5", "8.6", "8.7", "8.8", "8.9", "8.10",
                          "9.0", "9.1", "9.2", "9.3", "9.4", "9.5"]
            rhel:
                version: ["8.0", "8.1", "8.2", "8.3", "8.4", "8.5", "8.6", "8.7", "8.8", "8.9", "8.10",
                          "9.0", "9.1", "9.2", "9.3", "9.4", "9.5"]
            ubuntu:
                version: ["20.04", 20.10", "22.04", "22.10", "24.04", "24.10"]
compiler:
    clang:
        libcxx:
            # The versions here are meant to capture ABI compatibility / symbol
            # versioning so that libraries built against one version of the
            # stdlib and happen to use new symbols that are not present in
            # older versions can be keyed separately.
            libstdc++11:
                # See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
                # We capture the SO filename suffix.
                abiversion: &libstdcxx11_version
                            ["6.0.25", "6.0.26", "6.0.27", "6.0.28", "6.0.29", "6.0.30",
                             "6.0.31", "6.0.32", "6.0.33"]
            libc++:
                # See https://maskray.me/blog/2023-06-25-c++-standard-library-abi-compatibility.
                # We capture the ABI version together with the LLVM version.
                abiversion: &libcxx_version
                            ["1.16.0.6", "1.17.0.6"]
    gcc:
        libcxx:
            libstdc++11:
                abiversion: *libstdcxx11_version
```

This is our `detect_libcxx_abiversion.py` script:
```
#!/usr/bin/env python3
import os
import pathlib
import subprocess
import sys
import traceback
from typing import Any, Optional

_LIBSTDCXX_SO = "libstdc++.so"


def _eprint(*args: Any, **kwargs: Any) -> None:
    print(*args, file=sys.stderr, **kwargs)


def _invoke_proc(args: list[str], check: bool = True) -> str:
    return subprocess.run(
        args,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        check=check,
        encoding="utf-8",
    ).stdout


def _is_elf(path: pathlib.Path) -> bool:
    with path.open("rb") as f:
        return f.read(4) == b"\x7fELF"


def _find_libstdcxx_path(cc: str, cxx_exe: str) -> pathlib.Path:
    assert cc in (
        "gcc",
        "clang",
    ), "Only gcc and clang are currently supported for libstdc++ detection"
    args = [cxx_exe, "-Wl,--verbose"]
    if cc == "clang":
        args.append("-stdlib=libstdc++")
    stdout = _invoke_proc(args=args, check=False)
    for line in stdout.splitlines():
        if line.startswith("/") and " " not in line:
            path = pathlib.Path(line)
        elif line.startswith("ld.lld: /"):
            path = pathlib.Path(line[8:])
        elif line.startswith("-lstdc++ (") and line.endswith(")"):
            path = pathlib.Path(line[10:-1])
        else:
            continue

        if path.name.startswith(_LIBSTDCXX_SO) and _is_elf(path):
            return path

    raise FileNotFoundError("libstdc++.so not found")


def _detect_libstdcxx_abiversion(cc: str, cxx_exe: str) -> str:
    path = _find_libstdcxx_path(cc, cxx_exe).resolve()
    abiversion = path.name[len(_LIBSTDCXX_SO) + 1 :]
    _eprint(f"libstdc++.so found at '{path}' with abiversion '{abiversion}'")
    split = abiversion.split(".")
    assert len(split) == 3 and all(
        part.isdigit() for part in split
    ), f"Invalid abiversion: {abiversion}"
    return abiversion


def _detect_libcxx_abiversion(cc: str, cxx_exe: str) -> str:
    assert cc == "clang", "Only clang is currently supported for libc++ detection"

    # It might be even more robust to compute and retrieve _LIBCPP_ODR_SIGNATURE,
    # but _LIBCPP_ABI_VERSION + _LIBCPP_VERSION will suffice for now.
    defines_str = _invoke_proc(
        [
            cxx_exe,
            "-xc++",
            "-std=c++11",
            "-stdlib=libc++",
            "-Wno-everything",
            "-E",
            "-dM",
            "-include",
            "ciso646",
            os.devnull,
        ]
    )
    defines: dict[str, Optional[str]] = {}
    for line in defines_str.splitlines():
        start = "#define "
        assert line.startswith(start), f"Unexpected line: {line}"
        stripped = line[len(start) :]
        split = stripped.split(" ", 1)
        key = split[0]
        value = split[1] if len(split) == 2 else None
        defines[key] = value

    abi = defines["_LIBCPP_ABI_VERSION"]
    assert abi is not None, "_LIBCPP_ABI_VERSION not found"
    version = defines["_LIBCPP_VERSION"]
    assert version is not None and len(version) == 6, "_LIBCPP_VERSION not found"
    return f"{int(abi)}.{int(version[:2])}.{int(version[2:4])}.{int(version[4:])}"


def _detect_libcxx_abiversion_gcc_clang(
    cc: str, cxx_exe: str, compiler_libcxx: str
) -> str:
    match compiler_libcxx:
        case "libstdc++11":
            return _detect_libstdcxx_abiversion(cc, cxx_exe)

        case "libc++":
            return _detect_libcxx_abiversion(cc, cxx_exe)

        case _:
            raise ValueError(f"Unsupported libcxx for gcc/clang {compiler_libcxx!r}")


def main(args: list[str]) -> int:
    try:
        if len(args) != 3:
            _eprint(
                f"Usage: {os.path.basename(__file__)} <cc> <cxx_exe> <compiler_libcxx>"
            )
            return 1

        # The interface with this script is STDOUT.
        cc, cxx_exe, compiler_libcxx = args
        match cc:
            case "gcc" | "clang":
                print(_detect_libcxx_abiversion_gcc_clang(cc, cxx_exe, compiler_libcxx))
                return 0

            case _:
                _eprint(f"Unsupported compiler {cc!r}")
                return 1

    except Exception:
        _eprint(f"Error detecting libcxx abiversion: {traceback.format_exc()}")
        return 2


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
```

And this is how we call it from a profile template:
```
{% set check_output = os.sys.modules["builtins"].__import__("subprocess").check_output %}
{% set cc, cc_ver, cc_exe = detect_api.detect_default_compiler() %}
{% set cxx_exe = os.getenv("CXX", "c++") %}
{% set compiler_version = detect_api.default_compiler_version(cc, cc_ver) %}
{% set compiler_libcxx = detect_api.detect_libcxx(cc, cc_ver, cc_exe) %}
{% set detect_libcxx_abiversion = os.path.realpath(os.path.join(profile_dir, "..", "detect_libcxx_abiversion.py")) %}
{% set compiler_libcxx_abiversion = check_output([detect_libcxx_abiversion, cc, cxx_exe, compiler_libcxx]).decode("utf-8").strip() %}
```

I left in-band comments with links to documentation about the way I picked the strings that will represent the ABI version for each stdlib (libstdc++, libc++), but they are notably different in substance. Writing the above script was a bit finicky, as there were different behaviors with respect to some toolchain distributions, like inadvertently finding symlinks to `.la` files (which is why you see the ELF check above). Notably this is very ELF/Linux-centric, and I don't know how it will generalize to MachO and to MSVC/MSVCRuntime.
> {% set check_output = os.sys.modules["builtins"].__import__("subprocess").check_output %}

Oh, yes, that is right. We might consider exposing either ``subprocess`` or a way to run things and capture the output.

Thanks very much for sharing the detection code.
At first sight, it looks like a bit too much to make it built-in (I was hoping it was a bit simpler such detection), as maintenance can be relatively too high cost, but I'll check with the team.
I am doing PR https://github.com/conan-io/conan/pull/18244 that will automatically close this ticket, adding ``subprocess`` to the jinja context rendering.

I have been talking with the team, it seems the above detection of such libcxx version is a bit too specific/niche, and it might imply some extra future maintenance beyond the value for the larger community, so at the moment we are not considering to make it built-in in the ``detect_api``. Fortunately, as you have already seen, it is relatively straightforward from the Conan point of view, using the jinja syntax, to apply your own detection scripts logic. The complexity is just in knowing what/how to detect what you want, that is, the implementation of the ``detect_libcxx_abiversion.py`` file itself, and maintaining that within an organization is typically an order of magnitude simpler than trying to maintain that for the global community where the variety of compilers and toolchains is very large.

Thanks very much for all your feedback!
I would have guessed as much. Thank you for taking the time to evaluate this and for the followups!
Finally closed by https://github.com/conan-io/conan/pull/18244 for next 2.17.



Patch:
diff --git a/conan/internal/api/profile/profile_loader.py b/conan/internal/api/profile/profile_loader.py
index b0f67af1dc8..b9265b8c294 100644
--- a/conan/internal/api/profile/profile_loader.py
+++ b/conan/internal/api/profile/profile_loader.py
@@ -1,5 +1,6 @@
 import os
 import platform
+import subprocess
 from collections import OrderedDict, defaultdict
 
 from jinja2 import Environment, FileSystemLoader
@@ -118,6 +119,7 @@ def _load_profile(self, profile_name, cwd):
         file_path = os.path.basename(profile_path)
         context = {"platform": platform,
                    "os": os,
+                   "subprocess": subprocess,
                    "profile_dir": base_path,
                    "profile_name": file_path,
                    "conan_version": conan_version,


Test patch:
diff --git a/test/integration/configuration/test_profile_jinja.py b/test/integration/configuration/test_profile_jinja.py
index d8af563a135..ac3ca66af38 100644
--- a/test/integration/configuration/test_profile_jinja.py
+++ b/test/integration/configuration/test_profile_jinja.py
@@ -10,7 +10,9 @@
 
 def test_profile_template():
     client = TestClient()
-    tpl = textwrap.dedent("""
+    tpl = textwrap.dedent("""\
+        # This is just to check that subprocess is injected in the render and can be used
+        {% set check_output = subprocess.check_output %}
         [settings]
         os = {{ {"Darwin": "Macos"}.get(platform.system(), platform.system()) }}
         build_type = {{ os.getenv("MY_BUILD_TYPE") }}