Repository: mesonbuild/meson

Issue:
Underscore prefix detection broken if -g specified
This is not an issue currently, due to #5481 hiding this bug but once that is solved, it will become an issue again.

If the cflags contain `-g`, the test for the underscore symbol would incorrectly return as soon as it finds the symbol with or without underscore, but when debug symbols are enabled using `-g`, this will yield the false result. This can easily be illustrated by a small testcase:

```python
symbol_name = b'meson_uscore_prefix'
with open('out.obj', 'rb') as o:
    for line in o:
        # Check if the underscore form of the symbol is somewhere
        # in the output file.
        if b'_' + symbol_name in line:
            print('True')
        # Else, check if the non-underscored form is present
        elif symbol_name in line:
            print('False')
```

Would print:

```
False
True
```

as it first finds the symbol without the underscore, only later finds the symbol with underscore.

Of course this is not an issue when using the meson buildtype option, as meson will not use the `-g` argument from that for the test, but is an issue when `-g` is contained in externally provided flags by env variables or crossfile.


Discussion:
@jpakkane Possible solutions for this:

1. Search the whole file and if both the underscore and non-underscore variant are found, assume symbols have underscore prefix. (This might still produce wrong results in some cases)

2. Have the compiler output assembly and check the assembly. (This is probably the most reliable we can do without using external tools like `nm`)

3. Use `nm` on the resulting binary and parse the output of `nm`. (This is probably the slowest of all approaches, and most complex one as we would need to check for `nm` or equivalent tool and parse the output)

Personally I am in favor of the second option.
Or possibly filter out `-g` from feature test arguments?
@jpakkane That might work, although not sure whats the easiest way to do that? Especially if any compiler would use multiple args for that.

Is there a way to filter part fo a list with another list in order, like:

`['foo', 'bar', 'baz', 'xyz', 'abc']` filter with `['baz', 'xyz']` to get `['foo', 'bar', 'abc']`
```python
filtered = [x for x in original_list if x not in things_to_take_out]
```

That won't respect order though, so could lead to wrong, confusing, results:

```
>>> original_list = ['-x', 'foo', '-x', 'debug']
>>> filter_list = ['-x', 'debug']
>>> [x for x in original_list if x not in filter_list]
['foo']
```
@jpakkane So the filtering approach is not really that feasible given that there are other flags than `-g` that can enable debug symbols. So probably it needs a more complex check that is more reliable to debug symbols presence.
I just ran into this when updating from meson 0.51.0 and after 976c30360370354cdfd3222e6fb9e2e595b2a4c8 I had to strip `-g` from our cross file flags. Any updates to solving this?
`-g0` negates `-g` and handles ordering just fine. Can we know ahead of time which potential arguments need to be removed and can they all be negated?
I've also had to drop `-gdwarf-2` as that broke the build also
`flto` in CFLAGS also causes the problem. And @ePirat 's option 2 won't fix it.
@xry111 Why would 2. not work here?
@ePirat with `-flto` the outputed "assembly" contains strings like `.gnu.lto_foo.0` so `_foo` will be found.
Similar to the problem with GCC's `-flto` flag reported by @xry111, `symbols_have_underscore_prefix()` also returns True incorrectly if Clang's `-fprofile-instr-generate` flag is in `CFLAGS`. I think this is because the object file generated by the compiler contains strings like `__profc_foo`.

The following commands reproduce a linker error caused by this bug when building the dav1d library with Clang's `-fprofile-instr-generate` flag in `CFLAGS` on Linux:

```
$ git clone -b 0.9.2 --depth 1 https://code.videolan.org/videolan/dav1d.git
$ cd dav1d
$ mkdir build
$ cd build
$ export CC=clang
$ export CFLAGS=-fprofile-instr-generate
$ meson ..
$ ninja
```

You will see a lot of "undefined reference" linker errors. The problem is that the following code in dav1d/meson.build malfunctions and incorrectly adds an underscore prefix to symbols defined in assembly language files:

```
if cc.symbols_have_underscore_prefix()
    cdata.set10('PREFIX', true)
    cdata_asm.set10('PREFIX', true)
endif
```
upstream dav1d will move away from meson's `cc.symbols_have_underscore_prefix()`. See https://code.videolan.org/videolan/dav1d/-/merge_requests/1296
We need it only for external asm. The  systems with underscore symbol prefix are well known and have to be supported explicitly in the form of asm code anyway. Adding an additional condition to the OS/architecture list when asm is added for a new system should not be a problem. I don't expect that there will be new OS/architecture combinations which use underscore symbol prefixes.
As per discussion on IRC / Matrix with @ePirat, underscore detection will be moved to a hardcoded list based on toolchain and platform, and we will move the current broken compile check to a unit test that compares the compile check against the hardcoded list.

We could use things like `nm` or manually parsing object files, but that is actually harder to maintain than this system. Especially if new object formats / OSes / toolchains need to be added.
@ePirat do you have a draft branch / PR to implement the hard-coded list of underscore + toolchain + platform combination? We hit this in gstreamer again recently: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/978
@nirbheek Doing that is really tricky given the amount of platforms meson already supports, but I found a much better solution, will submit the PR for this probably tomorrow evening.

Resolving patch:
diff --git a/mesonbuild/compilers/mixins/clike.py b/mesonbuild/compilers/mixins/clike.py
--- a/mesonbuild/compilers/mixins/clike.py
+++ b/mesonbuild/compilers/mixins/clike.py
@@ -900,9 +900,10 @@ def has_type(self, typename: str, prefix: str, env: 'Environment',
         return self.compiles(t, env, extra_args=extra_args,
                              dependencies=dependencies)
 
-    def symbols_have_underscore_prefix(self, env: 'Environment') -> bool:
+    def _symbols_have_underscore_prefix_searchbin(self, env: 'Environment') -> bool:
         '''
-        Check if the compiler prefixes an underscore to global C symbols
+        Check if symbols have underscore prefix by compiling a small test binary
+        and then searching the binary for the string,
         '''
         symbol_name = b'meson_uscore_prefix'
         code = '''#ifdef __cplusplus
@@ -914,10 +915,10 @@ def symbols_have_underscore_prefix(self, env: 'Environment') -> bool:
         #endif
         '''
         args = self.get_compiler_check_args(CompileCheckMode.COMPILE)
-        n = 'symbols_have_underscore_prefix'
+        n = '_symbols_have_underscore_prefix_searchbin'
         with self._build_wrapper(code, env, extra_args=args, mode='compile', want_output=True, temp_dir=env.scratch_dir) as p:
             if p.returncode != 0:
-                raise RuntimeError(f'BUG: Unable to compile {n!r} check: {p.stdout}')
+                raise RuntimeError(f'BUG: Unable to compile {n!r} check: {p.stderr}')
             if not os.path.isfile(p.output_name):
                 raise RuntimeError(f'BUG: Can\'t find compiled test code for {n!r} check')
             with open(p.output_name, 'rb') as o:
@@ -925,13 +926,79 @@ def symbols_have_underscore_prefix(self, env: 'Environment') -> bool:
                     # Check if the underscore form of the symbol is somewhere
                     # in the output file.
                     if b'_' + symbol_name in line:
-                        mlog.debug("Symbols have underscore prefix: YES")
+                        mlog.debug("Underscore prefix check found prefixed function in binary")
                         return True
                     # Else, check if the non-underscored form is present
                     elif symbol_name in line:
-                        mlog.debug("Symbols have underscore prefix: NO")
+                        mlog.debug("Underscore prefix check found non-prefixed function in binary")
                         return False
-        raise RuntimeError(f'BUG: {n!r} check failed unexpectedly')
+        raise RuntimeError(f'BUG: {n!r} check did not find symbol string in binary')
+
+    def _symbols_have_underscore_prefix_define(self, env: 'Environment') -> T.Optional[bool]:
+        '''
+        Check if symbols have underscore prefix by querying the
+        __USER_LABEL_PREFIX__ define that most compilers provide
+        for this. Return if functions have underscore prefix or None
+        if it was not possible to determine, like when the compiler
+        does not set the define or the define has an unexpected value.
+        '''
+        delim = '"MESON_HAVE_UNDERSCORE_DELIMITER" '
+        code = f'''
+        #ifndef __USER_LABEL_PREFIX__
+        #define MESON_UNDERSCORE_PREFIX unsupported
+        #else
+        #define MESON_UNDERSCORE_PREFIX __USER_LABEL_PREFIX__
+        #endif
+        {delim}MESON_UNDERSCORE_PREFIX
+        '''
+        with self._build_wrapper(code, env, mode='preprocess', want_output=False, temp_dir=env.scratch_dir) as p:
+            if p.returncode != 0:
+                raise RuntimeError(f'BUG: Unable to preprocess _symbols_have_underscore_prefix_define check: {p.stdout}')
+            symbol_prefix = p.stdout.partition(delim)[-1].rstrip()
+
+            mlog.debug(f'Queried compiler for function prefix: __USER_LABEL_PREFIX__ is "{symbol_prefix!s}"')
+            if symbol_prefix == '_':
+                return True
+            elif symbol_prefix == '':
+                return False
+            else:
+                return None
+
+    def _symbols_have_underscore_prefix_list(self, env: 'Environment') -> T.Optional[bool]:
+        '''
+        Check if symbols have underscore prefix by consulting a hardcoded
+        list of cases where we know the results.
+        Return if functions have underscore prefix or None if unknown.
+        '''
+        m = env.machines[self.for_machine]
+        # Darwin always uses the underscore prefix, not matter what
+        if m.is_darwin():
+            return True
+        # Windows uses the underscore prefix on x86 (32bit) only
+        if m.is_windows() or m.is_cygwin():
+            return m.cpu_family == 'x86'
+        return None
+
+
+    def symbols_have_underscore_prefix(self, env: 'Environment') -> bool:
+        '''
+        Check if the compiler prefixes an underscore to global C symbols
+        '''
+        # First, try to query the compiler directly
+        result = self._symbols_have_underscore_prefix_define(env)
+        if result is not None:
+            return result
+
+        # Else, try to consult a hardcoded list of cases we know
+        # absolutely have an underscore prefix
+        result = self._symbols_have_underscore_prefix_list(env)
+        if result is not None:
+            return result
+
+        # As a last resort, try search in a compiled binary, which is the
+        # most unreliable way of checking this, see #5482
+        return self._symbols_have_underscore_prefix_searchbin(env)
+
 
     def _get_patterns(self, env: 'Environment', prefixes: T.List[str], suffixes: T.List[str], shared: bool = False) -> T.List[str]:
         patterns = []  # type: T.List[str]
diff --git a/mesonbuild/compilers/mixins/visualstudio.py b/mesonbuild/compilers/mixins/visualstudio.py
--- a/mesonbuild/compilers/mixins/visualstudio.py
+++ b/mesonbuild/compilers/mixins/visualstudio.py
@@ -383,6 +383,23 @@ def has_func_attribute(self, name: str, env: 'Environment') -> T.Tuple[bool, boo
     def get_argument_syntax(self) -> str:
         return 'msvc'
 
+    def symbols_have_underscore_prefix(self, env: 'Environment') -> bool:
+        '''
+        Check if the compiler prefixes an underscore to global C symbols.
+
+        This overrides the Clike method, as for MSVC checking the
+        underscore prefix based on the compiler define never works,
+        so do not even try.
+        '''
+        # Try to consult a hardcoded list of cases we know
+        # absolutely have an underscore prefix
+        result = self._symbols_have_underscore_prefix_list(env)
+        if result is not None:
+            return result
+
+        # As a last resort, try search in a compiled binary
+        return self._symbols_have_underscore_prefix_searchbin(env)
+
 
 class MSVCCompiler(VisualStudioLikeCompiler):
 
diff --git a/unittests/allplatformstests.py b/unittests/allplatformstests.py
--- a/unittests/allplatformstests.py
+++ b/unittests/allplatformstests.py
@@ -60,7 +60,7 @@
 from mesonbuild.wrap.wrap import PackageDefinition, WrapException
 
 from run_tests import (
-    Backend, exe_suffix, get_fake_env
+    Backend, exe_suffix, get_fake_env, get_convincing_fake_env_and_cc
 )
 
 from .baseplatformtests import BasePlatformTests
@@ -1546,6 +1546,32 @@ def test_prebuilt_shared_lib_rpath(self) -> None:
             self.build()
             self.run_tests()
 
+    def test_underscore_prefix_detection_list(self) -> None:
+        '''
+        Test the underscore detection hardcoded lookup list
+        against what was detected in the binary.
+        '''
+        env, cc = get_convincing_fake_env_and_cc(self.builddir, self.prefix)
+        expected_uscore = cc._symbols_have_underscore_prefix_searchbin(env)
+        list_uscore = cc._symbols_have_underscore_prefix_list(env)
+        if list_uscore is not None:
+            self.assertEqual(list_uscore, expected_uscore)
+        else:
+            raise SkipTest('No match in underscore prefix list for this platform.')
+
+    def test_underscore_prefix_detection_define(self) -> None:
+        '''
+        Test the underscore detection based on compiler-defined preprocessor macro
+        against what was detected in the binary.
+        '''
+        env, cc = get_convincing_fake_env_and_cc(self.builddir, self.prefix)
+        expected_uscore = cc._symbols_have_underscore_prefix_searchbin(env)
+        define_uscore = cc._symbols_have_underscore_prefix_define(env)
+        if define_uscore is not None:
+            self.assertEqual(define_uscore, expected_uscore)
+        else:
+            raise SkipTest('Did not find the underscore prefix define __USER_LABEL_PREFIX__')
+
     @skipIfNoPkgconfig
     def test_pkgconfig_static(self):
         '''


Test patch:
diff --git a/run_tests.py b/run_tests.py
--- a/run_tests.py
+++ b/run_tests.py
@@ -40,7 +40,7 @@
 from mesonbuild import mesonmain
 from mesonbuild import mtest
 from mesonbuild import mlog
-from mesonbuild.environment import Environment, detect_ninja
+from mesonbuild.environment import Environment, detect_ninja, detect_machine_info
 from mesonbuild.coredata import backendlist, version as meson_version
 from mesonbuild.mesonlib import OptionKey, setup_vsenv
 
@@ -153,6 +153,17 @@ def get_fake_env(sdir='', bdir=None, prefix='', opts=None):
     env.machines.host.cpu_family = 'x86_64' # Used on macOS inside find_library
     return env
 
+def get_convincing_fake_env_and_cc(bdir, prefix):
+    '''
+    Return a fake env and C compiler with the fake env
+    machine info properly detected using that compiler.
+    Useful for running compiler checks in the unit tests.
+    '''
+    env = get_fake_env('', bdir, prefix)
+    cc = compilers.detect_c_compiler(env, mesonlib.MachineChoice.HOST)
+    # Detect machine info
+    env.machines.host = detect_machine_info({'c':cc})
+    return (env, cc)
 
 Backend = Enum('Backend', 'ninja vs xcode')