Repository: mesonbuild/meson

Issue:
MSVC build fails owing to overly long build directory names
So I'm building `gst-build` on Windows, with glib as a subproject. The build fails with the following error message:

`[1/1857] Compiling C object subprojects/glib/gio/tests/gdbus-object-manager-example/subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha/meson-generated_.._gdbus-example-objectmanager-generated.c.obj.
FAILED: subprojects/glib/gio/tests/gdbus-object-manager-example/subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha/meson-generated_.._gdbus-example-objectmanager-generated.c.obj
cl @subprojects/glib/gio/tests/gdbus-object-manager-example/subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha/meson-generated_.._gdbus-example-objectmanager-generated.c.obj.rsp
subprojects/glib/gio/tests/gdbus-object-manager-example/gdbus-example-objectmanager-generated.c: fatal error C1041: cannot open program database 'subprojects\\glib\\gio\\tests\\gdbus-object-manager-example\\subprojects@glib@gio@tests@gdbus-object-manager-example@@libgdbus-example-objectmanager@sha\\meson-generated_.._gdbus-example-objectmanager-generated.c.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS
`

Turns out the problem is the overly long build directory paths here, exceeding the default 260 path limit.

The build works fine if I use `c:\build` as build directory instead.

I understand why these paths might be needed in the flat layout case, but Meson knows that we don't use a flat layout here, so I think it shouldn't be using these long cluttery paths in this case.


Discussion:
cc @sarum9in 
We've run into the in mesa as well.
I thought we were already using response files for these but apparently we only do that for linker invocations.
I think response files are already being used (I remember being irritated by `ninja -v` not showing any details), but won't help. They help bypass the max command line length aiui.

It looks to me like the args make it fine to `cl.exe` but something somewhere then constructs an absolute path and that blows the default 260 char path limit. And Meson makes the paths much longer than they need to be in this case.
So the problem is #3246.
This is the code that generates the target id, which is the major part of the name. 
```python
        name_part = self.name.replace('/', '@').replace('\\', '@')
        assert not has_path_sep(self.type_suffix())
        myid = name_part + self.type_suffix()
        if self.subdir:
            subdir_part = self.subdir.replace('/', '@').replace('\\', '@')
            myid = subdir_part + '@@' + myid
```

It was shorter in the past:
```python
        base = self.name + self.type_suffix()
        if self.subproject == '':
            return base
        return self.subproject + '@@' + base
```

The problem with previous approach is that it did not allow same ids in different subdirectories. New approach takes this into account, but unfortunately it effectively doubles the name size.

I have a possible workaround in mind: let's hash subdir_part of id with some short stable hashing function (e.g. sha). So the code would effectively be
```python
        name_part = self.name.replace('/', '@').replace('\\', '@')
        assert not has_path_sep(self.type_suffix())
        myid = name_part + self.type_suffix()
        if self.subdir:
            subdir_part = self.subdir.replace('/', '@').replace('\\', '@')
            myid = SOME_SHA_FUNC(subdir_part) + '@@' + myid
```

This will not look as nice unfortunately but will solve the problem.
Another option is to check the size of the subdir_part and hash it only if it exceeds N (e.g. 100) symbols. Or similar but for the whole id, which will provide more consistent experience.

Thoughts?
I don't mind the hash - and for me the paths are quite unwieldy anyway, so I don't think there's too much lost from an aesthetic point of view - perhaps an abbreviated/short hash would be enough here? (8 chars)

On a related note, using a hash could be needed to uniquify target ID per install directory. See https://github.com/mesonbuild/meson/issues/4019#issuecomment-420773340.
@tp-m I will prepare a patch today/tomorrow and create a PR. Let's give it a try if it works for your builds and see if we get any other feedback regarding more obfuscated IDs.

Resolving patch:
diff --git a/mesonbuild/build.py b/mesonbuild/build.py
--- a/mesonbuild/build.py
+++ b/mesonbuild/build.py
@@ -15,6 +15,7 @@
 import copy, os, re
 from collections import OrderedDict
 import itertools, pathlib
+import hashlib
 import pickle
 from functools import lru_cache
 
@@ -363,19 +364,38 @@ def get_basename(self):
     def get_subdir(self):
         return self.subdir
 
-    def get_id(self):
+    @staticmethod
+    def _get_id_hash(target_id):
+        # We don't really need cryptographic security here.
+        # Small-digest hash function with unlikely collision is good enough.
+        h = hashlib.sha256()
+        h.update(target_id.encode(encoding='utf-8', errors='replace'))
+        # This ID should be case-insensitive and should work in Visual Studio,
+        # e.g. it should not start with leading '-'.
+        return h.hexdigest()[:7]
+
+    @staticmethod
+    def construct_id_from_path(subdir, name, type_suffix):
+        """Construct target ID from subdir, name and type suffix.
+
+        This helper function is made public mostly for tests."""
         # This ID must also be a valid file name on all OSs.
         # It should also avoid shell metacharacters for obvious
         # reasons. '@' is not used as often as '_' in source code names.
         # In case of collisions consider using checksums.
         # FIXME replace with assert when slash in names is prohibited
-        name_part = self.name.replace('/', '@').replace('\\', '@')
-        assert not has_path_sep(self.type_suffix())
-        myid = name_part + self.type_suffix()
-        if self.subdir:
-            subdir_part = self.subdir.replace('/', '@').replace('\\', '@')
-            myid = subdir_part + '@@' + myid
-        return myid
+        name_part = name.replace('/', '@').replace('\\', '@')
+        assert not has_path_sep(type_suffix)
+        my_id = name_part + type_suffix
+        if subdir:
+            subdir_part = Target._get_id_hash(subdir)
+            # preserve myid for better debuggability
+            return subdir_part + '@@' + my_id
+        return my_id
+
+    def get_id(self):
+        return self.construct_id_from_path(
+            self.subdir, self.name, self.type_suffix())
 
     def process_kwargs(self, kwargs):
         if 'build_by_default' in kwargs:
diff --git a/run_unittests.py b/run_unittests.py
--- a/run_unittests.py
+++ b/run_unittests.py
@@ -48,6 +48,7 @@
 from mesonbuild.environment import detect_ninja
 from mesonbuild.mesonlib import MesonException, EnvironmentException
 from mesonbuild.dependencies import PkgConfigDependency, ExternalProgram
+from mesonbuild.build import Target
 import mesonbuild.modules.pkgconfig
 
 from run_tests import exe_suffix, get_fake_env, get_meson_script
@@ -1633,7 +1634,8 @@ def test_internal_include_order(self):
         incs = [a for a in shlex.split(execmd) if a.startswith("-I")]
         self.assertEqual(len(incs), 9)
         # target private dir
-        self.assertPathEqual(incs[0], "-Isub4/sub4@@someexe@exe")
+        someexe_id = Target.construct_id_from_path("sub4", "someexe", "@exe")
+        self.assertPathEqual(incs[0], "-I" + os.path.join("sub4", someexe_id))
         # target build subdir
         self.assertPathEqual(incs[1], "-Isub4")
         # target source subdir
@@ -2897,6 +2899,18 @@ def test_reconfigure(self):
         self.build()
         self.run_tests()
 
+    def test_target_construct_id_from_path(self):
+        # This id is stable but not guessable.
+        # The test is supposed to prevent unintentional
+        # changes of target ID generation.
+        target_id = Target.construct_id_from_path('some/obscure/subdir',
+                                                  'target-id', '@suffix')
+        self.assertEqual('5e002d3@@target-id@suffix', target_id)
+        target_id = Target.construct_id_from_path('subproject/foo/subdir/bar',
+                                                  'target2-id', '@other')
+        self.assertEqual('81d46d1@@target2-id@other', target_id)
+
+
 class FailureTests(BasePlatformTests):
     '''
     Tests that test failure conditions. Build files here should be dynamically
@@ -3709,8 +3723,10 @@ def test_compiler_cpp_stds(self):
     def test_unity_subproj(self):
         testdir = os.path.join(self.common_test_dir, '46 subproject')
         self.init(testdir, extra_args='--unity=subprojects')
-        self.assertPathExists(os.path.join(self.builddir, 'subprojects/sublib/subprojects@sublib@@simpletest@exe/simpletest-unity.c'))
-        self.assertPathExists(os.path.join(self.builddir, 'subprojects/sublib/subprojects@sublib@@sublib@sha/sublib-unity.c'))
+        simpletest_id = Target.construct_id_from_path('subprojects/sublib', 'simpletest', '@exe')
+        self.assertPathExists(os.path.join(self.builddir, 'subprojects/sublib', simpletest_id, 'simpletest-unity.c'))
+        sublib_id = Target.construct_id_from_path('subprojects/sublib', 'sublib', '@sha')
+        self.assertPathExists(os.path.join(self.builddir, 'subprojects/sublib', sublib_id, 'sublib-unity.c'))
         self.assertPathDoesNotExist(os.path.join(self.builddir, 'user@exe/user-unity.c'))
         self.build()
 


Test patch: