Repository: mesonbuild/meson

Issue:
cmake: subproject with dependencies do not build
**Describe the bug**
With CMake this project builds fine. 
But when I try to use it as Meson subproject it crashes on Meson side.

```
|Executing subproject jayatana method cmake 
|
|Found CMake: /usr/bin/cmake (3.15.5)
|
||Configuring the build directory with CMake version 3.15.5
||Running: /usr/bin/cmake -DCMAKE_C_COMPILER=cc -G Ninja -DCMAKE_INSTALL_PREFIX=/usr/local --trace --trace-expand
||
||Running with trace output on.
||Running with expanded trace output on.
||-- Could NOT find VCM (missing: VCM_DIR)
||-- Configuring done
||-- Generating done
||-- Build files have been written to: /home/rilian/Projects/vala-panel-appmenu/build/subprojects/jayatana/__CMake_build
|
|CMake configuration: SUCCEEDED
|CMake server handshake: SUCCEEDED
|CMake server configure: SUCCEEDED
|Generating build system files: SUCCEEDED
|Querying build system files: SUCCEEDED
|Querying the CMake code model: SUCCEEDED
|WARNING: CMake: path /usr/lib/jvm/default/include;/usr/lib/jvm/default/include/linux;/usr/lib/jvm/default/include does not exist. Ignoring. This can lead to build errors
|CMake project jayatana has 9 build targets.
|
||Processing generated meson AST
||Build file: /home/rilian/Projects/vala-panel-appmenu/build/subprojects/jayatana/meson.build
||
||Project name: jayatana
||Project version: undefined
||C compiler for the host machine: cc (gcc 9.2.0 "cc (GCC) 9.2.0")
||C linker for the host machine: GNU ld.bfd 2.33.1
||Program /usr/lib/python3.8/site-packages/mesonbuild/cmake/data/run_ctgt.py found: YES (/usr/lib/python3.8/site-packages/mesonbuild/cmake/data/run_ctgt.py)

subprojects/jayatana/meson.build:0:0: ERROR: Unknown variable "java-headers".

```

**To Reproduce**
Build [this project](https://gitlab.com/vala-panel-project/vala-panel-appmenu) with meson

**Expected behavior**
It should builds fine as in CMake alone.

**system parameters**
* Native build
* ArchLinux
* Python 3.8.0
* meson 0.52.0
* ninja 1.9.0



Discussion:
Since there is an `add_custom_target` somewhere in the relevant `create_javah` function, this probably depends on #5989. However, even with that PR, I am still getting an error.

The current state of CMake subprojects is, that we want to support *most* CMake projects, however, there will always be cases where this won't be possible. That being said, projects like this should be supported (especially with #5989 being on the way), but, unfortunately, this won't happen before meson 0.53.0.
@mensinda We use `create_javah` only on old CMake, but on new we use jar api.

Error is same?
Yes, the error is the same, but you are right, create_javah is not called. Guess it's time for me to figure out what's going wrong then.
OK, with #6213, this sort of works now, but meson still won't be able to detect the correct internal dependency structure. At least #5989 for `add_dependency()` support will be required to do this.

However, I can compile your project if I manually force a compilation order for ninja:
```
ninja -C build subprojects/jayatana/java_class_filelist
ninja -C build subprojects/jayatana/c0_java_class_filelist
ninja -C build # All the other stuff
```

It would be nice if you could test #6213 (with the ninja workaround) on your system if you have the time.
I will try it on weekend

Resolving patch:
diff --git a/mesonbuild/cmake/data/run_ctgt.py b/mesonbuild/cmake/data/run_ctgt.py
--- a/mesonbuild/cmake/data/run_ctgt.py
+++ b/mesonbuild/cmake/data/run_ctgt.py
@@ -41,6 +41,7 @@
         continue
 
     try:
+        os.makedirs(args.directory, exist_ok=True)
         subprocess.run(i, cwd=args.directory, check=True)
     except subprocess.CalledProcessError:
         exit(1)
diff --git a/mesonbuild/cmake/interpreter.py b/mesonbuild/cmake/interpreter.py
--- a/mesonbuild/cmake/interpreter.py
+++ b/mesonbuild/cmake/interpreter.py
@@ -128,16 +128,78 @@
 
 transfer_dependencies_from = ['header_only']
 
-# Utility functions to generate local keys
-def _target_key(tgt_name: str) -> str:
-    return '__tgt_{}__'.format(tgt_name)
+class OutputTargetMap:
+    rm_so_version = re.compile(r'(\.[0-9]+)+$')
+
+    def __init__(self, build_dir: str):
+        self.tgt_map = {}
+        self.build_dir = build_dir
+
+    def add(self, tgt: Union['ConverterTarget', 'ConverterCustomTarget']) -> None:
+        def assign_keys(keys: List[str]) -> None:
+            for i in [x for x in keys if x]:
+                self.tgt_map[i] = tgt
+        keys = [self._target_key(tgt.cmake_name)]
+        if isinstance(tgt, ConverterTarget):
+            keys += [tgt.full_name]
+            keys += [self._rel_artifact_key(x) for x in tgt.artifacts]
+            keys += [self._base_artifact_key(x) for x in tgt.artifacts]
+        if isinstance(tgt, ConverterCustomTarget):
+            keys += [self._rel_generated_file_key(x) for x in tgt.original_outputs]
+            keys += [self._base_generated_file_key(x) for x in tgt.original_outputs]
+        assign_keys(keys)
+
+    def _return_first_valid_key(self, keys: List[str]) -> Optional[Union['ConverterTarget', 'ConverterCustomTarget']]:
+        for i in keys:
+            if i and i in self.tgt_map:
+                return self.tgt_map[i]
+        return None
+
+    def target(self, name: str) -> Optional[Union['ConverterTarget', 'ConverterCustomTarget']]:
+        return self._return_first_valid_key([self._target_key(name)])
+
+    def artifact(self, name: str) -> Optional[Union['ConverterTarget', 'ConverterCustomTarget']]:
+        keys = []
+        candidates = [name, OutputTargetMap.rm_so_version.sub('', name)]
+        for i in lib_suffixes:
+            if not name.endswith('.' + i):
+                continue
+            new_name = name[:-len(i) - 1]
+            new_name = OutputTargetMap.rm_so_version.sub('', new_name)
+            candidates += ['{}.{}'.format(new_name, i)]
+        for i in candidates:
+            keys += [self._rel_artifact_key(i), os.path.basename(i), self._base_artifact_key(i)]
+        return self._return_first_valid_key(keys)
+
+    def generated(self, name: str) -> Optional[Union['ConverterTarget', 'ConverterCustomTarget']]:
+        return self._return_first_valid_key([self._rel_generated_file_key(name), self._base_generated_file_key(name)])
+
+    # Utility functions to generate local keys
+    def _rel_path(self, fname: str) -> Optional[str]:
+        fname = os.path.normpath(os.path.join(self.build_dir, fname))
+        if os.path.commonpath([self.build_dir, fname]) != self.build_dir:
+            return None
+        return os.path.relpath(fname, self.build_dir)
+
+    def _target_key(self, tgt_name: str) -> str:
+        return '__tgt_{}__'.format(tgt_name)
+
+    def _rel_generated_file_key(self, fname: str) -> Optional[str]:
+        path = self._rel_path(fname)
+        return '__relgen_{}__'.format(path) if path else None
+
+    def _base_generated_file_key(self, fname: str) -> str:
+        return '__gen_{}__'.format(os.path.basename(fname))
+
+    def _rel_artifact_key(self, fname: str) -> Optional[str]:
+        path = self._rel_path(fname)
+        return '__relart_{}__'.format(path) if path else None
 
-def _generated_file_key(fname: str) -> str:
-    return '__gen_{}__'.format(os.path.basename(fname))
+    def _base_artifact_key(self, fname: str) -> str:
+        return '__art_{}__'.format(os.path.basename(fname))
 
 class ConverterTarget:
     lang_cmake_to_meson = {val.lower(): key for key, val in language_map.items()}
-    rm_so_version = re.compile(r'(\.[0-9]+)+$')
 
     def __init__(self, target: CMakeTarget, env: Environment):
         self.env = env
@@ -204,7 +266,7 @@ def __repr__(self) -> str:
 
     std_regex = re.compile(r'([-]{1,2}std=|/std:v?|[-]{1,2}std:)(.*)')
 
-    def postprocess(self, output_target_map: dict, root_src_dir: str, subdir: str, install_prefix: str, trace: CMakeTraceParser) -> None:
+    def postprocess(self, output_target_map: OutputTargetMap, root_src_dir: str, subdir: str, install_prefix: str, trace: CMakeTraceParser) -> None:
         # Detect setting the C and C++ standard
         for i in ['c', 'cpp']:
             if i not in self.compile_opts:
@@ -242,29 +304,13 @@ def postprocess(self, output_target_map: dict, root_src_dir: str, subdir: str, i
         elif self.type.upper() not in ['EXECUTABLE', 'OBJECT_LIBRARY']:
             mlog.warning('CMake: Target', mlog.bold(self.cmake_name), 'not found in CMake trace. This can lead to build errors')
 
-        # Fix link libraries
-        def try_resolve_link_with(path: str) -> Optional[str]:
-            basename = os.path.basename(path)
-            candidates = [basename, ConverterTarget.rm_so_version.sub('', basename)]
-            for i in lib_suffixes:
-                if not basename.endswith('.' + i):
-                    continue
-                new_basename = basename[:-len(i) - 1]
-                new_basename = ConverterTarget.rm_so_version.sub('', new_basename)
-                new_basename = '{}.{}'.format(new_basename, i)
-                candidates += [new_basename]
-            for i in candidates:
-                if i in output_target_map:
-                    return output_target_map[i]
-            return None
-
         temp = []
         for i in self.link_libraries:
             # Let meson handle this arcane magic
             if ',-rpath,' in i:
                 continue
             if not os.path.isabs(i):
-                link_with = try_resolve_link_with(i)
+                link_with = output_target_map.artifact(i)
                 if link_with:
                     self.link_with += [link_with]
                     continue
@@ -297,9 +343,8 @@ def rel_path(x: str, is_header: bool, is_generated: bool) -> Optional[str]:
             return x
 
         def custom_target(x: str):
-            key = _generated_file_key(x)
-            if key in output_target_map:
-                ctgt = output_target_map[key]
+            ctgt = output_target_map.generated(x)
+            if ctgt:
                 assert(isinstance(ctgt, ConverterCustomTarget))
                 ref = ctgt.get_ref(x)
                 assert(isinstance(ref, CustomTargetReference) and ref.valid())
@@ -343,7 +388,7 @@ def check_flag(flag: str) -> bool:
 
         # Handle explicit CMake add_dependency() calls
         for i in self.depends_raw:
-            tgt = output_target_map.get(_target_key(i))
+            tgt = output_target_map.target(i)
             if tgt:
                 self.depends.append(tgt)
 
@@ -451,7 +496,7 @@ def __init__(self, target: CMakeGeneratorTarget):
     def __repr__(self) -> str:
         return '<{}: {} {}>'.format(self.__class__.__name__, self.name, self.outputs)
 
-    def postprocess(self, output_target_map: dict, root_src_dir: str, subdir: str, build_dir: str, all_outputs: List[str]) -> None:
+    def postprocess(self, output_target_map: OutputTargetMap, root_src_dir: str, subdir: str, build_dir: str, all_outputs: List[str]) -> None:
         # Default the working directory to the CMake build dir. This
         # is not 100% correct, since it should be the value of
         # ${CMAKE_CURRENT_BINARY_DIR} when add_custom_command is
@@ -498,11 +543,8 @@ def ensure_absolute(x: str):
             for j in i:
                 if not j:
                     continue
-                target_key = _target_key(j)
-                if target_key in output_target_map:
-                    cmd += [output_target_map[target_key]]
-                else:
-                    cmd += [j]
+                target = output_target_map.target(j)
+                cmd += [target] if target else [j]
 
             commands += [cmd]
         self.command = commands
@@ -516,15 +558,16 @@ def ensure_absolute(x: str):
         for i in self.depends_raw:
             if not i:
                 continue
-            tgt_key = _target_key(i)
-            gen_key = _generated_file_key(i)
-
-            if os.path.basename(i) in output_target_map:
-                self.depends += [output_target_map[os.path.basename(i)]]
-            elif tgt_key in output_target_map:
-                self.depends += [output_target_map[tgt_key]]
-            elif gen_key in output_target_map:
-                self.inputs += [output_target_map[gen_key].get_ref(i)]
+            art = output_target_map.artifact(i)
+            tgt = output_target_map.target(i)
+            gen = output_target_map.generated(i)
+
+            if art:
+                self.depends += [art]
+            elif tgt:
+                self.depends += [tgt]
+            elif gen:
+                self.inputs += [gen.get_ref(i)]
             elif not os.path.isabs(i) and os.path.exists(os.path.join(root_src_dir, i)):
                 self.inputs += [i]
             elif os.path.isabs(i) and os.path.exists(i) and os.path.commonpath([i, root_src_dir]) == root_src_dir:
@@ -544,10 +587,11 @@ def process_inter_target_dependencies(self):
         self.depends = list(set(new_deps))
 
     def get_ref(self, fname: str) -> Optional[CustomTargetReference]:
+        fname = os.path.basename(fname)
         try:
             if fname in self.conflict_map:
                 fname = self.conflict_map[fname]
-            idx = self.outputs.index(os.path.basename(fname))
+            idx = self.outputs.index(fname)
             return CustomTargetReference(self, idx)
         except ValueError:
             return None
@@ -592,6 +636,7 @@ def __init__(self, build: 'Build', subdir: str, src_dir: str, install_prefix: st
         self.targets = []
         self.custom_targets = []  # type: List[ConverterCustomTarget]
         self.trace = CMakeTraceParser()
+        self.output_target_map = OutputTargetMap(self.build_dir)
 
         # Generated meson data
         self.generated_targets = {}
@@ -770,29 +815,16 @@ def analyse(self) -> None:
             self.custom_targets += [ConverterCustomTarget(i)]
 
         # generate the output_target_map
-        output_target_map = {}
-        output_target_map.update({x.full_name: x for x in self.targets})
-        output_target_map.update({_target_key(x.cmake_name): x for x in self.targets})
-        for i in self.targets:
-            for j in i.artifacts:
-                output_target_map[os.path.basename(j)] = i
-        for i in self.custom_targets:
-            output_target_map[_target_key(i.cmake_name)] = i
-            for j in i.original_outputs:
-                output_target_map[_generated_file_key(j)] = i
-        object_libs = []
-
-        # Sometimes an empty string can be inserted (no full name, etc.)
-        # Delete the entry in this case
-        if '' in output_target_map:
-            del output_target_map['']
+        for i in [*self.targets, *self.custom_targets]:
+            self.output_target_map.add(i)
 
         # First pass: Basic target cleanup
+        object_libs = []
         custom_target_outputs = []  # type: List[str]
         for i in self.custom_targets:
-            i.postprocess(output_target_map, self.src_dir, self.subdir, self.build_dir, custom_target_outputs)
+            i.postprocess(self.output_target_map, self.src_dir, self.subdir, self.build_dir, custom_target_outputs)
         for i in self.targets:
-            i.postprocess(output_target_map, self.src_dir, self.subdir, self.install_prefix, self.trace)
+            i.postprocess(self.output_target_map, self.src_dir, self.subdir, self.install_prefix, self.trace)
             if i.type == 'OBJECT_LIBRARY':
                 object_libs += [i]
             self.languages += [x for x in i.languages if x not in self.languages]
@@ -806,12 +838,10 @@ def analyse(self) -> None:
             i.process_inter_target_dependencies()
         for i in self.custom_targets:
             i.process_inter_target_dependencies()
-            i.log()
 
         # Fourth pass: Remove rassigned dependencies
         for i in self.targets:
             i.cleanup_dependencies()
-            i.log()
 
         mlog.log('CMake project', mlog.bold(self.project_name), 'has', mlog.bold(str(len(self.targets) + len(self.custom_targets))), 'build targets.')
 


Test patch:
diff --git a/test cases/cmake/8 custom command/subprojects/cmMod/CMakeLists.txt b/test cases/cmake/8 custom command/subprojects/cmMod/CMakeLists.txt
--- a/test cases/cmake/8 custom command/subprojects/cmMod/CMakeLists.txt	
+++ b/test cases/cmake/8 custom command/subprojects/cmMod/CMakeLists.txt	
@@ -10,6 +10,7 @@ add_definitions("-DDO_NOTHING_JUST_A_FLAG=1")
 add_executable(gen main.cpp)
 add_executable(mycpy cp.cpp)
 
+# cpyBase
 add_custom_command(
   OUTPUT  "${CMAKE_CURRENT_BINARY_DIR}/genTest.cpp" "${CMAKE_CURRENT_BINARY_DIR}/genTest.hpp"
   COMMAND gen ARGS genTest
@@ -42,7 +43,53 @@ add_custom_command(
   DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/cpyBase.hpp.something"
 )
 
-add_library(cmModLib SHARED cmMod.cpp genTest.cpp cpyBase.cpp cpyBase.hpp)
+# cpyNext (out of order is on purpose)
+# -- first copy round
+add_custom_command(
+  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/s1_a_hpp/file.txt"
+  COMMAND mycpy "${CMAKE_CURRENT_SOURCE_DIR}/cpyNext.hpp.am" file.txt
+  DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/cpyNext.hpp.am"
+  WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/s1_a_hpp"
+)
+
+add_custom_command(
+  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/s1_b_cpp/file.txt"
+  COMMAND mycpy "${CMAKE_CURRENT_SOURCE_DIR}/cpyNext.cpp.am" file.txt
+  DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/cpyNext.cpp.am"
+  WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/s1_b_cpp"
+)
+
+# -- final cpy round
+add_custom_command(
+  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/cpyNext.hpp"
+  COMMAND mycpy "${CMAKE_CURRENT_BINARY_DIR}/s2_b_hpp/file.txt" cpyNext.hpp
+  DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/s2_b_hpp/file.txt"
+  WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
+)
+
+add_custom_command(
+  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/cpyNext.cpp"
+  COMMAND mycpy "${CMAKE_CURRENT_BINARY_DIR}/s2_a_cpp/file.txt" cpyNext.cpp
+  DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/s2_a_cpp/file.txt"
+  WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
+)
+
+# -- second copy round
+add_custom_command(
+  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/s2_b_hpp/file.txt"
+  COMMAND mycpy "${CMAKE_CURRENT_BINARY_DIR}/s1_a_hpp/file.txt" file.txt
+  DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/s1_a_hpp/file.txt"
+  WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/s2_b_hpp"
+)
+
+add_custom_command(
+  OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/s2_a_cpp/file.txt"
+  COMMAND mycpy "${CMAKE_CURRENT_BINARY_DIR}/s1_b_cpp/file.txt" file.txt
+  DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/s1_b_cpp/file.txt"
+  WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/s2_a_cpp"
+)
+
+add_library(cmModLib SHARED cmMod.cpp genTest.cpp cpyBase.cpp cpyBase.hpp cpyNext.cpp cpyNext.hpp)
 include(GenerateExportHeader)
 generate_export_header(cmModLib)
 
diff --git a/test cases/cmake/8 custom command/subprojects/cmMod/cmMod.cpp b/test cases/cmake/8 custom command/subprojects/cmMod/cmMod.cpp
--- a/test cases/cmake/8 custom command/subprojects/cmMod/cmMod.cpp	
+++ b/test cases/cmake/8 custom command/subprojects/cmMod/cmMod.cpp	
@@ -1,6 +1,7 @@
 #include "cmMod.hpp"
 #include "genTest.hpp"
 #include "cpyBase.hpp"
+#include "cpyNext.hpp"
 #include "cmModLib.hpp"
 
 #ifndef FOO
@@ -18,5 +19,5 @@ string cmModClass::getStr() const {
 }
 
 string cmModClass::getOther() const {
-  return getStr() + "  --  " + getStrCpy();
+  return "Srings:\n - " + getStrCpy() + "\n - " + getStrNext();
 }
diff --git a/test cases/cmake/8 custom command/subprojects/cmMod/cp.cpp b/test cases/cmake/8 custom command/subprojects/cmMod/cp.cpp
--- a/test cases/cmake/8 custom command/subprojects/cmMod/cp.cpp	
+++ b/test cases/cmake/8 custom command/subprojects/cmMod/cp.cpp	
@@ -12,6 +12,11 @@ int main(int argc, char *argv[]) {
   ifstream src(argv[1]);
   ofstream dst(argv[2]);
 
+  if(!src.is_open()) {
+    cerr << "Failed to open " << argv[1] << endl;
+    return 2;
+  }
+
   dst << src.rdbuf();
   return 0;
 }
diff --git a/test cases/cmake/8 custom command/subprojects/cmMod/cpyNext.cpp.am b/test cases/cmake/8 custom command/subprojects/cmMod/cpyNext.cpp.am
new file mode 100644
--- /dev/null
+++ b/test cases/cmake/8 custom command/subprojects/cmMod/cpyNext.cpp.am	
@@ -0,0 +1,5 @@
+#include "cpyNext.hpp"
+
+std::string getStrNext() {
+  return "Hello Copied File -- now even more convoluted!";
+}
diff --git a/test cases/cmake/8 custom command/subprojects/cmMod/cpyNext.hpp.am b/test cases/cmake/8 custom command/subprojects/cmMod/cpyNext.hpp.am
new file mode 100644
--- /dev/null
+++ b/test cases/cmake/8 custom command/subprojects/cmMod/cpyNext.hpp.am	
@@ -0,0 +1,5 @@
+#pragma once
+
+#include <string>
+
+std::string getStrNext();