Repository: mesonbuild/meson

Issue:
compile_commands.json is useless on Windows due to rsp files
The `compile_commands.json` is needed for tooling like clang-tidy or a language server like [cquery](https://github.com/cquery-project/cquery). But on Windows this file isn't very useful as the command line looks like this:

```json
"command": "c++ @foo.cpp.obj.rsp",
```

The unit tests know this and already extract the real command lines: https://github.com/mesonbuild/meson/blob/dfd368d4054bfe067545112717a5bd21bc37aca7/run_unittests.py#L583
But this is also needed for tooling, as there's no standard way for e.g. cquery to know that the `compile_commands.json` came from Meson+Ninja and to extract the real command line flags using a dry run.

I've thought of three possible solutions:

## 1. Let Meson fix up compile_commands.json after Ninja generated it

This basically means moving the stuff from run_unittests.py into: https://github.com/mesonbuild/meson/blob/6cc1849b8f7528fc713405f5f6b0cdebd810c088/mesonbuild/backend/ninjabackend.py#L211
I've tried this and this is what I've got so far:

```python
    def generate_compdb(self):
        pch_compilers = ['%s_PCH' % i for i in self.build.compilers]
        native_compilers = ['%s_COMPILER' % i for i in self.build.compilers]
        cross_compilers = ['%s_CROSS_COMPILER' % i for i in self.build.cross_compilers]
        ninja_compdb = [self.ninja_command, '-t', 'compdb'] + pch_compilers + native_compilers + cross_compilers
        builddir = self.environment.get_build_dir()
        jsondb = subprocess.check_output(ninja_compdb, cwd=builddir)
        contents = json.loads(jsondb)
        # If Ninja is using .rsp files, generate them, read their contents, and
        # replace it as the command for all compile commands in the parsed json.
        if len(contents) > 0 and contents[0]['command'].endswith('.rsp'):
            # Pretend to build so that the rsp files are generated
            subprocess.check_call([self.ninja_command, '-d', 'keeprsp', '-n'] + pch_compilers + native_compilers + cross_compilers)
            for each in contents:
                # Extract the actual command from the rsp file
                compiler, rsp = each['command'].split(' @')
                rsp = os.path.join(builddir, rsp)
                # Replace the command with its contents
                with open(rsp, 'r', encoding='utf-8') as f:
                    each['command'] = compiler + ' ' + f.read()
            jsondb = json.dumps(contents)
        with open(os.path.join(builddir, 'compile_commands.json'), 'wb') as f:
            f.write(jsondb)
```

Unfortunately running ninja again at this stage causes it to only run `[0/1] Regenerating build files.` :( Maybe there's a way to fix this?

## 2. Fix it in Ninja itself

There's actually a PR for this already! https://github.com/ninja-build/ninja/pull/1223
Unfortunately I don't think it will be merged any time soon. Ninja looks a bit unmaintained to be honest :(

## 3. Use something else instead of Ninja to generate compile_commands.json

This is what CMake does IIRC. I've created a PR for Shake, a Ninja-compatible build tool which can also generate `compile_commands.json`: https://github.com/ndmitchell/shake/issues/562


Discussion:
It really is a bit of a crappy situation, yes. One workaround for people reading this is to use `ninja -d keeprsp` which will keep them around, and you can then read the rsp file in whatever tool is parsing compile_commands.json.
Ninja has merged https://github.com/ninja-build/ninja/pull/1223 now with an option for generating the `compile_commands.json` with full commands, what needs to be done in Meson to use this?
I think we'll have to wait for a new Ninja release (probably 1.9.0) for the PR to be included. Then we can use the ninja version to see if we can add the `-x` flag.
Ninja 1.9.0 has been released https://github.com/ninja-build/ninja/releases/tag/v1.9.0

Resolving patch:
diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -285,9 +285,10 @@ def detect_prefix(out):
 
     def generate(self, interp):
         self.interpreter = interp
-        self.ninja_command = environment.detect_ninja(log=True)
-        if self.ninja_command is None:
+        ninja = environment.detect_ninja_command_and_version(log=True)
+        if ninja is None:
             raise MesonException('Could not detect Ninja v1.5 or newer')
+        (self.ninja_command, self.ninja_version) = ninja
         outfilename = os.path.join(self.environment.get_build_dir(), self.ninja_filename)
         tempfilename = outfilename + '~'
         with open(tempfilename, 'w', encoding='utf-8') as outfile:
@@ -342,7 +343,8 @@ def generate_compdb(self):
             for lang in self.environment.coredata.compilers[for_machine]:
                 rules += [self.get_compiler_rule_name(lang, for_machine)]
                 rules += [self.get_pch_rule_name(lang, for_machine)]
-        ninja_compdb = [self.ninja_command, '-t', 'compdb'] + rules
+        compdb_options = ['-x'] if mesonlib.version_compare(self.ninja_version, '>=1.9') else []
+        ninja_compdb = [self.ninja_command, '-t', 'compdb'] + compdb_options + rules
         builddir = self.environment.get_build_dir()
         try:
             jsondb = subprocess.check_output(ninja_compdb, cwd=builddir)
diff --git a/mesonbuild/environment.py b/mesonbuild/environment.py
--- a/mesonbuild/environment.py
+++ b/mesonbuild/environment.py
@@ -141,6 +141,10 @@ def find_coverage_tools():
     return gcovr_exe, gcovr_new_rootdir, lcov_exe, genhtml_exe
 
 def detect_ninja(version: str = '1.5', log: bool = False) -> str:
+    r = detect_ninja_command_and_version(version, log)
+    return r[0] if r else None
+
+def detect_ninja_command_and_version(version: str = '1.5', log: bool = False) -> (str, str):
     env_ninja = os.environ.get('NINJA', None)
     for n in [env_ninja] if env_ninja else ['ninja', 'ninja-build', 'samu']:
         try:
@@ -162,7 +166,7 @@ def detect_ninja(version: str = '1.5', log: bool = False) -> str:
                 if name == 'samu':
                     name = 'samurai'
                 mlog.log('Found {}-{} at {}'.format(name, found, quote_arg(n)))
-            return n
+            return (n, found)
 
 def get_llvm_tool_names(tool: str) -> typing.List[str]:
     # Ordered list of possible suffixes of LLVM executables to try. Start with


Test patch: