Repository: mesonbuild/meson

Issue:
Purge persistent Ninja state when doing a full reconfigure.



Discussion:
This seems like a terrible idea, because this persistent state is highly valuable and desirable. When a build is reconfigured but the command lines for many (but not all) outputs haven't changed as a result, ninja can tell that they don't need to be rebuilt because their currently built versions are identical to what would have been built.

If the command lines do change, then ninja compares the log lines that produced the output, detects that the reconfigure has resulted in changed build commands, and reruns the build edge.

The same basic logic occurs with regard to the deps file, except that those record known depfile states and ensure more reliable ordering based on the last good build, and are automatically recalculated and resaved the next time that build edge is run and outputs a depfile.
> This seems like a terrible idea, because this persistent state is highly valuable and desirable.

On the other hand I wrote this patch _exactly_ because this behaviour shot me in the leg quite badly and sent me on a multi-hour debugging session.  And in hindsight it did not even require deletion, just changing the build setup like so:

- have a shared library with no versioning info, it creates `foobar.so`
- add versioning info, the output is now `foobar.0.so` (or something)
- Meson creates the symlink `foobar.so -> foobar.0.so`
- Now on reconfigure we purge targets that used to be created but no longer are
- Ninja sees that `foobar.so` exists, used to be a build target but no longer is, so it promptly deletes it

The only reliable and simple way to do this is to delete stale data. This can get especially difficult if you first build one project and then delete the build dir and reuse it to build another. No, people should not be doing that. Yes, it will happen and we should guard against it.
This sounds like a job for `meson configure --clearcache`.
@jpakkane is there any way meson and/or ninja could detect that situation and do the right thing? I don't fully understand the consequence of removing that state, but would be sad to make rebuild after reconfigure slower for everyone just to catch one corner case. But I understand doing the build right is more important than doing it fast. Any chance we can have both?
> but would be sad to make rebuild after reconfigure slower for everyone just to catch one corner case.

No. It is only run on the first configuration. Subsequent runs does not attempt to delete the files. For them the cost is one `if` condition.

> But I understand doing the build right is more important than doing it fast. Any chance we can have both?

The "correct" fix that I can see is to make Ninja remove the files it deletes on `cleandead`  from its internal deps files. The fix is not perfect but at least it would be transient as opposed to permanent like it is now.
Oh, you do that only on first invocation indeed. But then how could you have stale ninja files, that requires manually messing with the build dir, right? In that case maybe we should error out and suggest using --wipe? Because if user manually messed, god knows what else could go wrong, better be safe and wipe, no?
The steps I needed to get hit by this are listed above. Don't know if that is common, but it did happen to me.
> On the other hand I wrote this patch *exactly* because this behaviour shot me in the leg quite badly and sent me on a multi-hour debugging session. 

*None* of which is explained in the commit message or the PR description.

> This can get especially difficult if you first build one project and then delete the build dir and reuse it to build another.

I do not know what you mean by this. If you delete a build dir and reuse it to build another project, that's a perfectly fine and valid use case, but it will already delete all files in the build directory leaving no stale data behind.

> Ninja sees that `foobar.so` exists, used to be a build target but no longer is, so it promptly deletes it

Okay, this is an explanation, and I can understand it. You're saying that because meson generates some build outputs at configure time, not at build time, those outputs aren't tracked by ninja. 

> The "correct" fix that I can see is to make Ninja remove the files it deletes on `cleandead` from its internal deps files. The fix is not perfect but at least it would be transient as opposed to permanent like it is now.

Actually I'd say that is very much perfect, with a small tweak. Only create symlinks from meson, after running cleandead

But also I'd like to clarify that the only thing at stake here is the internal logs file, not the internal deps file.

The other solution here is to resolve to never create output files from meson, and only create them by producing build edges. Unfortunately I can't seem to find a way to create a file in advance by meson and not have ninja rerun the build rule regardless... so that will be slightly slower in all cases, since all those .so symlinks will need to run additional build commands.

I have a patch to do this, if we think it's worth it.
> No. It is only run on the first configuration. Subsequent runs does not attempt to delete the files. For them the cost is one `if` condition.

Then this doesn't solve the issue. On first configuration, there are no stale files. On second configuration, i.e. "I did `git pull` and retrieved commits that modified meson.build; it regenerated the build directory automatically" this PR doesn't have any effect, and cleandead deletes the symlinks.

Resolving patch:
diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -488,6 +488,7 @@ def __init__(self, build: T.Optional[build.Build], interpreter: T.Optional[Inter
         self.introspection_data = {}
         self.created_llvm_ir_rule = PerMachine(False, False)
         self.rust_crates: T.Dict[str, RustCrate] = {}
+        self.implicit_meson_outs = []
 
     def create_phony_target(self, all_outputs, dummy_outfile, rulename, phony_infilename, implicit_outs=None):
         '''
@@ -3439,17 +3440,20 @@ def get_dependency_filename(self, t):
 
     def generate_shlib_aliases(self, target, outdir):
         for alias, to, tag in target.get_aliases():
-            aliasfile = os.path.join(self.environment.get_build_dir(), outdir, alias)
+            aliasfile = os.path.join(outdir, alias)
+            abs_aliasfile = os.path.join(self.environment.get_build_dir(), outdir, alias)
             try:
-                os.remove(aliasfile)
+                os.remove(abs_aliasfile)
             except Exception:
                 pass
             try:
-                os.symlink(to, aliasfile)
+                os.symlink(to, abs_aliasfile)
             except NotImplementedError:
                 mlog.debug("Library versioning disabled because symlinks are not supported.")
             except OSError:
                 mlog.debug("Library versioning disabled because we do not have symlink creation privileges.")
+            else:
+                self.implicit_meson_outs.append(aliasfile)
 
     def generate_custom_target_clean(self, trees: T.List[str]) -> str:
         e = self.create_phony_target(self.all_outputs, 'clean-ctlist', 'CUSTOM_COMMAND', 'PHONY')
@@ -3611,6 +3615,12 @@ def generate_ending(self):
         elem.add_item('pool', 'console')
         self.add_build(elem)
 
+        # If these files used to be explicitly created, they need to appear on the build graph somehow,
+        # otherwise cleandead deletes them. See https://github.com/ninja-build/ninja/issues/2299
+        if self.implicit_meson_outs:
+            elem = NinjaBuildElement(self.all_outputs, 'meson-implicit-outs', 'phony', self.implicit_meson_outs)
+            self.add_build(elem)
+
         elem = NinjaBuildElement(self.all_outputs, 'reconfigure', 'REGENERATE_BUILD', 'PHONY')
         elem.add_item('pool', 'console')
         self.add_build(elem)


Test patch: