Repository: mesonbuild/meson

Issue:
Changing contents of a wrap file doesn't trigger a rebuild of the subproject.
**Describe the bug**

If I have a project with a wrap file and I change some information in the file (such as version), that isn't picked up by ninja to make meson run configure again.

**To Reproduce**

I don't think a reproducer is needed, but if one is, I can cook it up on request.

**Expected behavior**

ninja will trigger a meson reconfiguration, which will pick up differences in the wrap file. 

**system parameters**
* Native build
* Ubuntu 20.04
* python 3.8
* meson master
* samurai

In case this is intended behavior, I think it should be documented (or I missed it in the docs :c )


Discussion:
Leaving aside the question of docs, the implementation itself currently assumes that ninja rebuilds are triggered when the subproject itself changes.

Wrap files aren't the subproject itself, though... They only describe how to download one (as well as which deps/progs it provides). To put it another way... Even if you manually reconfigure, for example by running `meson setup --wipe`, it won't actually update a wrap-git to the new revision, or re-extract the patch_directory of a wrap-file. There is a dedicated.command to do that though, `meson subprojects update`.

In the event that a reconfigure is triggered for some other reason, the subproject will be resolved again, and if the directory for it exists it will be considered to be cached and blindly reused. If the wrap file updates to a new version and that version is a tarball with a different directory key, Meson will treat it as missing (not know that the old directory name even exists) and run the download/extract stage of subproject resolution as a "missing" subproject, rather than an out of date one.

Any suggestions or PRs to make this clearer in the docs will be gratefully welcomed. I think you're not the first person who wasn't initially clear on how that works.
If you do `meson wrap update foo` then IIRC it will only touch the wrap file. Thus it won't trigger an update on recompilation, which would be unexpected.
> the implementation itself currently assumes that ninja rebuilds are triggered when the subproject itself changes.

@eli-schwartz: What is the definition of subproject here? For me the changes in `meson.build` inside `packagefiles` are also silently ignored. This made both me and other devs I work with waste many hours debugging problems which were non-existent, just because the subprojects builds got stale silently.

> Any suggestions or PRs to make this clearer in the docs will be gratefully welcomed. I think you're not the first person who wasn't initially clear on how that works.

I'm not sure this will help much, in our project we painfully rediscover this Meson quirk from time to time despite we all knew about this at some point in the past. It's just super unintuitive for us that Meson can sometimes generate incorrect build results if you don't run `meson subprojects purge --confirm`.
Couldn't Meson track these dependencies somehow?
One of the reason it's not updating the subproject checkout when changing the wrap or packagefiles is you could lose local changes to the subproject.  The command to ensure you build up to date subproject is `meson subprojects update --reset`.
> @eli-schwartz: What is the definition of subproject here? For me the changes in `meson.build` inside `packagefiles` are also silently ignored. This made both me and other devs I work with waste many hours debugging problems which were non-existent, just because the subprojects builds got stale silently.

The subproject is the files in `subprojects/<subproject_name>/`, note ending directory slash.

The changes inside `packagefiles/` are not "a subproject", they are instructions for how to download a subproject. In fact, they are *literally* an alternative way to provide the same content normally provided by patch_url, which is a network resource.

If the file `subprojects/<subproject_name>/meson.build` gets updated, then it triggers a buildsystem reconfigure, just like any other meson.build in any subdir which gets parsed to contribute its build rules.

...

If you want to apply updates to ***the source code*** of a subproject, then as I said above, there is a dedicated command for that, `meson subprojects update`, and as @xclaesse pointed out, this must be done manually, thereby acknowledging that you want to destroy any local changes. It won't be done automatically.

Note that this is NOT a buildsystem reconfigure. It is a source code update. It's the same codepath that will also run `git fetch` and re-checkout the latest upstream version of a branch.

> Couldn't Meson track these dependencies somehow?

What should it track, exactly? The input in question is your `packagefiles/` directory, and the *output* is not the build directory, but the subprojects/ source code. If dependencies *were* tracked, then the action they would trigger would be to run `meson subprojects update --reset`, a potentially destructive action that isn't related to building at all.
> One of the reason it's not updating the subproject checkout when changing the wrap or packagefiles is you could lose local changes to the subproject.

Couldn't this be solved by building subprojects inside build directory? This is another thing which surprised me after reading https://mesonbuild.com/Quick-guide.html#compiling-a-meson-project (I recently went through all the entry-level docs), it seemed to be Meson's philosophy to build everything in the build directory. But then I noticed that it changes stuff in `subprojects/` in-place.

> What should it track, exactly? The input in question is your `packagefiles/` directory, and the output is not the build directory, but the subprojects/ source code. If dependencies were tracked, then the action they would trigger would be to run `meson subprojects update --reset`, a potentially destructive action that isn't related to building at all.

Assuming you need to stick with in-tree build here for some reason: Maybe it could at least throw an error/warning in such case and suggest the `update` command, with a disclaimer that it may be destructive? Is there any case where the user modified `packagefiles/`, but doesn't want the change to be noticed by the buildsystem? At least in my usages this was always an accidental error and what I really wanted was a rebuild.
> it seemed to be Meson's philosophy to build everything in the build directory. But then I noticed that it changes stuff in subprojects/ in-place.

> stick with in-tree build here for some reason: 

> what I really wanted was a rebuild.

Correct terminology is relevant here. This isn't building. This is checking out or otherwise downloading source code.

...

> Couldn't this be solved by building subprojects inside build directory? This is another thing which surprised me after reading https://mesonbuild.com/Quick-guide.html#compiling-a-meson-project (I recently went through all the entry-level docs), it seemed to be Meson's philosophy to build everything in the build directory. But then I noticed that it changes stuff in subprojects/ in-place.

Multiple build directories may exist and should share source code. Additionally, subprojects should not have source code .c and build artifact .o in the same directory (even if it is a build directory).

And ***definitely*** you want downloaded tarballs to be cached in subprojects/packagecache/ and not builddir/subprojects/packagecache/


> Maybe it could at least throw an error/warning in such case and suggest the update command, with a disclaimer that it may be destructive?

Maybe? I don't know what that would look like. It would probably involve configure-time scanning of the packagefiles directory as we really have no idea when the subproject is downloaded, via automatic fallback or manually by the subprojects command or via `meson dist --include-subprojects`.
> This is checking out or otherwise downloading source code.

Hmm, but it also consist of applying the overlay? So it's not "just downloading"?

> Additionally, subprojects should not have source code .c and build artifact .o in the same directory (even if it is a build directory).

This seems to be rather a technicality? You could have 2 directories inside build directory, one with the subprojects' sources and the other for their build outputs?

> And _**definitely**_ you want downloaded tarballs to be cached in subprojects/packagecache/ and not builddir/subprojects/packagecache/

To be honest, I myself would prefer to have them copied a few times (and they are _usually_ rather small) than wasting time debugging binaries built incorrectly.

> Maybe? I don't know what that would look like. It would probably involve configure-time scanning of the packagefiles directory as we really have no idea when the subproject is downloaded, via automatic fallback or manually by the subprojects command or via `meson dist --include-subprojects`.

I'm more concerned about the case where the `packagefiles/` included in the main project has changed. If you change some remote resource then it's expected and (IMO) intuitive that you may need a forced clean rebuild. But if you change something in your own project then I'd expect that this will properly be noticed by dependency checking in the build system.

Anyways, I'm just saying this as a pretty new user which doesn't know Meson internals, who tries to migrate to Meson in a few projects. This is quite a pain point for us, as multiple people from our team hit this problem more than once.
>@xclaesse  One of the reason it's not updating the subproject checkout when changing the wrap or packagefiles is you could lose local changes to the subproject. 

That's a good point. Could it check to make sure there are no local changes and only update the subproject checkout in that case?

The most common user error I've seen people encounter when working with Meson is that they checkout a revision of the main project which includes changes to the revisions in the subproject wrap files and don't think to blow away their subprojects before running Meson. If Meson were to check the actual subproject revision against the revision in the wrap file it would fix a lot of user error.

If the Meson team would be open to some variant of that feature, I'd be happy to take a stab at implementing it.


Local changes is difficult to define.

I'd be more comfortable with somehow adding a warning message that a subproject might be out of date, and leave it to the user to actually update.

Shower thought time. We could create some sort of stamp file inside the subproject e.g.

```
subprojects/proj_name/.meson-subproject-extracted.stamp
```

containing a list of the files that was used to extract the subproject, e.g. the wrap file, any tarballs from packagecache/, any files in packagefiles/

This could then be checked at relevant times, for example when setting up a build directory we could issue a warning (not an error, but warnings become errors if you use --fatal-meson-warnings).

In theory we could also add a ninja rule that checks to see if the stamp is "out of date" and emits a notification message to stderr on every single build.
I like the general concept. If we're not trying to detect local changes, though, and just whether the user has modified any of the inputs, would it be sufficient to just check the `.wrap` file and the contents of `packagefiles`? I'm thinking it could store off a hash of the wrap file, as well as a hash of any packagefiles directory that the wrap points to. Then when meson runs it could recalculate the hashes and compare them against the stored hashes. If either don't match, it could log a warning like

> Subproject  <proj_name> may be out of date. Changes have been made to  <proj_name>.wrap or the packagefiles since the subproject was initially configured.
> would it be sufficient to just check the `.wrap` file and the contents of `packagefiles`? I'm thinking it could store off a hash of the wrap file, as well as a hash of any packagefiles directory that the wrap points to.

Maybe. I did offer the disclaimer that I was engaging in shower thoughts. I guess since the wrap file already contains a hash of any url-based tarballs, the packagecache/ tarball cannot really change without the wrap file changing.

> If the Meson team would be open to some variant of that feature, I'd be happy to take a stab at implementing it.

I am absolutely open to this idea, or any idea you have to improve the user experience, which based on this discussion I think I agree definitely has some sharp edges.

Specifically for one possible solution, I don't want it to automatically update/re-extract the subproject source files because I'm concerned that this cannot be done 100% safely, only 99.9% safely, and in that 0.1% case we will overwrite someone's work. I don't expect to hear arguments that will change my mind, but you're welcome to try...

Anything short of that is definitely fair game, and I am brainstorming possible solutions for you to run with -- you did say you'd be open to take a stab at implementing it, which means the exact details of how you implement it are up to you, and I will simply review it to see if it behaves as expected, potentially offer helpful suggestions, and ultimately almost certainly accept the feature you design if we both agree it helps your use case and doesn't regress the experience for other users.

Resolving patch:
diff --git a/mesonbuild/msubprojects.py b/mesonbuild/msubprojects.py
--- a/mesonbuild/msubprojects.py
+++ b/mesonbuild/msubprojects.py
@@ -394,19 +394,22 @@ def update_svn(self) -> bool:
 
     def update(self) -> bool:
         self.log(f'Updating {self.wrap.name}...')
+        success = False
         if self.wrap.type == 'file':
-            return self.update_file()
+            success = self.update_file()
         elif self.wrap.type == 'git':
-            return self.update_git()
+            success = self.update_git()
         elif self.wrap.type == 'hg':
-            return self.update_hg()
+            success = self.update_hg()
         elif self.wrap.type == 'svn':
-            return self.update_svn()
+            success = self.update_svn()
         elif self.wrap.type is None:
             self.log('  -> Cannot update subproject with no wrap file')
         else:
             self.log('  -> Cannot update', self.wrap.type, 'subproject')
-        return True
+        if success:
+            self.wrap.update_hash_cache(self.wrap_resolver.dirname)
+        return success
 
     def checkout(self) -> bool:
         options = T.cast('CheckoutArguments', self.options)
diff --git a/mesonbuild/wrap/wrap.py b/mesonbuild/wrap/wrap.py
--- a/mesonbuild/wrap/wrap.py
+++ b/mesonbuild/wrap/wrap.py
@@ -127,17 +127,19 @@ def __init__(self, fname: str, subproject: str = ''):
         self.directory = self.name
         # must be lowercase for consistency with dep=variable assignment
         self.provided_deps[self.name.lower()] = None
+        # What the original file name was before redirection
         self.original_filename = fname
         self.redirected = False
         if self.has_wrap:
             self.parse_wrap()
-        self.directory = self.values.get('directory', self.name)
+            self.directory = self.values.get('directory', self.name)
+            with open(fname, 'r', encoding='utf-8') as file:
+                self.wrapfile_hash = hashlib.sha256(file.read().encode('utf-8')).hexdigest()
         if os.path.dirname(self.directory):
             raise WrapException('Directory key must be a name and not a path')
         if self.type and self.type not in ALL_TYPES:
             raise WrapException(f'Unknown wrap type {self.type!r}')
         self.filesdir = os.path.join(os.path.dirname(self.filename), 'packagefiles')
-        # What the original file name was before redirection
 
     def parse_wrap(self) -> None:
         try:
@@ -221,6 +223,14 @@ def get(self, key: str) -> str:
         except KeyError:
             raise WrapException(f'Missing key {key!r} in {self.basename}')
 
+    def get_hashfile(self, subproject_directory: str) -> str:
+        return os.path.join(subproject_directory, '.meson-subproject-wrap-hash.txt')
+
+    def update_hash_cache(self, subproject_directory: str) -> None:
+        if self.has_wrap:
+            with open(self.get_hashfile(subproject_directory), 'w', encoding='utf-8') as file:
+                file.write(self.wrapfile_hash + '\n')
+
 def get_directory(subdir_root: str, packagename: str) -> str:
     fname = os.path.join(subdir_root, packagename + '.wrap')
     if os.path.isfile(fname):
@@ -367,6 +377,7 @@ def resolve(self, packagename: str, method: str) -> str:
 
         # The directory is there and has meson.build? Great, use it.
         if method == 'meson' and os.path.exists(meson_file):
+            self.validate()
             return rel_path
         if method == 'cmake' and os.path.exists(cmake_file):
             return rel_path
@@ -403,6 +414,11 @@ def resolve(self, packagename: str, method: str) -> str:
         if method == 'cmake' and not os.path.exists(cmake_file):
             raise WrapException('Subproject exists but has no CMakeLists.txt file')
 
+        # At this point, the subproject has been successfully resolved for the
+        # first time so save off the hash of the entire wrap file for future
+        # reference.
+        self.wrap.update_hash_cache(self.dirname)
+
         return rel_path
 
     def check_can_download(self) -> None:
@@ -504,6 +520,26 @@ def get_git(self) -> None:
             if push_url:
                 verbose_git(['remote', 'set-url', '--push', 'origin', push_url], self.dirname, check=True)
 
+    def validate(self) -> None:
+        # This check is only for subprojects with wraps.
+        if not self.wrap.has_wrap:
+            return
+
+        # Retrieve original hash, if it exists.
+        hashfile = self.wrap.get_hashfile(self.dirname)
+        if os.path.isfile(hashfile):
+            with open(hashfile, 'r', encoding='utf-8') as file:
+                expected_hash = file.read().strip()
+        else:
+            # If stored hash doesn't exist then don't warn.
+            return
+
+        actual_hash = self.wrap.wrapfile_hash
+
+        # Compare hashes and warn the user if they don't match.
+        if expected_hash != actual_hash:
+            mlog.warning(f'Subproject {self.wrap.name}\'s revision may be out of date; its wrap file has changed since it was first configured')
+
     def is_git_full_commit_id(self, revno: str) -> bool:
         result = False
         if len(revno) in (40, 64): # 40 for sha1, 64 for upcoming sha256
diff --git a/unittests/allplatformstests.py b/unittests/allplatformstests.py
--- a/unittests/allplatformstests.py
+++ b/unittests/allplatformstests.py
@@ -3733,10 +3733,26 @@ def test_wrap_git(self):
                   patch_directory = wrap_git_builddef
                   revision = master
                 '''.format(upstream_uri)))
-            self.init(srcdir)
+            out = self.init(srcdir)
             self.build()
             self.run_tests()
 
+            # Make sure the warning does not occur on the first init.
+            out_of_date_warning = 'revision may be out of date'
+            self.assertNotIn(out_of_date_warning, out)
+
+            # Change the wrap's revisions, reconfigure, and make sure it does
+            # warn on the reconfigure.
+            with open(os.path.join(srcdir, 'subprojects', 'wrap_git.wrap'), 'w', encoding='utf-8') as f:
+                f.write(textwrap.dedent('''
+                  [wrap-git]
+                  url = {}
+                  patch_directory = wrap_git_builddef
+                  revision = not-master
+                '''.format(upstream_uri)))
+            out = self.init(srcdir, extra_args='--reconfigure')
+            self.assertIn(out_of_date_warning, out)
+
     def test_extract_objects_custom_target_no_warning(self):
         testdir = os.path.join(self.common_test_dir, '22 object extraction')
 


Test patch: