Repository: mesonbuild/meson

Issue:
Wish: add ability to use local patch files
It is impossible to use cmake projects if they placed not in root directory of subproject.

For example look into LLVM project repository:
https://github.com/llvm/llvm-project/
Each of its subdir is cmake projects but it is impossible to use them from Meson if llvm-project will be wrapped as subproject.



Discussion:
You can work around this limitation by creating a dummy `CMakeLists.txt` that calls `add_subdirectory` and patch them with the [wrap patch mechanism](https://mesonbuild.com/Wrap-dependency-system-manual.html#wrapfile-with-meson-build-patch).
Uhh. Yes... But I prefer to avoid it by all means - wrap patch mechanism is hard to maintain: it assumes strict URL to archieve with original source and set up my own storage for patch URL. Each change will cause also changing hashes of source and/or patch files.

In general patching is a good thing, but since we have a cmake module and cmake system itself allows such kind of subdirectories it will be better to have native support for this.

I more generally think we should allow subprojects anywhere in the source tree, as long as that's code checked-in the main git tree and not something downloaded by a .wrap. But that idea got strongly rejected by @jpakkane so I got discouraged to work on it. I still believe that should be done.
Being able to have subprojects anywhere is one of those things that is easy and "fun" in the immediate short term but which becomes a complete maintenance nightmare in the long term. The main advantage of the current approach is that no matter which Meson project you use, you _know_ where the external deps are. They are always in the same place. You can rely on that. It reduces the time you need to understand any given code base noticeably.

In practice what happens is that every project of non-trivial size that has external dependencies comes up with their own rules of where to put things. These rules are pretty much arbitrary, you just have to pick one and stick with it. Since this will pretty much always happen, Meson has made the choice already and sticks with it. This is the only way to get consistency among projects, which is the true end goal.
@jpakkane external dependencies gets pulled by wrap files, that always must go to subprojects/ I totally agree. But here we are not talking about that. There are cases where a single project is split into independant parts, which is litterally sub-projects. Just like GStreamer has gst-plugins-good, gst-plugins-bad, etc. There are parts of the same project, not external dependencies.
At least that's what I understand from the LLVM project, it is split into sub-projects that are not external.
> Uhh. Yes... But I prefer to avoid it by all means - wrap patch mechanism is hard to maintain: it assumes strict URL to archieve with original source and set up my own storage for patch URL. Each change will cause also changing hashes of source and/or patch files.

In that case, would adding an (official, non-hacky) way to specify local patch archives work for you?
@denizzzka one work around would be to have a patch in your llvm.wrap that would only add a meson.build file at the top source dir. That way llvm becomes a *meson* subproject. In that meson.build you do:
```meson
foreach sub : ['llvm', ...]
  # See https://gitlab.freedesktop.org/gstreamer/gst-build/-/blob/master/meson.build#L78
  run_command(<python script to create symlink to subprojects/>)
  cmake.subproject(sub)
endforeach
```
@xclaesse This does not solve the main problem of having to patch the SP. And if you are patching the SP, it is *far* easier to add a dummy `CMakeLists.txt` with `add_subdirectory`.
@mensinda ah, misunderstood then, I though that it would have to change each CMakeLists.txt in subdirs which is not acceptable because wrap patch are replacing files, so you would have to regen the patch for each upstream change.
@mensinda 
> In that case, would adding an (official, non-hacky) way to specify local patch archives work for you?

Yes

upd: This is much more preferable!
@mensinda According to this, do we need to replace this issue title and badges?
@denizzzka Yes. Maybe something like "add ability to use local patch files".

Resolving patch:
diff --git a/mesonbuild/interpreter.py b/mesonbuild/interpreter.py
--- a/mesonbuild/interpreter.py
+++ b/mesonbuild/interpreter.py
@@ -2675,7 +2675,7 @@ def do_subproject(self, dirname: str, method: str, kwargs):
             return subproject
 
         subproject_dir_abs = os.path.join(self.environment.get_source_dir(), self.subproject_dir)
-        r = wrap.Resolver(subproject_dir_abs, self.coredata.get_builtin_option('wrap_mode'))
+        r = wrap.Resolver(subproject_dir_abs, self.coredata.get_builtin_option('wrap_mode'), current_subproject=self.subproject)
         try:
             resolved = r.resolve(dirname, method)
         except wrap.WrapException as e:
diff --git a/mesonbuild/wrap/wrap.py b/mesonbuild/wrap/wrap.py
--- a/mesonbuild/wrap/wrap.py
+++ b/mesonbuild/wrap/wrap.py
@@ -27,6 +27,7 @@
 import configparser
 import typing as T
 
+from pathlib import Path
 from . import WrapMode
 from ..mesonlib import git, GIT, ProgressBar, MesonException
 
@@ -126,7 +127,7 @@ def get(self, key: str) -> str:
             raise WrapException(m.format(key, self.basename))
 
     def has_patch(self) -> bool:
-        return 'patch_url' in self.values
+        return 'patch_filename' in self.values
 
 def load_wrap(subdir_root: str, packagename: str) -> PackageDefinition:
     fname = os.path.join(subdir_root, packagename + '.wrap')
@@ -146,10 +147,12 @@ def get_directory(subdir_root: str, packagename: str):
     return wrap, directory
 
 class Resolver:
-    def __init__(self, subdir_root: str, wrap_mode=WrapMode.default):
+    def __init__(self, subdir_root: str, wrap_mode=WrapMode.default, current_subproject: str = ''):
         self.wrap_mode = wrap_mode
         self.subdir_root = subdir_root
+        self.current_subproject = current_subproject
         self.cachedir = os.path.join(self.subdir_root, 'packagecache')
+        self.filesdir = os.path.join(self.subdir_root, 'packagefiles')
 
     def resolve(self, packagename: str, method: str) -> str:
         self.packagename = packagename
@@ -363,7 +366,9 @@ def get_data(self, urlstring: str) -> T.Tuple[str, str]:
             hashvalue = h.hexdigest()
         return hashvalue, tmpfile.name
 
-    def check_hash(self, what: str, path: str) -> None:
+    def check_hash(self, what: str, path: str, hash_required: bool = True) -> None:
+        if what + '_hash' not in self.wrap.values and not hash_required:
+            return
         expected = self.wrap.get(what + '_hash')
         h = hashlib.sha256()
         with open(path, 'rb') as f:
@@ -393,17 +398,28 @@ def download(self, what: str, ofname: str, fallback=False) -> None:
 
     def get_file_internal(self, what: str) -> str:
         filename = self.wrap.get(what + '_filename')
-        cache_path = os.path.join(self.cachedir, filename)
+        if what + '_url' in self.wrap.values:
+            cache_path = os.path.join(self.cachedir, filename)
 
-        if os.path.exists(cache_path):
-            self.check_hash(what, cache_path)
-            mlog.log('Using', mlog.bold(self.packagename), what, 'from cache.')
+            if os.path.exists(cache_path):
+                self.check_hash(what, cache_path)
+                mlog.log('Using', mlog.bold(self.packagename), what, 'from cache.')
+                return cache_path
+
+            if not os.path.isdir(self.cachedir):
+                os.mkdir(self.cachedir)
+            self.download(what, cache_path)
             return cache_path
+        else:
+            from ..interpreterbase import FeatureNew
+            FeatureNew('Local wrap patch files without {}_url'.format(what), '0.55.0').use(self.current_subproject)
+            path = Path(self.filesdir) / filename
+
+            if not path.exists():
+                raise WrapException('File "{}" does not exist'.format(path))
+            self.check_hash(what, path.as_posix(), hash_required=False)
 
-        if not os.path.isdir(self.cachedir):
-            os.mkdir(self.cachedir)
-        self.download(what, cache_path)
-        return cache_path
+            return path.as_posix()
 
     def apply_patch(self) -> None:
         path = self.get_file_internal('patch')


Test patch:
diff --git a/test cases/common/157 wrap file should not failed/meson.build b/test cases/common/157 wrap file should not failed/meson.build
--- a/test cases/common/157 wrap file should not failed/meson.build	
+++ b/test cases/common/157 wrap file should not failed/meson.build	
@@ -3,7 +3,11 @@ project('mainproj', 'c',
 )
 
 subproject('zlib')
-subproject('foo')
+foo = subproject('foo')
+bar = subproject('bar')
+
+libfoo = foo.get_variable('libfoo')
+libbar = bar.get_variable('libbar')
 
 executable('grabprog', files('src/subprojects/prog.c'))
 executable('grabprog2', files('src/subprojects/foo/prog2.c'))
diff --git a/test cases/common/157 wrap file should not failed/src/meson.build b/test cases/common/157 wrap file should not failed/src/meson.build
--- a/test cases/common/157 wrap file should not failed/src/meson.build	
+++ b/test cases/common/157 wrap file should not failed/src/meson.build	
@@ -1,2 +1,6 @@
 executable('grabprog3', files('subprojects/prog.c'))
 executable('grabprog4', files('subprojects/foo/prog2.c'))
+
+texe = executable('testexe', files('test.c'), link_with: [libfoo, libbar])
+
+test('t1', texe)
diff --git a/test cases/common/157 wrap file should not failed/src/test.c b/test cases/common/157 wrap file should not failed/src/test.c
new file mode 100644
--- /dev/null
+++ b/test cases/common/157 wrap file should not failed/src/test.c	
@@ -0,0 +1,9 @@
+#include <stdio.h>
+
+int bar_dummy_func(void);
+int dummy_func(void);
+
+int main(void) {
+    printf("Hello world %d\n", bar_dummy_func() + dummy_func());
+    return 0;
+}
diff --git a/test cases/common/157 wrap file should not failed/subprojects/.gitignore b/test cases/common/157 wrap file should not failed/subprojects/.gitignore
new file mode 100644
--- /dev/null
+++ b/test cases/common/157 wrap file should not failed/subprojects/.gitignore	
@@ -0,0 +1,2 @@
+/foo-1.0
+/bar-1.0
diff --git a/test cases/common/157 wrap file should not failed/subprojects/bar.wrap b/test cases/common/157 wrap file should not failed/subprojects/bar.wrap
new file mode 100644
--- /dev/null
+++ b/test cases/common/157 wrap file should not failed/subprojects/bar.wrap	
@@ -0,0 +1,8 @@
+[wrap-file]
+directory = bar-1.0
+lead_directory_missing = true
+
+source_filename = bar-1.0.tar.xz
+source_hash     = f0f61948530dc0d33e3028cd71a9f8ee869f6b3665960d8f41d715cf4aed6467
+
+patch_filename  = bar-1.0-patch.tar.xz
diff --git a/test cases/common/157 wrap file should not failed/subprojects/foo-1.0/foo.c b/test cases/common/157 wrap file should not failed/subprojects/foo-1.0/foo.c
deleted file mode 100644
--- a/test cases/common/157 wrap file should not failed/subprojects/foo-1.0/foo.c	
+++ /dev/null
@@ -1,3 +0,0 @@
-int dummy_func(void) {
-    return 42;
-}
diff --git a/test cases/common/157 wrap file should not failed/subprojects/foo-1.0/meson.build b/test cases/common/157 wrap file should not failed/subprojects/foo-1.0/meson.build
deleted file mode 100644
--- a/test cases/common/157 wrap file should not failed/subprojects/foo-1.0/meson.build	
+++ /dev/null
@@ -1,2 +0,0 @@
-project('shared lib', 'c')
-libfoo = shared_library('foo', 'foo.c')
diff --git a/test cases/common/157 wrap file should not failed/subprojects/foo.wrap b/test cases/common/157 wrap file should not failed/subprojects/foo.wrap
--- a/test cases/common/157 wrap file should not failed/subprojects/foo.wrap	
+++ b/test cases/common/157 wrap file should not failed/subprojects/foo.wrap	
@@ -3,9 +3,9 @@ directory = foo-1.0
 
 source_url = http://something.invalid
 source_filename = foo-1.0.tar.xz
-source_hash = ae5fc03185654f76b459db16ca25809703f8821aeb39a433902244bb479c4b79
+source_hash = 9ed8f67d75e43d3be161efb6eddf30dd01995a958ca83951ea64234bac8908c1
 lead_directory_missing = true
 
 patch_url = https://something.invalid/patch
 patch_filename = foo-1.0-patch.tar.xz
-patch_hash = 8f2e286a4b190228d4e0c25ddc91195449cfb5e5c52006355838964b244037da
+patch_hash = d0ddc5e60fdb27d808552f5ac8d0bb603ea2cba306538b4427b985535b26c9c5
diff --git a/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0-patch.tar.xz b/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0-patch.tar.xz
Binary files a/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0-patch.tar.xz and b/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0-patch.tar.xz differ
diff --git a/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0.tar.xz b/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0.tar.xz
Binary files a/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0.tar.xz and b/test cases/common/157 wrap file should not failed/subprojects/packagecache/foo-1.0.tar.xz differ
diff --git a/test cases/common/157 wrap file should not failed/subprojects/packagefiles/bar-1.0-patch.tar.xz b/test cases/common/157 wrap file should not failed/subprojects/packagefiles/bar-1.0-patch.tar.xz
new file mode 100644
Binary files /dev/null and b/test cases/common/157 wrap file should not failed/subprojects/packagefiles/bar-1.0-patch.tar.xz differ
diff --git a/test cases/common/157 wrap file should not failed/subprojects/packagefiles/bar-1.0.tar.xz b/test cases/common/157 wrap file should not failed/subprojects/packagefiles/bar-1.0.tar.xz
new file mode 100644
Binary files /dev/null and b/test cases/common/157 wrap file should not failed/subprojects/packagefiles/bar-1.0.tar.xz differ