Repository: mesonbuild/meson

Issue:
0.36.0 test regression test cases/frameworks/13 yelp
Hi, after trying to package the new 0.36.0 version i have noticed a test regression in ```test cases/frameworks/13 yelp```. I have used ```MESON_PRINT_TEST_OUTPUT=1``` but not quote sure what the problem is:

```
Failed test: test cases/frameworks/13 yelp
Reason: Running install failed.
The Meson build system
Version: 0.36.0
Source dir: /build/meson/src/meson-0.36.0/test cases/frameworks/13 yelp
Build dir: /build/meson/src/meson-0.36.0/b 9cu54r21
Build type: native build
Project name: yelp
Native c compiler: cc (gcc 6.2.1)
Appending CFLAGS from environment: '-march=i686 -mtune=generic -O2 -pipe -fstack-protector-strong'
Appending LDFLAGS from environment: '-Wl,-O1,--sort-common,--as-needed,-z,relro'
Appending CPPFLAGS from environment: '-D_FORTIFY_SOURCE=2'
Build machine cpu family: x86
Build machine cpu: i686
Build targets in project: 2
ninja: no work to do.
No tests defined.
No benchmarks defined.
[0/1] Installing files.
Running custom install script /usr/bin/python3
Failed to run install script: /usr/bin/python3 /build/meson/src/meson-0.36.0/meson.py --internal yelphelper install --subdir=help --id=meson --installdir=share/help --sources=index.page --media=media/test.txt --langs=de@@es
FAILED: install 
'/usr/bin/python3' '/build/meson/src/meson-0.36.0/meson.py' '--internal' 'install' '/build/meson/src/meson-0.36.0/b 9cu54r21/meson-private/install.dat'
ninja: build stopped: subcommand failed.
```


Discussion:
Actually it seems to be because of missing `itstool`, when installing it into the clean chroot before building it passes. Shouldn't this be skipped in case something is missing?

It looks like the yelp helper doesn't really check for any of its required executables at all.

Neither the i18n or gnome module check for the tools they just fail. Was on my todo list to clean up the error messages.

See #821 for the i18n check issue.

This seems to be fixed with: 
https://github.com/mesonbuild/meson/pull/3060
https://github.com/mesonbuild/meson/commit/d78f88160a9ee1cc8d75274599d6c45c7e678d9a

See also:
https://github.com/mesonbuild/meson/issues/2325

But in my opinion it would be better if the gnome.yelp module would check for the required dependencies so that its not needed to manually check for them every time


Resolving patch:
diff --git a/mesonbuild/modules/gnome.py b/mesonbuild/modules/gnome.py
--- a/mesonbuild/modules/gnome.py
+++ b/mesonbuild/modules/gnome.py
@@ -39,6 +39,7 @@
     MachineChoice, MesonException, OrderedSet, Popen_safe, join_args,
 )
 from ..programs import ExternalProgram, OverrideProgram, EmptyExternalProgram
+from ..scripts.gettext import read_linguas
 
 if T.TYPE_CHECKING:
     from typing_extensions import Literal, TypedDict
@@ -1154,47 +1155,99 @@ def yelp(self, state: 'ModuleState', args: T.Tuple[str, T.List[str]], kwargs: 'Y
                 raise MesonException('Yelp requires a list of sources')
         elif args[1]:
             mlog.warning('"gnome.yelp" ignores positional sources arguments when the "sources" keyword argument is set')
-        source_str = '@@'.join(sources)
+        sources_files = [mesonlib.File.from_source_file(state.environment.source_dir,
+                                                        os.path.join(state.subdir, 'C'),
+                                                        s) for s in sources]
 
         langs = kwargs['languages']
+        if not langs:
+            langs = read_linguas(os.path.join(state.environment.source_dir, state.subdir))
+
+        media = kwargs['media']
+        symlinks = kwargs['symlink_media']
+        targets: T.List[T.Union['Target', build.Data, build.SymlinkData]] = []
+        potargets: T.List[build.RunTarget] = []
+
+        itstool = state.find_program('itstool')
+        msgmerge = state.find_program('msgmerge')
+        msgfmt = state.find_program('msgfmt')
+
+        install_dir = os.path.join(state.environment.get_datadir(), 'help')
+        c_install_dir = os.path.join(install_dir, 'C', project_id)
+        c_data = build.Data(sources_files, c_install_dir, c_install_dir,
+                            mesonlib.FileMode(), state.subproject)
+        targets.append(c_data)
+
+        media_files: T.List[mesonlib.File] = []
+        for m in media:
+            f = mesonlib.File.from_source_file(state.environment.source_dir,
+                                               os.path.join(state.subdir, 'C'), m)
+            media_files.append(f)
+            m_install_dir = os.path.join(c_install_dir, os.path.dirname(m))
+            m_data = build.Data([f], m_install_dir, m_install_dir,
+                                mesonlib.FileMode(), state.subproject)
+            targets.append(m_data)
+
+        pot_file = os.path.join('@SOURCE_ROOT@', state.subdir, 'C', project_id + '.pot')
+        pot_sources = [os.path.join('@SOURCE_ROOT@', state.subdir, 'C', s) for s in sources]
+        pot_args = [itstool, '-o', pot_file] + pot_sources
+        pottarget = build.RunTarget(f'help-{project_id}-pot', pot_args, [],
+                                    os.path.join(state.subdir, 'C'), state.subproject)
+        targets.append(pottarget)
+
+        for l in langs:
+            l_subdir = os.path.join(state.subdir, l)
+            l_install_dir = os.path.join(install_dir, l, project_id)
+
+            for i, m in enumerate(media):
+                m_dir = os.path.dirname(m)
+                m_install_dir = os.path.join(l_install_dir, m_dir)
+                if symlinks:
+                    link_target = os.path.join(os.path.relpath(c_install_dir, start=m_install_dir), m)
+                    l_data = build.SymlinkData(link_target, os.path.basename(m),
+                                               m_install_dir, state.subproject)
+                else:
+                    try:
+                        m_file = mesonlib.File.from_source_file(state.environment.source_dir, l_subdir, m)
+                    except MesonException:
+                        m_file = media_files[i]
+                    l_data = build.Data([m_file], m_install_dir, m_install_dir,
+                                        mesonlib.FileMode(), state.subproject)
+                targets.append(l_data)
+
+            po_file = l + '.po'
+            po_args = [msgmerge, '-q', '-o',
+                       os.path.join('@SOURCE_ROOT@', l_subdir, po_file),
+                       os.path.join('@SOURCE_ROOT@', l_subdir, po_file), pot_file]
+            potarget = build.RunTarget(f'help-{project_id}-{l}-update-po',
+                                       po_args, [pottarget], l_subdir, state.subproject)
+            targets.append(potarget)
+            potargets.append(potarget)
+
+            gmo_file = project_id + '-' + l + '.gmo'
+            gmo_kwargs = {'command': [msgfmt, '@INPUT@', '-o', '@OUTPUT@'],
+                          'input': po_file,
+                          'output': gmo_file,
+            }
+            gmotarget = build.CustomTarget(f'help-{project_id}-{l}-gmo', l_subdir, state.subproject, gmo_kwargs)
+            targets.append(gmotarget)
+
+            merge_kwargs = {'command': [itstool, '-m', os.path.join(l_subdir, gmo_file),
+                                        '-o', '@OUTDIR@', '@INPUT@'],
+                            'input': sources_files,
+                            'output': sources,
+                            'depends': gmotarget,
+                            'install': True,
+                            'install_dir': l_install_dir,
+            }
+            mergetarget = build.CustomTarget(f'help-{project_id}-{l}', l_subdir, state.subproject, merge_kwargs)
+            targets.append(mergetarget)
 
-        script = state.environment.get_build_command()
-        inscript_args = ['--internal',
-                'yelphelper',
-                'install',
-                '--subdir=' + state.subdir,
-                '--id=' + project_id,
-                '--installdir=' + os.path.join(state.environment.get_datadir(), 'help'),
-                '--sources=' + source_str]
-        if kwargs['symlink_media']:
-            inscript_args.append('--symlinks=true')
-        if kwargs['media']:
-            inscript_args.append('--media=' + '@@'.join(kwargs['media']))
-        if langs:
-            inscript_args.append('--langs=' + '@@'.join(langs))
-        inscript = state.backend.get_executable_serialisation(script + inscript_args)
-
-        potargs = state.environment.get_build_command() + [
-            '--internal', 'yelphelper', 'pot',
-            '--subdir=' + state.subdir,
-            '--id=' + project_id,
-            '--sources=' + source_str,
-        ]
-        pottarget = build.RunTarget('help-' + project_id + '-pot', potargs,
-                                    [], state.subdir, state.subproject)
-
-        poargs = state.environment.get_build_command() + [
-            '--internal', 'yelphelper', 'update-po',
-            '--subdir=' + state.subdir,
-            '--id=' + project_id,
-            '--sources=' + source_str,
-            '--langs=' + '@@'.join(langs),
-        ]
-        potarget = build.RunTarget('help-' + project_id + '-update-po', poargs,
-                                   [], state.subdir, state.subproject)
-
-        rv: T.List[T.Union[build.ExecutableSerialisation, build.RunTarget]] = [inscript, pottarget, potarget]
-        return ModuleReturnValue(None, rv)
+        allpotarget = build.AliasTarget(f'help-{project_id}-update-po', potargets,
+                                        state.subdir, state.subproject)
+        targets.append(allpotarget)
+
+        return ModuleReturnValue(None, targets)
 
     @typed_pos_args('gnome.gtkdoc', str)
     @typed_kwargs(
diff --git a/mesonbuild/scripts/yelphelper.py b/mesonbuild/scripts/yelphelper.py
deleted file mode 100644
--- a/mesonbuild/scripts/yelphelper.py
+++ /dev/null
@@ -1,133 +0,0 @@
-# Copyright 2016 The Meson development team
-
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-
-#     http://www.apache.org/licenses/LICENSE-2.0
-
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-import subprocess
-import shutil
-import argparse
-from .. import mlog
-from ..mesonlib import has_path_sep
-from . import destdir_join
-from .gettext import read_linguas
-import typing as T
-
-parser = argparse.ArgumentParser()
-parser.add_argument('command')
-parser.add_argument('--id', dest='project_id')
-parser.add_argument('--subdir', dest='subdir')
-parser.add_argument('--installdir', dest='install_dir')
-parser.add_argument('--sources', dest='sources')
-parser.add_argument('--media', dest='media', default='')
-parser.add_argument('--langs', dest='langs', default='')
-parser.add_argument('--symlinks', type=bool, dest='symlinks', default=False)
-
-def build_pot(srcdir: str, project_id: str, sources: T.List[str]) -> None:
-    # Must be relative paths
-    sources = [os.path.join('C', source) for source in sources]
-    outfile = os.path.join(srcdir, project_id + '.pot')
-    subprocess.call(['itstool', '-o', outfile] + sources)
-
-def update_po(srcdir: str, project_id: str, langs: T.List[str]) -> None:
-    potfile = os.path.join(srcdir, project_id + '.pot')
-    for lang in langs:
-        pofile = os.path.join(srcdir, lang, lang + '.po')
-        subprocess.call(['msgmerge', '-q', '-o', pofile, pofile, potfile])
-
-def build_translations(srcdir: str, blddir: str, langs: T.List[str]) -> None:
-    for lang in langs:
-        outdir = os.path.join(blddir, lang)
-        os.makedirs(outdir, exist_ok=True)
-        subprocess.call([
-            'msgfmt', os.path.join(srcdir, lang, lang + '.po'),
-            '-o', os.path.join(outdir, lang + '.gmo')
-        ])
-
-def merge_translations(blddir: str, sources: T.List[str], langs: T.List[str]) -> None:
-    for lang in langs:
-        subprocess.call([
-            'itstool', '-m', os.path.join(blddir, lang, lang + '.gmo'),
-            '-o', os.path.join(blddir, lang)
-        ] + sources)
-
-def install_help(srcdir: str, blddir: str, sources: T.List[str], media: T.List[str], langs: T.List[str], install_dir: str, destdir: str, project_id: str, symlinks: bool) -> None:
-    c_install_dir = os.path.join(install_dir, 'C', project_id)
-    for lang in langs + ['C']:
-        indir = destdir_join(destdir, os.path.join(install_dir, lang, project_id))
-        os.makedirs(indir, exist_ok=True)
-        for source in sources:
-            infile = os.path.join(srcdir if lang == 'C' else blddir, lang, source)
-            outfile = os.path.join(indir, source)
-            mlog.log(f'Installing {infile} to {outfile}')
-            shutil.copy2(infile, outfile)
-        for m in media:
-            infile = os.path.join(srcdir, lang, m)
-            outfile = os.path.join(indir, m)
-            c_infile = os.path.join(srcdir, 'C', m)
-            if not os.path.exists(infile):
-                if not os.path.exists(c_infile):
-                    mlog.warning('Media file "%s" did not exist in C directory' % m)
-                    continue
-                elif symlinks:
-                    srcfile = os.path.join(c_install_dir, m)
-                    mlog.log(f'Symlinking {outfile} to {srcfile}.')
-                    if has_path_sep(m):
-                        os.makedirs(os.path.dirname(outfile), exist_ok=True)
-                    try:
-                        try:
-                            os.symlink(srcfile, outfile)
-                        except FileExistsError:
-                            os.remove(outfile)
-                            os.symlink(srcfile, outfile)
-                        continue
-                    except (NotImplementedError, OSError):
-                        mlog.warning('Symlinking not supported, falling back to copying')
-                        infile = c_infile
-                else:
-                    # Lang doesn't have media file so copy it over 'C' one
-                    infile = c_infile
-            mlog.log(f'Installing {infile} to {outfile}')
-            if has_path_sep(m):
-                os.makedirs(os.path.dirname(outfile), exist_ok=True)
-            shutil.copyfile(infile, outfile)
-            shutil.copystat(infile, outfile)
-
-def run(args: T.List[str]) -> int:
-    options = parser.parse_args(args)
-    langs = options.langs.split('@@') if options.langs else []
-    media = options.media.split('@@') if options.media else []
-    sources = options.sources.split('@@')
-    destdir = os.environ.get('DESTDIR', '')
-    src_subdir = os.path.join(os.environ['MESON_SOURCE_ROOT'], options.subdir)
-    build_subdir = os.path.join(os.environ['MESON_BUILD_ROOT'], options.subdir)
-    abs_sources = [os.path.join(src_subdir, 'C', source) for source in sources]
-
-    if not langs:
-        langs = read_linguas(src_subdir)
-
-    if options.command == 'pot':
-        build_pot(src_subdir, options.project_id, sources)
-    elif options.command == 'update-po':
-        build_pot(src_subdir, options.project_id, sources)
-        update_po(src_subdir, options.project_id, langs)
-    elif options.command == 'build':
-        if langs:
-            build_translations(src_subdir, build_subdir, langs)
-    elif options.command == 'install':
-        install_dir = os.path.join(os.environ['MESON_INSTALL_PREFIX'], options.install_dir)
-        if langs:
-            build_translations(src_subdir, build_subdir, langs)
-            merge_translations(build_subdir, abs_sources, langs)
-        install_help(src_subdir, build_subdir, sources, media, langs, install_dir,
-                     destdir, options.project_id, options.symlinks)
-    return 0


Test patch:
diff --git a/test cases/frameworks/13 yelp/help/C/index2.page b/test cases/frameworks/13 yelp/help/C/index2.page
new file mode 100644
--- /dev/null
+++ b/test cases/frameworks/13 yelp/help/C/index2.page	
@@ -0,0 +1,8 @@
+<page xmlns="http://projectmallard.org/1.0/"
+      xmlns:its="http://www.w3.org/2005/11/its"
+      type="guide"
+      id="index2">
+  <title>
+        Hello!
+  </title>
+</page>
diff --git a/test cases/frameworks/13 yelp/help/C/index3.page b/test cases/frameworks/13 yelp/help/C/index3.page
new file mode 100644
--- /dev/null
+++ b/test cases/frameworks/13 yelp/help/C/index3.page	
@@ -0,0 +1,8 @@
+<page xmlns="http://projectmallard.org/1.0/"
+      xmlns:its="http://www.w3.org/2005/11/its"
+      type="guide"
+      id="index3">
+  <title>
+        Hello!
+  </title>
+</page>
diff --git a/test cases/frameworks/13 yelp/help/meson.build b/test cases/frameworks/13 yelp/help/meson.build
--- a/test cases/frameworks/13 yelp/help/meson.build	
+++ b/test cases/frameworks/13 yelp/help/meson.build	
@@ -8,14 +8,14 @@ gnome.yelp('meson',
 )
 
 gnome.yelp('meson-symlink',
-   sources: 'index.page',
+   sources: 'index2.page',
    media: 'media/test.txt',
    symlink_media: true,
    languages: ['de', 'es'],
 )
 
 gnome.yelp('meson-linguas',
-   sources: 'index.page',
+   sources: 'index3.page',
    media: 'media/test.txt',
    symlink_media: false,
 )
diff --git a/test cases/frameworks/13 yelp/test.json b/test cases/frameworks/13 yelp/test.json
--- a/test cases/frameworks/13 yelp/test.json	
+++ b/test cases/frameworks/13 yelp/test.json	
@@ -6,17 +6,17 @@
     {"type": "file", "file": "usr/share/help/es/meson/media/test.txt"},
     {"type": "file", "file": "usr/share/help/de/meson/index.page"},
     {"type": "file", "file": "usr/share/help/de/meson/media/test.txt"},
-    {"type": "file", "file": "usr/share/help/C/meson-symlink/index.page"},
+    {"type": "file", "file": "usr/share/help/C/meson-symlink/index2.page"},
     {"type": "file", "file": "usr/share/help/C/meson-symlink/media/test.txt"},
+    {"type": "file", "file": "usr/share/help/es/meson-symlink/index2.page"},
     {"type": "file", "file": "usr/share/help/es/meson-symlink/media/test.txt"},
-    {"type": "file", "file": "usr/share/help/es/meson-symlink/index.page"},
-    {"type": "file", "file": "usr/share/help/de/meson-symlink/index.page"},
+    {"type": "file", "file": "usr/share/help/de/meson-symlink/index2.page"},
     {"type": "file", "file": "usr/share/help/de/meson-symlink/media/test.txt"},
-    {"type": "file", "file": "usr/share/help/C/meson-linguas/index.page"},
+    {"type": "file", "file": "usr/share/help/C/meson-linguas/index3.page"},
     {"type": "file", "file": "usr/share/help/C/meson-linguas/media/test.txt"},
+    {"type": "file", "file": "usr/share/help/es/meson-linguas/index3.page"},
     {"type": "file", "file": "usr/share/help/es/meson-linguas/media/test.txt"},
-    {"type": "file", "file": "usr/share/help/es/meson-linguas/index.page"},
-    {"type": "file", "file": "usr/share/help/de/meson-linguas/index.page"},
+    {"type": "file", "file": "usr/share/help/de/meson-linguas/index3.page"},
     {"type": "file", "file": "usr/share/help/de/meson-linguas/media/test.txt"}
   ],
   "skip_on_jobname": ["azure", "cygwin", "macos", "msys2"]