Repository: SCons/scons

Issue:
packaging tests fail on newer rpm
A set of packaging tests which build rpm packages - obviously due to availability only run on systems which have an `rpm`/`rpmbuild` command - fail on "new" rpm. Fail as in the `rpmbuild` itself fails to complete. There are multiple issues.

    test/packaging/option--package-type.py
    test/packaging/rpm/cleanup.py
    test/packaging/rpm/internationalization.py
    test/packaging/rpm/multipackage.py
    test/packaging/rpm/package.py
    test/packaging/rpm/tagging.py

The error looks like this:

    error: Empty %files file /tmp/testcmd.247674.0oeSTZ/rpmtemp/BUILD/foo-1.2.3/debugsourcefiles.list

I don't have a precise definition for "new", but certainly current Fedora instances (26/27/28) do this and now I recognize what is happening I know it goes back a lot further than that.  Package building in the Red Hat / Fedora / Centos world wants to build separate debug support packages by default now, and runs, behind the scenes, a script which tries to extract things, building file lists which are used, again behind the scenes, in building additional packages besides ones you thought you were asking for (that is, beyond the base package name plus any subpackages you define with `%package` directives - the specfile has no hint these extra packages are being built). All this is stuff the packaging tests in scons do not expect.  `debugsourcefiles.list`?  How to deal with that?

So you can turn off debug building. I am not clear if it is the right thing to do unconditionally; certainly the tests as written need to do so, or else be reworked a lot, risking not working on systems which do not have this rpm implementation. It could be done conditionally - other rpm settings exist in defined construction environment variables, one could add another to control debug-package generation, letting the tests avoid it but packages actually being built which might expect these extra packages can leave it undisturbed.

More fun - packages may also have unexpected contents. In the test code, the generated specfile contains a deduced files section:

    %files
    %defattr (-,root,root)
     /bin/main
(the initial space is probably a bug in the code which deduces the files). But running `rpm -qlp` on the package generated during the same test shows these contents:

    $ rpm -qlp ./foo-1.2.3-0.x86_64.rpm
    /bin/main
    /usr/lib/.build-id
    /usr/lib/.build-id/05
    /usr/lib/.build-id/05/3cbb2cec927c20f70df854a21b3cf1d17e4c34

so the test which tries an exact match with what has been generated in scons, namely the single line `"/bin/main"` still fails, even after you make fixes to make the package build.  There is a different test routine to make sure it contains that line, but could contain other things - the test can easily use this routine.

An additional problem is that there is code purporting to allow specifying an alternate build architecture, by setting a value for the (documented) `ARCHITECTURE` key in the construction envirionment. While this value is used in constructing the target filename, it is never actually given to `rpmbuild` - it has to either be a command-line argument `--target ARCHITECTURE`, or added as a line `BuildArch: ARCHITECTURE` in the specfile, scons does neither.   This is an actual scons problem, not a test problem; probably the alternate-architecture stuff (with 'noarch' being the interesting one) has only lightly been exercised.

This issue is scons version-agnostic (confirmed on 3.0.1) and python version-agnostic (confirmed on 2.7 and 3.6); not certain the rpm-build version but definitely a problem on 4.14 and the Fedora versions mentioned.

Reproducer: given a suitable Linux host such as Fedora 27, just run one of the tests above:

    python runtest.py test/packaging/rpm/package.py


Discussion:

Resolving patch:
diff --git a/src/engine/SCons/Tool/packaging/rpm.py b/src/engine/SCons/Tool/packaging/rpm.py
--- a/src/engine/SCons/Tool/packaging/rpm.py
+++ b/src/engine/SCons/Tool/packaging/rpm.py
@@ -51,10 +51,9 @@ def package(env, target, source, PACKAGEROOT, NAME, VERSION,
     if str(target[0])!="%s-%s"%(NAME, VERSION):
         raise UserError( "Setting target is not supported for rpm." )
     else:
-        # This should be overridable from the construction environment,
-        # which it is by using ARCHITECTURE=.
+        # Deduce the build architecture, but allow it to be overridden
+        # by setting ARCHITECTURE in the construction env.
         buildarchitecture = SCons.Tool.rpmutils.defaultMachine()
-
         if 'ARCHITECTURE' in kw:
             buildarchitecture = kw['ARCHITECTURE']
 
@@ -126,20 +125,18 @@ def build_specfile(target, source, env):
     """ Builds a RPM specfile from a dictionary with string metadata and
     by analyzing a tree of nodes.
     """
-    file = open(target[0].get_abspath(), 'w')
-
-    try:
-        file.write( build_specfile_header(env) )
-        file.write( build_specfile_sections(env) )
-        file.write( build_specfile_filesection(env, source) )
-        file.close()
+    with open(target[0].get_abspath(), 'w') as file:
+        try:
+            file.write(build_specfile_header(env))
+            file.write(build_specfile_sections(env))
+            file.write(build_specfile_filesection(env, source))
 
-        # call a user specified function
-        if 'CHANGE_SPECFILE' in env:
-            env['CHANGE_SPECFILE'](target, source)
+            # call a user specified function
+            if 'CHANGE_SPECFILE' in env:
+                env['CHANGE_SPECFILE'](target, source)
 
-    except KeyError as e:
-        raise SCons.Errors.UserError( '"%s" package field for RPM is missing.' % e.args[0] )
+        except KeyError as e:
+            raise SCons.Errors.UserError('"%s" package field for RPM is missing.' % e.args[0])
 
 
 #
@@ -201,7 +198,8 @@ def build_specfile_header(spec):
         'PACKAGEVERSION' : '%%define release %s\nRelease: %%{release}\n',
         'X_RPM_GROUP'    : 'Group: %s\n',
         'SUMMARY'        : 'Summary: %s\n',
-        'LICENSE'        : 'License: %s\n', }
+        'LICENSE'        : 'License: %s\n',
+    }
 
     str = str + SimpleTagCompiler(mandatory_header_fields).compile( spec )
 
@@ -211,6 +209,7 @@ def build_specfile_header(spec):
         'X_RPM_URL'           : 'Url: %s\n',
         'SOURCE_URL'          : 'Source: %s\n',
         'SUMMARY_'            : 'Summary(%s): %s\n',
+        'ARCHITECTURE'        : 'BuildArch: %s\n',
         'X_RPM_DISTRIBUTION'  : 'Distribution: %s\n',
         'X_RPM_ICON'          : 'Icon: %s\n',
         'X_RPM_PACKAGER'      : 'Packager: %s\n',
@@ -229,19 +228,33 @@ def build_specfile_header(spec):
         'X_RPM_PREFIX'        : 'Prefix: %s\n',
 
         # internal use
-        'X_RPM_BUILDROOT'     : 'BuildRoot: %s\n', }
+        'X_RPM_BUILDROOT'     : 'BuildRoot: %s\n',
+    }
 
     # fill in default values:
-    # Adding a BuildRequires renders the .rpm unbuildable under System, which
+    # Adding a BuildRequires renders the .rpm unbuildable under systems which
     # are not managed by rpm, since the database to resolve this dependency is
     # missing (take Gentoo as an example)
-#    if not s.has_key('x_rpm_BuildRequires'):
-#        s['x_rpm_BuildRequires'] = 'scons'
+    #if 'X_RPM_BUILDREQUIRES' not in spec:
+    #    spec['X_RPM_BUILDREQUIRES'] = 'scons'
 
     if 'X_RPM_BUILDROOT' not in spec:
         spec['X_RPM_BUILDROOT'] = '%{_tmppath}/%{name}-%{version}-%{release}'
 
     str = str + SimpleTagCompiler(optional_header_fields, mandatory=0).compile( spec )
+
+    # Add any extra specfile definitions the user may have supplied.
+    # These flags get no processing, they are just added.
+    # github #3164: if we don't turn off debug package generation
+    # the tests which build packages all fail.  If there are no
+    # extra flags, default to adding this one. If the user wants
+    # to turn this back on, supply the flag set to None.
+
+    if 'X_RPM_EXTRADEFS' not in spec:
+        spec['X_RPM_EXTRADEFS'] = ['%global debug_package %{nil}']
+    for extra in spec['X_RPM_EXTRADEFS']:
+        str += extra + '\n'
+
     return str
 
 #
diff --git a/src/engine/SCons/Tool/packaging/tarbz2.py b/src/engine/SCons/Tool/packaging/tarbz2.py
--- a/src/engine/SCons/Tool/packaging/tarbz2.py
+++ b/src/engine/SCons/Tool/packaging/tarbz2.py
@@ -32,7 +32,7 @@
 
 def package(env, target, source, PACKAGEROOT, **kw):
     bld = env['BUILDERS']['Tar']
-    bld.set_suffix('.tar.gz')
+    bld.set_suffix('.tar.bz2')
     target, source = putintopackageroot(target, source, env, PACKAGEROOT)
     target, source = stripinstallbuilder(target, source, env)
     return bld(env, target, source, TARFLAGS='-jc')


Test patch:
diff --git a/test/packaging/option--package-type.py b/test/packaging/option--package-type.py
--- a/test/packaging/option--package-type.py
+++ b/test/packaging/option--package-type.py
@@ -26,6 +26,9 @@
 
 """
 Test the --package-type option.
+
+Side effect: also tests that we can produce a noarch package
+by supplying the ARCHITECTURE tag.
 """
 
 import TestSCons
@@ -63,12 +66,13 @@
              X_RPM_INSTALL  = r'%(_python_)s %(scons)s --install-sandbox="$RPM_BUILD_ROOT" "$RPM_BUILD_ROOT"',
              DESCRIPTION    = 'this should be really long',
              source         = [ prog ],
-             SOURCE_URL     = 'http://foo.org/foo-1.2.3.tar.gz'
+             SOURCE_URL     = 'http://foo.org/foo-1.2.3.tar.gz',
+             ARCHITECTURE   = 'noarch'
             )
 """ % locals())
 
 src_rpm = 'foo-1.2.3-0.src.rpm'
-machine_rpm = 'foo-1.2.3-0.%s.rpm' % SCons.Tool.rpmutils.defaultMachine()
+machine_rpm = 'foo-1.2.3-0.noarch.rpm'
 
 test.run(arguments='package PACKAGETYPE=rpm', stderr = None)
 
diff --git a/test/packaging/rpm/cleanup.py b/test/packaging/rpm/cleanup.py
--- a/test/packaging/rpm/cleanup.py
+++ b/test/packaging/rpm/cleanup.py
@@ -28,6 +28,7 @@
 Assert that files created by the RPM packager will be removed by 'scons -c'.
 """
 
+import os
 import TestSCons
 import SCons.Tool.rpmutils
 
@@ -47,13 +48,8 @@
 rpm_build_root = test.workpath('rpm_build_root')
 
 test.subdir('src')
-
-test.write( [ 'src', 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-  return 0;
-}
-""")
+mainpath = os.path.join('src', 'main.c')
+test.file_fixture(mainpath, mainpath)
 
 test.write('SConstruct', """
 env=Environment(tools=['default', 'packaging'])
diff --git a/test/packaging/rpm/explicit-target.py b/test/packaging/rpm/explicit-target.py
--- a/test/packaging/rpm/explicit-target.py
+++ b/test/packaging/rpm/explicit-target.py
@@ -28,6 +28,7 @@
 Test the ability to create a rpm package from a explicit target name.
 """
 
+import os
 import TestSCons
 
 _python_ = TestSCons._python_
@@ -44,13 +45,8 @@
 rpm_build_root = test.workpath('rpm_build_root')
 
 test.subdir('src')
-
-test.write( [ 'src', 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-  return 0;
-}
-""")
+mainpath = os.path.join('src', 'main.c')
+test.file_fixture(mainpath, mainpath)
 
 test.write('SConstruct', """
 import os
diff --git a/test/packaging/rpm/internationalization.py b/test/packaging/rpm/internationalization.py
--- a/test/packaging/rpm/internationalization.py
+++ b/test/packaging/rpm/internationalization.py
@@ -33,7 +33,6 @@
 
 import os
 import SCons.Tool.rpmutils
-
 import TestSCons
 
 _python_ = TestSCons._python_
@@ -52,12 +51,7 @@
 #
 # test INTERNATIONAL PACKAGE META-DATA
 #
-test.write( [ 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-  return 0;
-}
-""")
+test.file_fixture('src/main.c', 'main.c')
 
 test.write('SConstruct', """
 # -*- coding: utf-8 -*-
@@ -123,13 +117,8 @@
 #
 # test INTERNATIONAL PACKAGE TAGS
 #
-
-test.write( [ 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-  return 0;
-}
-""")
+mainpath = os.path.join('src', 'main.c')
+test.file_fixture(mainpath)
 
 test.write( ['man.de'], '' )
 test.write( ['man.en'], '' )
diff --git a/test/packaging/rpm/multipackage.py b/test/packaging/rpm/multipackage.py
--- a/test/packaging/rpm/multipackage.py
+++ b/test/packaging/rpm/multipackage.py
@@ -47,13 +47,8 @@
 rpm_build_root = test.workpath('rpm_build_root')
 
 test.subdir('src')
-
-test.write( [ 'src', 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-  return 0;
-}
-""")
+mainpath = os.path.join('src', 'main.c')
+test.file_fixture(mainpath, mainpath)
 
 test.write('SConstruct', """
 import os
@@ -109,8 +104,10 @@
 test.must_exist( src_rpm2 )
 
 test.must_not_exist( 'bin/main' )
-test.fail_test( not os.popen('rpm -qpl %s' % machine_rpm).read()=='/bin/main\n')
-test.fail_test( not os.popen('rpm -qpl %s' % src_rpm).read()=='foo-1.2.3.spec\nfoo-1.2.3.tar.gz\n')
+out = os.popen( 'rpm -qpl %s' % machine_rpm).read()
+test.must_contain_all_lines( out, '/bin/main')
+out = os.popen( 'rpm -qpl %s' % src_rpm).read()
+test.fail_test( not out == 'foo-1.2.3.spec\nfoo-1.2.3.tar.gz\n')
 
 test.pass_test()
 
diff --git a/test/packaging/rpm/package.py b/test/packaging/rpm/package.py
--- a/test/packaging/rpm/package.py
+++ b/test/packaging/rpm/package.py
@@ -46,13 +46,8 @@
 rpm_build_root = test.workpath('rpm_build_root')
 
 test.subdir('src')
-
-test.write( [ 'src', 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-  return 0;
-}
-""")
+mainpath = os.path.join('src', 'main.c')
+test.file_fixture(mainpath, mainpath)
 
 test.write('SConstruct', """
 import os
@@ -88,8 +83,10 @@
 test.must_exist( machine_rpm )
 test.must_exist( src_rpm )
 test.must_not_exist( 'bin/main' )
-test.fail_test( not os.popen('rpm -qpl %s' % machine_rpm).read()=='/bin/main\n')
-test.fail_test( not os.popen('rpm -qpl %s' % src_rpm).read()=='foo-1.2.3.spec\nfoo-1.2.3.tar.gz\n')
+out = os.popen( 'rpm -qpl %s' % machine_rpm).read()
+test.must_contain_all_lines( out, '/bin/main')
+out = os.popen( 'rpm -qpl %s' % src_rpm).read()
+test.fail_test( not out == 'foo-1.2.3.spec\nfoo-1.2.3.tar.gz\n')
 
 test.pass_test()
 
diff --git a/test/packaging/rpm/src/main.c b/test/packaging/rpm/src/main.c
new file mode 100644
--- /dev/null
+++ b/test/packaging/rpm/src/main.c
@@ -0,0 +1,5 @@
+
+int main( int argc, char* argv[] )
+{
+  return 0;
+}
diff --git a/test/packaging/rpm/tagging.py b/test/packaging/rpm/tagging.py
--- a/test/packaging/rpm/tagging.py
+++ b/test/packaging/rpm/tagging.py
@@ -30,7 +30,6 @@
 
 import os
 import SCons.Tool.rpmutils
-
 import TestSCons
 
 _python_ = TestSCons._python_
@@ -50,13 +49,8 @@
 # Test adding an attr tag to the built program.
 #
 test.subdir('src')
-
-test.write( [ 'src', 'main.c' ], r"""
-int main( int argc, char* argv[] )
-{
-return 0;
-}
-""")
+mainpath = os.path.join('src', 'main.c')
+test.file_fixture(mainpath, mainpath)
 
 test.write('SConstruct', """
 import os
@@ -91,9 +85,12 @@
 machine_rpm = 'foo-1.2.3-0.%s.rpm' % SCons.Tool.rpmutils.defaultMachine()
 
 test.must_exist( machine_rpm )
+out = os.popen('rpm -qpl %s' % machine_rpm).read()
+test.must_contain_all_lines( out, '/bin/main')
+
 test.must_exist( src_rpm )
-test.fail_test( not os.popen('rpm -qpl %s' % machine_rpm).read()=='/bin/main\n')
-test.fail_test( not os.popen('rpm -qpl %s' % src_rpm).read()=='foo-1.2.3.spec\nfoo-1.2.3.tar.gz\n')
+out = os.popen('rpm -qpl %s' % src_rpm).read()
+test.fail_test( not out == 'foo-1.2.3.spec\nfoo-1.2.3.tar.gz\n')
 
 expect = '(0755, root, users) /bin/main'
 test.must_contain_all_lines(test.read('foo-1.2.3.spec',mode='r'), [expect])