Cross-compilation in pkgsrc
Taylor ‘Riastradh’ Campbell
campbell@mumble.net
riastradh@NetBSD.org

AsiaBSDcon 2015
Tokyo, Japan
March 15, 2015

pkgsrc: portable package build system
I https://www.pkgsrc.org/
I Framework for building third-party software on Unix-like

operating systems.
I > 15, 000 packages.
I Supported platforms:

NetBSD (first platform, based on mid-’90s FreeBSD ports)
GNU/Linux, GNU/kFreeBSD
I FreeBSD, OpenBSD, DragonflyBSD, MirBSD
I Haiku, MINIX 3
I Solaris / SmartOS / illumos
I OS XI IRIX, AIX, OSF/1, HP-UX, QNX, Cygwin
IIpkgsrc: portable package build system
I https://www.pkgsrc.org/
I Framework for building third-party software on Unix-likeoperating systems.
I > 15, 000 packages.
I Supported platforms:

NetBSD (first platform, based on mid-’90s FreeBSD ports)
GNU/Linux, GNU/kFreeBSD
I FreeBSD, OpenBSD, DragonflyBSD, MirBSD
I Haiku, MINIX 3
I Solaris / SmartOS / illumos
I OS XI IRIX, AIX, OSF/1, HP-UX, QNX, Cygwin
II

I Works unprivileged, so you can use it in your home directoryon a server you don’t administer.

Anatomy of a pkgsrc package

I DESCR – Human-readable description.
I Makefile – Machine-readable description.

Tells where to download source code.
Rules for how to configure, build, install.
I Etc.
II

I distinfo – Names, sizes, and hashes of source distribution.

Provides cryptographic integrity check.
I PLIST – Packing list: lists files installed by package.
I /usr/pkg/etc/mk.conf – Site configuration for package

options.

pkgsrc example: security/nettle, part 1# $NetBSD: Makefile,v 1.13 2013/11/26 09:22:19 martin Exp $DISTNAME= nettle-2.7.1
PKGREVISION= 1
CATEGORIES= devel security
MASTER_SITES= http://www.lysator.liu.se/~nisse/archive/ \ftp://ftp.lysator.liu.se/pub/security/lsh/
MAINTAINER= pkgsrc-users@NetBSD.org
HOMEPAGE= http://www.lysator.liu.se/~nisse/nettle/
COMMENT= Cryptographic library
LICENSE= gnu-lgpl-v2.1
USE_LANGUAGES= c
USE_LIBTOOL= yes
USE_TOOLS+= gm4 gmake
GNU_CONFIGURE= yes
SET_LIBDIR= yes
CONFIGURE_ARGS+= --disable-openssl --disable-sharedpkgsrc example: security/nettle, part 2.include "../../mk/bsd.prefs.mk".if !empty(USE_CROSS_COMPILE:M[yY][eE][sS])
CONFIGURE_ENV+= CC_FOR_BUILD=${NATIVE_CC:Q}.endif
INFO_FILES= yes
TEST_TARGET= check
PKGCONFIG_OVERRIDE= hogweed.pc.in
PKGCONFIG_OVERRIDE+= nettle.pc.in.include "../../devel/gmp/buildlink3.mk".include "../../mk/bsd.pkg.mk"Building and installing a package1# which socat
socat not found# cd /usr/pkgsrc/net/socat# bmake install=> Bootstrap dependency digest>=20010302: found digest-20121220=> Fetching socat-1.7.2.4.tar.gz...
=> Checksum SHA1 OK for socat-1.7.2.4.tar.gz...
===> Installing dependencies for socat-1.7.2.4...
=> Tool dependency checkperms>=1.1: found checkperms-1.11=> Full dependency readline>=6.0: found readline-6.3nb3...
=> Creating binary package /tmp/.../socat-1.7.2.4.tgz===> Install binary package of socat-1.7.2.4# which socat/usr/pkg/bin/socat
1

On NetBSD, can use base system’s make, but everywhere else we bootstrap
devel/bmake for pkgsrc.

Binary packages: build once, install many times

I Building from source is necessary: verify source, audit

programs, modify, etc.
I Building from source is slow: run compiler on lots of sourcecode.
I Do it once, save the result, install binary packages after.

builder# cd /usr/pkgsrc/net/socat
builder# bmake package
client# PKG_PATH=/nfs/builder/usr/pkgsrc/packages
client# export PKG_PATH
client# pkg_add socat

Binary package bulk buildsI NetBSD provides binary packages for NetBSD on manyarchitectures2 .
I Joyent provides binary packages for OS X3 and illumos4 .
I I build binary packages for my own machines.
I You can too!

2

ftp://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/
http://www.perkin.org.uk/pages/pkgsrc-binary-packages-for-osx.html
4
http://www.perkin.org.uk/pages/pkgsrc-binary-packages-for-illumos.html
3

Cross-compiling NetBSD

I Every NetBSD build is a cross-build.
I build.sh

tools builds cross-toolchain.

kernel=GENERIC distribution builds NetBSD
with the cross-toolchain.

I build.shCross-compiling pkgsrc
I Use NetBSD build.sh
started.5

tools distribution to get

I USE CROSS COMPILE=yes
I MACHINE ARCH=powerpc
I TOOLDIR=/usr/obj.evbppc/tooldir.NetBSD-6.1.amd64
I CROSS DESTDIR=/usr/obj.evbppc/destdir.evbppc# uname -m
amd64# cd /usr/pkgsrc/net/socat# bmake package...
# cd /usr/pkgsrc/packages.powerpc/All# pkg_info -Q MACHINE_ARCH socat-1.7.2.4.tgz
powerpc
5

See doc/HOWTO-use-crosscompile for details.

Dependencies

I Some packages depend on other packages.
Itor program uses libevent library at run-time.
IIII

net/tor also build-depends on devel/libevent.

Compiling libxcb requires turning XML into C header files
with xsltproc.
II

net/tor depends on devel/libevent.

Compiling tor program requires event.h at build-time

x11/libxcb tool-depends on textproc/xsltproc.

Also bootstrap-depends, like tool-depends but for parts of the pkgsrc
infrastructure.

Cross-compiling dependencies

I Use Intel Xeon to build x11/xterm, run on yourpowerpc-based thin client.
I x11/xterm must be built for MACHINE ARCH=powerpc.
I x11/xterm depends on x11/libxcb6 .
I

x11/libxcb must be built for MACHINE ARCH=powerpc.

I x11/libxcb tool-depends on textproc/xsltproc.
I6

textproc/libxsltproc must be built for
MACHINE ARCH=x86 64.

Via x11/libX11.

Build-depends vs tool-depends

I Both build-depends and tool-depends need to exist atbuild-time.
I Build-depends are cross-built and installed into/usr/obj.evbppc/destdir.evbppc/usr/pkg/...
I

Example: C libraries, needed for linker.

I Tool-depends are natively built and installed into/usr/pkg/...
I
I

Example: xsltproc, cross-compiler.
When built, TARGET ARCH set to cross-compilation target.

Complications part 1: mixing up build-depends andtool-depends

I Originally, pkgsrc had only build-depends.
I x11/libxcb build-depended on textproc/xsltproc.
I Solution: change build-depends to tool-depends whereappropriate.

Complications part 2: package builds tools internally

I Some packages depend on external tools like x11/libxcb

depends on textproc/xsltproc.
I Others use internal tools, like security/nettle above.
I These try to use CC, which may be powerpc--netbsd-gcc

for cross-compilation.
I Can’t run the result on x86!
I Solution: set CC FOR BUILD, maybe patch package to use it

instead.
.if !empty(USE_CROSS_COMPILE:M[yY][eE][sS])
CONFIGURE_ENV+= CC_FOR_BUILD=${NATIVE_CC:Q}.endifComplications part 3: file existence tests
I Package wants to know whether /dev/urandom will exist

when run.
I Uses GNU autoconf to ask whether /dev/urandom exists

now, when built.
I Build machine and target system may be different!
I But we know /dev/urandom will exist.
I Solution: tell autoconf up front.

.if !empty(USE_CROSS_COMPILE:M[yY][eE][sS]).if ${OPSYS} == "NetBSD" || ${OPSYS} == "OpenBSD" || ..
CONFIGURE_ENV+= ac_cv_file__dev_urandom=yes.endif.endifComplications part 3’: file existence tests in pkgsrc

I From x11/libdrm in the past:.if !exists(/usr/include/sys/atomic.h)# libdrm won’t find system atomic ops, use a package.
. include "../../devel/libatomic_ops/buildlink3.mk".endif
I Solution: don’t look in /usr/include — look in/usr/obj.evbppc/destdir.evbppc:.if !exists(${CROSS_DESTDIR}/usr/include/sys/atomic.h)# libdrm won’t find system atomic ops, use a package.
. include "../../devel/libatomic_ops/buildlink3.mk".endifComplications part 4a: configure run-tests

I Similar to file existence tests.
I Program wants to know sizeof(long) at compile-time.
I Compiles a test program to print it, runs test program.
I Can’t do that if building on 64-bit amd64 for 32-bit powerpc!
I Solution: binary search with compile-time assertions using

cross-compiler.
I (Yes, seriously! GNU autoconf supports this withAC CHECK SIZEOF.)

Complications part 4b: configure run-tests

I Some are harder to replace.
I Tell the answers up front, maybe with patches.
I From shells/zsh:.if !empty(USE_CROSS_COMPILE:M[yY][eE][sS]).if ${OPSYS} == "NetBSD"
CONFIGURE_ENV+= zsh_cv_shared_environ=yes
CONFIGURE_ENV+= zsh_cv_shared_tgetent=yes
CONFIGURE_ENV+= zsh_cv_shared_tigetstr=yes
CONFIGURE_ENV+= zsh_cv_sys_dynamic_execsyms=yes.endif.endifComplications part 5: problem children

I Some packages go to great effort to resist cross-compilation.
IIPerlPython

I Workaround: just build on your powerpc thin client and shipbinary packages back to x86 build machine to continue.
I (Solution: chainsaws and rototillers. Fix the build systems!7 )7It can be done: OpenWrt does it, with a lot of work. If you would like tohelp adapt their approach to pkgsrc, talk to me!

Related work
I OpenWrt: cross-compiled packages for Linux-based network

appliances.
Linux-only.
Not general-purpose package system.
I Much smaller than pkgsrc.
II

I distcc: run pkgsrc on thin client, run compiler remotely on

x86 build machine.
Complex to set up: many moving parts (literally).
Hard to parallelize.
I Compiler is a big part but not all of run-time.
II

I FreeBSD ports: run native compiler in user-mode emulator.

Many moving parts (figuratively).
Emulators are slow.
I Less clean separation between host and target.
IIFuture work
I Cross-OS compilation. Use SmartOS x86 cloud cluster tobuild for MACHINE PLATFORM=NetBSD-7.0-powerpc.
I User interface improvements.

Can’t do bmake package MACHINE ARCH=powerpc for stupid
reasons.
I (When we switch to MACHINE PLATFORM reasons will go away.)
I Setting up cross-compiling requires a manual step to workaround broken GNU libtool.

II Bulk builds.
Ipbulk doesn’t understand build-depends vs tool-depends.

I Unprivileged builds for privileged installs.
II

Native and cross packages must both point at /usr/pkg.
(Unprivileged builds for unprivileged installs work fine — not a
problem with privileges, just with different paths.)

Thank you!

Questions?