Bulk building in the many core era
Jörg Sonnenberger ⟨joerg@NetBSD.org⟩
February 4, 2017

Abstract

As a result both performance and correctness issues
have been the major design guide lines for pbulk,
with a general preference for the latter.

pbulk, the current generation of the pkgsrc bulk
build infrastructure, was created during the GSoC
2007 and remained mostly unchanged. Increased
processing power, aﬀordable parallelization, significantly more RAM and SSDs for storage have allchanged the environment. This paper investigatesdiﬀerent configuration choices and their performance and management impact. The current development towards looser coupling between clientsand the build master are presented as well as thenecessary changes for a secure integration of package signatures. The impact of the changes is quantified for diﬀerent native and virtualised setups.
The measurements are used to identify improvements for the NetBSD kernel and pkgsrc.

1

In the past decade, the computing environment
for bulk building has changed significantly. The
disk I/O associated with preparing and tearing
down the per-package build environment has always been a major time factor, but the recent availability of multiple GB RAM per core and SSDs as
backing store has shifted the hot spots to other areas.
The budget requirements for a bulk build running entirely in RAM would have required at leasta five digit budget per machine; today typical oﬀthe-shelf dedicated servers in the mid-price range
already fit the bill. If a build node had two sockets
with a dual core CPU a decade ago, they would
now have one or two sockets with eight to sixteen
cores each.

Introduction

The pbulk framework for bulk building was created during the GSoC 2007 to address a number
Exploiting the available resources eﬃciently asof concerns with the old bulk build infrastructure,
well as building cost eﬀective systems requires
including:greater care than before. This paper investigates• No support for building diﬀerent packages con- the build time of several natural system configurations. Based on the initial findings, a number ofcurrently on the same machine.
less obvious refinements are compared in order to• No support for distributing the build across mitigate severe contention issues. The results pin
point issues as well as provide estimations for permultiple machines.
formance gains from future fixes.
• Diﬃcult to debug build problems due to partial cleanup between builds.

The paper starts with an overview of the pbulk
framework and how it extends to a build cluster.
• Inconsistent rules for when a package has to Benchmarks for builds of NetBSD/AMD64 follow
with configurations ranging from native NetBSD
be rebuilt due to dependency changes.
over para-virtualised NetBSD under Xen to fully• Inconsistent handling of redistribution of prob- virtualised configurations under Xen and SmartOS.
lematic packages.
Finally, the on-going work is presented.
1

2

Design overview for pbulk

the scan phase is responsible for approx. 5% of thetotal build time. For incremental builds e.g. on aquarterly branch, few rebuilds are typically necessary and the scan phase can dominate the totalbuild time. The scan time can be further reducedin those situations by enabling reuse of old scan results2 . This will skip the extraction if Makefile (orany of the files included by it) has a newer modification time than the scan result from the lastrun. This option is not enabled by default as it cangive false results if the base system or the build
configuration changes. As such, it requires manualintervention from the user to clean the stale files inthose situations.
After the extraction of the packages and theirdependencies, precise dependencies according to
the rules pkg add uses during installation are computed. This means that discrepancies between scanand build phase show up as errors due to missing
dependencies in the latter phase. Some diagnostic data is also provided for maintainers, e.g. if thepreferred dependency is not found in the expected
location. This can happen when packages share thesame name root like PHP versions, but also whenalternative patterns are used (e.g. to depend on either an OSS or ALSA plug-in to be installed).
One important consideration for the scan phaseis that it is eﬀectively read-only. No packages haveto be installed and no temporary files in the build
area are created. This allows sharing the same virtual machine or chroot with multiple scan jobs. As
the memory requirements per scan jobs are also
very moderate, it is easy to run one scan job per
core.
The third phase is responsible for building theindividual binary packages. A build job in a virtual machine or chroot will first decide whether apre-existing binary package is good enough to bereused. This check normally includes:

The pbulk framework separates the build into four
diﬀerent phases:
1. Build preparation
2. Tree scanning
3. Package building
4. Post-processing, reporting and uploading
The first phase prepares everything for the newbulk build. This includes the removal of leftovers
from earlier builds. Depending on the specificsetup, it can also be used to create chroots, spinup virtual machines or synchronize pkgsrc trees.
The second phase is responsible for determining
the list of packages and their dependencies. The
tree scanning exists in two variations: partial andcomplete tree. The former option is typically used
when packages for a specific set of machines areneeded. The user provides a list of directories thatshould be built and the scan process will iteratively
add dependencies as necessary. For the complete
tree, a list of all directories is extracted from the
top and second level Makefiles and then processed.
While processes similar to the tree scanning canbe found in build systems other than pkgsrc, it
is normally much easier. Dependencies in pkgsrc
are often not specified in the main Makefile, but
distributed across a tree of included buildlink3.mk
files. Since pkgsrc supports picking base system dependencies on various platforms, additional computations are often needed. If a package like devel/cmake wants to use the Curses library, pkgsrc has
to check if the base system provides a compatible
implementation. If it finds one, no additional dependency has to be installed, otherwise a fallback
to devel/ncurses is added. Other complications are
optional features in one package changing the behavior of dependees, like the list of visible headersand libraries.
To mitigate the overhead of the build option
extraction, pkgsrc recently gained an option1 forcaching the result of those build option extractions.
For a typical bulk build, this reduces the time spentin the second phase by 50%. For a complete fromscratch bulk build without reusing earlier packages,• The exact dependencies for the current build
match the state of the old build.
• The recorded RCS IDs of important files likethe main Makefile or distinfo match the stateof the pkgsrc tree.
• The binary package is newer the packages it
requires.

1 Added 2016-12-18, set PBULK CACHE DIRECTORY
in mk.conf and create/empty it in the preparation step.

2 Added 2012-11-23, set reuse scan results to yes inpbulk.conf.

2

If all checks pass, no further work is necessary. If
there is no pre-existing binary package or it is considered out-dated, a full package build is started.
The build job will first tear down the package prefix (e.g. /usr/pkg per default on NetBSD).
The actual build processes in stages:clients can be added on-demand. The protocol currently provides no mechanism to restart a failedbuild of an individual package, even if it is a side
eﬀect of a machine crash.

3

Build cluster infrastructure

1. pre-cleanup
The pbulk framework depends on a number ofshared resources for eﬃcient builds:

2. installation of listed dependencies• distribution files and patches,

3. fetching distribution sources and patches andmatching the checksums against distinfo• logs for failing builds,

4. configuration• binary packages and5. build• the pkgsrc tree itself.

6. installation to staging directory

The past recommendation for bulk builders wasto use shared file systems like null mounts or NFS.
While this works reasonably, it is also regular asource of headaches in some cases. For the distribution files, if two build jobs try to fetch the same
file at the same time, they can stumble across eachother, making one of them fail. As the build order of variant packages like Python modules tend
to schedule diﬀerent variants of the same module
near each other, this has been a regular problem.
In the case of the log directory on the other hand,
NFS is known to create build failures when intermediate file operations fail like attempt at locking.
This section includes a new set of recommendations based on those experiences.

7. creation of binary package
8. test installation
The output of the stages is logged separate for easier debugging and preserved for failing builds. Depending on the result, either a fresh binary package
exists or build log for the problems. After notifying
the build scheduler, the next build is executed.
The fourth and final phase is responsible forbuilding the SHA512 index over all packages as wellas uploading the results (if applicable) and providing a human and machine readable report.
Both scan phase and build phase provide a basic scheduler mode to process jobs sequentiallyfrom an integrated client. This mode is functionally very similar to the previous generation of thebulk build system. It is reasonable when doing asmall set of packages or when operating on constrained hardware (e.g. with only one or two cores).
Pre-populating a local copy of distribution sources
should still be considered to avoid the delays fromfetching during the build from the internet.
Alternatively, scan and build jobs can run in parallel in their respective phase, either across a diﬀerent chroots or across diﬀerent (virtual) machines.
The framework provides a pull-based scheduler. A
hook is provided to execute a script after the master process is ready and this in turn will start thescan or build clients via chroot3 or ssh. Additional

3.1

Distribution files and patches

For a single machine build, having a local directorynull mounted into each build chroot is still a reasonable low-overhead option. The danger of the occasional failures has been mentioned. Sharing acrossmachines with NFS is no longer recommended.
A scalable and reliable mechanism is to use a local mirror with automatic upload of new files. For
this purpose, MASTER SITE OVERRIDE should
point to the local HTTP or HTTPS server. It
can run on the same machine as the build master, but any other local server works as well. The
second part of the puzzle is POST FETCH HOOK,a script that is called with path of the newly ob3 Added 2016-12-18, by specifying an absolute path astained file and the URL the file was obtained from.
client in pbulk.conf.
The author uses WebDAV for uploading new files,3

as requires less configuration than sftp or scp only
SSH upload. An example script can be foundin listing 18. Finally, DISTDIR can be set to$WRKDIR/.distdir, so that files are removed automatically once the build job is finished. In thiscase, use unprivileged checksum 4 should be set to‘yes‘ in pbulk.conf as well.
Some package have distribution files which can’t
be obtained automatically. A typical example is
the EULA for Oracle’s JRE and JDK builds. If
the user uploads those files to the local mirror byhand, the IGNORE INTERACTIVE FETCH option can be used to skip the check, so that MASTER SITE OVERRIDE is actually tried.

3.2

pbulk.conf. The primary use case is allowing package signing from a separate environment, so thatkeying material is never accessible to the builds.
The basic approach is illustrated by listing 10, but
the given script shouldn’t be used in production
due to the obvious security concerns.
The long-term plan is to ingrate the uploading
into the pbulk protocol to reduce the number ofmoving parts as well as to allow tighter constraints
on the package content. The build master knows
the package that is supposed to be signed and can
detect fancy attempts for creating signed core packages when a minor source has been trojaned. Thishas been deferred until the infrastructure side formulti-package builds (e.g. one build creating multiple sub-packages) has been resolved. The work wasstarted by Leonardo Taccari’s GSoC 2016 project.

Build logs

Similar to distribution files, having a directory null
mounted into each build chroot is reasonable forsingle-machine builds. The file system is primarymodified by appending to files, so lock contention
not a big factor. For cross-machine machine builds,
it is recommended to use either scp/sftp based
uploads or WebDAV. NFS comes with a noticeable performance penalty and the earlier mentioned
mysterious error cases. The sync buildlog option5
in pbulk.conf can be pointed to either a shell script
or function to provide the necessary functionality.
An example can be found in listing 7.
The long-term plan is to integrate the uploading of the build logs into pbulk itself to reduce thenumber of moving parts.

3.3

3.4

Sharing of the pkgsrc tree

The pkgsrc tree is read-only during the build andespecially during the scan phase highly contended.
The combination of those properties make it attractive to distribute a copy to all build nodes e.g. via
rsync in the first phase of the build. Even for diskless clients like building on small ARM boards, it isoften better to have a copy of the pkgsrc tree on a
local file system over iSCSI compared to plain NFS,
since NFS provides only limited caching. Multiple scan/build jobs on a single node can share thesame copy via null mounts. No noticeable lock contention has been observed on NetBSD at this point
for shared pkgsrc trees. The client prepare chroot
and client prepare action options7 can be used forthis purpose.

Binary packages

The constraints for binary packages are a bit different. Unlike build logs or distribution files, thebinary packages are accessed directly as part of thecheck for skipping rebuilds. It is diﬃcult to reproduce this logic for packaged served via HTTPS.
As such, a shared file system like NFS is unavoidable for the cluster case. Since the check uses timestamps, use of NTP is strongly recommended forall build machines.
It is possible to use a read-only NFS file system
in combination with the sync package option6 in

4Comparative benchmarks ofdiﬀerent configurations

In this section, diﬀerent configurations will be evaluated for a whole tree from-scratch bulk build.
Common base configuration is a NetBSD current
on AMD64 from mid-2016 with clang as systemcompiler. Some of the Xen and SmartOS based
tests use a newer kernel from around November
2016 for technical reasons, but it is believed to
not have any significant impact on the build times.

4 Added 2015-09-08
5 Added 2015-09-13

7 Added 2016-12-18

6 Added 2015-09-13

4

are not relevant for package builds. The resulting
tmpfs requires 518MB per chroot.
Timing results can be found in figure 1. The dif1
97 2667
102 2111
ference in the scan time is within the noise. This2
113 2862
119 2211
was unexpected as the scan phase normally hits
3
112 2862
98 2211the file system and memory subsystem much more
Avg
107 2797
106 2178 -1% -22%
often due to the high fork/exec rate. An easy exσ
4.2
53
5.3 27.2
planation like “the scan phase is read-only” doesn’t
work as the file system itself is read-only in bothFigure 1: Scan and total build time (in minutes)
cases. The lock protocol for nullfs clearly needs tofor nullfs root vs tmpfs root
be investigated.
The tmpfs root numbers will be used as base lineforall further comparisons
A number of serializing packages are explicitly
marked as broken like math/openaxiom, as theywould skew numbers artificially
4.2 Para-virtualisation with Xen
Test platform is a dual Intel Xeon E5-2630v3
with 64GB RAM and SSD for persistent storage. The first alternative to running NetBSD on bareHyper-threading is active. Persistent file systems metal hardware is para-virtualisation with Xen.
use FFS with WAPBL enabled. MAKE JOBS is This is used on the build cluster of The NetBSD
set to 16, developer checks are enabled. The use of Foundation for the oﬃcial bulk builds, so the overhead here is direct interest. The reference configcwrappers is enabled as well.
All distribution files are served by a HTTP server uration is a NetBSD DOM0 configured with 1GB
in the local network, connected by Gigabit Ether- RAM running under Xen 4.6.3 as hyper-visor In
net. Build logs are pushed via rsync similar to list- the single DOMU case, one NetBSD guest is usinging 7. Packages are written directly to nullfs or the remaining 62GB RAM and 31 cores. For the
NFS (for multi-VM configurations). The state of four DOMUs case, all DOMUs have 15GB RAM
the pkgsrc tree is identical across all builds. Para- assigned and eight cores, except one DOMU with
virtualised Xen builds have a couple of additional only seven cores.
Timing results can be found in figure 2. The
failures related to Emacs, but this is believed to be
single DOMU case was expected from unpublisheda minor impact.
benchmarks by the author in the 2007-2008 timeframe. The real surprise was the four DOMUs case
4.1 Base line with native kernel
though. An investigation with lockstat showed avery high contention on the page queues. When
The original configuration used a FFS file system
splitting the system, this contention has cut down
for the root template of the chroots. Each chroot
significantly and compensates for the system call
has a read-only null mount of this root templateand context switch overhead of Xen. Since thiswith tmpfs instances mounted over it for /etc, /var,
overhead is still present in the four DOMUs num/tmp, /usr/pkg and the work area. Eight chrootsbers, they provide a good lower bound for the exare used for build jobs and each chroot played hostpected gains from more granular page queue lockto four scan jobs. Two additional chroots are proing.
visioned for the build master and potential testing,but are otherwise not used during the builds.
Investigations of lockstat output suggested a high 4.3 Full virtualised builds with Xencontention of the vnode for the top level of the
HVM and SmartOS-KVM
root template. This lead to the first variation witha read-only tmpfs copy of the root template for The reference configuration for Xen-HVM uses aevery chroot. Root template contains the ‘base’, NetBSD DOM0 configured with 1GB RAM and‘comp’, ‘etc’, ‘games’, ‘misc’ and ‘text’ sets with a four HVM guests with 15GB RAM each. The
small pruning by removing the lib* pic.a files which hyper-visor is Xen 4.6.3. The guests all have eightRunnullfs root
scan total

tmpfs root
scan total

Diﬀerencescan total

5

Run

Base line
scan total

1 DOMUscan total

1
2
3
Avg
σ

102
119
98
106
4.2

185
188
186
186.3
0.72

2111
2211
2211
2178
53

2897
2894
2909
2900
3.7

Diﬀerence
scan
total

+75%

+33%

4 DOMUs
scan total
34
33
33
33
0.27

1917
1913
1909
1913
1.9

Diﬀerencescan total

-69%

-12%

Figure 2: Scan and total build time (in minutes) for base line vs 1 DOMU vs 4 DOMUs
Run

Base line
scan total

Xen-HVM
scan total

1
2
3
Avg
σ

102
119
98
106
4.2

39
36
38
38
0.72

2111
2211
2211
2178
53

2880
2748
2927
2852
44

Diﬀerence
scan
total

-65%

+31%

SmartOS KVM
scan
total
35
48
43
42
3.1

2721
2727
2774
2741
14

Diﬀerence
scan
total

-61%

+26%

Figure 3: Scan and total build time (in minutes) for base line vs Xen-HVM vs SmartOS KVM

5cores assigned each, except one with only sevencores.

Work in progress

There are currently five on-going projects for the
pbulk framework:
In the SmartOS case, the KVM guests were assigned 14GB each. A stable build was not possible
with 4GB RAM reserved for the SmartOS kernel,even with manual tuning to limit the size of the
ARC buﬀer. NetBSD uses virtio for both disk andnetwork I/O under KVM.

1. Integration of a better process monitor for
BSD systems
2. Rebuilding the storage mechanism by moving
to SQLite
3. Extending the communication protocol to allow retrying failed individual builds.

Timing results can be found in figure 3. The
scan time results can be readily explained by thepage queue contention as discussed in the last section. Both systems have comparable results with
SmartOS winning over all. The diﬀerences canlikely be attributed to optimizations in the pagetable trapping or context switching, but has notbeen analyzed in detail. The cost of complex device emulation vs the simpler virtio protocols isa factor that may favor SmartOS for the longer
builds though. The virtualisation overhead on sixth
generation Xeons with Extended Page Tables andother features is significantly lower compared to thethird generation Xeons, where a bulk build took
over 100% longer in Xen-HVM compared to native.

4. Extending the communication protocol to separate up-to-date check and build jobs.
5. Investigate metrics for resource consumption
and build load for improved scheduling.
Long-term a move to a new HTTP(S)-based communication protocol is planned, but no details exist
yet.

5.1

Process monitoring with kqueue

Build jobs sometimes hang for a long time. There
are two common problematic situations:61. Infinite loops resulting in wasted CPU time.

Switching pbulk to using SQLite allows further
developer and simplifies improvements to the data
2. Dead locks and other conditions where no model. When the scan phase was extended to allow
progress is made.
reusing old results, it had the side eﬀect of increasThe first category can be solved easily with process ing the data volume by a factor of six. This directly
limits on the CPU time, but this doesn’t work for aﬀects the memory requirement accordingly. Norprocesses that wait for a very long time on events malization would avoid most of the overhead.
that might not never happen. Regular occurrenceswith pkgsrc are Qt’s MOC with bugs in the process 5.3 Protocol extensions for retrying
cleanup, Mono with internal dead locks or the job
individual builds
accounting problems in GNU make 4.2.1.
The author has created a new process monitor for The build scheduling is currently a one-way street.
this purpose. It uses the kqueue event notification Every package is scheduled for build at some point
system to listen for process creation and termina- after its dependencies are built successfully. The
tion. If no process is created within a given time build status is appended to one of two files (error
period, the job is considered to hang. It can also or success). This makes it diﬃcult to provide a
specify a fixed hard limit for the total time a job is mechanism for developers to say “I fixed foo-1.2,
try again please”. The only way is currently toallowed to take.
As a side eﬀect of monitoring the process tree, stop the build, edit the error file by hand and thenit can also detect leftover process. At least one restart the build. As this doesn’t magically stop
package in the pkgsrc tree is known to leak ‘bonobo- current builds and those can take a long time toactivation’ instances, but it is not yet known which. finish, it is clearly suboptimal. With the new storThe processor monitor would allow hunting for this age backed, it is much easier to extend the communication protocol with an option for rescheduling akind of garbage.
The tool is currently tested internally and will failing package or just doing the manipulation dibe published after some more polishing. It should rectly in a way that it will be picked up on the next
work on all systems with a kqueue implementa- decision point.
tion, assuming the kernel implementation works
correctly.
5.4 Up-to-date-check nodes

5.2

As detailed in section 3.3 pbulk checks whether binary packages can be reused or not. For this checkdirect access to the binary packages is necessary
and this is the only part of a bulk build still depending on shared file systems.
Three possible solutions exist:

Moving towards SQLite storage

When pbulk was originally written in 2007, only
the old Berkeley Database 1.85 was used by pkgsrc
infrastructure. Experience with SQLite at the time
was too limited for taking it into consideration. As
a consequence pbulk is using flat files in appendonly mode. After crashes, manual cleanup was often necessary to resume a build and code flexibility
was also limited.
Since then, SQLite has seen use in diﬀerent
NetBSD and pkgsrc related projects:

1. Rewrite the check to deal with protocols like
HTTP.
2. Move the check into the build master.
3. Create a new set of jobs that can be run fromthe machine hosting the packages.

• NetBSD’s apropos uses the Full Text Search of
SQLite.

The first option is somewhat diﬃcult as it requiresa custom client code and the protocol is not thatfriendly towards such time checks. The second option has been considered, but the build master currently needs no access to the actual pkgsrc tree andit would be detrimental to change that back. This

• The repository mirror of NetBSD’s CVS tree
for Fossil and Git depend on SQLite for storage
and query logic.
• The pkgin front-end depends on it.
7Full virtualised solutions like Xen-HVM andKVM on SmartOS have been shown to have a reasonable overhead of 25% to 30% on Haswell Xeons,compared to native NetBSD builds. This showsthat cloud-based bulk builds are technically feasible without wasting too much in terms of resource
utilization The missing cooperation between guest
OS and hyper-visor clearly comes at a significant
price.
The integration of better process monitoring
5.5 Tracing resource consumption
tools for the build is an on-going project. The
From a theoretical perspective, an optimal build pbulk framework will see changes to its communicascheduling should take the availability of restricted tion protocol and storage mechanisms in the nearresources and optimize the job scheduling to min- future. This will ease the deployment on smaller
imize the target function (here: the total build machines and improve the experience for developtime). For bulk builds, CPU and total build time, ers.
concurrency, memory requirements and volume ofdisk I/O are all possible factors for the build time.
The prediction of the interaction of multiple builds
on the same machine is highly complicated. During
the configure phase, many package will utilize onlya single core and little memory. During the build
phase, many cores can be used, but compiler andlinker can require a lot of memory for a single task.
The installation phase, but also linking of debug
builds, can create a lot of disk I/O. For a bulk build,many of those interactions will even out towards amean. From regular observations of bulk builds one
pattern for improvement is obvious: avoiding building of large packages near the end. This normallyhappens naturally as some very large packages like
LibreOﬃce are not necessary for building anythingelse. Building a database of at least the total CPU
time, the wall time and the peak RSS would allow
making better decisions. Some of those measurements can be obtained easily. The move towards
SQLite will also simplify integration of such data.
leaves the last option of splitting the build jobs into
two parts. This requires extending the protocol tosupport three diﬀerent clients: build-only, checkonly, combined build and check. That’s a moderately small change. The master needs to keep
track of a second to-do list and some additional
book keeping. Overall, this is the most promising
approach.

6

Summary and outlook

The paper presented modern recommendations forconfiguring bulk builds for pkgsrc with a focus on
NetBSD. While some of the details will diﬀer onother operating systems, the core tenets remains.
Real-world performance tests have shown clear
scalability issues in NetBSD. Para-virtualisation
can provide a 10% boost to the over-all build timeon the tested hardware. It is expected that thebenefit grows for systems with even more cores.
8

#! / b i n / shm i r r o r= d i s t f i l e s . example . com
case ” $2 ” inhttps : // ${ mirror }/∗);;∗)
# c a c e r t . pem i s not i n t h e d e f a u l t s e a r c h p a t h .
# Pick password from d i s t f i l e n e t r c , i . e . i n t h e form :# machine d i s t f i l e s . example . com# l o g i n myuser# password m y s e c r e t/ pbulk / b i n / c u r l −T ” $1 ” \−−c a c e r t / pbulk / s h a r e / m o z i l l a −r o o t c e r t s / c a c e r t . pem \−−n e t r c − f i l e / pbulk / e t c / d i s t f i l e n e t r c \” h t t p s : / / $ { m i r r o r }/ $1 ” ; ;esac
Listing 1: Sample upload script

push buildlog () {# Don ’ t depend on r s y n c i n PATH
/ pbulk / b i n / r s y n c −−rsync −path / pbulk / b i n / r s y n c \−−d e l e t e −−a r c h i v e −e s s h ” $1 / $2 ” $ { m a s t e r i p } : $1 /}s y n c b u i l d l o g=p u s h b u i l d l o g
Listing 2: pbulk.conf fragment for build logs

p u sh pa c k a g e ( ) {s s h $ { m a s t e r i p } mkdir −p / p a c k a g e s / A l ls c p ” $1 ” $ { m a s t e r i p } : / tmp/ $ {pkgname} ${ p k g s u f x } . tmp
s s h $ { m a s t e r i p } / pbulk / s b i n / pkg admin gpg−s i g n −package \/tmp/ $ {pkgname} ${ p k g s u f x } \/ p a c k a g e s / A l l /$ {pkgname} ${ p k g s u f x }s s h $ { m a s t e r i p } rm /tmp/ $ {pkgname}$ { p k g s u f x }}s y n c p a c k a g e=p u s h p a c k a g e
Listing 3: Insecure proof of concept for signing package remotely

9