Package: binary-gentoo
binary-gentooAboutbinary-gentoois a collection of
simple
CLI tools
to helpbuild Gentoo packages on a non-Gentoo Linux host, primarily.
A typical scenario is operation of anactiveGentoo binary package host— an active "binhost".Secondarily,binary-gentoocan also build Gentoo packages on a Gentoo host
with Docker isolation and a fullemergeperspective
(while not affecting your host system).There currently four CLI tools
that follow theUnix philosophyand are meant to be combined using a glue language like Bash:gentoo-build– Builds a Gentoo package with Docker isolationgentoo-clean— Clean Gentoo pkgdir/distdir files usingecleanofapp-portage/gentoolkitwith Docker isolationgentoo-local-queue– Manages simple file-based push/pop build task queuesgentoo-packages— Do operations on pkgdir (other thanemaint --fix binhost)gentoo-tree-diff– Lists packages/versions/revisions that one portdir has over anothergentoo-tree-sync– Brings a given portdir directory (and its backup) up to datebinary-gentoois software libre licensed under theGNU Affero GPL version 3 or laterlicense.InstallationInstall Docker, Python >=3.8, pip and rsync and then:#pipinstallbinary-gentooActive Binhost OperationTo set-up an active binhost usingbinary-gentooyou will likely need
a virtual machine with…Enough resources, i.e. probably>=2 virtual cores>=8 GB RAM>=100 GB disk spaceSome distribution of LinuxAuto-updates enabled to stay reasonably secure over timeRuntime dependencies installed:DockerPython >=3.8rsyncAn SSH or an HTTP daemon
(to serve the produced binaries to your consuming Gentoo boxes,
unless that is done from another machine)An up-to-date copy of the/etc/portagefolder to useAn up-to-date text file with the names of all packages to build,
e.g. generated byEIX_LIMIT=0 eix -I --only-namesA set of compile flags (CFLAGS,CXXFLAGS,LDFLAGS,CPU_FLAGS_X86)
that works forboththe producing machine and the consuming machine(s)
and ideallyresolve-march-nativeinstalled — more on finding the ideal flags belowTwo portdir copies — one synced withgentoo-tree-sync(or some form ofgit pull),
the other brought back in sync usingrsyncafter a call togentoo-tree-diffA glue script that…respects your inclusion list
(e.g.grep -q -f installed.txt -F <<<"${atom}" || continueif a few false positives are okay)respects your exclusion list
(e.g. something like*sys-kernel/*|*-bin-*|*-9999*|*acct-*/*)pushesgentoo-tree-diffnews onto the queuepops off the queue,
extracts the atom using e.g.jq -r .atom,
and then let'sgentoo-builddo a buildloops foreversleeps every now and thenrunsgentoo-cleanevery now and thenDetermining Ideal Build FlagsLet me give a concrete example.Consumer MachineOn the consumer machine, we have this hardware situation:#lscpu|fgrep'Model name'|sed's,^.\{33\},,'Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz#resolve-march-native-march=sandybridge -maes --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=3072 -O2 -pipe#cpuid2cpuflagsCPU_FLAGS_X86: aes avx mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3Producer MachineOn theproducer machine,
we have this hardware situation:#lscpu|fgrep'Model name'|sed's,^.\{33\},,'QEMU Virtual CPU version 2.5+#resolve-march-native-march=k8-sse3 -maes -mcx16 -mno-3dnow -mno-3dnowa -mpclmul -mpopcnt -mrdrnd -msahf -msse4 -msse4.1 -msse4.2 -mssse3 -mtune=k8 --param=l1-cache-line-size=64 --param=l1-cache-size=64 --param=l2-cache-size=512 -O2 -pipe#cpuid2cpuflagsCPU_FLAGS_X86: aes mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3Resulting Producer Build FlagsSimplified, we are looking for flags
that areas close to the consumer machines as possiblewhile still resulting in binaries,
thatboth the consumer machines and the producer machine can execute(so that the producer machine is be able to install and run the binaries ofdependenciesof the package of interest).For the example above, I ended up with these values for the producer machine:CFLAGS:-march=x86-64 -mtune=sandybridge -maes --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=3072 -mpclmul -mpopcnt -msse4.1 -msse4.2 -mssse3  -O2 -pipe(note-marchand-mtunein particular)CXXFLAGS: same asCFLAGSLDFLAGS: whateverportageq envvar LDFLAGSon the consumer machine returnsCPU_FLAGS_X86:aes mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3(i.e. withoutavxand withoutrdrand)Design DecisionsAll code inbinary-gentoomust work on a non-Gentoo Linux machine,
provided that it has Docker installed and working internet access.Big packages like Chromium need a pile of RAM and CPU time.
Therefore, the build defaults toMAKETOPTS=-j1the package of interest is emerged separate from it dependencies.
That allows to build e.g. a package of Chromium in a VM with only 8 GB of RAM.With regard to dependency constraints,
some packages can bebuiltwithout conflicts but not beinstalledwithout conflicts.
Hence the default is to only install dependencies, but not the package of interest.