nyuuzyou/wikis huggingface dataset:
TipsAndTricks/InvestigatingBuildTime

== Draft: Investigating Build Time ==
(Placeholder. Demonstrate use of buildstats and pybootchartgui)
Current status: The intention is to use the scripts under scripts/contrib/bb-perf to parse
buildstat data and produce useful plots.
The buildstats data produced by the class with the same name provides process statistics for every recipe task.
This data can offer insight of the time spent by the builder (bitbake) when
constructing a particular target at process level. This article focus more on profiling than optimization, the former being
the base for the latter.
=== Setup ===
 $ cd poky
 $ git checkout -b morty origin/morty
 $ . oe-init-build-env
 $ bitbake world
=== Buildstats ===
buildstats.bbclass stores mainly two set of process' stats: time and IO stats. For the moment, we are currently focusing on the following
time data:
* /proc/[pid]/stat: utime, stime
* /proc/[pid]/stat: cutime, cstime
* python resource module: rusage ru_utime, rusage ru_stime
* python resource module: Child rusage ru_utime, Child rusage ru_stime
and the IO stats:
* /proc/[pid]/io: IO rchar, IO wchar
* /proc/[pid]/io: IO read_bytes, IO write_bytes
With this data, the following script (once the bitbake world command has finished) is used
to either plot the top N recipes-task stat values
 $ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -n $N | gnuplot -p
or sum stats values per task for all (world) recipes
 $ ../scripts/contrib/bb-perf/buildstats-plot.sh -s <BUILDSTATS> -S | gnuplot -p
The former produces a more granular plot and second a more general idea where the type is spent per task set.
=== Results ===
Preliminar results can be found here
TipsAndTricks/LinuxKernelAndSDKs

This is the start of what I hope becomes a longer article. This early stage is to answer a particular question. 
 Issue: Currently make menuconfig fails in the sdk. This is an active bug and can be tracked here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11155 You can work around this currently by running make menuconfig in a different bash shell from the one you sourced the sdk environment into.
== Kernel Building w/o Bitbake ==
Suppose you'd like to build a kernel but you don't care about having it packed into a wic image. Maybe you are loading the kernel on the command line in qemu, or maybe you are netbooting it. But, you'd like to use the Yocto toolchains to insure that it will work with an eventual Yocto image... How do you do that?
* make the sdk or download it.
** if you are targeting a release, you can download the sdk from http://downloads.yoctoproject.org/
*** For example for YP 2.2.1 you could use http://downloads.yoctoproject.org/releases/yocto/yocto-2.2.1/toolchain/x86_64/poky-glibc-x86_64-core-image-sato-core2-64-toolchain-2.2.1.sh
** if you need to build it do 
*** $ bitbake core-image-minimal && bitbake core-image-minimal -c populate_sdk 
**** you would only need to do this ONCE
**** the result will be in tmp/deploy/sdk/ and will be named something like poky-glibc-x86_64-core-image-minimal-core2-64-toolchain-2.2+snapshot.sh
**** This assumes you have set your MACHINE to whatever your target is (for example MACHINE=qemux86-64, in your conf/local.conf) to get the correct cross toolchains.
* install the sdk $. poky-glibc-x86_64-core-image-sato-core2-64-toolchain-2.2.1.sh for example. Or, you could use the one you built. 
** Since we are focused on the kernel, the image that the toolchain was built for isn't important, we want to use the cross toolchains.
** from here on, we'll assume you installed it into /opt/poky/2.2 
* source the environment for the sdk from a fresh shell (i.e. not from the shell that has all the bitbake variables set in it). $. /opt/poky/2.2/environment-setup-x86_64-poky-linux 
* then just do $ make bzImage in your kernel directory. The kernel Makefile respects the env variables that were set in the environment script your sourced.
== But, I want the config that linux-yocto would use... ==
* If you want to start with the config that linux-yocto would use, do $ bitbake -c kernel_configme linux-yocto 
** The resulting .config file will be somewhere like tmp/work/qemux86_64-poky-linux/linux-yocto/...
** honestly, it's easier to do $ find . -name .config| grep linux-yocto
* then just cp that .config file to your kernel tree and make like above. You should probably to a make oldconfig if you aren't using the linux-yocto kernel source itself.
== Kernel Modules == 
* Addressed Later
TipsAndTricks/Netconsole

= Using Netconsole =
If your board doesn't have a traditional serial port it can be difficult to see early boot logs, but if you have ethernet then the netconsole can be used instead. The netconsole is a virtual serial console which outputs over UDP to a specific IP address.
The prerequisites are that netconsole support is enabled, and your ethernet driver is built in to the kernel and not a module. Luckily, the stock Yocto kernels have netconsole enabled and the atom-pc machine integrates the driver for my hardware.
Then, on the target machine, you pass netconsole=... to the kernel. The kernel documentation explains this quite well:
 netconsole=[src-port]@[src-ip]/[],[tgt-port]@/[tgt-macaddr]
 where
 src-port source for UDP packets (defaults to 6665)
 src-ip source IP to use (interface address)
 dev network interface (eth0)
 tgt-port port for logging agent (6666)
 tgt-ip IP address for logging agent
 tgt-macaddr ethernet MAC address for logging agent (broadcast)
The biggest gotcha is that you (obviously) need a source IP address, and netconsole starts before the networking normally comes up. Apart from that you can generally get away with minimal settings:
 netconsole=@192.168.1.21/,@192.168.1.17/
Note that apparently some routers may not forward the broadcast packets correctly, so you may need to specify the target MAC address.
On the target machine run something like netcat to capture the packets:
 $ netcat -l -u -p 6666 | tee console.log
If you get the options wrong the kernel will tell you why, so if you don’t get any logging iterate a working argument on an image that works, using dmesg to see what the problem is.
Finally instead of typing in this argument every time you boot, you can add it to the boot loader in your local.conf:
 APPEND += "netconsole=@192.168.1.21/,@192.168.1.17/"
(APPEND being the name of the variable that is passed to the kernel by the boot loader.)
Note that under systemd when journald starts up it will stop logging to the console, so if you want to get all systemd messages also pass systemd.log_target=kmsg to the kernel.
= To Do =
* change machine name to eg intel-corei7-64
* test it still works
* redraft
TipsAndTricks/ParsingProfiling

Ever wondered what bitbake does when parsing?
If you've ever wondered where bitbake spends its time during parsing, its now quite easy to find out!
Start with a clean cache by "rm tmp/cache -rf". You can then trigger bitbake's profiling mode with the "-P" option, so "bitbake -p -P". Bitbake will parse the recipes, then print a message saying "Processed parsing statistics saved to profile-parse.log.processed".
Internally, it enables profiling in each separate parsing thread, saving the data to files called profile-parse-Parser-1:X.log where X is the thread number. It then merges all the files together to form the processed output.
There are four sections to profile-parse.log.processed, the first shows a list of functions, ordered by "internal time" which tells us which function consumed the most time. Underneath this is a callgraph section, which shows which functions called those functions. This is useful where you see a function called many times and want to know where it was called from.
These two data items are then repeated but ordered by cumulative function time. This allows you to answer a different kind of question, where you're interested to learn more about the critical path and which functions all the time is spent in.
Interestingly, on my highly parallel machine I just tested this on, I see "389110629 function calls in 1334.486 seconds" and the top of the function time table reads:
 ncalls tottime percall cumtime percall filename:lineno(function)
 6415 450.867 0.070 450.867 0.070 {method 'acquire' of '_multiprocessing.SemLock' objects}
 3403372 56.846 0.000 350.207 0.000 bitbake/lib/bb/data.py:332(build_dependencies)
which suggests in machines with many cores, we start to need to look at optimising the parsing queue lock since we're spending around a third of the overall time in there on this machine.
TipsAndTricks/Patching the source for a recipe

= Patching the source for a recipe =
One of the useful things about OpenEmbedded building everything from source is that it's fairly easy to make changes to anything that gets built, but doing so for the first time can be a bit daunting.
As part of building a recipe, OE creates a tmp/work/<architecture>/<recipe>/<version> directory, known as the "work directory". This is where all of the work done to build a recipe takes place. One of the things you'll find in this directory is the source, usually under a subdirectory named <recipename>-<version> or "git" (depending on how the fetched source is provided). The temptation (and what people used to do in the past) is to simply make changes here and then recompile, but there are several reasons why that's not a good idea:
* It's awkward - you have to use bitbake -c compile -f or bitbake -C compile to force recompilation, since the build system doesn't know that you've made any changes
* You can easily lose your changes if you're not careful e.g. running bitbake -c clean will wipe the directory out
Luckily if you're using the fido (1.8) or later release, there's a much better method using the devtool command:
# Run devtool modify <recipename>. This will fetch the sources for the recipe and unpack them to a workspace/sources/<recipename> directory and initialise it as a git repository if it isn't already one. If you prefer you can specify your own path, or if you already have your own existing source tree you can specify the path along with the -n option to use that instead of unpacking a new one.
# Make the changes you want to make to the source
# Run a build to test your changes - you can just devtool build <recipename> or even build an entire image incorporating the changes assuming a package produced by the recipe is part of an image. There's no need to force anything - the build system will detect changes to the source and recompile as necessary.
# If you wish, test your changes on the target. There's a "devtool deploy-target" command which will copy the files installed at do_install over to the target machine assuming it has network access, and any dependencies are already present in the image.
# Repeat from step 2 as needed until you're happy with the results.
# At this point you will almost certainly want to place your changes in the form of a patch to be applied from the metadata - devtool provides help with this as well. Commit your changes using "git commit" (as many or as few commits as you'd like) and then run either:
#* devtool update-recipe <recipename> to update the original recipe - usually appropriate if it's your own recipe or you're submitting the changes back to the upstream layer
#* devtool update-recipe -a <layerpath> <recipename> to put your changes in the form of a bbappend to be applied by a different layer. This is usually the desired method if your changes are customisations rather than bugfixes.
# If you're finished working on the recipe, run devtool reset <recipename>.
This is just one of the things that devtool can do - it provides some powerful tools to help you maintain recipes and make changes to source code.
TipsAndTricks/ResolvingLocaleIssues

= What to do when bitbake says " Sad Locale, Need UTF-8" =
If bitbake says:
 
 Please use a locale setting which supports utf-8.
 Python can't change the filesystem locale after loading so we need a utf-8 when python starts or things won't work.
 
You need a locale. My example will target en_US.UTF-8. You can choose your favorite (native?) locale as long as it's UTF8.
== Ubuntu/Debian ==
 
 
 tt@7e801839f06a:~/build$ sudo apt-get install locales
 tt@7e801839f06a:~/build$ sudo dpkg-reconfigure locales 
 <pick your favorite, in my case en_US.UTF-8 UTF-8, then default the language to en_US.UTF-8>
 tt@7e801839f06a:~/build$ sudo locale-gen en_US.UTF-8
 Generating locales (this might take a while)...
 en_US.UTF-8... done
 Generation complete.
 tt@7e801839f06a:~/build$ sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
 tt@7e801839f06a:~/build$ export LANG=en_US.UTF-8
 
Alternately
 
 sudo apt-get install locales
 sudo locale-gen en_US.UTF-8
 sudo localectl set-locale LANG=en_US.UTF-8
== Fedora ==
=== Fedora:24 ===
 
 sudo dnf install glibc-langpack-en
 sudo dnf reinstall glibc-common 
 export LANG=en_US.UTF-8
 
== OpenSuse ==
=== OpenSuse:42.1 ===
 
 bash-4.2$ sudo zypper install glibc-locale
 bash-4.2$ export LANG=en_US.UTF-8
TipsAndTricks/RunningQemuOnMacOSX

If you'd like to forward X11 from a build machine to a machine running OSX, you'll need to install XQuartz. 
Sadly the keyboard mapping may be incorrect. To solve this simply add these qemu params to your runqemu command:
 - $runqemu qemux86 core-image-sato qemuparams="-k en-us"
A couple of notes:
 1. keyboard mapping will be incorrect for all builds (core-image-minimal, core-image-sato)
 2. for images without a GUI, you can also use the 'nographic' option: $runqemu nographic qemux86 core-image-minimal
Also, when running a forwarded X11 qemu graphical display over the network, especially over WiFi, the output of the serial port and graphics may be painfully slow to respond. One way to fix this is to use is to use QEMU's vnc option:
 - $runqemu qemux86 core-image-sato qemuparams="-vnc :66"
This will start a vnc server at 127.0.0.1 on display 66. You can then use an X11 forwarded vncviewer. Of course, you could connect remotely without X11 forwarding using:
 - $runqemu qemux86 core-image-sato qemuparams="-vnc 192.168.0.123:66"
Where the IP matches that of your build machine.
TipsAndTricks/TeamWorkflows

Note: there's a bug with PREMIRRORS handling. For optimal use of this workflow you should be running with either Morty 2.2.2+/1.32, Pyro 2.3.1+/1.34
-----
= OE-Core image builder =
How to configure an OE-Core system to build images from shared artefacts.
== Goals ==
* Quickly build images from pre-existing artefacts on minimal hardware (e.g. laptops).
* Avoid networking issues fetching from some upstream hosts (i.e. great firewall)
== Requirements ==
* A mirror of sstate objects (i.e. contents of SSTATE_DIR) which the build machine can access via a uri (file://, http://, etc)
** if using the example poky distro you can use the mirrors on the Yocto Project infrastructure at sstate.yoctoproject.org
* A fixed version of the metadata which matches the sstate mirror (if the metadata isn't identical there's a strong chance of sstate checksum mismatches which result in being unable to use the mirrored objects and having to build recipes and their dependencies)
** i.e. if using poky and the Yocto Project mirrors as above, use a tagged release
* Make use of uninative, this fixed libc implementation ensures greater shared state object reuse for native recipes
** the poky distro makes use of uninative by default, it's highly recommended that you use uninative in your custom distro
* A mirror of the download sources (i.e. contents of DL_DIR) to use as a premirror
** the sources used by the core Yocto Project layers are mirrored on Yocto Project infrastructure at downloads.yoctoproject.org
** Note: the poky distro uses the Yocto Project download mirror as a premirror for non-SCM fetchers by default
== Configuration steps ==
# Use own-mirrors to configure PREMIRRORS for all supported fetchers with a single mirror URL
# (optional) add extra protocol-url pairs to PREMIRRORS
# Add SSTATE_MIRRORS
== Example site.conf ==
# Use Yocto Project source mirror for all sources
SOURCE_MIRROR_URL ?= "http://downloads.yoctoproject.org/mirror/sources/"
INHERIT += "own-mirrors"
# Add extra PREMIRRORS for non-upstream (Yocto Project) components
PREMIRRORS_prepend = "\
 https?$://.*/.* http://my-mirrors.example.com/sources/ \n \
 git://.*/.* http://my-mirrors.example.com/sources/ \n \
"
# Use Yocto Project sstate mirror and project-specific sstate mirror
SSTATE_MIRRORS = "\
 file://.* http://my-mirrros.example.com/sstate/PATH;downloadfilename=PATH \n \
 file://.* http://sstate.yoctoproject.org/PATH;downloadfilename=PATH \n \
"
= eSDK image builder =
How to generate an eSDK to facilitate application development and custom image builds from an existing set of artefacts.
== Goals ==
* Fully contained Yocto Project environment
* go from eSDK install to generated image in <10 minutes
TipsAndTricks/UnconventionalImages

Images don't have to contain a full root file system and kernel, as they're constructed from packages they can contain anything in a package.
For example, if you were publishing the API documentation for your SDK online somewhere then generating a tarball of all the API docs would be a good start. This would be an image which just contained the -doc packages, and nothing else:
 # Use PACKAGE_INSTALL directly to bypass the convenience layers and specify exactly what packages should be installed
 PACKAGE_INSTALL = "gtk+3-doc glib-2.0-doc"
 # Don't want any image features such as a package manager
 IMAGE_FEATURES = ""
 # Don't want any locales to be installed
 IMAGE_LINGUAS = ""
 # Just generate a tarball
 IMAGE_FSTYPES = "tar.gz"
 # Don't generate the ldconfig cache
 LDCONFIGDEPEND = ""
 # This is an image
 inherit image
This will create an image which just contains the packages listed in PACKAGE_INSTALL (and their dependencies), and nothing else.
TipsAndTricks/UseRamdisikToSpeedUpBuilds

At some point your build system may become IO bound, when your hard disk(s) cannot keep up with CPUs running various tasks. This is frequently the case when your systems has numerous CPUs, but only a single hard drive. Providing you have plenty of RAM, it is possible to improve
the build performance by using a ramdisk. It is probably a good idea not to keep DEPLOY_DIR in ramdisk. Placing something like the following: 
TMPDIR="/tmp/ramdisk/build-ramdisk"
DEPLOY_DIR="/work/poky/build-ramdisk/tmp/deploy"
in your local.conf would keep the built images on your harddrive, everything else in a volatile ramdisk.
Some build times as observed on my system (in minutes), YMMV.
Builds are clean builds, no SSTATE, with data already downloaded:
----------------------- HDD -------- RAMDISK ---
core-image-minimal 24 20
core-image-sato 47 39
If running multiple builds at the same time, the benefits become even more obvious.
Setting up a ramdisk is trivial, the following example creates 64G ramdisk:
$ sudo mkdir /tmp/ramdisk
$ sudo chmod 777 /tmp/ramdisk
$ sudo mount -t tmpfs -o size=64G ramdisk /tmp/ramdisk
If you want the ramdisk mounted automatically at boot time, add this into /etc/fstab:
ramdisk /tmp/ramdisk tmpfs defaults,size=64G,x-gvfs-show 0 0
(Another advantage of ramdisk is that it is much faster to delete the files.)
TipsAndTricks/UseRamdiskToSpeedUpBuilds

At some point your build system may become IO bound, when your hard disk(s) cannot keep up with CPUs running various tasks. This is frequently the case when your systems has numerous CPUs, but only a single hard drive. Providing you have plenty of RAM, it is possible to improve
the build performance by using a ramdisk. It is probably a good idea not to keep DEPLOY_DIR in ramdisk. Placing something like the following: 
TMPDIR="/tmp/ramdisk/build-ramdisk"
DEPLOY_DIR="/work/poky/build-ramdisk/tmp/deploy"
in your local.conf would keep the built images on your harddrive, everything else in a volatile ramdisk.
Some build times as observed on my system (in minutes), YMMV.
Builds are clean builds, no SSTATE, with data already downloaded:
----------------------- HDD -------- RAMDISK ---
core-image-minimal 24 20
core-image-sato 47 39
If running multiple builds at the same time, the benefits become even more obvious.
Setting up a ramdisk is trivial, the following example creates 64G ramdisk:
$ sudo mkdir /tmp/ramdisk
$ sudo chmod 777 /tmp/ramdisk
$ sudo mount -t tmpfs -o size=64G ramdisk /tmp/ramdisk
If you want the ramdisk mounted automatically at boot time, add this into /etc/fstab:
ramdisk /tmp/ramdisk tmpfs defaults,size=64G,x-gvfs-show 0 0
(Another advantage of ramdisk is that it is much faster to delete the files.)
TipsAndTricks/UsingRPM

The RPM command allows for a number of query operations. The format of the query operations can be used for a number things, such as debugging a package or even exporting information into another format, such as XML.
First in order to perform a query, you want to tell the system to go into query mode:
 rpm -q
You then need to add the option to tell it WHAT to query:
To query a single package that has been installed into the database, just add the package name, such as:
 rpm -q bash
To query all of the installed packages in the system, add the '-a' option, such as:
 rpm -q -a
To query a single package that is not yet installed, add '-p <package name>'
 rpm -q -p bash-3.0-r3.i386.rpm
To query multiple packages, just list the packages after the -p:
 rpm -q -p bash-3.0-r3.i386.rpm libc6-7-r3.i386.rpm
You can include lists of packages instead:
 echo "bash-3.0-r3.i386.rpm" > mymanifest.txt
 echo "libc6-7-r3.i386.rpm" >> mymanifest.txt
 rpm -q -p mymanifest.txt
(Note: there is a minimum size of the file for RPM to determine this is a manifest file. So it's recommended if this is used in script to always include a header that says something like "# This is a manifest file".
The above operations simply control the 'what', by default the system will just print the package name. (Some versions will print the name and package architecture.)
In order to select the exact items you want to query, you want to use the '--queryformat' (or '--qf') to specify a format. Such as:
 rpm -q -p mymanifest.txt --qf "[%{NAME}\n]"
The above duplicates the default. The special characters or parts being used are:
[ ] - this indicates that it should be applied on each package or item (if the item being queried is an iterator, like a filelist.) Including '\n' inside of the brackets means adding a line feed at the end of every item that is queried.
%{...} -- this is how you specify the exact field to be queried. To get a list of valid options, you can run: 'rpm --querytags'.
The format of %{...} also allows for select 'type' conversions. Some of the valid types are: octal, shescape, deptype, pgpsig, day, date, and others.
For instance, the typical, 'rpm -q bash --requires' is actually implemented as:
 rpm -q bash --qf "[%|VERBOSE?{%{REQUIREFLAGS:deptype}: }:{}|%{REQUIRENEVRS}\n]"
The above indicate that based on the value 'VERBOSE', either %{REQUIREFLAGS:deptype} followed by either {} | %{REQUIRENEVRS}.
| ? and other special operators exist to permit you to do some basic boolean cases. See: /usr/lib/rpm/rpmpopt-* for a bunch of examples.
Another popular example is how to export the whole DB to XML:
 rpm -qa --qf "[%{*:xml}\n]"
The '*' says to not only query the main thing, but also iterate over the matching entries (all of them), with the :xml indicating to dump the contents in XML format.