nyuuzyou/wikis huggingface dataset:
Third Party BSP Release Process

==Overview==
Third Party BSPs may be released with any major/minor Yocto release by the maintainers of the various BSP layers. The process for including a BSP release into a yocto release has a few routes, some less complicated than others.
===Release Requirements===
* If your DL_DIR contains source which we are required to provide and that source package/revision does not exist at http://downloads.yoctoproject.org/mirror/sources/ you MUST provide the source, as generated from your DL_DIR.
* md5sums must be generated for EVERY file to be included in the release. Please use something similar to find . -type f -exec md5sum {} \; >> ../mybsp-poky-edison-6.0.md5sum to generate.
** This should include additional files needed from your DL_DIR. 
** I do validate every file's md5sum prior to release and will not gpg sign an invalid .md5sum 
* You must include release notes.
* You should include some sort of QA report, preferably a link to the QA report on the yocto wiki.
* You should inform the Yocto Release Engineer (elizabeth.flanagan@intel.com) at least 2 weeks prior to your intended release date (We prefer 4 weeks so we don't surprise those kind folks who mirror yocto). 
* Everything required for release must available to the Yocto Release Engineer at least 48 hours prior to release. This time is needed for mirror propagation.
* Please do NOT mix major releases (e.g. building a meta-foo bsp layer with edison 6.0 and meta-qt3 5.0.1 ).
* Please ensure that your release is built with a known poky release.
* While you can include the poky release with your BSP, it is *STRONGLY* suggested that you do not, but rather provide a link to it. See BKM:_Packaging_a_BSP_for_Release
===Release Checklist===
2-4 Weeks Prior:
* You have emailed the Yocto Release Engineer with your intention to release. Please try to give an estimate size of the release and against what released version of yocto you are building against.
Prior to the Release:
* You have generated a tested BSP release against a known released version of yocto. Please include the git hash for EVERY official yocto layer you have built against. (meta-qt3, etc). 
* You have created a publicly accessible QA report that the release may link against.
* You have generated an md5sum file of all files needed.
* You have validated that all required source files are either in http://downloads.yoctoproject.org/sources or in the area to be used for the release.
* You have created a valid and documented BSP layer tarball named LAYERNAME-RELEASENAME-M.m.p.tar.bz2
48 Hours Prior to your release:
* You have emailed the Yocto Release Engineer a link to download all files needed for the release
* I validate the md5sums
* I gpg sign the .md5sum file and prepare the release for the release directory/mirrors
Hours Prior to release:
* Download links are made live and verified by both the Yocto Release Engineer and You.
* If you would like the release announcement to be sent out to yocto-announce you have provided the text to the Yocto Community Manager (Jefro) and the Yocto Release Engineer.
===Release Checklist===
Tinfoil Test Plan

Category:Tinfoil
This article is the test plan for Tinfoil.
= About Tinfoil=
Tinfoil is an API that allows you to write simple utilities that call into bitbake code to read variables, parse recipes, etc
= Objectives =
Verify the functionality of Tinfoil API
= Contacts =
 Jose, Perez Carranza
Paul, Eggleton
= Scope =
Test Cases will be designed to verify the different functionality of the API, all test cases should be designed to be executed in automated way.
- The first stage of test is a set of basic test listed on Features already tested
- The subsequent improvements to the Test Suite should be listed on Features to be tested
= Test Strategy =
<TBD>
== Test Automation ==
All the test should be automated and executed under oe-selftest suite
== Test Approach ==
Design of test are intended to cover the functionality of Tinfoil API, all the test will be executed every milestone on GDC AB and frequently on main AB
== Test Cases Maintenance ==
Test cases are located 
 poky/meta/lib/oeqa/selftest/tinfoil.py
- To create a new test case a Testopia instance should be created on OE-CORE under "Test Cases" the number given by Testopia should be added prior to the test case name as below:
 @testcase(<TESTOPIA_ID>)
 def TEST_CASE_NAME(self):
- To send the patches of created or modified test cases rules defined here should be followed.
=Features=
Test are under poky/meta/lib/oeqa/selftest/tunfoil.py
==Features already tested==
 - test_getvar
 - test_expand
 - test_getvar_bb_origenv
 - test_parse_recipe
 - test_parse_recipe_copy_expand
 - test_parse_recipe_initial_datastore
 - test_list_recipes
 - test_wait_event
 - test_setvariable_clean
==Features to be tested==
* parseRecipes
* get_overlayed_recipes
* get_skipped_recipes
* get_all_providers
* find_providers
* get_runtime_providers
* get_recipe_file
* get_file_appends
* build_file
* parseCommandLine
TipsAndTricks/Building core-image-minimal on AWS

= Sometimes, a Cloud Server is just easier =
I won't rehash the already very good instructions on how to create an EC2 instance. Instead, I'll outline what I did and how long it took.
Here's a good starting point to get an EC2 instance up and connect to it: 
= My Setup = 
Machine (AMI) Chosen - Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-6e1a0117
Instance Type - m4.xlarge (4 cpu, 16 Gb memory, high network performance )
=== Changes from Default ===
* I have a security group that allows ssh/http/https/icmp from my ip into my AWS instances, I chose this security group.
* I upped storage to 90 GB; note since everything was setscened using sstate, I only ended up using 1.7Gb total ,
 
 du -sh ~/work
 1.7G	/home/ubuntu/work 
 
= After you have successfully ssh'ed into your EC2 Instance, what now? =
* From the quick start guide, you can just cut and paste the sudo apt-get install line for ubuntu.
* clone poky as usual: $ git clone git://git.yoctoproject.org/poky and checkout the release: $ git checkout yocto-2.4 
** Note: the sstate for the rolling branch (in this case rocko) may not be available, but the tagged sstate will be.
* set up the environment $ . ./poky/oe-init-build-env 
* create (or cp ) a site.conf into the conf directory that leverages the sstate YP publishes. This is key to making a quick build! Here's a sample site.conf, (this was done for the 2.4 release):
 
 SSTATE_MIRRORS = "\
 file://.* http://sstate.yoctoproject.org/2.4/PATH;downloadfilename=PATH \n \
 file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH \n \
 "
 
* build it $ time bitbake core-image-minimal 
 
 ... many setscene tasks ...
 NOTE: Executing SetScene Tasks
 NOTE: Executing RunQueue Tasks
 NOTE: Tasks Summary: Attempted 2384 tasks of which 2177 didn't need to be rerun and all succeeded.
 real	2m26.947s
 user	0m5.832s
 sys	0m0.924s
 
** Note, time to image is only 2 1/2 minutes . Pretty nice!
* you could then rsync the images or whatever else you'd like to grab from the instance. or store it on S3, or ...
* make sure to terminate the instance when you are done with it. this will delete the files, so make sure you have what you want saved.
= What's the Big Deal? =
Nothing, really. The point of this very short article is that it is trivial to get an EC2 instance up that can build a YP image and do so in a very short time, assuming you leverage our sstate mirrors.
Note: The free tier instance type <italic>t2.micro</italic> can NOT build core-image-minimal as it runs out of ram in it's default configuration.
= Results = 
* Time to build core-image-minimal leveraging a release sstate-mirror - 2.5 minutes
* Cost on AWS (on demand cost which is higher than spot) - $0.29
** this includes me pulling in the necessary build dependencies, cloning poky, and building.
TipsAndTricks/CliBuildsInToasterBuilddir

= How to Take Over a Toaster Build Dir from the Command Line =
Suppose you have used toaster to add layers and do some builds in a couple of toaster projects. This results in a directory structure that could look something like:
 
 build 
 build-toaster-1 
 build-toaster-2 
 build-toaster-3 
 downloads 
 sstate-cache 
 _toaster_clones
 
In this example, we'll assume that build-toaster-1 is a "master" project, build-toaster-2 is a "local" project, and build-toaster-3 is a "morty" project. Let's further assume you've used the Toaster web ui to configure the layers and their dependencies to have what you'd like in them, but would now like to continue to do your work by hand. If you start a new shell that does not have the env variable pollution of the shell from which you started Toaster, you can do a command line build in the build-toaster-XX directory just as you normally would.
A couple of caveats:
* You must start from a new shell, otherwise the environment variables the Toaster startup sets will cause issues.
* You need to set up the env in the new shell as if that toaster build dir were the main build dir. e.g. 
 cd build-toaster-1;. <path>/poky/oe-init-build-env 
*You need to run bitbake like 
 bitbake -R conf/toaster.conf -r conf/toaster-bblayers.conf <target> 
if you have added additional toaster layers to the project or 
 bitbake -R conf/toaster.conf <target> 
if you have not.
* Since Toaster was not really defined for this workflow, the layers you have added are not written out to disk until a build is run from the Web ui. The smallest build you can run from the Web ui that will force all your configuration to be written to disk is quilt-native. 
* If you edit the toaster*conf files by hand, and go back the the ui, it will break. 
* If you made a project type other than local you may run into issues since the command line build will be using the same bitbake that you started toaster from (the definition of a local project) , but if you made a past release project, this command line approach would be trying to use those layers targeted to an older release from a newer bitbake; which may or may not work.
** For example, building a morty project from a master at 2.3 M2 did not work. If I wanted to set up a Morty project from toaster, I would need to start with a Morty Toaster...
** But, since the poky I started toaster from was within 100 commits of master, the master project did work...
TipsAndTricks/Cmake,Eclipse, and SDKS

= Cmake,Eclipse, and SDKS: Oh My!=
Suppose you would like to use cmake in an sdk &/or would like to use cmake from the Yocto Eclipse plugin. There's a couple of tricks/issues to be aware of.
== Getting cmake into your sdk ==
Normally, if you do 
 $ bitbake core-image-sato -c populate_sdk 
the generated sdk does not include the build system created cmake. Using the host cmake, cross compilation will often work, but you will not have all the cmake options set to the generated cmake.
If you would like to include the generated cmake in your sdk , you need to add the following to your local.conf fille:
 
 TOOLCHAIN_HOST_TASK_append = "${SDK_EXTRA_TOOLS}" 
 SDK_EXTRA_TOOLS = " nativesdk-cmake " 
 
== Using the Eclipse plugin with the cmake from the sdk ==
This can get confusing because it may be unclear which cmake eclipse is using, if you have one in the sdk and one installed on the host. To make sure you get the cmake from the sdk, launch eclipse from a terminal that has already sourced the environment script instead of launching it by clicking on an icon (old school, I know...). For example, :
 
 $ . /home/me/poky-qemuarm-2.2-cmake/environment-setup-armv5e-poky-linux-gnueabi
 $ /home/me/eclipse/eclipse
 
That way, eclipse will find the "correct" cmake and wont complain about either CMAKE_ROOT or "cmake not installed properly" 
 Note: the current sdk's published on downloads.yoctoproject.org as of release 2.3 do NOT include cmake 
== Using the Eclipse plugin with the cmake from the build directory == 
This is a bit harder, since we won't have the sdk environment setup script to help us. To get cmake to work in this case, you need to point the Yocto Eclipse plugin at your build environment as described in TipsAndTricks/RunningEclipseAgainstBuiltImage and then go to 
 Project->Properties->C/C++ Build/Setting/CMake configure 
and change the 
 Command: cmake 
to 
 Command: <path to builddir>/tmp/sysroots/x86_64-linux/usr/bin/cmake 
This will let Eclipse find the cmake in your build sysroot and use it.
== Using host installed cmake ==
This seems to often work since cmake respects the cross compile variables fairly well. So, you can download an sdk without cmake in it, source the env script and then have a cmake project successfully build using the host cmake and the sdk toolchains. This becomes less likely to work in the case where we throw Eclipse into the mix, thus the above suggestions.
TipsAndTricks/DebugNativeRecipeWithGdb

=How to debug native recipes with gdb=
I recently had the need to debug a crash in a native binary that a layer I was working on invokes during the build.
To debug the crash with gdb I had to do two things; 1) have the build system generate a binary with debug symbols and 2) not have them stripped during installation to sysroot.
To enable debug symbols, usually one needs to add "-g" to the list of compiler flags. Unforunately how this is done is specific to each project - you need to inspect the
source code tree and look for clues. For autoconf-based projects adding to CFLAGS environment variable is enough.
After the build has completed, inspect the compilation log to verify that -g is indeed there.
To ensure that the binaries in the sysroot don't have their debugging symbols stripped you need to set INHIBIT_SYSROOT_STRIP = "1" in your local.conf
Alternatively you might want to only inhibit stripping for specific recipes, this can be achieved using recipe OVERRIDES such as:
 INHIBIT_SYSROOT_STRIP_pn-swupd-server-native = "1"
I used the former as it allowed me to be certain of having all debugging symbols available.
Note: you'll need to ensure any binaries you wish to debug are (re-)staged into the sysroot after changing this setting.
For more information on the gdb commands above see the gdb manual section "Commands to Specify Files"
TipsAndTricks/DebuggingAvoidingRebuilds

Imagine you want to debug which change in some early dependency is causing a particular build issue. You could bisect that early dependency and then wait for a complete rebuild each time and debug that way. There is a slightly more risky but faster approach you can take.
For this example, the early dependency is pseudo-native and the recipe producing the failure is e2fsprogs. We want to find out which change in pseudo-native causes a packaging issue in e2fsprogs.
First, we set the SRCREV we want to test in the pseudo recipe. We then build it with "bitbake -b virtual:native:pseudo_1.8 -c populate_sysroot". Here, we're telling it to build the 'native' BBCLASSEXTEND version of the recipe, we add _1.8 to the filename to differentiate the recipe we want compared to the _1.6.5 and _git versions and we specify which task we want. Since we changed SRCREV, we don't need to clean it, even the -b version uses checksums, just with truncated dependency chains limited to the extent of the recipe only. 
With the pseudo=native version we want now built, you might think "bitbake e2fsprogs" would work however that has the full dependency tree which wouldn't match the truncated dependency tree checksum we just built so it would cause a rebuild of pseudo-native and all the dependencies of e2fsprogs. We can however run "bitbake -b e2fsprogs -c package_qa" which does do what we want.
Assuming we want to test another pseudo-native version, we can change SRCREV again and it will build the new version. This time however we need to run "bitbake -b e2fsprogs -c cleansstate" first since the checksum for e2fsprogs will not have changed and we don't want to use the existing build or any data from sstate.
Incidentally, to debug pseudo, you can change FAKEROOTENV in bitbake.conf to include PSEUDO_DEBUG=nfoPdeViDxy (or whichever flags you want, see pseudo -h output for more info).
TipsAndTricks/DebuggingBitbakeInPudb

=How to debug bitbake itself with pudb=
I recently had the need to debug bitbake and toaster to figure out why toaster was no longer able to save event information to it's database.
It turned out that I wanted to step through a number of the lib/bb/* files to see how the server was being setup.
==What I did==
 
 . ./poky/oe-init-build-env
 . toaster start webport=0.0.0.0:8000
 
This set up the toaster webserver to run.
Then my debug test:
 
 python3 -m pudb.run $(which bitbake) quilt-native
 
I was looking at changes to bitbake/lib/bb/main.py. So, in that file I could set up breakpoints in code like so:
 
 from pudb import set_trace as bp
 ...
 ...
 bp()
 ...
 ...
 
This allows me to sprinkle programmatic breakpoints wherever I want.
If you have a better/preferred way to step through bitbake, feel free to add/change this document.
= Debugging python in metadata with pudb =
Now, this is kinda out there but you can do it.
My example is definitely a tad contrived but here goes. Suppose you want to see the state of things in the pudb debugger from a .inc file. As our example let's put the following at the top of quilt-native.inc, just after the line EXTRA_OECONF = "--disable-nls":
 
 python () {
 import sys
 from pudb import set_trace as bp
 i=5
 for j in range(i):
 print("\nCOWZ IN QUILT_NATIVE.INC\n")
 bp()
 print("\n VERSION=",sys.version,"\n")
 }
 
If I then run 
 
 python3 -m pudb.run $(which bitbake) quilt-native
 
I will break in the python in the metadata and be able to stop through it and watch j=0,1,... up to 4.
Looks like this:
TipsAndTricks/DeployWithLocalPackageServer

Imagine you want to test some changes to a piece of software and already have an image you've built running on the target. You can set:
PACKAGE_FEED_URIS = "http://<ip-address>:8000"
PRSERV_HOST = "localhost:0"
which sets up the image to refer to that package feed and ensures package versions increment.
Update the index with:
bitbake package-index
Start an http server with:
python3 -mhttp.server /path/to/deploy/rpms
Then you can do something like:
"bitbake <recipe> -C compile; bitbake package-index"
to cause recipe to recompile and then be available to update on the target using the package manager of choice (should work for opkg or dnf).
TipsAndTricks/GitBisectABitbake

= Using Git to bisect a build problem =
If you have a regression in your build, for example "master" builds fine but your feature branch "devel" is broken in some way, then you can use git bisect to identify the commit which introduced the problem.
'git bisect' in it's most basic form can be used to help you identify which commit is broken by keeping track of known good and known bad commits, and automatically checking out the commit between the two so it can be tested. When the problem is a broken build or something which can be examined without user interaction then git bisect can automate both the search and test.
For example, if at some point between master and devel your recipe foo stopped building, then you could drive the bisection like this:
 $ git bisect start
 $ git bisect good master
 $ git bisect bad devel
 $ git bisect run sh -c "cd build ; bitbake foo"
"git bisect start" says that you want to start a bisection run.
"git bisect good master" says that master is known to be good.
"git bisect bad devel" says that master is known to be bad.
"git bisect run ..." tells git bisect to run the script to determine if the commit it is testing is good or bad, and automatically proceed to the next commit or finish depending on the exit code. For this we need to cd into build (as git bisect needs to run at the top level) and bitbake foo. If the build is successful then the script exits with success and the commit is marked as good, if the build fails then the commit is marked as bad. Either way the bisection the continues, a new commit is selected, and the script ran again until the problematic commit is identified.
The test script can be as complicated as required. I was debugging a problem where python was no longer shipping the zlib module, so I used this command:
 $ git bisect run sh -c "cd build && bitbake python -C unpack && oe-pkgdata-util list-pkg-files -p python | grep python-zlib"
This meant that for every iteration python was forcibly built (python -C unpack says to build python and forcibly run unpack, so it always does a clean build) and then oe-pkgdata-util used to list the packages that python generated, which was then grepped for python-zlib. The final grep provided the exit code, so if python-zlib was present then the commit was good and if it was missing then the commit was bad.
TipsAndTricks/Incorporating closed source components

= Incorporating closed source components =
As users of open source tools, we'd often like all of the software we use to be distributed in source form; however that's not always possible. Particularly in a commercial setting where there are third parties involved it may not be practical, in which case if you're using OpenEmbedded you may need a way to both produce and subsequently consume the binary-only components.
Create a file describing the redistribution terms for the binaries in plain text form.
Create a separate build directory where you do have recipe which compiles the component you want to deliver from source. This recipe would go into a separate layer that you do not use as part of normal image/SDK builds and that is not released publicly. Within do_install the recipe should install all files as normal, plus install the license text created in step 1 to some location e.g. ${docdir} so that it gets distributed along with the output.
Build the recipe and take the package (rpm/ipk) produced from this as what you deliver for use in the next step. Alternatively if you would prefer a plain tarball instead, add the following to the recipe (not to your local.conf/distro config): 
IMAGE_CLASSES += "package_tar"
Store the package created in step 3 somewhere such that it can be fetched by any user wishing to use it, e.g. an http/https server, and obtain a URL for the package.
Create a recipe in one of the layers you do use for image/SDK builds that points at the URL and installs the files in the package verbatim given that they are already in the structure matching the target. There is a shortcut for this - to create each recipe use devtool add -b, or recipetool create -b if you just want the recipe with less hand-holding. LICENSE should be set to "CLOSED" for this recipe or alternatively make up some name that you can use to refer to the license text you prepared in step 1 e.g. "<softwarename>-Proprietary". You may want to name this recipe slightly differently to the original recipe used in step 1 for clarity, e.g. if the original was called widget-driver then you could name this one widget-driver-bin.
Add the packages produced from the recipe created in step 4 to the image recipe so they end up in the image and corresponding SDK.
Of course, in a typical scenario, steps 1-4 and 5-6 may be done by developers in different organisations. As the developer incorporating the closed source components you will want to ensure reproducible builds in future, so you may need to mirror the binary package file(s) locally. You can use MIRRORS or PREMIRRORS to point to this mirror.