Yocto Architecture

= Yocto Project Architecture Whitepaper =
== Embedded Market ==
The embedded market is characterised by its wide range of devices which 
operate in many different environments e.g. home users, industrial devices, 
medical use or data centres. These devices are tailored to their environments 
and vastly differ in connectivity capabilities, input and output methods and 
features like processing power, memory capacity and available data storage. 
These systems need software to function and Linux is desirable due to its 
adaptability, its functionality, quality and no royalties model.
The market is highly fragmented with many different approaches to software 
being adopted using both open source and proprietary solutions. Each market 
sub-segment tends to have a preferred approach but there are no accepted 
standards.
Anyone working in this market faced some difficult issues trying to deal with this 
fragmentation. One approach would be to take each sub-segment in turn and try 
and create solutions targeting that segment. This would be costly in resources. 
An alternative is that offered by Yocto Linux - to create a common best practises 
infrastructure which can fulfill the software needs of most embedded market segments.
Building a common base set of tools and a reference embedded Linux stack means this 
non-differentiation work which everyone currently does themselves can be shared in 
the form of a common project. Organisations can then focus where their effort is best 
spent, on their differentiation and project features.
== Yocto Project ==
The objective is to look at all the solutions currently in the embedded space 
and then take the best of what is available, fill in any missing functionality 
and create something which can lead the market. 
By doing this, the project users share the cost of the base stack and there is
a common standard set of tools used by the marketplace rather than everyone having 
to do this themselves.
== Existing Build System Solutions ==
Some kind of build system is the cornerstone of most embedded projects and 
the Yocto Project is no different. There are many such solutions in the embedded space. In 
order to capture the attention of the industry this project requires something 
open that people can see and use without royalties, much like Linux itself. 
For this reason only open source systems are contenders for this project. 
There are a variety of possible existing contenders and its worth evaluating 
these for their strengths and weaknesses:
Buildroot - One of the older embedded build systems, this is makefile driven 
and is a natural evolution of the scripts based build solutions many people 
start with when starting an embedded project. The general feeling of this 
system is that it has had its day, it was a good learning experience but 
lacked the fundamental structure to its processes to grow into anything bigger 
than what it is. It lacks fundamental features such as packaging.
E2-Factory - A more modern from scratch solution that it very structured and 
rigorous in its build processes. Sadly whilst the core tool is open source, no 
metadata is available so its ruled out. It does have some neat features like 
dependency checksumming which can be learnt from though. 
Desktop (PC) Distribution Systems - Whilst not embedded solutions, these are 
widely used for PCs so should be considered. The fundamental problem is that 
these systems all promote a "one size fits all" approach to software and do 
not support customisation for multiple targets easily or well. You cannot 
build two distributions of different character from one set of metadata with 
these solutions. In the embedded space this rules the solution out but again, 
there are things that can be learnt from like the OBS web interfaces.
PTXdist - Makefile driven, more advanced than buildroot but is small and only 
caters for a small segment of the market.
LTIB - This is a system developed and pushed by Freescale which combines spec 
files with a kconfig menu front end. Its nice to have the menu control of the 
system but it has limited package support, limited platform support and is 
very tied to the rpm format and is not as structured, mature or comprehensive 
as other solutions in the marketplace.
OpenEmbedded - If there were an industry standard, this would be it. It was 
written to learn from all the mistakes in buildroot, has a well defined 
structure to its metadata and has by far the largest coverage of open source 
software. It dropped Makefiles in favour of a custom data file format of its 
own with tools to handle them (bitbake). Feature wise its also the most 
advanced system in the market and it was specifically designed for the 
customisation needs of the embedded market space. It does lack some features
such as checksumming but the structure would allow them to be added.
Poky - Created by Openedhand as a commercial version of OpenEmbedded and the 
rights to it passed to Intel. It has a good reputation in the market as 
lots of technical developments in OpenEmbedded have come via Poky.
LDAT - The build system developed and used by Wind River. This is not an open 
solution at this time but could become open so is worth evaluating. Its 
strength is that it has a large feature set and is widely used by Wind River 
customers. Its makefile derived and shows signs of strain due to being 
constrained by the file format. It has been developed behind closed doors with 
a narrow focus and as such does not stand a good chance of wide community 
adoption if opened and hence is unlikely to become an industry standard.
== Yocto Linux build system ==
Acceptance of whichever build system is used by the wider community is 
essential to the project. We're in the fortunate position that Poky is an 
available option. The advantages of this choice include:
* Its already well established and accepted in the community
* It has commercial leanings
* Intel can donate Poky to the Yocto Project
* Its based on OpenEmbedded - industry leading
On this basis it has been decided that Poky makes the best choice of build 
system for the Yocto Project.
Where there are gaps in functionality we have the ability to change this.
== Yocto Project Components ==
If Poky is the build system for the Yocto Project, what other components are there? Should 
the Yocto Project perhaps be Poky?
There is much more to the idea of the Yocto Project than just the build system. It makes sense to have 
namespaces for the components which is why Poky should remain just the build 
system. The other features/components that make up the Yocto Project include:
* A place of collaboration - There are very effective strong communities in the opensource world such as xorg.org, kernel.org and gnome.org but there is no such place for embedded. The Yocto Projectcan provide this, primarily by providing a place where embedded focused software projects can be housed and discussions can be had.
* A reference implementation: Yocto Linux - Any reference distribution/images created by Poky are something that is provided by the Yocto Project . This aims to reinforce the idea that Poky is the build system, not the end result of any given build of the metadata which instead would be Yocto Linux.
* Board Support Packages - The Yocto Project can provide a place for people to publish its SoC BSPs.
* Testing infrastructure - The Yocto Project can provide hardware to allow regular regression testing of the Poky metadata and demonstrate this publicly.
* Partners - The Yocto Project also provides the entity which other parties can associate with to jointly improve and advanced the embedded Linux software position.
* Standards - The Yocto Project can become the entity which sets the standards for the industry, aiming to be the model of best practises.
* Documentation - Another output possible from the build infrastructure is documentation of all the components where the source contains these. As part of the reference implementation it would be desirable and extremely useful for the community to have access to a set of documentation all in one place.
More details on the individual components follows below.
== Yocto Project Partners ==
To succeed, Yocto Project needs to ensure people will use it and one way of doing this 
is through partners.
Discussions between Wind River and Intel revealed that they share frustration 
over the structure of the Embedded Linux market as mentioned above. The 
fragmented nature of the market has a cost to them and also presents a barrier 
to entry and their build system LDAT is different to any other on the market. 
All their BSPs effectively have to be developed and maintained in-house at 
significant cost.
Wind River see the Yocto Project as very desirable as it means everyone would share 
the non-differentiation and become more standardised. Creating a BSP standard
reduces the amount of work they have to do in integrating BSPs themselves. For 
these reasons Wind River are keen to be a founding partner of Yocto Project. Where 
the Yocto Project solutions lack the features they need for this there is agreement 
to enhance it and its components like Poky.
As the Yocto Project grows and becomes public, its likely more partners will emerge and 
this will make the project stronger.
== Yocto Project Components: A place of collaboration ==
The Yocto Project website is intended to house embedded related software projects since 
there currently isn't a good home for a lot of these. We already have a number 
of different projects which could be used to seed this site:
* Poky (from Intel)
* Poky Anjuta SDK integration (from Intel)
* MeeGo Eclipse SDK integration (from MeeGo)
* Pseudo - A replacement for fakeroot (from Wind River)
* qemuGL - A project to allow GL passthrough in virtualsation (from Intel)
* OProfileUI - an embedded targeted client/server profiling tool (from Intel)
* Cross platform prelink tools (from Wind River)
Anyone wishing to house the source code/mailing list/bug-tracking for an open 
source project furthering embedded technology would be free to do so on this 
site.
== Yocto Project Components: Reference Implementation ==
A reference implementation will be available from the Yocto Project website under the name of Yocto Linux. These will 
be packages, images and an SDK/toolchain generated by Poky supporting the range of 
reference hardware we're promoting. For IA, this means we'd have images 
capable of being run under emulation in QEMU as well as having support for a 
suitable selection of our reference boards, particularly the SoCs chosen from 
our BSP selection. 
For other architectures we will provide reference images suitable for running 
under emulation as proof we are architecture independent but Intel will not 
provide builds for other specific platforms. If other partners join the 
project and undertake to maintain platforms on other architectures they are 
free to do so however.
It is worth highlighting that the reference distribution and images created by 
Poky are something that is provided by the Yocto Project. We should reinforce the idea that 
Poky is the build system, not the end result of any given build of the 
metadata. The images and packages on the Yocto Project website are also a reference 
implementation which others are then free to build off.
At any given time there is likely to be a stable release version of Yocto Linux 
available and also packages and images build from the master branch as part of 
automated testing. These are provided to demonstrate the status of the 
project, that everything is working and allow people to test new features. Its 
expected that product development be done based on stable release branches and 
if cutting edge features are required, these will be ported to the last stable 
release either by Yocto Linux maintainers if there is a widespread need, or on an 
individual basis if not.
The benefit of having this reference implementation available is that anyone 
wishing to do embedded development has a known good place to start it from. By 
sharing this common base, solutions to any issues can be shared, faster 
development and time to market is possible, a strong community is formed and 
there is a common future forward migration path.
== Yocto Project Components: Board Support Packages (BSPs) ==
Currently there is no standardised way of publishing a BSP for a given 
platform either in the location its made public or the format. It is a 
massive problem in the embedded marketplace and nobody has as yet found a way 
to deal with this well. ARM is a particularly bad example with hundreds of 
different kernel trees, let alone userspaces. A given ARM platform is unlikely 
to work well with a standard mainline Linux kernel and whilst effort is being made to change this, it has not yet succeeded.
The ideal situation would be having one central place where up to date code for 
a given platform can be found, both in the form of a kernel but also the support 
utilities and extras that platforms have.
This needs to provide complete support for that platform including firmware and 
other userspace components that might be needed to make best use of the hardware features.
The Yocto Project proivdes a standardised place and standardised format to do 
this. The exact format will be 
determined by the needs to the platforms but will be generic enough that an 
industry BSP standard could be established as a result.
== Yocto Project Components: Testing Infrastructure ==
One area many embedded build systems fall down is that their core developers 
can make them work but a newcomer to the project cannot. These systems by 
their nature are extremely complex and its essential they are regularly and 
extensively tested. There are two forms of testing that are desirable:
Incremental - This provides an "acid test" of the last change made to the 
metadata. Its designed to run relatively quickly and test the change but not 
exhaustively or from scratch. If the tests succeed, its expected that there is 
a 95% chance the change did not cause any regressions. It also checks this 
change builds against an existing build. The developer can expect a warning 
about problems with a change quickly from these tests.
Full - This is a test of the metadata as a whole from the ground up and 
ensures that the system is self consistent. Its designed to prove there was no 
regression but obviously the tests take longer to run.
Its expected that incremental tests will happen on every commit to the 
metadata and that full testing will happen once every 24 hours at a minimum. 
Its also desirable that in future, separate branches can be tested outside of master and only merged into the main branch when a test suite completes.
By doing this publicly, we're forced to keep the metadata buildable, 
consistent and people can see that we do this and care about the quality of 
what we're doing.
Eventually it would be desirable to also test the build output of the system 
either under emulation or on the real hardware in the form of "board labs". 
There is an significant capital expense of setting these up as well as a 
current lack of software to support their use. It would however fit well in 
the Yocto Project testing infrastructure if we can reach a point where this is 
practical.
== Yocto Project Components: Partners ==
Partners are an important part of the Yocto Project as we want to defragment the embedded 
market and we can't do this if we don't build relationships with others. Its 
one of the key areas many other efforts in this marketplace fail as they tend 
not to cover the needs of the majority of the community.
The main criteria for becoming a supporting partner will be agreeing to 
standardise working practices in line with the Yocto Project's objectives and most likely 
starting to use it as a standard. This may also involve offering support and 
development of its tools and infrastructure.
In return partners benefit from the defragmented embedded market and jointly 
improve and advanced the embedded Linux software position.
== Yocto Project Components: Documentation ==
Whilst there is a wealth of information online about given projects, nobody 
has managed to effectively document a complete Linux software stack. The 
reasons is related to the size of the undertaking and the effort required to 
maintain it.
Through the Yocto Project build infrastructure its possible that documentation 
generation could be largely automated. We'd have to find a way to bind all the 
separate pieces of documentation together but I think this is solvable 
problem.
This coherent and up to date source of documentation would be another feature 
of the project provided to the community as part of our reference 
implementation. There would be a documentation collection corresponding to 
each release as well as a version corresponding to the latest master branch.
An added benefit of this would be that individual software projects would be 
encouraged to put their documentation into order if they could see a high 
profile consumer of it which is inline with the Yocto Project's objectives of 
standardisation and improving the embedded ecosystem.
== Build System - More details ==
The structure of OpenEmbedded and Poky are specifically tailored to the needs 
of the embedded market space. One of the main ways this happens is the 
powerful way the metadata is split into separate configuration areas:
: “Recipe” - details of a software package
The recipe represents individual software components of the system, usually 
corresponding to a project which then goes on to make a release tarball. The 
recipe contains information on where to get this source code, how to extract 
it, information on any tweaks it may need for known issues (patches), what 
configuration options this software has and how to apply them, how to compile 
it and how to extract and split the output into packages.
There can be one definition of a software project which is them shared by 
definitions for specific version releases of the software.
: “Distro” - policy of the resulting system
The distribution configuration is concerned with policy decisions such as 
which ABI to use, which versions of software should be preferred and is a 
representation of choices available and the values selected.
: “Machine” - details of a specific hardware target
The machine entities are information about particular hardware targets. They 
can be more general such as "netbook" or provide information about specific 
hardware like a Sharp Zaurus handheld SL-C3000 (spitz). These given 
information about the capabilities of the hardware such as the presence of a 
keyboard, touchscreens, displays, audio capabilities, I/O interfaces and so 
on. Also details about the best formats to provide images for the device can 
be contained here.
This information is used by the system to customise output to work well on 
specific hardware.
: “Image” - groups of software packages
These are package groups which provide a working end result. "Sato" images 
contain all the functionality required to boot the Sato UI for example. A 
console image contains enough to boot a system to a console shell prompt. The 
contents of an image will depend on information from both the distribution and 
machine configurations.
Each of these functionality areas are independent of the others and one can be 
changed or customised separate from the others. Bitbake through the structure 
of the metadata is able to combine the information from each section to 
provide a specifically tailored end result.
The second powerful feature of the metadata structure is the overlay or layer 
capability. By combining several layers of metadata, a basic set of 
functionality can be extended or customised at multiple levels allowing a 
basic core to be extended and customised for different uses.
For example we could have the following metadata stack:
* Core Poky Metadata
* MeeGo specific Overlay (adding the MeeGo UI software packages)
* Vendor specific BSP (adding support for a particular hardware platform)
* Commercial Overlay (provided by say Wind River adding commercialisation to the underlying layers)
* Customer Specific Overlay (adding customisations only relevant to the customer)
At each level, features can be added and underlying layers can be customised 
without duplicating code. The changes anyone makes also remain clearly 
isolated from the underlying system meaning they're clearly identifiable and 
can easily be ported to newer versions of other layers such as the core 
metadata.
This structure encourages reuse of code and sharing and collaboration of 
metadata. An objective is to make the core as useful as possible so over time 
code initially developed as upper layers will tend to head towards the core 
where it makes sense. There will be functionality blocks such as "meego" which 
make sense as an overlay in their own right and customisations which will only 
ever apply to an end user's application too and this structure supports them 
all well.
== Build System - Configurability ==
Due to its modular nature, OE and Poky are very much infrastructures rather 
than monolithic projects. Any one component is usually isolated from the rest 
of the system meaning that if some area is found to need improvement, this can 
happen and the technology can move forwards without impacting other areas of 
the system. Pretty much everything is configurable.
Implicit assumptions in the system have been carefully avoided. Issues such as 
output package format(s) or output image format(s) are abstracted such that 
multiple outputs are available and future developments can be added too, very 
easily. This makes the system very adaptable to almost any situation at a 
relatively low cost compared to other systems.
== Build System - Cross compiling and multi architecture ==
Firstly, it needs to be realised that the build system used has to be a multi architecture 
system. If it is specific to a single platform such as IA, it will 
automatically be ruled out of the embedded market and the community focus 
which we'd like to see will not happen. It therefore makes sense to have a 
multi architecture system.
Regardless of architecture issues, for embedded systems its always likely 
there will be some differences between the build system's configuration and 
that of the target systems. This may be different configuration options, 
compiler flags, package versions and many other possible differences, it's 
impractical to even list them all.
As such some way of isolating the target system from the build system is 
essential. The ways of doing this have been a topic of discussion for many 
years and will continue to be. Various approaches such as chroots, emulated 
environments, bare cross compilation and cross compilation using sysroots are 
all possibilities.
In the general multi architecture case the build system is most likely 
different from the target architecture. This can apply on IA with a 64 bit 
build system building for a 32 bit target and vice versa. Since emulation and 
chroot aren't always possible or reliable, OE and Poky are a cross compiling 
build system making the best use of new cross compilation technology such as 
sysroots as it becomes available.
In the embedded space, cross compilation is considered standard practise and 
widely accepted. The difficulties of cross compiling are also continually 
being reduced with new technology.
== Build System - Any downsides? ==
The biggest downside of OE/Poky by far is the shear complexity of the inner 
workings of the systems. They have been designed to be powerful and 
comprehensive which means there is a steep learning curve to a true deep level 
understanding of them. There are also several concepts which are different to 
a traditional 'desktop' Linux system.
Poky has been at the leading edge of making the systems easier to use and is 
designed such that anyone with some basic Linux knowledge should be able to 
download it and build a Linux system from scratch. It also has a comprehensive 
manual covering many of the questions a new user to the system has. As always 
there is room for improvement but there is a strong foundation to build on.
== Build System - Vendor Use Cases and Development Process ==
The end result most vendors require are software "images" which then can 
install onto the end products. In some cases these will be field upgradable 
either through placing a new image onto the device in totality or in some 
cases though package upgrades in a more traditional PC Linux distribution 
style. In order to reach this goal there are several stages of development and 
the Yocto Project infrastructure needs to support them through them all:
: Prototyping - The end hardware is probably not available yet but a quick 
mockup needs to be possible. This may be using CRBs, possibly customised to 
demonstrate key hardware features of the end device or it could be done using 
virtualisation.
: Hardware Development - The system needs to allow for testing of new hardware as its developed.
: Software Development - The system needs to allow development of the software 
that will be used on the final device, often before the final hardware is 
available. Some developers will be interested in the broad system and how all 
the pieces build and fit together, others could just be application developers 
interested in developing a single application. Both sets of developers must be 
supported.
Poky itself fulfils some of these needs but not all of them. There are some 
cases having the full build system available is overkill for a specific 
developers needs. The counterpart is the SDK.
== Software Development Kit (SDK) ==
The SDK is a standalone output from Poky which consists of a sysroot 
representing a target system and some tools which allow the developer to build 
further software to work with that sysroot. Since we have little control over 
the system the developer will be using the SDK on, a standalone cross compiler 
and cross build tools are provided with the SDK to isolate the two systems in 
the same way Poky itself does.
Using this toolchain the developer can build applications and libraries that 
will work with their target platform. Whilst usable from a shell, the SDKs are 
also capable of being used from within IDEs such as Eclipse and Anjuta at the 
preference of the developer.
By having the SDK it further reduces the barriers to entry of developing 
applications for embedded systems since the developer does not have to use the 
build system itself. There is however a migration path should the developer 
require to become more involved with the build system itself.
== Current Status and Areas for Development ==
The Yocto Project itself does not currently exist and needs to be established as detailed 
above. OE and Poky do exist and are powerful and already extensively developed 
but there are some gaps in functionality which there are some gains by 
addressing:
* By filling functionality gaps it promotes OE/Poky/Yocto Linux as the standard in the marketplace
* By providing this technology first in Poky, it promotes Poky and Yocto Linux 
* Some functionality is required to enable adoption of commercial solutions vendors such as Wind River where features they currently enjoy in systems like LDAT are unavailable in Poky.
The Yocto Project itself goes a long way to solving a number of problems by pulling pieces 
like the various SDK plugins and other technologies into one place which can 
then be consistently documented. There is also a list of features specifically being worked on for Poky and Bitbake.
== Yocto Linux Development Team Structure ==
To fulfil its objectives we anticipate the need for 3 different teams of 
people working on the project:
=== Embedded Core Team ===
This team provides the core guidance on the project and consists of the 
architect and technical direction, programme management, some developers 
working on the overall picture of the project along with people responsible 
for overall documentation and QA.
=== Embedded Tools and SDK Team ===
This team is responsible for improvements to the tools and infrastructure. 
They would be implementing features like the server mode of bitbake and its 
integration to something with similarities to Buildbot, the Eclipse and Anjuta 
plugins and SoC support through the Yocto Linux SDK
=== Distribution Engineering ===
This team is much more focused on the day to day maintenance of the metadata, 
upgrading versions when required to keep Poky at the forefront of technology, 
watching for security updates, integrating any relevant technology and 
managing metadata releases and QA.
The Yocto Project team does not directly cover development of BSPs and SoC drivers. The teams 
will advise on integration of code into Yocto Linux in parallel with its submission 
of upstream and assist with any issues that arise however.
Author: Richard Purdie, November 2009
(names have been adapted in the document subsequently for clarity)