Distro Testing Plan

This article is the test plan for enabling OS distributions at the Yocto Project Autobuilder 
= About Distro Testing =
Distro Testing is part of the process of enabling an OS distribution at the Yocto Project Autobuilder. It is intended to catch bugs that are distribution specific and would prevent an Autobuilder worker to use such distribution.
= Test Objectives =
* Verify that Distro executes oe-selftest 
* Verify that Distro is able to execute components (Eclipse, Toaster, WIC)
* Veirfy that Distro is able to build per package type (IPK, DEB, RPM)
* Verify that Distro is able to build different image types (LSB, non LSB)
* Verify that Distro is able to build per architecture (arm, x86)
* Verify that Distro is able to build per bootloader (systed, init)
* Verify that Distro is able to build poky-tiny
* Verify that Distro executes multilib
= Test Strategy =
The strategy will be divided in two groups: 
* Review the test objectives once in an Autobuilder instance when the OS distribution is first-time enabled (one time testing) 
* SWAT monitoring at the Autobuilders
== Staging AutoBuilder ==
When an OS Distribution is chosen for inclusion, it will be setup as a worker of an Autobuilder and a series of build steps will run. Once every Buildset shows no failures, that distro will be added to the list of supported Distros by QA, this is a one-time testing activity, not executed in every milestone o release test cycle.
== Public Autobuilder ==
Once the distro was first time validated by QA team it will be running on public autobuilder different build sets already defined , after that it going to be under the focus of SWAT team
= Process =
This section describes the process to add a new Distro as supported on Yocto Project
File:Distro Steps_v2.PNG
== Open a Bug ==
When a new Distro is identified to be added as a supported candidate a bug should be filed on Bugzilla and has to be assigned to the QA Responsible of the Distro Testing. 
*example of a bug bug 11005
== Install Distro ==
QA responsible should ensure that the new Distro is installed properly on the local AB
=== Process To Install New Distro ===
1. Install the new distro
2. Install the requirements from the Yocto Project Quickstart
3. Start an Autobuilder staging instance
 
 $ git clone http://git.yoctoproject.org/git/yocto-autobuilder 
 
 # Ensure the owner of cloned repo is different than root (Poky does not work with root user)
 $ chown -R ab:ab yocto-autobuilder/
 $ cd yocto-autobuilder
 # Load buildset configuration with nightly-qa-distro 
 $ ln -sf buildset-config.yocto-qa/ buildset-config
 # setup the autobuilder
 $ . yocto-autobuilder-setup
 
 # Generates password for AB Web
 $ htpasswd -b .htpasswd distrouser distropass
 # start autobuilder
 $ ./yocto-start-autobuilder both
4. Open a browser and enter to http://localhost:8010 
 Note: If the browser fails to open the URL review AB logs for errors yocto-controller/twistd.log and yocto-worker/twistd.log
5. Log in with the distrouser and distropass 
6. Enter to http://localhost:8010/builders/nightly-qa-distro and start (force) a build 
7. Wait to the buildsets to end and review the output, load bugs if failures occur
8. Stop the staging Autobuilder
 $ ./yocto-stop-autobuilder both
== Execute Build Sets ==
Build sets to be executed are defined in below table, to add a Distro as supported all the build sets should be PASS on local AB
RPM 	DEB 	IPK 	Component 	Multilib 	World 	oe-selftest
ARM 	x86_64 	x86_32 	Eclipse 	x86 64/32 	x86_64 	
init 	systemd 	systemd 	Toaster 	x86 64/x32 	systemd 	
poky_lsb 	poky 	poky_tiny 	buildtools 	core-image-sato	poky 	
core-image-lsb	core-image-sato-sdk	core-image-minimal	read_only rootfs	poky 	core-image-sato-sdk	
 	 	 	pam 	init/systemd 	 	
 	 	 	Log rotate 	RPM,DEB,IPK 	 	
== Results ==
The QA Responsible should ensure that all the defined Build Sets are executed correctly and raise the proper bugs for the failures that each of the build sets is showing.
== Update Supported Distro List ==
Once all the build sets were executed and all were PASS, then then it can be added to the list of supported Distros, also the unsupported distros should be removed from the list.
To accomplish above step a patch should be sent to the poky mailing list with Sanity Tested Distro File updated, you can refer to the following guide to send a patch, remember top use the correct list. 
=== Sanity Tested Distro File (Snapshot)===
 
 meta-poky/conf/distro/poky.conf
 -----
 SANITY_TESTED_DISTROS ?= " \
 poky-1.7 \n \
 poky-1.8 \n \
 poky-2.0 \n \
 Ubuntu-14.04 \n \
 Ubuntu-14.10 \n \
 Ubuntu-15.04 \n \
 Ubuntu-15.10 \n \
 Ubuntu-16.04 \n \
 Fedora-21 \n \
 Fedora-22 \n \
 Fedora-23 \n \
 CentOS-6.* \n \
 CentOS-7.* \n \
 Debian-7.* \n \
 Debian-8.* \n \
 openSUSE-project-13.2 \n \
=Requirements=
==HW Requirements==
Item 	Hardware requirements
Processor 	Intel Core i5-6260U (1.9 GHz Turbo, Dual Core, 4 MB Cache, 15W TDP)
Memory 	DDR4 32GB
Disk space	2 TB of free disk space is required.
==Software Requirements==
To make sure all those packages are installed on the new distro.
Packages 	Packages 	Packages 	Packages
gawk 	make 	wget 	tar
bzip2 	gzip 	python 	unzip
perl 	patch 	diffutils 	diffstat
git 	cpp 	gcc 	gcc-c++
glibc-devel 	texinfo 	chrpath 	ccache
perl-Data-Dumper	perl-Text-ParseWords	perl-Thread-Queue	socat
findutils 	which 	SDL-devel 	xterm
tigervnc 	tigervnc-server 	python3 	tunctl
gitpython 	python3-pip 	python3-devel 	qemu-kvm
perl-bignum 	 	 	
= Test execution Cycle =
== QA Responsible ==
TBD
meanwhile you can contact José Pérez Carranza
== Public Autobuilder ==
* There will be a continuous execution of random build sets defined on autobuilder, results under foucs of SWAT team
= Exit Criteria for Enabling an OS Distribution =
* All the defined build steps are PASS Execute Build Sets
* Distro was added to the supported list Update Supported Distro
= Supported OS Distributions =
The distributions used are: 
*Poky
*Fedora 
*Ubuntu 
*CentOS 
*OpenSuse 
*Debian 
== Supported OS Distributions Criteria ==
List of supported OS Distributions should be the same at Mega Manual and in variable mentioned in Distro File if is not equal should a bug should be filled.
 
The following criteria is used to see if a Distro and his specific version is supported
*Poky 
** Most 2 recent releases
*Fedora 
** Most 2 recent releases
*Ubuntu 
** Most recent release
** Most recent LTS release
** When above are the same, use most recent 2 releases
*CentOS
** Most recent release
*OpenSuse
** Most recent release
** Most recent LEAP release
*Debian
** Most recent release 
BETA version are considered to enter the cycle of validation as a new distro, but The goal here is to provide some pre-release testing of major OS when their release comes soon after YP release. Once the major release is published the process defined above will be applied and then that distro can be added to the supported list. 
=== List ===
OS Version 	Status
Ubuntu-16.04 LTS	PASS
Ubuntu-17.04 	PASS
Fedora-25 	PASS
Fedora-26 	PASS
CentOS-7 	PASS
Debian-9 	PASS
openSUSE-13.2 	PASS
openSUSE-42.2 	PASS
== Supported OS Distributions Release Calendar ==
This section contains the links to the supported distros schedules, should be tracked at beginning of every YP release to check which DIstro may be candidate to be added during that release cycle.
*Ubuntu 
*Fedora
*Debian
*OpenSuse
*CentOS