BSP Test Plan

= About BSP =
A Board Support Package (BSP) is the collection of software and data that allows an operating system to run in a specific hardware. It contains the drivers, routines and information that defines how to support a particular hardware device, set of devices, or hardware platform.
= Objectives =
Verify different supported images generated by Yocto Project build system on main AB, manual and automated test are applied in both physical HW and emulated with qemu.
= Team members =
Member	Team
Ee Peng Yeoh	Intel
Sangeeta Jain	Intel
Yi Zhao	Wind River
= Scope =
Types of Tests 
* Manual tests on different platforms
* Automated test on different platforms
Platforms, type of images and type of tests are showed detailed on BSP Test Matrix. 
Features tested
* Manual Tests
** Boot image 
** Install Image 
** Multimedia
** Graphical interface components 
** System reboot and shutdown recovery and standby 
** X server tests 
** rpm packages 
** File system (reading/writing)
** Mont / Umont 
** Network
complete test steps of these manual tests are on poky repo under meta/lib/oeqa/manual/ <>
* Automated Tests 
** buildcpio
** buildgalculator
** buildlzip
** connman
** date
** df
** dnf
** gcc
** kernelmodule
** ldd
** logrotate
** multilib
** oe_syslog
** pam
** parselogs
** perl
** ping
** ptest
** python
** rpm
** scanelf
** scp
** skeletoninit
** ssh
** systemd
** x32lib
** xorg
full code of this automated test are on poky repo under meta/lib/oeqa/runtime/cases <>
= Test Strategy =
As mentioned below there are 2 types of test Automated an Manual tests, below a description of how those test are applied.
==Manual==
Test cases executed fully manually on real HW and on a qemu instance, these tests are focused on user scenarios using the image and also ensuring that graphical interface and multimedia are properly working on the image. 
==Automated Tests==
Test cases execute automatically used testimage suite as well as 2 special test called pTest and Compliance, these tests are written in Python using oeqa framework and check different content and packages behavior of the booted image by a series of commands executed by a ssh communication.
==Test Process==
 1. Prepare the image and device under test
 
 1.1 Download Artifacts
 there are 2 links available from Public AutoBuilder to download images 
 http://autobuilder.yoctoproject.org/pub/releases/
 https://autobuilder.yocto.io/pub/releases/
 
 Artifacts to be downloaded are: 
 
 Real HW
 under folder machines/<MACHINE_UNDER_TEST>
 - <IMAGE_TO_BE_TESTED>.hddimg
 - <IMAGE_TO_BE_TESTED>.wic
 - <IMAGE_TO_BE_TESTED>.wic.bmap
 - <IMAGE_TO_BE_TESTED>.manifest
 - <IMAGE_TO_BE_TESTED>.testdata.json
 example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-genericx86.*
 
 QEMU
 under folder machines/qemu/<MACHINE_UNDER_TEST>
 - <BZIMAGE>.bin
 - <IMAGE_TO_BE_TESTED>.ext4
 - <IMAGE_TO_BE_TESTED>.qemuboot.conf
 - <IMAGE_TO_BE_TESTED>.manifest
 - <IMAGE_TO_BE_TESTED>.testdata.json
 example of <IMAGE_TO_BE_TESTED> name core-image-sato-sdk-qemux86.*
 
 1.2 For real hardware devices:
 - Write the image to a bootable media (USB key, SD Card, etc.) using the dd command:
 $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=TARGET_DEVICE
 example with usb key connected: $ sudo dd if=tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.wic of=/dev/sdb
 - Attach the bootable device to the DUT
 - Turn the DUT on, and use its boot options to select the correct media for booting
 - After the DUT boots to image in USB stick, the default username is "root". (This is because the yocto image by default uses "debug-tweaks" feature)
 - Ensure the device has a network connection and a valid IP address
 2. For Manual Test Cases:
 
 2.1 - Go to the proper Test Run assigned on the execution and start executing test cases 
 3. For automated Test Cases:
 
 3.1 Clone poky repo $git clone git://git.yoctoproject.org/poky
 3.2 $cd poky 
 3.3 Checkout to the commit specified on the release information $git checkout <commit-id>
 3.4 If testing meta-intel, clone the layer repo inside poky $git clone git://git.yoctoproject.org/meta-intel
 3.4.1 $cd meta-intel
 3.4.2 Checkout to the meta-intel commit specified on the release information $git checkout <commit-id>
 3.4.3 Return to the poky root path $cd ..
 3.5 $source oe-init-build-env
 3.6 If testing meta-intel, add the layer to conf/bblayers.conf. It may be done with:
 $bitbake-layers add-layer "../meta-intel"
 3.7 For real HW edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding: 
 Install / Boot the image on the DUT and check the IP
 INHERIT += "testimage"
 TEST_TARGET = "simpleremote"
 TEST_SERVER_IP = "HOST ip" --> IP of the machine being used to launch tests
 TEST_TARGET_IP = "DUT ip" --> IP of the device under test
 if image is LSB 
 DISTRO ?= "poky-lsb"
 3.8 For QEMU edit local.conf. PATH: ~/poky/build/conf, at the end of the file adding:
 INHERIT += "testimage"
 if image is LSB 
 DISTRO ?= "poky-lsb"
 3.9 Make sure conf only set DISTRO once, if DISTRO was set more than once, during testimage you could face error: 
 > ERROR: Nothing PROVIDES 'core-image-lsb-sdk'
 > core-image-lsb-sdk was skipped: missing required distro feature 'pam'
 > (not in DISTRO_FEATURES)
 For releases before 2.3 also add
 TEST_SUITES = " auto" 
 
 Remember to add the machine to local.conf, for example:
 If genericx86 is needed, MACHINE ?= "genericx86"
 If qemux86 is needed, MACHINE ?= "qemux86"
 If intel-corei7-64 is needed, MACHINE = "intel-corei7-64"
 4. Prepare the environment to execute test suite 
 
 4.1 $bitbake rpm busybox curl run-postinsts
 4.2 $bitbake package-index
 4.3 Ensure that a folder is created, otherwise create it, under
 tmp/deploy/images/<MACHINE>, where <MACHINE> corresponds to the variable defined on
 local.conf on previous steps
 4.4 Paste artifacts downloaded on previously created folder 
 5. Execute the test suite
 
 5.1 $bitbake <IMAGE_TESTED> -c testimage
 example bitbake core-image-sato-sdk -c testimage -v
 6. Collect the test result from <build>/tmp/log/oeqa/testresults.json
 Note: Collect test results after one test run completed.
 7. Verify that all test cases PASSED if not raise bugs properly 
 [Deprecated] 
 Update results to corresponding Testopia Test Run 
 There is a script available to upload results form the log automatically on QA-Tools and follow this example Update TR by ID
==Bug Reporting==
For correctly report the bugs follow the already defined steps here
= Special Tests =
There are 2 special test that also applied as BSP Components those are:
 
== pTest ==
For detailed information about behavior and characteristics check Ptest
=== Test Process ===
This required 2 machine: 
a) host machine - to start/control poky/oeqa/runtime tests
b) device under test (DUT) - to boot up the target ptest image and to run the ptest trigger by host machine
 1. Download pTest image from autobuilder (core-image-sato-sdk-ptest-genericx86-64.hddimg)
 2. On DUT, install or write the image on a media connected to the DUT 
 Configure the network so it be able to work externally:
 - Export the proxy using "export http_proxy=<add your proxy link>" command eg
 $ export http_proxy=http://<my_proxy>:<port>
 - And check if you have internet connection, typing on terminal:
 $ wget www.google.com
 3. On host machine
 Configure for running oeqa/runtime test (refer to above BSP runtime test instruction)
 Add to local.conf, 
 INHERIT += "testimage"
 TEST_SUITES = "ping ssh ptest"
 Then, start the runtime test and collect the test result from <build>/tmp/log/oeqa/testresults.json
=== Reporting Process ===
1. Generate the report of pTests as follows: 
 1.1 Collect the testresults.json files for both current release (base) and previous release (target). 
 1.2 Use "resulttool report" to compute the ptest summary result using current release (base). 
 $ resulttool report <file_path_to_current_ptest_testresultsjson>
 1.3 Use "resulttool regression" to compute the ptest regression as below
 $ resulttool regression <file_path_to_current_ptest_testresultsjson> <file_path_to_previous_ptest_testresultsjson>
 1.4 Copy and paste the report and regression report to wiki page https://wiki.yoctoproject.org/wiki/Ptest/archive
== Compliance ==
In this section, 3 Test suites are executed on NUC HW with a core-image-lsb-sdk-genericx86-64 image.
*LSB tests
*POSIX tests
*LTP tests
=== Test Process ===
 1. Download the core-image-lsb-sdk-genericx86-64.hddimg image from Autobuilder (same image as in LSB automated testing for genericx86-64-lsb bsp)
 2. Install the image on the DUT
 * We test compliance on a NUC
 3. Configure the network so it be able to work externally:
 * edit /etc/resolv.conf and add the gateway ip_address
 * add the ip and netmask using "ifconfig" command
 * add the route using "route add default gw <ip_address>" 
 * export the proxy using "export http_proxy=<add your proxy link>" command
 : there is a bug and if you make these steps in another order than above, it may be possible not work
 4. Copy "compliance_test.py" script on DUT
 script can be found here
 5. Make sure that your network connection is working
 6. Run the script like this:
 * make the script executable: "chmod a+x compliance_local.py"
 * run in command line the following command "./compliance_test.py <milestone> <date>"
 
 7. Get the logs from DUT:
 - result-<milestone>-data.fulllog
 - result-<milestone>-data.log
 - result-<milestone>-data.fail
 - posix.log (can be found in: /opt/ltp/testcases/open_posix_testsuite)
 - the three others are found in /opt/ltp directory, in output, temp, result folders . The logs need to be sent to yi.zhao@windriver.com specifying the
 version and the type of image
 - in /var/opt/lsb/test/manager/results/x86.../x86....tar.gz (you can find it with auto-complete(tab) easily)
 -- it has like 18 Mb
 -- upload this file on drive and send the link to yi.zhao@windriver.com
 -- also I'll fwd an email to see how it looks
 8. Put the tests from Testopia - Runtime test run on passed
=== Reporting Process ===
<To Be Updated>
= Add New Tests =
<To be Updated>
=Test Approach=
=Test Matrix=
Yocto QA Cycle
 
Release 	CPU Class	HW Platform 	BSP Name 	linux-yocto	Image-type 	AUTO	MANUAL	Compliance	pTest
generic BSPs (x86) 	Big Core 	MinwMax 32bit 	genericx86 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	Y 	 	 	
 	 	 	genericx86-wic 	4.x 	core-image-sato-sdk WIC	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	MinwMax 64bit 	genericx86-64 	4.x 	core-image-sato-sdk 	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	genericx86-64-wic	4.x 	core-image-sato-sdk WIC	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	NUC 	genericx86-64 	4.x 	core-image-sato-sdk 	Y 	Y 	 	Y
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	Y 	 	Y 	
 	 	 	genericx86-64-wic	4.x 	core-image-sato-sdk WIC	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	VM 	QEMU 	qemux86 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	 	4,x 	core-image-sato-sdk WIC	Y 	 	 	
 	 	 	qemux86-64 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	 	4,x 	core-image-sato-sdk WIC	Y 	 	 	
generic BSPs (n-x86)	MIPS 	EdgeRouter 	EdgeRouter 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	 	4,x 	core-image-sato-sdk WIC	Y 	 	 	
 	PPC 	MPC8315e-rdb 	MPC8315e-rdb 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	 	4,x-ltsi 	core-image-sato-sdk WIC	Y 	 	 	
 	ARM 	Beaglebone Black	beaglebone 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	 	4,x-ltsi 	core-image-sato-sdk WIC	Y 	 	 	
 	VM 	QEMU 	qemuarm 	4.x 	core-image-sato-sdk 	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	qemuarm64 	4.x 	core-image-sato-sdk 	Y 	Y 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	qemuppc 	4.x 	core-image-sato-sdk 	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	qemumips 	4.x 	core-image-sato-sdk 	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
 	 	 	qemumips64 	4.x 	core-image-sato-sdk 	Y 	 	 	
 	 	 	 	4,x-ltsi 	core-image-lsb-sdk 	 	 	 	
Meta-Intel
Release 	CPU Class	HW Platform 	BSP Name 	Image-type 	Auto	Manual
meta-intel	Big Core 	MinnowMax 64bit 	corei7-64 	core-image-sato-sdk 	Y 	
meta-intel	Big Core 	MinnowMax 64bit 	corei7-64 	core-image-lsb-sdk 	Y 	
meta-intel	Big Core 	MinnowMax 64bit 	corei7-64 	core-image-sato-sdk WIC	Y 	
meta-intel	Big Core 	NUC 	corei7-64 	core-image-sato-sdk 	Y 	Y
meta-intel	Big Core 	NUC 	corei7-64 	core-image-lsb-sdk 	Y 	
meta-intel	Big Core 	NUC 	corei7-64 	core-image-sato-sdk WIC	Y 	
meta-intel	Big Core 	CherryHill 	corei7-64 	core-image-sato-sdk 	Y 	
meta-intel	Big Core 	CherryHill 	corei7-64 	core-image-lsb-sdk 	Y 	
meta-intel	Big Core 	CherryHill 	corei7-64 	core-image-sato-sdk WIC	Y 	
meta-intel	Big Core 	Joule 	corei7-64 	core-image-sato-sdk 	Y 	
meta-intel	Big Core 	Joule 	corei7-64 	core-image-lsb-sdk 	 	
meta-intel	Big Core 	Joule 	corei7-64 	core-image-sato-sdk WIC	Y 	
meta-intel	Big Core 	MinnowMax 32bit 	core2-32 	core-image-sato-sdk 	Y 	Y
meta-intel	Big Core 	MinnowMax 32bit 	core2-32 	core-image-lsb-sdk 	Y 	
meta-intel	Big Core 	MinnowMax 32bit 	core2-32 	core-image-sato-sdk WIC	Y 	
meta-intel	Big Core 	meta-intel-quark 32bit	intel-quark	core-image-sato-sdk 	Y 	
meta-intel	Big Core 	meta-intel-quark 32bit	intel-quark	core-image-lsb-sdk 	 	
meta-intel	Big Core 	meta-intel-quark 32bit	intel-quark	core-image-sato-sdk WIC	Y 	
=Not Tested=
=Requirements=
=Tools=