Yocto Project 2.5 Release Test Plan

= Reversion history =
Version	Modifier 	Comments
1.0 	Jose Perez Carranza	First draft
=References=
Documents that support this test plan 
= Introduction =
This is the the overall test plan for version 2.5 of the , for all the generic information this document will be pointing to QA Master Test Plan
= Objectives =
This test plan for Yocto Project 2.5 release supports the following objectives:
*To define the the tools to be used throughout the testing process.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
*To define how the tests will be conducted.
*Release Criteria
*Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
= Risk Analysis & Contingency Plan =
This is the document for Risk Analysis https://docs.google.com/spreadsheets/d/1K-MZYo5sfabwI2UozGp6Uuh2sw5y309bXkyEZRPEEgk/edit#gid=0
= Test Strategy =
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.5 Release
== Bugs Tracking ==
*Bugs that need to be implemented by QA Team
** 2.4 QA Assigned Bugs
*Bugs that need to be verified by QA Team 
**2.5_QA_Bugs_To_Verify
*Old Bugs that need to be Verified
**Old resolved bugs and features
= Test Items =
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. 
 * centos 7 
 * Debian 9
 * Fedora 26
 * Fedora 27
 * OpenSuse42.3
 * Tumbleweed (dropped)
 * ubuntu 16.04 
 * ubuntu 17.10
= Test Cases =
*Yocto 2.5 Test Cases
= Test Matrix =
The components to be tested for version 2.5 of the against the different milestones are included in this section:
Code of Colors
Active
Tracked on Testpoia
Active
Non Tracked on Testopia
In Design
Will be tracked on Tesopia
Inactive
== ADT ==
Component	Environment 	Auto	Manual	Owner
ADT 	Fedora 25 i686 	Y 	Y 	PG
ADT 	Ubuntu 16.04 x86_64	Y 	Y 	PG
== Eclipse Plugin ==
Component 	Environment	Auto	Manual	Owner
Neon + Any Distro 	 	 	Y 	PG
Oxygen + Any Distro	 	 	Y 	PG
== Kernel ==
Component	Environment	Auto	Manual	Owner
Kernel 	Any Distro 	 	Y 	PG
== Meta-Yocto ==
Component 	Environment	Auto	Manual	Owner
Meta-Yocto	Any Distro 	 	Y 	PG
== Toaster ==
Component	Environment	Auto	Manual	Owner
Toaster 	Any Distro 	Y 	Y 	PG
== eSDK ==
Component	Environment 	Auto	Manual	Owner
eSDK 	Any Distro + runtime 	Y 	 	PG
eSDK 	Any Distro + selftest	Y 	 	PG
== Selftest ==
Component	Environment 	Auto	Manual	Owner
OE-CORE 	Ubuntu 16.10 	Y 	 	PG
OE-CORE 	Ubuntu 16.04 	Y 	 	PG
OE-CORE 	OpenSuse 42.2	Y 	 	PG
OE-CORE 	Opensuse 13.2	Y 	 	PG
OE-CORE 	Debian 8 	Y 	 	PG
OE-CORE 	Centos 7 	Y 	 	PG
OE-CORE 	Fedora 24 	Y 	 	PG
OE-CORE 	Fedora 25 	Y 	 	PG
== Bitbake ==
Component	Environment	Auto	Manual	Owner
Bitbake 	Any Distro 	 	Y 	PG
== Build - Apliance ==
Component 	Environment 	Auto	Manual	Owner
Build - Apliance	VM - Any Distro	 	Y 	PG
== Package Manager ==
Component 	Environment	Auto	Manual	Owner
Package Manager	Any Distro 	 	Y 	PG
== Performance == 
Component 	Environment	Auto	Manual	Owner
Performance	Fedora 23 	Y 	 	GDC
Performance	Ubuntu 15 	Y 	 	GDC
== BSP/QEMU ==
Component	Arch 	HW 	Environment 	Image 	Auto	Manual	pTest	Compliance
BSP 	x86 	MinwMax 32bit 	genericx86 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	x86 	MinwMax 32bit 	genericx86 	core-image-lsb-sdk 	Y 	 	 	
BSP 	x86 	MinwMax 64bit 	genericx86-64	core-image-sato-sdk 	Y 	 	 	
BSP 	x86 	MinwMax 64bit 	genericx86-64	core-image-lsb-sdk 	Y 	 	 	
BSP 	x86 	MinwMax 64bit 	genericx86-64	core-image-tiny-initramfs	Y 	 	 	
BSP 	x86 	NUC 64 	genericx86-64	core-image-sato-sdk 	Y 	Y 	Y 	
BSP 	x86 	NUC 64 	 	core-image-lsb-sdk 	Y 	 	 	Y
BSP 	x86 	QEMU 	qemux86 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	x86 	QEMU 	qemux86 	core-image-lsb-sdk 	Y 	 	 	
BSP 	x86 	QEMU 	qemux86-64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	x86 	QEMU 	qemux86-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	non x86 	EdgeRouter 	EdgeRouter 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	MPC8315e-rdb 	MPC8315e-rdb 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	Beaglebone Black 	Beaglebone 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemuarm 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemuarm-64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemuppc 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemumips 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemumips64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	Meta-Intel	MinnowMax 32bit 	core2-32 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	Meta-Intel	MinnowMax 32bit 	core2-32 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	MinnowMax 64bit 	corei7-64 	core-image-sato-sdk 	Y 	 	 	
BSP 	Meta-Intel	MinnowMax 64bit 	corei7-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	NUC 	corei7-64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	Meta-Intel	NUC 	corei7-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	CherryHill 	corei7-64 	core-image-sato-sdk 	Y 	 	 	
BSP 	Meta-Intel	CherryHill 	corei7-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	Joule 	corei7-64 	core-image-sato-sdk 	Y 	 	 	
BSP 	Meta-Intel	meta-intel-quark 32bit	intel-quark 	Core-image-sato-sdk 	Y 	 	 	
All component to be tested on 2.5 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
= Testopia Templates =
COMPONENT 	TEMPLATE
BitBake 	TRTEMP_2.4_MANUAL_BitBake
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto 	TRTEMP_2.4_MANUAL_Meta-yocto
OE-CORE 	TRTEMP_2.4_AUTO_OE-Core
Build-Appliance 	TRTEMP_2.4_MANUAL_Build-Appliance
Runtime 	TRTEMP_2.4_AUTO_Runtime
Toaster 	TRTEMP_2.4_AUTO_Toaster_UI
Toaster 	TRTEMP_2.4_MANUAL_Toaster_UI-managed-mode
ADT 	TRTEMP_2.4_MANUAL_ADT
ADT 	TRTEMP_2.4_AUTO_ADT
CROPS 	TRTEMP_2.4_AUTO_CROPS_Poky
CROPS 	TRTEMP_2.4_MANUAL_CROPS_eSDK
Eclipse 	TRTEMP_2.4_MANUAL_Eclipse
eSDK 	TRTEMP_2.4_AUTO_eSDK_sdkext
eSDK 	TRTEMP_2.4_AUTO_eSDK_oeselftest
Kernel 	TRTEMP_2.4_MANUAL_Kernel
Package Management	TRTEMP_2.4_MANUAL_Package-Management-Updates
= Features pending to be tested =
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
ADD FEATURES !!!!!!!!!!
The complete List of QA Implementations for YP 2.4 can be found at Features Implementation by QA Team
= Features Not to be tested =
*
=Approach=
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible. 
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
= Pass / Fail Criteria =
This criteria is defined in Yocto_Project_v2.4_Status#Yocto_Project_v2.4_Release_Criteria
= Exit Criteria =
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
= Test Deliverables =
* Live Status 2.5_QA_Status
* Final execution Report #Execution History
* Mail sent to the mailing list
= Test Environment =
There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.
= Schedule =
== M1 (May 8, 2017 to Jun. 23, 2017 ==
=== M1 Feature/Tasks Development / Cutoff is Jun. 12, 2017 ===
=== M1 Bugs ===
The planed schedule for 2.4 can be found at Yocto_2.4_Schedule
== Live Schedule & Execution History ==
MILESTONE 	STARTED 	FINISHED 	STATUS	REPORT
M1 rc1 	12/27/2017	1/8/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW02_-_2018-1-8-_Full_Test_Cycle_-_2.5_M1_rc1
M1 rc3 	1/8/2018 	1/18/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW03_-_2018-1-18-_Full_Test_Cycle_-_2.5_M1_rc3
M1 rc1 meta-intel 	2/1/2018 	2/7/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW06_-_2018-02-07-_Full_Test_Cycle_-_2.5_M1_rc1_-_meta-intel
M2 rc1 	2/7/2018 	2/14/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW07_-_2018-02-14-_Full_Test_Cycle_-_2.5_M2_rc1
M3 rc1 	3/22/2018 	4/9/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW15_-_2018-04-09-_Full_Test_Cycle_-_2.5_M3_rc1
M4 rc1 	4/25/2018 	5/10/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW19_-_2018-05-10-_Full_Test_Cycle_-_2.5_M4_rc1
M4 rc1 meta-intel 	5/18/2018 	5/31/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW22_-_2018-05-31-_Full_Test_Cycle_-_2.5_M4_rc1_-_meta-intel
2.5.1.rc1 	8/7/2018 	8/16/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW33_-_2018-08-16-_Full_Test_Cycle_-_2.5.1_rc1
2.5.1.rc1 meta-intel	9/25/2018 	9/28/2018 	DONE 	https://wiki.yoctoproject.org/wiki/WW39_-_2018-09-28-_Full_Test_Cycle_-_2.5.1_rc1_-_meta-intel
2.5.2.rc1 	12/06/2018	12/14/2018	DONE 	https://wiki.yoctoproject.org/wiki/WW49_-_2018-12-06-_Full_Test_Cycle_-_2.5.2_rc1