Yocto Project 2.4 Release Test Plan

= Reversion history =
Version	Modifier 	Comments
1.0 	Jose Perez Carranza	First draft
=References=
Documents that support this test plan 
= Introduction =
This is the the overall test plan for version 2.4 of the , for all the generic information this document will be pointing to QA Master Test Plan
= Objectives =
This test plan for Yocto Project 2.4 release supports the following objectives:
*To define the the tools to be used throughout the testing process.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
*To define how the tests will be conducted.
*Release Criteria
*Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
= Test Strategy =
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.4 Release
== Bugs Tracking ==
*Bugs that need to be implemented by QA Team
** 2.4 QA Assigned Bugs
*Bugs that need to be verified by QA Team 
**2.4_QA_Bugs_To_Verify
*Old Bugs that need to be Verified
**Old resolved bugs and features
= Test Items =
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. 
 * centos 7 
 * Debian 8
 * Fedora 24 
 * Fedora 25
 * OpenSuse132 
 * OpenSuse422 
 * ubuntu 16.04 
 * ubuntu 16.10
= Test Cases =
*Yocto 2.4 Test Cases
= Test Matrix =
The components to be tested for version 2.4 of the against the different milestones are included in this section:
Code of Colors
Active
Tracked on Testpoia
Active
Non Tracked on Testopia
In Design
Will be tracked on Tesopia
Inactive
== ADT ==
Component	Environment 	Auto	Manual	Owner
ADT 	Fedora 25 i686 	Y 	Y 	WR
ADT 	Ubuntu 16.04 x86_64	Y 	Y 	WR
== Eclipse Plugin ==
Component 	Environment 	Auto	Manual	Owner
Eclipse Plugin	Mars + Any Distro 	 	Y 	GDC
Eclipse Plugin	Neon + Any Distro 	 	Y 	GDC
Eclipse Plugin	Oxygen + Any Distro	 	Y 	GDC
== Kernel ==
Component	Environment	Auto	Manual	Owner
Kernel 	Any Distro 	 	Y 	GDC
== Meta-Yocto ==
Component 	Environment	Auto	Manual	Owner
Meta-Yocto	Any Distro 	 	Y 	WR
== Toaster ==
Component	Environment	Auto	Manual	Owner
Toaster 	Any Distro 	Y 	Y 	GDC
== eSDK ==
Component	Environment 	Auto	Manual	Owner
eSDK 	Any Distro + runtime 	Y 	 	GDC
eSDK 	Any Distro + selftest	Y 	 	GDC
== Selftest ==
Component	Environment 	Auto	Manual	Owner
OE-CORE 	Ubuntu 16.10 	Y 	 	GDC
OE-CORE 	Ubuntu 16.04 	Y 	 	GDC
OE-CORE 	OpenSuse 42.2	Y 	 	GDC
OE-CORE 	Opensuse 13.2	Y 	 	GDC
OE-CORE 	Debian 8 	Y 	 	GDC
OE-CORE 	Centos 7 	Y 	 	GDC
OE-CORE 	Fedora 24 	Y 	 	GDC
OE-CORE 	Fedora 25 	Y 	 	GDC
== Bitbake ==
Component	Environment	Auto	Manual	Owner
Bitbake 	Any Distro 	 	Y 	GDC
== Build - Apliance ==
Component 	Environment 	Auto	Manual	Owner
Build - Apliance	VM - Any Distro	 	Y 	GDC
== Package Manager ==
Component 	Environment	Auto	Manual	Owner
Package Manager	Any Distro 	 	Y 	GDC
== Performance == 
Component 	Environment	Auto	Manual	Owner
Performance	Fedora 23 	Y 	 	GDC
Performance	Ubuntu 15 	Y 	 	GDC
== BSP/QEMU ==
Component	Arch 	HW 	Environment 	Image 	Auto	Manual	pTest	Compliance
BSP 	x86 	MinwMax 32bit 	genericx86 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	x86 	MinwMax 32bit 	genericx86 	core-image-lsb-sdk 	Y 	 	 	
BSP 	x86 	MinwMax 64bit 	genericx86-64	core-image-sato-sdk 	Y 	 	 	
BSP 	x86 	MinwMax 64bit 	genericx86-64	core-image-lsb-sdk 	Y 	 	 	
BSP 	x86 	MinwMax 64bit 	genericx86-64	core-image-tiny-initramfs	Y 	 	 	
BSP 	x86 	NUC 64 	genericx86-64	core-image-sato-sdk 	Y 	Y 	Y 	
BSP 	x86 	NUC 64 	 	core-image-lsb-sdk 	Y 	 	 	Y
BSP 	x86 	QEMU 	qemux86 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	x86 	QEMU 	qemux86 	core-image-lsb-sdk 	Y 	 	 	
BSP 	x86 	QEMU 	qemux86-64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	x86 	QEMU 	qemux86-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	non x86 	EdgeRouter 	EdgeRouter 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	MPC8315e-rdb 	MPC8315e-rdb 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	Beaglebone Black 	Beaglebone 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemuarm 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemuarm-64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemuppc 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemumips 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	non x86 	QEMU 	qemumips64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	Meta-Intel	MinnowMax 32bit 	core2-32 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	Meta-Intel	MinnowMax 32bit 	core2-32 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	MinnowMax 64bit 	corei7-64 	core-image-sato-sdk 	Y 	 	 	
BSP 	Meta-Intel	MinnowMax 64bit 	corei7-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	NUC 	corei7-64 	core-image-sato-sdk 	Y 	Y 	 	
BSP 	Meta-Intel	NUC 	corei7-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	CherryHill 	corei7-64 	core-image-sato-sdk 	Y 	 	 	
BSP 	Meta-Intel	CherryHill 	corei7-64 	core-image-lsb-sdk 	Y 	 	 	
BSP 	Meta-Intel	Joule 	corei7-64 	core-image-sato-sdk 	Y 	 	 	
BSP 	Meta-Intel	meta-intel-quark 32bit	intel-quark 	Core-image-sato-sdk 	Y 	 	 	
All component to be tested on 2.4 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
= Testopia Templates =
COMPONENT 	TEMPLATE
BitBake 	TRTEMP_2.4_MANUAL_BitBake
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86_MMAX32
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_core2-32_MMAX32
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Genericx86-64_NUC
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_NUC
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_corei7-64_MMAX64
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_wic_ANYBSP
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_Beaglebone-Black
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_EdgeRouter
BSP/QEMU 	TRTEMP_2.4_MANUAL_BSP/QEMU_core-image-sato-sdk_MPC8315e-rdb
BSP/QEMU 	TRTEMP_2.4_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYQEMU
Meta-Yocto 	TRTEMP_2.4_MANUAL_Meta-yocto
OE-CORE 	TRTEMP_2.4_AUTO_OE-Core
Build-Appliance 	TRTEMP_2.4_MANUAL_Build-Appliance
Runtime 	TRTEMP_2.4_AUTO_Runtime
Toaster 	TRTEMP_2.4_AUTO_Toaster_UI
Toaster 	TRTEMP_2.4_MANUAL_Toaster_UI-managed-mode
ADT 	TRTEMP_2.4_MANUAL_ADT
ADT 	TRTEMP_2.4_AUTO_ADT
CROPS 	TRTEMP_2.4_AUTO_CROPS_Poky
CROPS 	TRTEMP_2.4_MANUAL_CROPS_eSDK
Eclipse 	TRTEMP_2.4_MANUAL_Eclipse
eSDK 	TRTEMP_2.4_AUTO_eSDK_sdkext
eSDK 	TRTEMP_2.4_AUTO_eSDK_oeselftest
Kernel 	TRTEMP_2.4_MANUAL_Kernel
Package Management	TRTEMP_2.4_MANUAL_Package-Management-Updates
= Features pending to be tested =
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
ADD FEATURES !!!!!!!!!!
The complete List of QA Implementations for YP 2.4 can be found at Features Implementation by QA Team
= Features Not to be tested =
*
=Approach=
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible. 
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
= Pass / Fail Criteria =
This criteria is defined in Yocto_Project_v2.4_Status#Yocto_Project_v2.4_Release_Criteria
= Exit Criteria =
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
= Test Deliverables =
* Live Status 2.4_QA_Status
* Final execution Report #Execution History
* Mail sent to the mailing list
= Test Environment =
There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.
= Schedule =
== M1 (May 8, 2017 to Jun. 23, 2017 ==
=== M1 Feature/Tasks Development / Cutoff is Jun. 12, 2017 ===
=== M1 Bugs ===
The planed schedule for 2.4 can be found at Yocto_2.4_Schedule
== Live Schedule ==
MILESTONE 	STARTED 	FINISHED 	STATUS
M1 rc1 	06/15/2017	06/22/2017	DONE
M1 rc1 meta-intel 	06/23/2017	06/27/2017	DONE
M2 rc2 	07/26/2017	08/02/2017	DONE
M2 rc3 	08/03/2017	08/10/2017	DONE
M2 rc2 meta-intel 	08/11/2017	08/16/2017	DONE
M3 	08/23/2017	08/30/2017	DONE
M3 rc1 meta-intel 	09/01/2017	09/05/2017	DONE
M4 RC1 	09/26/2017	10/06/2017	DONE
M4 RC2 	10/09/2017	10/13/2017	DONE
M4 RC3 	10/17/2017	10/23/2017	DONE
M4 rc1 meta-intel 	10/26/2017	10/31/2017	DONE
M4 rc2 meta-intel 	11/03/2017	11/06/2017	DONE
2.4.1.rc1 	12/08/2017	12/22/2017	DONE
2.4.2.rc2 	2/21/2018 	3/5/2018 	DONE
2.4.2.rc2 meta-intel	3/5/2018 	3/15/2018 	DONE
2.4.3.rc2 	5/25/2018 	6/12/2018 	DONE
2.4.4.rc1 	11/12/2018	11/23/2018	DONE
= Execution History =
MILESTONE 	REPORT
M1 	https://wiki.yoctoproject.org/wiki/WW25_-_2017-06-22_-_Full_Test_Cycle_2.4_M1_rc1
M1 meta-intel 	https://wiki.yoctoproject.org/wiki/WW26_-_2017-06-27-_Full_Test_Cycle_-_2.4_M1_rc1_-_meta-intel
M2 	https://wiki.yoctoproject.org/wiki/WW31_-_2017-07-31_-_Full_Test_Cycle_2.4_M2_rc2
M2 - rc3 	https://wiki.yoctoproject.org/wiki/WW32_-_2017-08-09_-_Full_Test_Cycle_2.4_M2_rc3
M2 - rc2 meta-intel 	https://wiki.yoctoproject.org/wiki/WW33_-_2017-08-15-_Full_Test_Cycle_-_2.4_M2_rc2_-_meta-intel
M3 	https://wiki.yoctoproject.org/wiki/WW35_-_2017-08-30_-_Full_Test_Cycle_2.4_M3
M3 - rc1 meta-intel 	https://wiki.yoctoproject.org/wiki/WW36_-_2017-09-05-_Full_Test_Cycle_-_2.4_M3_rc1_-_meta-intel
M4 - rc1 	https://wiki.yoctoproject.org/wiki/WW40_-_2017-10-06_-_Full_Test_Cycle_2.4_RC1
M4 - rc2 	https://wiki.yoctoproject.org/wiki/WW41_-_2017-10-13_-_Full_Test_Cycle_2.4_RC2
M4 - rc3 	https://wiki.yoctoproject.org/wiki/WW43_-_2017-10-23_-_Full_Test_Cycle_2.4_RC3
M4 - rc1 meta-intel 	https://wiki.yoctoproject.org/wiki/WW44_-_2017-10-31-_Full_Test_Cycle_-_2.4_M4_rc1_-_meta-intel
M4 - rc2 meta-intel 	https://wiki.yoctoproject.org/wiki/WW45_-_2017-11-06-_Full_Test_Cycle_-_2.4_M4_rc2_-_meta-intel
2.4.1.rc1 	https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-21-_Full_Test_Cycle_-_2.4.1_rc1
2.4.2.rc2 	https://wiki.yoctoproject.org/wiki/WW10_-_2018-03-05-_Full_Test_Cycle_-_2.4.2_rc2
2.4.2.rc2 meta-intel	https://wiki.yoctoproject.org/wiki/WW11_-_2018-3-15-_Full_Test_Cycle_-_2.4.2_rc2_-_meta-intel
2.4.3.rc2 	https://wiki.yoctoproject.org/wiki/WW24_-_2018-06-12-_Full_Test_Cycle_-_2.4.3_rc2
2.4.4.rc1 	https://wiki.yoctoproject.org/wiki/WW46_-_2018-11-14_-_Full_Test_Cycle_2.4.4_RC1