Yocto Project 2.3 Release Test Plan

= Reversion history =
Version	Modifier 	Comments
1.0 	Jose Perez Carranza	First draft
=References=
Documents that support this test plan 
= Introduction =
This is the the overall test plan for version 2.3 of the , for all the generic information this document will be pointing to QA Master Test Plan
= Objectives =
This test plan for Yocto Project 2.3 release supports the following objectives:
*To define the the tools to be used throughout the testing process.
*To communicate to the responsible parties the items to be tested, set expectations around schedule, and define environmental needs.
*To define how the tests will be conducted.
*Release Criteria
*Risks and contingencies
QA_Master_Test_Plan#Objectives_and_Tasks
= Test Strategy =
Is followed the general strategy defined on QA_Master_Test_Plan#Testing_Strategy and below is showed specific information applicable only for 2.3 Release
== Bugs Tracking ==
*Bugs that need to be implemented by QA Team
** 2.3 QA Assigned Bugs
*Bugs that need to be verified by QA Team 
**2.3_QA_Bugs_To_Verify
*Old Bugs that need to be Verified
**Old resolved bugs and features
= Test Items =
The Yocto Project QA team verifies each release against recent versions of the most popular Linux distributions that provide stable releases. 
 * centos 7 
 * Debian 8
 * Fedora 23 
 * Fedora 24
 * OpenSuse132 
 * OpenSuse421 
 * ubuntu 14.04 
 * ubuntu 16.04
= Test Cases =
*Yocto 2.3 Test Cases
= Components to be tested =
The components to be tested for version 2.3 of the against the different milestones are included in this section:
Code of Colors
Active
Tracked on Testpoia
Active
Non Tracked on Testopia
In Design
Will be tracked on Tesopia
Inactive
COMPONENT 	TEMPLATE
ADT 	TRTEMP_2.3_MANUAL_ADT
ADT 	TRTEMP_2.3_AUTO_ADT
BSP 	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYQEMU
BSP 	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_ANYBSP
BSP 	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-lsb-sdk_ANYBSP
BSP 	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk-wic_ANYBSP
BSP 	TRTEMP_2.3_AUTO_BSP/QEMU_core-image-sato-sdk_intel-quark
BSP 	TRTEMP_2.3_MANUAL_BSP/QEMU_core-image-sato-sdk_ANYQEMU
BSP 	TRTEMP_2.3_MANUAL_BSP/QEMU_core2-32_MinnowMax
BSP 	TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_MinnowMax
BSP 	TRTEMP_2.3_MANUAL_BSP/QEMU_corei7-64_NUC
BSP 	TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86-64_NUC
BSP 	TRTEMP_2.3_MANUAL_BSP/QEMU_GenericX86_MMAX32
CROPS 	TRTEMP_2.3_AUTO_CROPS
CROPS 	TRTEMP_2.3_MANUAL_CROPS
Eclipse-Plugin 	TRTEMP_2.3._MANUAL_Eclipse-Plugin
Kernel 	TRTEMP_2.3_MANUAL_Kernel
Meta-Yocto 	TRTEMP_2.3_MANUAL_Meta-yocto
Toaster 	TRTEMP_2.3_AUTO_TOASTER_UI
Toaster 	TRTEMP_2.3_MANUAL_TOASTER_UI-managed-mode
eSDK 	TRTEMP_2.3_AUTO_eSDK
eSDK 	TRTEMP_2.3_MANUAL_eSDK
OE-CORE / BitBake 	TRTEMP_2.3_AUTO_OE-Core
OE-CORE / BitBake 	TRTEMP_2.3_MANUAL_BitBake
Runtime (Compliance) 	TRTEMP_2.3_AUTO_Runtime
Build Apliance 	TRTEMP_2.3_MANUAL_Build-Appliance
pTest 	
Performance 	Performance Test
WUpd (Software Updater)	
Distributed Builds 	
MRAA 	
Meta-SwUpd 	
All component to be tested on 2.3 are in #Test Items a detailed list of features per component can be found in Test Plan of the components QA_Master_Test_Plan#Component
= Features pending to be tested =
Here is a list of areas that are not yet covered and need some work, implement test cases and add it to a component that belongs or create a new test plan for new component.
Features 	Features
Bug ID 	Description
Bug 10723	Build-Appliance - Run builds in the build-appliance image
Bug 10724	Verify if -c menuconfig work
Bug 10724	Verify if -c devshell work
Bug 10725	Verify if BBPOSTCONF and BBPRECONF work in bitbakek
Bug 10726	Verify if oe-pkgdata-util print help when no parameters specified
Bug 10727	Verify if buildhistory_analsysis.py is tested
Bug 10728	Verify combo-layer
Bug 6926 	Verify yocto-bsp/yocto-layer
Bug 10724	Verify bitbake -u depexp -g XXX
The complete List of QA Implementations for YP 2.3 can be found at Features Implementation by QA Team
= Features Not to be tested =
*
=Approach=
Tests will be conducted per the documented test cases stored in Testopia. The test manager will create test runs for each tester. The tester will execute the tests in Testopia and mark each case as Pass / Fail / Skip /Blocked.
The tester should leave notes on actual results and any other relevant details when possible. 
Once complete, the test manager should review the test run reports in Testopia and report back to the team by mail and in the wiki as a test report.
= Pass / Fail Criteria =
This criteria is defined in Yocto_Project_v2.3_Status#Yocto_Project_v2.3_Release_Criteria
= Exit Criteria =
As defined in QA_Master_Test_Plan#Entry and Exit Criteria
= Test Deliverables =
* Live Status 2.3_QA_Status
* Final execution Report #Execution History
* Mail sent to the mailing list
= Test Environment =
There are different environments and tools for every type of testing refer to QA_Master_Test_Plan#Test_Areas for more detailed information.
= Schedule =
== M1 (Oct. 31, 2016 to Dec. 23, 2016 -- 9 weeks: planning done in week 1, development week 2-6, stabilization week 7-8, release week 9) ==
=== M1 Feature/Tasks Development / Cutoff is Dec. 12, 2016 ===
=== M1 Bugs ===
=== M1 Stabilize schedule ===
Release Candidate and Milestone Release schedule
* RC1: 12/12/16
* M1 release: 12/23/16
The planed schedule for 2.3 can be found at Yocto_2.3_Schedule
== Live Schedule ==
MILESTONE 	STARTED 	FINISHED 	STATUS
M1 rc1 	12/19/2016	12/23/2016	DONE
M2 rc1 	01/24/2017	01/31/2017	DONE
M2 rc3 	02/03/2017	02/08/2017	DONE
M3 rc1 	03/15/2017	03/23/2017	DONE
M3 rc2 	03/28/2017	04/05/2017	DONE
M3 rc1 Meta-Intel 	04/05/2017	04/12/2017	DONE
M4 rc2 	04/21/2017	04/29/2017	DONE
M4 rc6 	05/02/2017	05/08/2017	DONE
2.3 Meta-Intel rc1	05/11/2017	05/16/2017	DONE
2.3 Meta-Intel rc4	05/22/2017	05/25/2017	DONE
2.3.1 rc2 	07/12/2017	07/18/2017	DONE
2.3.2 rc1 	09/12/2017	09/20/2017	DONE
2.3.3 rc1 	12/15/2017	12/20/2017	DONE
2.3.4 rc1 	6/13/2018 	7/10/2018 	DONE
= Execution History =
MILESTONE 	REPORT
M1 rc1 	WW52 - 2016-12-19 - Full Test Cycle 2.3 M1 rc1
M2 rc1 	WW05 - 2017-01-24 - Full Test Cycle 2.3_M2 rc1
M2 rc3 	WW06 - 2017-02-03 - Full Test Cycle 2.3_M2 rc3
M3 rc1 	WW11 - 2017-03-15 - Full Test Cycle 2.3 M3 rc1
M3 rc2 	WW14 - 2017-03-28 - Full Test Cycle 2.3 M3 rc2
M3 rc1 Meta-Intel 	WW15 - 2017-04-05 - Full Test Cycle - 2.3 M3 - meta-intel
M4 rc2 (2.3 release - pyro) 	WW17 - 2017-04-21 - Full Test Cycle 2.3 rc2
M4 rc4 (2.3 release - pyro) 	WW19 - 2017-05-02 - Full Test Cycle 2.3 rc4
Meta-Intel rc1 (2.3 release - pyro)	WW20 - 2017-05-11 - Full Test Cycle - 2.3 rc1 - meta-intel
Meta-Intel rc4 (2.3 release - pyro) 	WW21 - 2017-05-22 - Full Test Cycle - 2.3 rc4 - meta-intel
2.3.1 rc2 	WW29 - 2017-07-18- Full Test Cycle - 2.3.1 rc2
2.3.2 rc1 	WW38_-_2017-09-20-_Full_Test_Cycle_-_2.3.2_rc1
2.3.3 rc1 	WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1
2.3.4 rc1 	WW25_-_2018-06-22-_Full_Test_Cycle_-_2.3.4_rc1