Wiki Test Report Format

This page is to define Test Report Format for Yocto
==== Test Summary ====
Here is a test result summary for a build.
==== Test Result Matrix ====
A general Test Result Summary and a detailed Test Result Summary should be provided here.
Example
Test Result Summary	Test Result Summary	Test Result Summary	Test Result Summary 	Test Result Summary
Component 	Target 	Status 	Comments 	
BSP 	SugarBay 	GOOD 	Zypper install/remove has issue; 3D game running failed; 	
　 	CrownBay 	BLOCK 	emgd crownbay build failed; matchbox-panel segfault; S3 fail on B0 board 	
　 	JasperForest 	GOOD 	The only issue is zypper install/remove bug 	
　 	Blacksand 	Buggy 	mouse/keyboard do not work in X window; zypper install/remove not work well 	
　 	Beagleboard 	GOOD 	Only one RTC issue exists 	
　 	Routerstationpro 	GOOD 	Only zypper install has issue 	
　 	Mpc8315e-rdb 	GOOD 	Only zypper install has issue 	
QEMU 	qemux86 	Buggy 	Only zypper install/remove has issue 	
　 	qemux86-64 	Buggy 	Only zypper install/remove has issue 	
　 	qemuarm 	Buggy 	zypper could not work well 	
　 	qemuppc 	Buggy 	zypper could not work well 	
　 	qemumips 	Buggy 	Only zypper install has issue 	
ADT 	　 	BLOCK 	i686 toolchain not copied to sharing folder; unfs does not work with qemuppc;	
Core Build System 	　 	BUGGY 	sstate and non-gplv3 build do not work correctly 	
Stress 	　 	GOOD 	Sugarbay pass 18 hours stress and Jasperforest pass 12 hours stress 	
Compliance 	　 	GOOD 	Sugarbay pass LTP and POSIX test 	
Power/Performance 	　 	GOOD 	　 	
 	 	 	 	
Note:
BLOCK	Critical bugs, more than 50% test cases are blocked
GOOD 	Only Normal, Minor or Enhancement bugs, less than 10% test cases failed
BUGGY	Normal, Major and Critical bugs, more than 10% test cases failed
 	
Detailed Test Result for each component	Detailed Test Result for each component	Detailed Test Result for each component	Detailed Test Result for each component	Detailed Test Result for each component	Detailed Test Result for each component
Target 	Total TCs 	Not Run 	Passed 	Failed 	Not testable (Blocked)
Sugarbay Sato-SDK 	64 	0 	60 	1 (bug 1101) 	3 (bug 1112, 883)
Crownbay Sato-SDK 	60 	0 	55 	1 (bug 1099) 	4
Jasperforest LSB-SDK 	34 	0 	34 	0 	0
n450 Sato-SDK 	60 	0 	52 	2 (bug 613, 1099) 	6 (bug 613)
eMenlow Sato-SDK 	60 	0 	48 	2 (bug 1099, 503) 	6 (bug 503)
Routerstationpro Sato-SDK 	29 	0 	29 	0 	0
Mpc8315e-rdb Sato-SDK 	30 	0 	26 	4 (bug 766) 	0
Qemux86-64 Sato 	28 	0 	28 	0 	0
Qemux86-64 Sato-SDK 	31 	0 	31 	0 	0
Qemux86 Sato 	28 	0 	28 	0 	0
Qemux86 Sato-SDK 	31 	0 	31 	0 	0
Qemumips Sato 	26 	0 	26 	0 	0
Qemumips Sato-SDK 	32 	0 	32 	0 	0
Qemuppc Sato 	20 	0 	19 	0 	1 (bug 414)
Qemuppc Sato-SDK 	26 	0 	25 	0 	1 (bug 414)
Qemuarm Sato 	26 	0 	26 	0 	0
Qemuarm Sato-SDK 	32 	0 	32 	0 	0
Core Build System 	8 	0 	7 	1 (bug 1133) 	0
ADT 	6 	0 	5 	1 (bug 414) 	0
Power/Performance 	4 	0 	4 	0 	0
Stress 	2 	0 	2 	0 	0
Total 	637 	0 	600 	12 	21
 	 	 	 	 	
* You can check the detailed test result in attachment for each target.
* The failed/blocked case number is listed with failed cases’ bug number.
==== Commit Information ====
This section is to provide tree/branch commit information for testing.
Example:
Tree/Branch: Poky/1.1_M1
Poky Commit: 4ff7af11ef69849ef9c16f585eae58ac920b222b
Meta Branch: 1.1_M1
Meta Commit: fc719f0cd6530ce15148b4aa274f1644b461b298
==== Issue Summary ====
This section shows a list of open bugs for each components.
Example
Component 	Component 	Bug Number 	Target Milestone
System & Core OS 	Installation & Boot	Bug 1006: [mpc8315e-rdb & routerstationpro] minimal,sato,sato-sdk images boot failed (nightly build 20110423-1) 	1.1
System & Core OS 	Installation & Boot	Bug 1099: [blacksand/emenlow/crownbay] grub install failed with sato-sdk live image 	1.1 M2
System & Core OS 	Power Mangement 	Bug 613: [Blacksand]system cannot enter S3 standby mode 	1.1
System & Core OS 	Power Mangement 	Bug 503: [emenlow] system cannot enter S3 standby mode 	1.1
System & Core OS 	Others 	Bug 766: [mpc8315e-rdb] USB does not work on mpc8315e-rdb 	1.0 M4
System & Core OS 	Others 	Bug 1101: keyboard pop up when clicking terminal icon 	1.0 M2
System & Core OS 	Others 	Bug 1107: Open or add person in contacts may segfault on sugarbay/blacksand 	1.1 M2
System & Core OS 	Others 	Bug 1097: The mouse can not click accurate to the icons in qemumips X 	1.1 M2
Graphics 	Graphics 	Bug 1112: [sugarbay] mesa-demos should be added into sato image for graphics testing 	1.1 M2
Graphics 	Graphics 	Bug 883: [Sugarbay] GFX 3D game test fail to run with sugarbay BSP image due to libsdl.so 	1.1
Multimedia 	Multimedia 	Works well in sugarbay/n450/eMenlow/crownbay 	　
Core Build System	Core Build System 	New! Bug 1136: do_menuconfig for linux-yocto needs ${B} exists 	1.1 M2
Core Build System	Core Build System 	New! Bug 1133: No provider for core-image-minimal/basic for non-GPLv3 build 	1.1 M2
Compliance 	Compliance 	Compliance testing is finished, but result is still under analysis. LTP test result is on https://wiki.pokylinux.org/wiki/LTP_result   POSIX test result is on https://wiki.pokylinux.org/wiki/Posix_result   A new LTP failure is found and reported as bug 1138: [LTP] Some growfiles cases failed with Yocto 1.1 M1 build.	　
Stress 	Stress 	Helltest and Crashme could pass 24 hours testing 	　
ADT 	ADT 	Bug 414: [PPC] kernel panic when booting poky-image-sdk-qemuppc through UNFS 	1.1 M3