Resulttool

= Summary =
The resulttool is command line tool available inside poky/scripts, it was developed to manage test result files for both automated tests (eg. oeqa/runtime, oeqa/selftest, oeqa/sdk, etc) and manual tests (eg. oeqa/manual). It was developed as an alternative tool to replace test result storage and reporting using Testopia and Wiki Page. 
To view the features available for resulttool, git clone poky repository, source poky environment, type "resulttool --help".
= List of features = 
The resulttool provide plugin to:
 - "manualexecution": helper script for results populating during manual test execution
 - "merge": merge test result files/directories
 - "store": store test results into a git repository
 - "regression": regression file/directory analysis
 - "regression-git": regression git analysis
 - "report": summarise test results
 - "log": show logs
== manualexecution ==
The manualexecution plugin was developed to provide command line interface to execute manual tests inside oeqa/manual and it will create testresults.json file that contain the configurations and status of the testing (result format compatible to store, report, regression).
Current "Branch" and "Commit" used by the poky repository to run resulttool will be used to create the layers metadata inside testresults.json configuration. The layers metadata were important data required by the "resulttool store" for creating the git tag. Where the git tag will be used for "resulttool report" and "resulttool regression-git". 
The "MACHINE" and "IMAGE_BASENAME" configurations were mandatory inputs for manualexecution as these configurations will be used for "resulttool regression" and "resulttool regression-git", where regression used the configurations data to find matching result set for regression comparison if specific result id was not provided. 
=== manualexecution --help ===
helper script for results populating during manual test execution. You can
find manual test case JSON file in meta/lib/oeqa/manual/
arguments:
 file specify path to manual test case JSON file.Note: Please use "" to encapsulate the file path.
options:
 -h, --help show this help message and exit
 -c CONFIG_OPTIONS_FILE, --config-options-file CONFIG_OPTIONS_FILE
 the config options file to import and used as
 available configuration option selection or make
 config option file
 -m, --make-config-options-file
 make the configuration options file based on provided
 inputs
 -t TESTCASE_CONFIG_FILE, --testcase-config-file TESTCASE_CONFIG_FILE
 the testcase configuration file to enable user to run
 a selected set of test case or make a testcase
 configuration file
 -d, --make-testcase-config-file
 make the testcase configuration file to run a set of
 test cases based on user selection
=== manualexecution example ===
To run manual sdk testcases inside oeqa/manual/sdk.json, use "resulttool manualexecution": 
 $ cd poky
 $ source oe-init-build-env
 $ resulttool manualexecution ../meta/lib/oeqa/manual/sdk.json
== merge ==
The merge plugin was developed to facilitate test result file(s) merging. 
=== merge --help ===
merge the results from multiple files/directories into the target file or directory
arguments:
 base_results the results file/directory to import
 target_results the target file or directory to merge the base_results with
options:
 -h, --help show this help message and exit
 -t, --not-add-testseries
 do not add testseries configuration to results
 -x EXECUTED_BY, --executed-by EXECUTED_BY
 add executed-by configuration to each result file
=== merge example ===
To merge base testresults.json file to target directory. 
Assume base at tmp/log/manual/testresults.json and target directory at <target-testresults-dir>.
 $ cd poky
 $ source oe-init-build-env
 $ resulttool merge tmp/log/manual/testresults.json <target-testresults-dir>
== store ==
The store plugin was developed to store testresults.json file and other artifacts files to local git repository (https://git.yoctoproject.org/git/yocto-testresults). Where the local git repository serves as a database containing test result information to be used by "report", "regression" and "regression-git" plugin. 
=== store --help ===
takes a results file or directory of results files and stores them into the
destination git repository, splitting out the results files as configured
arguments:
 source source file or directory that contain the test result
 files to be stored
 git_dir the location of the git repository to store the results
 in
options:
 -h, --help show this help message and exit
 -a, --all include all files, not just testresults.json files
 -e, --allow-empty don't error if no results to store are found
 -x EXECUTED_BY, --executed-by EXECUTED_BY
 add executed-by configuration to each result file
 -t EXTRA_TEST_ENV, --extra-test-env EXTRA_TEST_ENV
 add extra test environment data to each result file
 configuration
=== store example ===
To store tmp/log/manual/testresults.json to local result repository. 
Assume "resulttool manualexecution" create the testresults.json file inside tmp/log/manual/testresults.json and yocto-testresults local git repository was clone at <yocto-testresults-dir>.
 $ cd poky
 $ source oe-init-build-env
 $ resulttool store tmp/log/manual/ <yocto-testresults-dir>
== regression ==
The regression plugin was developed to provide regression analysis between two set of results (eg. compare base set results to target set results). This "regression" shall be applied to specific test result file or directory. 
=== regression --help ===
regression analysis comparing the base set of results to the target results
arguments:
 base_result base result file/directory for the comparison
 target_result target result file/directory to compare with
options:
 -h, --help show this help message and exit
 -b BASE_RESULT_ID, --base-result-id BASE_RESULT_ID
 (optional) filter the base results to this result ID
 -t TARGET_RESULT_ID, --target-result-id TARGET_RESULT_ID
 (optional) filter the target results to this result ID
=== regression example ===
To have regression analysis on base result set and target result set.
Assume base test result files was at <base-result-dir>. Assume target test result files was at <target-result-dir>. 
 $ cd poky
 $ source oe-init-build-env
 $ resulttool regression <base-result-dir> <target-result-dir>
== regression-git ==
The regression-git plugin was developed to provide regression analysis to test result available inside a local git repository. User need to clone yocto-testresults (https://git.yoctoproject.org/git/yocto-testresults) to local machine. 
=== regression-git --help ===
regression analysis comparing base result set to target result set
arguments:
 repo the git repository containing the data
options:
 -h, --help show this help message and exit
 -b BASE_RESULT_ID, --base-result-id BASE_RESULT_ID
 (optional) default select regression based on
 configurations unless base result id was provided
 -t TARGET_RESULT_ID, --target-result-id TARGET_RESULT_ID
 (optional) default select regression based on
 configurations unless target result id was provided
 --branch BRANCH, -B BRANCH
 Branch to find commit in
 --branch2 BRANCH2 Branch to find comparision revisions in
 --commit COMMIT Revision to search for
 --commit-number COMMIT_NUMBER
 Revision number to search for, redundant if --commit
 is specified
 --commit2 COMMIT2 Revision to compare with
 --commit-number2 COMMIT_NUMBER2
 Revision number to compare with, redundant if
 --commit2 is specified
=== regression-git example ===
To perform regression analysis on local result repository. 
Assume base test result set and target result set was stored inside local git repository at <yocto-testresults-dir>. 
 $ cd poky
 $ source oe-init-build-env
 $ resulttool regression-git <yocto-testresults-dir> --commit <commit-of-base> --commit2 <commit-of-target>
== report ==
The report plugin was developed to summarise test result available in a file/directory/git repository. 
=== report --help ===
print a text-based summary of the test results
arguments:
 source_dir source file/directory that contain the test result files
 to summarise
options:
 -h, --help show this help message and exit
 --branch BRANCH, -B BRANCH
 Branch to find commit in
 --commit COMMIT Revision to report
 -t TAG, --tag TAG source_dir is a git repository, report on the tag
 specified from that repository
 -m, --use_regression_map
 instead of the default "store_map", use the
 "regression_map" for report
 -r RAW_TEST_ONLY, --raw_test_only RAW_TEST_ONLY
 output raw test result only for the user provided test
 result id
 -s SELECTED_TEST_CASE_ONLY, --selected_test_case_only SELECTED_TEST_CASE_ONLY
 output selected test case result for the user provided
 test case id, if both test result id and test case id
 are provided then output the selected test case result
 from the provided test result id
=== report example ===
To get summary report for test result stored inside local git repository. 
 $ cd poky
 $ source oe-init-build-env
 $ resulttool report <yocto-testresults-dir>
== log ==
The log plugin was developed to print log available in a file/directory/URL. 
=== log --help ===
show the logs from test results
arguments:
 source_dir the results file/directory/URL to import
options:
 -h, --help show this help message and exit
 --ptest PTEST show logs for a ptest
 --dump-ptest DIR Dump all ptest log files to the specified directory.
 --reproducible REPRODUCIBLE
 show logs for a reproducible test
 --prepend-run Dump ptest results to a subdirectory named after the
 test run when using --dump-ptest. Required if more
 than one test run is present in the result file
 --raw show raw (ptest) logs. Deprecated. Alias for "--raw-
 ptest"
 --raw-ptest show raw ptest log
 --raw-reproducible show raw reproducible build logs
=== log example ===
To dump ptest log to a directory. 
 $ cd poky
 $ source oe-init-build-env
 $ resulttool log <yocto-testresults-dir> --dump-ptest <output-dir-for-ptest>
To print custom ptest log from a custom ptest. 
 $ cd poky
 $ source oe-init-build-env
 $ resulttool log <yocto-testresults-dir> --ptest <package-name-that-provide-ptest>