Package: project-summarizer
Project SummarizerTL;DRProject Summarizer is a tool used to summarize various files produced by other tools during a build or test process.
This tool is intended to be executed after running a test script or a build script.
The benefit to using the Project Summarizer tool is a quick summary of more terse information provided by other tools.
The goal is to provide the minimum level of feedback on changes to the project, avoiding a more costly lookup of the summarized information.
If that goal is met, then a simple look at that summarized information can replace a more costly lookup, such as having to switch focus to a locally hosted web page to determine impact.Our hope is that we can help developers achieve that goal at least 50 percent of the time.Supported SummarizersThe currently supported summarizers are:JUnitfor test resultssupported by JUnit andPyTestCoberturafor code coveragesupported bypytest-covThere are plans for an extension mechanism to support other summarizers soon.Getting StartedThewhatis reported on and thehowthat information is generated is up to the developer team for every project.
The Project Summarizer tool aims to condense that reported information into a simple, glanceable report.Normally a project will have a build/test framework or a build/test script to perform this action.
The Project Summarizer tool can then be added to the end of that process, to be executed on success or failure.
If the project uses a test framework that exports a JUnit compatible results file, then the argument--junit <file>is used when calling the Project Summarizer tool.
If the project uses a coverage framework that exports a Cobertura compatible coverage file, then the argument--cobertura <file>is used when calling the Project Summarizer tool.When setup this way, the tool will present a quick summary of the contents of those two files.
In addition, the tool will create summary files in thereportdirectory of the project.
While the report files are not intended for human consumption, their summarized information should be easy enough to read and understand, if needed.Normal UsageWhile complete information on the current state of the project is useful, our development team finds that most often that they are looking for what has changed.
That is where the Project Summarizer tool shines.
But, to understand what has changed, a benchmark or snapshot of the previous "current" state must be placed somewhere.
For the Project Summarizer tool, those summary files in thereportdirectory arepublishedto thepublishdirectory using the--publishargument.
In our team, publishing is performed as the last action before committing changes to a project's repository.
The intent of that action is that we can always determine what changes have occurred since the last commit.
If we have any doubts about the integrity of that information, we can publish the summaries at the start of working on a new issue, just to get the confidence that we have the right summaries.Once the project's summaries have been published, the--only-changesargument can then be used to report only on the changes that have occurred since the last published summaries.
With that argument present, the summaries will not only display values that have changed since the published summaries, along with the amount of change that has occurred.
If adding, removing, or enabling tests, this is useful to make sure that the count of changed tests is as expected.
If making any changes to the code that the tests are covering, this is useful to see what effect the change has on code coverage metrics.Our team uses test driven development and maintains a high code coverage metric for all our projects.
The Project Summarizer tool allows us to see the impact of our current changes on the current existing tests, enabled and disabled.
It also allows us to keep track of the impact of any code changes on our coverage metrics.
With both summaries, if the reported information is outside of our expectations, we can then look at the more comprehensive reports to find that needed information.
But over half of the time, the summary information alone is enough to answer our questions about the changes we have made.It is recommended that projects do not commit the contents of thereportdirectory to a repository, and only commit the contents of thepublishdirectory.
While the decision to follow that recommendation is up to development teams, our team has found that it provides a particularly useful summary of what has changed since the last commit.
That information has helped our team ensure that the right tests have changed and that our code coverage is not negatively affected, all with a simple glance.
To enforce this in out projects, we added the following line to our.gitignorefiles:report/Sample OutputThese samples were captured against the Project Summarizer project itself and have not been modified.Full OutputGenerated againstthis commit.Test Results Summary
--------------------


  CLASS NAME                    TOTAL TESTS  FAILED TESTS  SKIPPED TESTS

  test.test_coverage_model                6             0              0
  test.test_coverage_profiles             3             0              0
  test.test_coverage_scenarios           12             0              0
  test.test_main                          4             0              0
  test.test_publish_scenarios             9             0              0
  test.test_results_scenarios            19             0              0
  test.test_scenarios                     1             0              0
  ---                                    --             -              -
  TOTALS                                 54             0              0

Test Coverage Summary
---------------------


  TYPE          COVERED  MEASURED  PERCENTAGE

  Instructions      ---       ---      ------
  Lines             563       563      100.00
  Branches          184       184      100.00
  Complexity        ---       ---      ------
  Methods           ---       ---      ------
  Classes           ---       ---      ------Only Output Changes with No ChangesGenerated againstthis commitwith no observable changes.Test Results Summary
--------------------

Test results have not changed since last published test results.

Test Coverage Summary
---------------------

Test coverage has not changed since last published test coverage.Expected ChangeGenerated againstthis commit.
Test functiontest_junit_jacoco_profilerenamed toxxtest_junit_jacoco_profileto not be executed.Test Results Summary
--------------------


  CLASS NAME                   TOTAL TESTS  FAILED TESTS  SKIPPED TESTS

  test.test_coverage_profiles       2 (-1)             0              0
  ---                              --                  -              -
  TOTALS                           53 (-1)             0              0


Test Coverage Summary
---------------------


  TYPE       COVERED  MEASURED     PERCENTAGE

  Lines     557 (-6)       563  98.93 (-1.07)
  Branches  178 (-6)       184  96.74 (-3.26)NoteOur team's development is primarily done on Windows systems.
As such, any examples that we present will typically be Windows CMD scripts.
We have a project note to provide Bash scripts for the project soon.Using This Project On Itself - SetupFor the Project Summarizer project itself, there is aptest.cmdscript that allows for various modes to be used in executing PyTest against the project.
These different modes allow for more focused testing depending on the needs of the developer at the time.
Those different modes use environment variables to specify how to execute PyTest, and for the purpose of this documentation, those variables are ignored to provide a clearer picture of how the Project Summarizer tool works.With those other environment variables out of the way, the heart of the script is thePYTEST_ARGSenvironment variable.
Under normal operation, that environment variable is set to:--timeout=10 -ra --strict-markers --junitxml=report/tests.xml --html=report/report.htmlWhen testing with code coverage applied, the following text is appended to the contents of that variable:--cov --cov-branch --cov-report xml:report/coverage.xml --cov-report html:report/coverageAs the project usespipenv, when the tests are executed, they are effectively executed with the following command line:pipenv run pytest %PYTEST_ARGS%The important parts here are the--junitxmlargument and the--cov-report xml:argument to PyTest.
The first argument specifies the location where the JUnit compatible report of test results will be written.
Similarly, the second argument specifies the location where the Cobertura compatible report of test coverage will be written.Using This Project On Itself - Adding In Project SummarizerGiven that setup, adding in the Project Summarizer to theptest.cmdscript is easy.
As code coverage can sometimes slow test execution, the following section of the script is used to setup the execution of the tool:set PTEST_REPORT_OPTIONS=--junit %PTEST_TEST_RESULTS_PATH%
if defined PTEST_COVERAGE_MODE (
    if not defined TEST_EXECUTION_FAILED (
        set PTEST_REPORT_OPTIONS=%PTEST_REPORT_OPTIONS% --cobertura %PTEST_TEST_COVERAGE_PATH%
    )
)
pipenv run %PTEST_PROJECT_SUMMARIZER_SCRIPT_PATH% %PTEST_PROJECT_SUMMARIZER_OPTIONS% %PTEST_REPORT_OPTIONS%As the summarizer is being used in the same project as it is created in, thePTEST_PROJECT_SUMMARIZER_SCRIPT_PATHvariable is set to point to the project'smain.pymodule.
Normally, that value would be the name of the Python packageproject_summarizer.
ThePTEST_PROJECT_SUMMARIZER_OPTIONSvariable contains the--only-changesargument by default but can be turned off if desired.
ThePTEST_TEST_RESULTS_PATHvariable is set toreport\tests.xmlto correspond with the--junitxml=report/tests.xmlargument passed to PyTest.
ThePTEST_TEST_COVERAGE_PATHvariable is set toreport\coverage.xmlto match the--cov-report xml:report/coverage.xmlargument passed to PyTest.Because our team wanted publishing summaries to be a function of testing, there is an alternate flow in theptest.cmdscript that published the summaries.
This is easy accomplished by invoking the summarizer with only the--publishargument.Using This Project On Itself - Generating The Above SamplesAll the samples in theSample Outputsection are generated using theptest.cmdscript with the Project Summarizer tool added in.used as part of the normal development process,ptest.cmd --publishwas previously invoked to benchmark the summaries for the last committhis effectively invokedproject_summarizer --publishptest -c -fwas invoked to run tests with coverage and provide theFull Output samplethis ran PyTest with coverage enabled, and effectively invokedproject_summarizer --junit report\tests.xml --cobertura report\coverage.xmlptest -cwas invoked to run tests with coverage and present theNo Change samplethis ran PyTest with coverage enabled, and effectively invokedproject_summarizer --only-changes --junit report\tests.xml --cobertura report\coverage.xmlptest -cwas invoked to run tests with coverage and present theExpected Change samplesame as above, but with the test "missing", it reported that test as not being executed and reported the coverage that was missingFuture GoalsThe initial 0.5.0 release is to get this project on the board.
Once that is done, we have plans to implement an extension mechanism to provide for customized summaries.When Did Things Change?The changelog for this project is maintainedat this location.Still Have Questions?If you still have questions, please consult ourFrequently Asked Questionsdocument.Contact InformationIf you would like to report an issue with the tool or the documentation, please file an issueusing GitHub.If you would like to us to implement a feature that you believe is important, please file an issueusing GitHubthat includes what you want to add, why you want to add it, and why it is important.
Please note that the issue will usually be the start of a conversation and be ready for more questions.If you would like to contribute to the project in a more substantial manner, please contact me atjack.de.winteratoutlook.com.