saltstack/salt — pull request #2135 by s0undt3ch (2012-09-29 11:54:29+00:00)
Title: Provide an overall tests report.

Previously we have to go through the scroll history to see which tests failed, which were skipped, etc. For example:
```
~ .~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting Module Tests
~ ~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
........................................................ssssssss...............................................
----------------------------------------------------------------------
Ran 111 tests in 73.123s

OK (skipped=8)
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting State Tests
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..............................sss..ssss...s.
----------------------------------------------------------------------
Ran 44 tests in 36.256s

OK (skipped=8)
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting Client Tests
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.s........
----------------------------------------------------------------------
Ran 10 tests in 1.867s

OK (skipped=1)
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting Shell Tests
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
......................[INFO ] Loaded configuration file: /home/vampas/projects/salt/salt/tests/integration/files/conf/minion
..[INFO ] Loaded configuration file: /home/vampas/projects/salt/salt/tests/integration/files/conf/minion
...............
----------------------------------------------------------------------
Ran 39 tests in 40.598s

OK
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting Unit Tests
~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
......................................................s........FEE.........
======================================================================
ERROR: test_assert_2 (unit.config_test.ConfigTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vampas/projects/salt/salt/tests/unit/config_test.py", line 47, in test_assert_2
1/0
ZeroDivisionError: integer division or modulo by zero

======================================================================
ERROR: test_assert_3 (unit.config_test.ConfigTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vampas/projects/salt/salt/tests/unit/config_test.py", line 50, in test_assert_3
1/0
ZeroDivisionError: integer division or modulo by zero

======================================================================
FAIL: test_assert_1 (unit.config_test.ConfigTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/vampas/projects/salt/salt/tests/unit/config_test.py", line 44, in test_assert_1
self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 75 tests in 0.285s

FAILED (failures=1, errors=2, skipped=1)
```
Now, besides test failures, errors and skips being reported like they always were, we also provide a nice overall report and the end.

An example report output:
```
================================================================== Overall Tests Resume ===================================================================
⋆⋆⋆ Module Tests ⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
-------- Skipped Tests -----------------------------------------------------------------------------------------------------------------------------------
→ integration.modules.supervisord.SupervisordModuleTest.test_restart_all → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_restart_one → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_start_all → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_start_one → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_status_all → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_status_one → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_stop_all → you must be this root to run this test
→ integration.modules.supervisord.SupervisordModuleTest.test_stop_one → you must be this root to run this test
------------------------------------------------------------------------------------------------------------------------------------------------------------
⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
⋆⋆⋆ State Tests ⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
-------- Skipped Tests -----------------------------------------------------------------------------------------------------------------------------------
→ integration.states.supervisord.SupervisordTest.test_restart → you must be this root to run this test
→ integration.states.supervisord.SupervisordTest.test_start → you must be this root to run this test
→ integration.states.supervisord.SupervisordTest.test_stop → you must be this root to run this test
→ integration.states.user.UserTest.test_user_not_present → you must be this root to run this test
→ integration.states.user.UserTest.test_user_present_gid_from_name → you must be this root to run this test
→ integration.states.user.UserTest.test_user_present_gid_from_name_default → you must be this root to run this test
→ integration.states.user.UserTest.test_user_present_nondefault → you must be this root to run this test
→ integration.states.pip.PipStateTest.test_pip_installed_weird_install → You don't have the required permissions to run this test
------------------------------------------------------------------------------------------------------------------------------------------------------------
⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
⋆⋆⋆ Client Tests ⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
-------- Skipped Tests -----------------------------------------------------------------------------------------------------------------------------------
→ integration.client.syndic.TestSyndic.test_ping → WARNING(SHOULD NOT HAPPEN #1935): Failed to get a reply from the minion. Received: '{}'
------------------------------------------------------------------------------------------------------------------------------------------------------------
⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
⋆⋆⋆ Unit Tests ⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
-------- Skipped Tests -----------------------------------------------------------------------------------------------------------------------------------
→ unit.utils.path_join_test.PathJoinTestCase.test_windows_paths → Non windows platform found. not running non patched os.path path_join tests
------------------------------------------------------------------------------------------------------------------------------------------------------------
-------- Tests with Errors -------------------------------------------------------------------------------------------------------------------------------
→ unit.config_test.ConfigTestCase.test_assert_2 .........................................................................................................
Traceback (most recent call last):
File "/home/vampas/projects/salt/salt/tests/unit/config_test.py", line 47, in test_assert_2
1/0
ZeroDivisionError: integer division or modulo by zero
..........................................................................................................................................................
→ unit.config_test.ConfigTestCase.test_assert_3 .........................................................................................................
Traceback (most recent call last):
File "/home/vampas/projects/salt/salt/tests/unit/config_test.py", line 50, in test_assert_3
1/0
ZeroDivisionError: integer division or modulo by zero
..........................................................................................................................................................
------------------------------------------------------------------------------------------------------------------------------------------------------------
-------- Failed Tests ------------------------------------------------------------------------------------------------------------------------------------
→ unit.config_test.ConfigTestCase.test_assert_1 .........................................................................................................
Traceback (most recent call last):
File "/home/vampas/projects/salt/salt/tests/unit/config_test.py", line 44, in test_assert_1
self.assertTrue(False)
AssertionError: False is not true
..........................................................................................................................................................
------------------------------------------------------------------------------------------------------------------------------------------------------------
⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆
================================================================== Overall Tests Resume ===================================================================
```