Repository: ActivisionGameScience/assertpy

Issue:
correct implementation of soft assertions
Hi!
   This is not a bug report, but more like a discussion kick-starter regarding soft assertions. And if we happen to agree on a different implementation, I'll be more than happy to create a PR.

What I suggest is soft assertions to be implemented as in other languages libraries. E.g [soft assertions in AssertJ](http://joel-costigliola.github.io/assertj/assertj-core-features-highlight.html#soft-assertions).

Soft assertions usually have a special value in higher levels of testing than unit, e.g. integration or system tests, when the result feedback is not as fast as with unit test. 
So basically a test is more "expensive" to run in terms of resources like CPU, memory, and specially time. And we want to take the most value out of each execution.

Let's assume that I have a test that:
 - Logs in
 - Creates a new user in the system with default settings
 - And verifies that the user has a
   * default locale = X
   * default timezone = Y
   * default privileges = Z
   * etc, etc

If any of these are missing or wrong, I want the test to fail. However, with regular assertions if the locale is missing the test will fail as expected but I won't have any information whether the system meets the other requirements.

As you know, that's when soft assertions come handy. The problem I see though, is that in your implementation, you silently pass, I mean.. you print a warning in stdout, but the test will pass.  And that's a wrong approach IMO, as it requires human intervention (someone reading the screen), so those assertions won't have any effect if tests are run, for instance, in jenkins as part of CI/CD pipeline.

What I suggest is what AssertJ does. You run assertions in a group, so even if the "locale" assertion fails, you still run the "timezone" and the "privileges" assertions. After all the assertions have been executed an AssertionError is raised if at least one assertion in the group failed. The error will contain the details of all those assertions in the group that failed.

Does all this make sense to you? WDYT?

Regards!

Discussion:

Patch:
diff --git a/README.md b/README.md
index 91b1eb5..99edf06 100644
--- a/README.md
+++ b/README.md
@@ -282,7 +282,7 @@ Fluent assertions against the value of a given key can be done by prepending `ha
 
 ```py
 fred = {'first_name': 'Fred', 'last_name': 'Smith', 'shoe_size': 12}
- 
+
 assert_that(fred).has_first_name('Fred')
 assert_that(fred).has_last_name('Smith')
 assert_that(fred).has_shoe_size(12)
@@ -534,7 +534,7 @@ As noted above, dynamic assertions also work on dicts:
 
 ```py
 fred = {'first_name': 'Fred', 'last_name': 'Smith'}
- 
+
 assert_that(fred).has_first_name('Fred')
 assert_that(fred).has_last_name('Smith')
 ```
@@ -613,24 +613,24 @@ Expected <3> to be equal to <2>, but was not.
 The `described_as()` helper causes the custom message `adding stuff` to be prepended to the front of the second error.
 
 
-#### Soft Assertions
+#### Just A Warning
 
-There are times when you don't want to a test to fail at all, instead you only want a warning message. In this case, just replace `assert_that` with `assert_soft`.
+There are times when you only want a warning message instead of an failing test. In this case, just replace `assert_that` with `assert_warn`.
 
 ```py
-assert_soft('foo').is_length(4)
-assert_soft('foo').is_empty()
-assert_soft('foo').is_false()
-assert_soft('foo').is_digit()
-assert_soft('123').is_alpha()
-assert_soft('foo').is_upper()
-assert_soft('FOO').is_lower()
-assert_soft('foo').is_equal_to('bar')
-assert_soft('foo').is_not_equal_to('foo')
-assert_soft('foo').is_equal_to_ignoring_case('BAR')
+assert_warn('foo').is_length(4)
+assert_warn('foo').is_empty()
+assert_warn('foo').is_false()
+assert_warn('foo').is_digit()
+assert_warn('123').is_alpha()
+assert_warn('foo').is_upper()
+assert_warn('FOO').is_lower()
+assert_warn('foo').is_equal_to('bar')
+assert_warn('foo').is_not_equal_to('foo')
+assert_warn('foo').is_equal_to_ignoring_case('BAR')
 ```
 
-The above soft assertions print the following warning messages (but an `AssertionError` is never raised):
+The above assertions just print the following warning messages, and an `AssertionError` is never raised:
 
 ```
 Expected <foo> to be of length <4>, but was <3>.
diff --git a/assertpy/__init__.py b/assertpy/__init__.py
index 385ab52..f673973 100644
--- a/assertpy/__init__.py
+++ b/assertpy/__init__.py
@@ -1,2 +1,2 @@
 from __future__ import absolute_import
-from .assertpy import assert_that, assert_soft, contents_of, fail, __version__
+from .assertpy import assert_that, assert_warn, soft_assertions, contents_of, fail, __version__
diff --git a/assertpy/assertpy.py b/assertpy/assertpy.py
index 462eee4..a1a644d 100644
--- a/assertpy/assertpy.py
+++ b/assertpy/assertpy.py
@@ -36,6 +36,7 @@ import datetime
 import numbers
 import collections
 import inspect
+from contextlib import contextmanager
 
 __version__ = '0.9'
 
@@ -48,14 +49,43 @@ else:
     xrange = xrange
     unicode = unicode
 
+
+### soft assertions ###
+_soft_ctx = False
+_soft_err = []
+
+@contextmanager
+def soft_assertions():
+    global _soft_ctx
+    global _soft_err
+
+    _soft_ctx = True
+    _soft_err = []
+
+    yield
+
+    if _soft_err:
+        out = 'soft assertion failures:'
+        for i,msg in enumerate(_soft_err):
+            out += '\n%d. %s' % (i+1, msg)
+        raise AssertionError(out)
+
+    _soft_err = []
+    _soft_ctx = False
+
+
+### factory methods ###
 def assert_that(val, description=''):
     """Factory method for the assertion builder with value to be tested and optional description."""
+    global _soft_ctx
+    if _soft_ctx:
+        return AssertionBuilder(val, description, 'soft')
     return AssertionBuilder(val, description)
 
-def assert_soft(val, description=''):
+def assert_warn(val, description=''):
     """Factory method for the assertion builder with value to be tested, optional description, and
-       just print assertion failures, don't raise exceptions."""
-    return AssertionBuilder(val, description, True)
+       just warn on assertion failures instead of raisings exceptions."""
+    return AssertionBuilder(val, description, 'warn')
 
 def contents_of(f, encoding='utf-8'):
     """Helper to read the contents of the given file or path into a string with the given encoding.
@@ -96,14 +126,15 @@ def fail(msg=''):
     else:
         raise AssertionError('Fail: %s!' % msg)
 
+
 class AssertionBuilder(object):
     """Assertion builder."""
 
-    def __init__(self, val, description, soft=False, expected=None):
+    def __init__(self, val, description='', kind=None, expected=None):
         """Construct the assertion builder."""
         self.val = val
         self.description = description
-        self.soft = soft
+        self.kind = kind
         self.expected = expected
 
     def described_as(self, description):
@@ -833,7 +864,7 @@ class AssertionBuilder(object):
                 else:
                     raise ValueError('val does not have property or zero-arg method <%s>' % name)
             extracted.append(tuple(items) if len(items) > 1 else items[0])
-        return AssertionBuilder(extracted, self.description)
+        return AssertionBuilder(extracted, self.description, self.kind)
 
 ### dynamic assertions ###
     def __getattr__(self, attr):
@@ -878,7 +909,7 @@ class AssertionBuilder(object):
             raise TypeError('val must be function')
         if not issubclass(ex, BaseException):
             raise TypeError('given arg must be exception')
-        return AssertionBuilder(self.val, self.description, expected=ex)
+        return AssertionBuilder(self.val, self.description, self.kind, ex)
 
     def when_called_with(self, *some_args, **some_kwargs):
         """Asserts the val function when invoked with the given args and kwargs raises the expected exception."""
@@ -889,7 +920,7 @@ class AssertionBuilder(object):
         except BaseException as e:
             if issubclass(type(e), self.expected):
                 # chain on with exception message as val
-                return AssertionBuilder(str(e), self.description)
+                return AssertionBuilder(str(e), self.description, self.kind)
             else:
                 # got exception, but wrong type, so raise
                 self._err('Expected <%s> to raise <%s> when called with (%s), but raised <%s>.' % (
@@ -908,9 +939,13 @@ class AssertionBuilder(object):
     def _err(self, msg):
         """Helper to raise an AssertionError, and optionally prepend custom description."""
         out = '%s%s' % ('[%s] ' % self.description if len(self.description) > 0 else '', msg)
-        if self.soft:
+        if self.kind == 'warn':
             print(out)
             return self
+        elif self.kind == 'soft':
+            global _soft_err
+            _soft_err.append(out)
+            return self
         else:
             raise AssertionError(out)
 


Test patch:
diff --git a/tests/test_readme.py b/tests/test_readme.py
index 2ad4554..2179166 100644
--- a/tests/test_readme.py
+++ b/tests/test_readme.py
@@ -29,7 +29,7 @@
 import sys
 import os
 import datetime
-from assertpy import assert_that, assert_soft, contents_of, fail
+from assertpy import assert_that, assert_warn, contents_of, fail
 
 class TestReadme(object):
 
@@ -382,16 +382,16 @@ class TestReadme(object):
             assert_that(str(e)).is_equal_to('[adding stuff] Expected <3> to be equal to <2>, but was not.')
 
     def test_soft_assertions(self):
-        assert_soft('foo').is_length(4)
-        assert_soft('foo').is_empty()
-        assert_soft('foo').is_false()
-        assert_soft('foo').is_digit()
-        assert_soft('123').is_alpha()
-        assert_soft('foo').is_upper()
-        assert_soft('FOO').is_lower()
-        assert_soft('foo').is_equal_to('bar')
-        assert_soft('foo').is_not_equal_to('foo')
-        assert_soft('foo').is_equal_to_ignoring_case('BAR')
+        assert_warn('foo').is_length(4)
+        assert_warn('foo').is_empty()
+        assert_warn('foo').is_false()
+        assert_warn('foo').is_digit()
+        assert_warn('123').is_alpha()
+        assert_warn('foo').is_upper()
+        assert_warn('FOO').is_lower()
+        assert_warn('foo').is_equal_to('bar')
+        assert_warn('foo').is_not_equal_to('foo')
+        assert_warn('foo').is_equal_to_ignoring_case('BAR')
 
     def test_chaining(self):
         fred = Person('Fred','Smith')
diff --git a/tests/test_soft.py b/tests/test_soft.py
index 8731c99..ca39c30 100644
--- a/tests/test_soft.py
+++ b/tests/test_soft.py
@@ -28,48 +28,37 @@
 
 import sys
 
-from assertpy import assert_that, assert_soft, fail
+from assertpy import assert_that, soft_assertions, fail
 
-class TestSoft(object):
-
-    def test_success(self):
-        assert_soft('foo').is_length(3)
-        assert_soft('foo').is_not_empty()
-        assert_soft('foo').is_true()
-        assert_soft('foo').is_alpha()
-        assert_soft('123').is_digit()
-        assert_soft('foo').is_lower()
-        assert_soft('FOO').is_upper()
-        assert_soft('foo').is_equal_to('foo')
-        assert_soft('foo').is_not_equal_to('bar')
-        assert_soft('foo').is_equal_to_ignoring_case('FOO')
-
-    def test_failures(self):
-        if sys.version_info[0] == 3:
-            from io import StringIO
-        else:
-            from StringIO import StringIO
-
-        # capture stdout
-        old = sys.stdout
-        sys.stdout = StringIO()
-
-        assert_soft('foo').is_length(4)
-        assert_soft('foo').is_empty()
-        assert_soft('foo').is_false()
-        assert_soft('foo').is_digit()
-        assert_soft('123').is_alpha()
-        assert_soft('foo').is_upper()
-        assert_soft('FOO').is_lower()
-        assert_soft('foo').is_equal_to('bar')
-        assert_soft('foo').is_not_equal_to('foo')
-        assert_soft('foo').is_equal_to_ignoring_case('BAR')
-
-        # stop capturing stdout
-        out = sys.stdout.getvalue()
-        sys.stdout.close()
-        sys.stdout = old
+def test_success():
+    with soft_assertions():
+        assert_that('foo').is_length(3)
+        assert_that('foo').is_not_empty()
+        assert_that('foo').is_true()
+        assert_that('foo').is_alpha()
+        assert_that('123').is_digit()
+        assert_that('foo').is_lower()
+        assert_that('FOO').is_upper()
+        assert_that('foo').is_equal_to('foo')
+        assert_that('foo').is_not_equal_to('bar')
+        assert_that('foo').is_equal_to_ignoring_case('FOO')
 
+def test_failure():
+    try:
+        with soft_assertions():
+            assert_that('foo').is_length(4)
+            assert_that('foo').is_empty()
+            assert_that('foo').is_false()
+            assert_that('foo').is_digit()
+            assert_that('123').is_alpha()
+            assert_that('foo').is_upper()
+            assert_that('FOO').is_lower()
+            assert_that('foo').is_equal_to('bar')
+            assert_that('foo').is_not_equal_to('foo')
+            assert_that('foo').is_equal_to_ignoring_case('BAR')
+        fail('should have raised error')
+    except AssertionError as e:
+        out = str(e)
         assert_that(out).contains('Expected <foo> to be of length <4>, but was <3>.')
         assert_that(out).contains('Expected <foo> to be empty string, but was not.')
         assert_that(out).contains('Expected <False>, but was not.')
@@ -81,3 +70,41 @@ class TestSoft(object):
         assert_that(out).contains('Expected <foo> to be not equal to <foo>, but was.')
         assert_that(out).contains('Expected <foo> to be case-insensitive equal to <BAR>, but was not.')
 
+def test_failure_chain():
+    try:
+        with soft_assertions():
+            assert_that('foo').is_length(4).is_empty().is_false().is_digit().is_upper()\
+                .is_equal_to('bar').is_not_equal_to('foo').is_equal_to_ignoring_case('BAR')
+        fail('should have raised error')
+    except AssertionError as e:
+        out = str(e)
+        assert_that(out).contains('Expected <foo> to be of length <4>, but was <3>.')
+        assert_that(out).contains('Expected <foo> to be empty string, but was not.')
+        assert_that(out).contains('Expected <False>, but was not.')
+        assert_that(out).contains('Expected <foo> to contain only digits, but did not.')
+        assert_that(out).contains('Expected <foo> to contain only uppercase chars, but did not.')
+        assert_that(out).contains('Expected <foo> to be equal to <bar>, but was not.')
+        assert_that(out).contains('Expected <foo> to be not equal to <foo>, but was.')
+        assert_that(out).contains('Expected <foo> to be case-insensitive equal to <BAR>, but was not.')
+
+def test_expected_exception_success():
+    with soft_assertions():
+        assert_that(func_err).raises(RuntimeError).when_called_with('foo').is_equal_to('err')
+
+def test_expected_exception_failure():
+    try:
+        with soft_assertions():
+            assert_that(func_err).raises(RuntimeError).when_called_with('foo').is_equal_to('bar')
+            assert_that(func_ok).raises(RuntimeError).when_called_with('baz')
+        fail('should have raised error')
+    except AssertionError as e:
+        out = str(e)
+        assert_that(out).contains('Expected <err> to be equal to <bar>, but was not.')
+        assert_that(out).contains("Expected <func_ok> to raise <RuntimeError> when called with ('baz').")
+
+def func_ok(arg):
+    pass
+
+def func_err(arg):
+    raise RuntimeError('err')
+
diff --git a/tests/test_warn.py b/tests/test_warn.py
new file mode 100644
index 0000000..6de80c2
--- /dev/null
+++ b/tests/test_warn.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2015-2016, Activision Publishing, Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification,
+# are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+# list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# 3. Neither the name of the copyright holder nor the names of its contributors
+# may be used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import sys
+
+from assertpy import assert_that, assert_warn, fail
+
+class TestSoft(object):
+
+    def test_success(self):
+        assert_warn('foo').is_length(3)
+        assert_warn('foo').is_not_empty()
+        assert_warn('foo').is_true()
+        assert_warn('foo').is_alpha()
+        assert_warn('123').is_digit()
+        assert_warn('foo').is_lower()
+        assert_warn('FOO').is_upper()
+        assert_warn('foo').is_equal_to('foo')
+        assert_warn('foo').is_not_equal_to('bar')
+        assert_warn('foo').is_equal_to_ignoring_case('FOO')
+
+    def test_failures(self):
+        if sys.version_info[0] == 3:
+            from io import StringIO
+        else:
+            from StringIO import StringIO
+
+        # capture stdout
+        old = sys.stdout
+        sys.stdout = StringIO()
+
+        assert_warn('foo').is_length(4)
+        assert_warn('foo').is_empty()
+        assert_warn('foo').is_false()
+        assert_warn('foo').is_digit()
+        assert_warn('123').is_alpha()
+        assert_warn('foo').is_upper()
+        assert_warn('FOO').is_lower()
+        assert_warn('foo').is_equal_to('bar')
+        assert_warn('foo').is_not_equal_to('foo')
+        assert_warn('foo').is_equal_to_ignoring_case('BAR')
+
+        # stop capturing stdout
+        out = sys.stdout.getvalue()
+        sys.stdout.close()
+        sys.stdout = old
+
+        assert_that(out).contains('Expected <foo> to be of length <4>, but was <3>.')
+        assert_that(out).contains('Expected <foo> to be empty string, but was not.')
+        assert_that(out).contains('Expected <False>, but was not.')
+        assert_that(out).contains('Expected <foo> to contain only digits, but did not.')
+        assert_that(out).contains('Expected <123> to contain only alphabetic chars, but did not.')
+        assert_that(out).contains('Expected <foo> to contain only uppercase chars, but did not.')
+        assert_that(out).contains('Expected <FOO> to contain only lowercase chars, but did not.')
+        assert_that(out).contains('Expected <foo> to be equal to <bar>, but was not.')
+        assert_that(out).contains('Expected <foo> to be not equal to <foo>, but was.')
+        assert_that(out).contains('Expected <foo> to be case-insensitive equal to <BAR>, but was not.')
+