Repository: simplistix/testfixtures (language: python)

Issue:
ShouldWarn option for disregarding order of warnings
I'd like to use `ShouldWarn` decorator to compare list of warnings disregarding their order. I imagined it could be done like this:

```python
import warnings
from testfixtures import SequenceComparison as S, ShouldWarn

with ShouldWarn(S(UserWarning("X"), UserWarning("Y"), ordered=False)):
    warnings.warn(UserWarning("Y"))
    warnings.warn(UserWarning("X"))
```

However, `ShouldWarn` does not accept `SequenceComparison`. And it also doesn't offer `ordered` kwargs, which would allow us to disregard order. When I do this, the code fails, as it should:

```python
import warnings
from testfixtures import ShouldWarn

with ShouldWarn(UserWarning("X"), UserWarning("Y")):
    warnings.warn(UserWarning("Y"))
    warnings.warn(UserWarning("X"))
```

I see two possible solutions:
1. allow passing of `SequenceComparison` objects directly to `ShouldWarn`
2. add `ordered` kwarg to `ShouldWarn` (default `True`, to maintain the current behaviour)

I'm willing to submit a PR, if we agree on the proper solution.


Discussion / hints:
I'd like to see this happen and great if you're up for working on it!

> However, ShouldWarn does not accept SequenceComparison

What happens when you try?

This feels like less API change, so maybe the way to go?
If you run the first code example I provided, it outputs this:

```
Traceback (most recent call last):
  File "/home/jmusilek/mwe/testfixtures/test.py", line 4, in <module>
    with ShouldWarn(S(UserWarning("X"), UserWarning("Y"), ordered=False)):
  File "/home/jmusilek/mwe/testfixtures/venv/lib/python3.11/site-packages/testfixtures/shouldwarn.py", line 55, in __exit__
    compare(self.expected, actual=[wm.message for wm in self.recorded])
  File "/home/jmusilek/mwe/testfixtures/venv/lib/python3.11/site-packages/testfixtures/comparison.py", line 758, in compare
    raise AssertionError(message)
AssertionError: sequence not as expected:

same:
[]

expected:
[<C:testfixtures.comparison.SequenceComparison(failed)>wrong type</>]

actual:
[UserWarning('Y'), UserWarning('X')]
```

This is because of `ShouldWarn.__init__`:

```python
class ShouldWarn(warnings.catch_warnings):
    def __init__(self, *expected: WarningOrType, **filters):
        super(ShouldWarn, self).__init__(record=True)
        self.expected = [C(e) for e in expected]
        self.filters = filters

    ...

    def __exit__(self, exc_type, exc_val, exc_tb):
        ...
        compare(self.expected, actual=[wm.message for wm in self.recorded])
```

The key element is `[C(e) for e in expected]`. `SequenceComparison` is just one element, which results in `C(S(...))` being compared with list of emmited warnings. It should be `S(C(...), C(...), ...)` instead.

When I went deeper and examined this, I'm not sure how to do such change without breaking the existing behavior. If we went for `ordered` arg on the `ShouldWarn`, that would be easier to implement although it may complicate the the error messages a bit (just because we would be comparing with `SequenceComparison` rather than plain `list`). The proposed interface would look like this:

```python
with ShouldWarn(UserWarning("X"), UserWarning("Y"), ordered=False):
    warnings.warn(UserWarning("Y"))
    warnings.warn(UserWarning("X"))
```

If we used `ordered=True` (or ommited the argument), the error would look like this:
```
<SequenceComparison(ordered=True, partial=False)(failed)>
same:
[]

expected:
[
<C:builtins.UserWarning(failed)>
attributes differ:
'args': ('X',) (Comparison) != ('Y',) (actual)
</C:builtins.UserWarning>,
 <C:builtins.UserWarning>args: ('Y',)</>]

actual:
[UserWarning('Y'), UserWarning('X')]
</SequenceComparison(ordered=True, partial=False)> (expected) != [UserWarning('Y'), UserWarning('X')] (actual)
```

Compare with current error message:

```
AssertionError: sequence not as expected:

same:
[]

expected:
[
<C:builtins.UserWarning(failed)>
attributes differ:
'args': ('X',) (Comparison) != ('Y',) (actual)
</C:builtins.UserWarning>,
 <C:builtins.UserWarning>args: ('Y',)</>]

actual:
[UserWarning('Y'), UserWarning('X')]
```

Not great, not terrible. I could also only create `SequenceComparison` inside `ShouldWarn` only when `ordered=False`. That way, the current behavior would stay 100% the same, even the error messages. We'd just add a new option. This change would be fairly minimal. I'll prepare a PR and we'll see what you think of it.

Fix patch:
diff --git a/docs/warnings.txt b/docs/warnings.txt
index 439834c..f14c9ef 100644
--- a/docs/warnings.txt
+++ b/docs/warnings.txt
@@ -26,8 +26,8 @@ causing the test in which it occurs to fail:
 ...     warn("sorry dave, I can't let you do that")
 Traceback (most recent call last):
 ...
-AssertionError: sequence not as expected:
-<BLANKLINE>
+AssertionError:...
+<SequenceComparison(ordered=True, partial=False)(failed)>
 same:
 []
 <BLANKLINE>
@@ -40,6 +40,7 @@ attributes differ:
 <BLANKLINE>
 actual:
 [UserWarning("sorry dave, I can't let you do that"...)]
+</SequenceComparison(ordered=True, partial=False)> (expected) != [UserWarning("sorry dave, I can't let you do that"...)] (actual)
 
 You can check multiple warnings in a particular piece of code:
 
@@ -50,6 +51,16 @@ You can check multiple warnings in a particular piece of code:
 ...     warn('you should fix that')
 ...     warn('and that too')
 
+If you don't care about the order of issued warnings, you can use ``order_matters=False``:
+
+>>> from warnings import warn
+>>> from testfixtures import ShouldWarn
+>>> with ShouldWarn(UserWarning('you should fix that'),
+...                 UserWarning('and that too'),
+...                 order_matters=False):
+...     warn('and that too')
+...     warn('you should fix that')
+
 If you want to inspect more details of the warnings issued, you can capture
 them into a list as follows:
 
@@ -78,8 +89,8 @@ context it manages, it will raise an :class:`AssertionError` to indicate this:
 ...     warn("woah dude")
 Traceback (most recent call last):
 ...
-AssertionError: sequence not as expected:
-<BLANKLINE>
+AssertionError:...
+<SequenceComparison(ordered=True, partial=False)(failed)>
 same:
 []
 <BLANKLINE>
@@ -88,3 +99,4 @@ expected:
 <BLANKLINE>
 actual:
 [UserWarning('woah dude'...)]
+</SequenceComparison(ordered=True, partial=False)> (expected) != [UserWarning('woah dude'...)] (actual)


Test patch:
diff --git a/testfixtures/shouldwarn.py b/testfixtures/shouldwarn.py
index 08ee630..165a23b 100644
--- a/testfixtures/shouldwarn.py
+++ b/testfixtures/shouldwarn.py
@@ -1,7 +1,7 @@
 import warnings
 from typing import Union, Type
 
-from testfixtures import Comparison as C, compare
+from testfixtures import Comparison, SequenceComparison, compare
 
 
 WarningOrType = Union[Warning, Type[Warning]]
@@ -26,6 +26,13 @@ class ShouldWarn(warnings.catch_warnings):
                      If no expected warnings are passed, you will need to inspect
                      the contents of the list returned by the context manager.
 
+
+    :param order_matters:
+
+      A keyword-only parameter that controls whether the order of the
+      captured entries is required to match those of the expected entries.
+      Defaults to ``True``.
+
     :param filters:
       If passed, these are used to create a filter such that only warnings you
       are interested in will be considered by this :class:`ShouldWarn`
@@ -36,9 +43,10 @@ class ShouldWarn(warnings.catch_warnings):
 
     _empty_okay = False
 
-    def __init__(self, *expected: WarningOrType, **filters):
+    def __init__(self, *expected: WarningOrType, order_matters: bool = True, **filters):
         super(ShouldWarn, self).__init__(record=True)
-        self.expected = [C(e) for e in expected]
+        self.order_matters = order_matters
+        self.expected = [Comparison(e) for e in expected]
         self.filters = filters
 
     def __enter__(self):
@@ -52,7 +60,10 @@ def __exit__(self, exc_type, exc_val, exc_tb):
             return
         if not self.expected and self.recorded and not self._empty_okay:
             return
-        compare(self.expected, actual=[wm.message for wm in self.recorded])
+        compare(
+            expected=SequenceComparison(*self.expected, ordered=self.order_matters),
+            actual=[wm.message for wm in self.recorded]
+        )
 
 
 class ShouldNotWarn(ShouldWarn):
diff --git a/testfixtures/tests/test_shouldwarn.py b/testfixtures/tests/test_shouldwarn.py
index eb4d1a1..2685873 100644
--- a/testfixtures/tests/test_shouldwarn.py
+++ b/testfixtures/tests/test_shouldwarn.py
@@ -25,10 +25,12 @@ def test_warn_expected(self):
 
     def test_warn_not_expected(self):
         with ShouldAssert(
-            "sequence not as expected:\n\n"
+            "\n<SequenceComparison(ordered=True, partial=False)(failed)>\n"
             "same:\n[]\n\n"
             "expected:\n[]\n\n"
-            "actual:\n[UserWarning('foo'"+comma+")]"
+            "actual:\n[UserWarning('foo'"+comma+")]\n"
+            "</SequenceComparison(ordered=True, partial=False)> (expected) "
+            "!= [UserWarning('foo'"+comma+")] (actual)"
         ):
             with warnings.catch_warnings(record=True) as backstop:
                 with ShouldNotWarn():
@@ -41,10 +43,11 @@ def test_no_warn_expected(self):
 
     def test_no_warn_not_expected(self):
         with ShouldAssert(
-            "sequence not as expected:\n\n"
+            "\n<SequenceComparison(ordered=True, partial=False)(failed)>\n"
             "same:\n[]\n\n"
             "expected:\n[<C:builtins.UserWarning>args: ('foo',)</>]"
-            "\n\nactual:\n[]"
+            "\n\nactual:\n[]\n"
+            "</SequenceComparison(ordered=True, partial=False)> (expected) != [] (actual)"
         ):
             with ShouldWarn(UserWarning('foo')):
                 pass
@@ -64,17 +67,42 @@ def test_multiple_warnings(self):
         self.assertTrue('foo' in content)
         self.assertTrue('bar' in content)
 
+    def test_multiple_warnings_ordered(self):
+        with warnings.catch_warnings(record=True) as backstop:
+            with ShouldWarn(UserWarning('foo'), UserWarning('bar')):
+                warnings.warn('foo')
+                warnings.warn('bar')
+        compare(len(backstop), expected=0)
+
+    def test_multiple_warnings_wrong_order(self):
+        with ShouldRaise(AssertionError) as s:
+            with ShouldWarn(UserWarning('foo'), UserWarning('bar')):
+                warnings.warn('bar')
+                warnings.warn('foo')
+        content = str(s.raised)
+        self.assertTrue('foo' in content)
+        self.assertTrue('bar' in content)
+
+    def test_multiple_warnings_ignore_order(self):
+        with warnings.catch_warnings(record=True) as backstop:
+            with ShouldWarn(UserWarning('foo'), UserWarning('bar'), order_matters=False):
+                warnings.warn('bar')
+                warnings.warn('foo')
+        compare(len(backstop), expected=0)
+
     def test_minimal_ok(self):
         with ShouldWarn(UserWarning):
             warnings.warn('foo')
 
     def test_minimal_bad(self):
         with ShouldAssert(
-            "sequence not as expected:\n\n"
+            "\n<SequenceComparison(ordered=True, partial=False)(failed)>\n"
             "same:\n[]\n\n"
             "expected:\n"
             "[<C:builtins.DeprecationWarning(failed)>wrong type</>]\n\n"
-            "actual:\n[UserWarning('foo'"+comma+")]"
+            "actual:\n[UserWarning('foo'"+comma+")]\n"
+            "</SequenceComparison(ordered=True, partial=False)> (expected) "
+            "!= [UserWarning('foo'"+comma+")] (actual)"
         ):
             with ShouldWarn(DeprecationWarning):
                 warnings.warn('foo')
@@ -87,14 +115,16 @@ def test_maximal_ok(self):
 
     def test_maximal_bad(self):
         with ShouldAssert(
-            "sequence not as expected:\n\n"
+            "\n<SequenceComparison(ordered=True, partial=False)(failed)>\n"
             "same:\n[]\n\n"
             "expected:\n[\n"
             "<C:builtins.DeprecationWarning(failed)>\n"
             "attributes differ:\n"
             "'args': ('bar',) (Comparison) != ('foo',) (actual)\n"
             "</C:builtins.DeprecationWarning>]\n\n"
-            "actual:\n[DeprecationWarning('foo'"+comma+")]"
+            "actual:\n[DeprecationWarning('foo'"+comma+")]\n"
+            "</SequenceComparison(ordered=True, partial=False)> (expected) "
+            "!= [DeprecationWarning('foo'"+comma+")] (actual)"
         ):
             with ShouldWarn(DeprecationWarning('bar')):
                 warnings.warn_explicit(
@@ -130,10 +160,11 @@ def test_filter_present(self):
 
     def test_filter_missing(self):
         with ShouldAssert(
-            "sequence not as expected:\n\n"
+            "\n<SequenceComparison(ordered=True, partial=False)(failed)>\n"
             "same:\n[]\n\n"
             "expected:\n[<C:builtins.DeprecationWarning>]\n\n"
-            "actual:\n[]"
+            "actual:\n[]\n"
+            "</SequenceComparison(ordered=True, partial=False)> (expected) != [] (actual)"
         ):
             with ShouldWarn(DeprecationWarning,
                             message="This function is deprecated."):