Repository: pytest-dev/pytest-django

Issue:
3.1.0 not backward compatible with 3.0.0
Does this project use semver? It looks like there's a backward-incompatible change between 3.0.0 and 3.1.0 that ended up failing any tests that relied on `mail.outbox`. What versions should we be tracking to avoid backward-incompatible changes?


Discussion:
Thanks for the report! In the case with mail.outbox it was never documented that it was actually cleared. We figured it was not covered by the normal backward compatibility guarantees. However we preferred to make it clear that it was not cleared and raising errors instead of silently stop cleaning it and potentially having tests pass that really should fail when we introduced the mailoutbox API.

I'm sorry it caused you trouble. I've amended the changelog to include a simple workaround to keep the old behavior until you can upgrade to the mailoutbox fixture: http://pytest-django.readthedocs.io/en/latest/changelog.html#compatibility
Thanks for providing the workaround.

I think the project should be upgraded to 4.0 because this will break many users' projects and I was surprised to see a breaking change in a point release. 

Alternatively, an option should be used to opt-in to this behaviour and a warning added that this will change in the next major version.

It's true the feature was never documented, but Django's `TestCase` does this by default and the current behaviour, while perhaps not "pytestic", is what is expected by many users of the plugin.
Thanks for the quick response. I have to agree with @ryankask above, if this project uses semantic versioning (which it seems to?) I think backward-incompatible changes should bump the major version rather than the minor.

However we preferred to make it clear that it was not cleared and raising errors instead of silently stop cleaning it and potentially having tests pass that really should fail when we introduced the mailoutbox API.

I understand your point that the behavior of `mail.outbox` is a new feature, and I agree that _not_ throwing an error would be silly, but if the new feature is backward-incompatible it seems that it should bump the major version rather than the minor. I get that `mail.outbox` wasn't explicitly part of your API, but unless the 3.0.0 API said "don't use `mail.outbox`, it's unreliable" I think it would be reasonable to expect 3.1.0 to continue working the same way that 3.0.0 did. It isn't that `mailoutbox` is hard to implement, it's that I'd like to trust that `pytest-django>=3,<4` isn't going to break our already-existing code.

Thanks for your work on this project, cheers!
Clearly this caused more problems than I anticipated. :-)

I propose that we add back the automatic clearing of mail.outbox and release 3.1.1.

Then if anyone is interested in pushing it, pytest-django 4.0 could emit a warning about mail.outbox being deprecated and in pytest-django 5.0 we can raise explicit errors. However, I think we should never remove the explicit errors since it would be very easy to accidentally screw up by relying on an unreliable outbox and having broken tests. And then, what's the point of going through all the pain of deprecating it just to end up with a more complicated solution that guards the outbox? :-)

So, let's revert to clearing the `mail.outbox` reliably while still keeping the mailoutbox fixture, then we can continue the discussion from there. :-)

If anyone is interested in working on a PR to bring it back that would be really great, otherwise I will try to find some time during the weekend.

/cc @peterlauri 
@pelme I'll do that, if you assign it to me ;)

Resolving patch:
diff --git a/pytest_django/plugin.py b/pytest_django/plugin.py
--- a/pytest_django/plugin.py
+++ b/pytest_django/plugin.py
@@ -415,44 +415,22 @@ def teardown():
         request.addfinalizer(teardown)
 
 
-class _DirectMailboxAccessProtector(list):
-
-    def _raise_assertion(*args, **kwargs):
-        __tracebackhide__ = True
-        raise AssertionError('''To access mail.outbox, use the mailoutbox fixture.
-See http://pytest-django.readthedocs.io/en/latest/helpers.html#mailoutbox for more information.''')
-
-    __len__ = _raise_assertion
-    __getitem__ = _raise_assertion
-    __nonzero__ = _raise_assertion
-    __bool__ = _raise_assertion
-    __eq__ = _raise_assertion
-    __ne__ = _raise_assertion
-    __iter__ = _raise_assertion
-
-
-@pytest.fixture(autouse=True)
-def _error_on_direct_mail_outbox_access(monkeypatch):
+@pytest.fixture(scope='function', autouse=True)
+def _dj_autoclear_mailbox():
     if not django_settings_is_configured():
         return
 
     from django.core import mail
-
-    outbox = _DirectMailboxAccessProtector()
-    monkeypatch.setattr(mail, 'outbox', outbox)
-    return outbox
+    del mail.outbox[:]
 
 
 @pytest.fixture(scope='function')
-def mailoutbox(monkeypatch, _error_on_direct_mail_outbox_access):
+def mailoutbox(monkeypatch, _dj_autoclear_mailbox):
     if not django_settings_is_configured():
         return
 
     from django.core import mail
-
-    outbox = list()
-    monkeypatch.setattr(mail, 'outbox', outbox)
-    return outbox
+    return mail.outbox
 
 
 @pytest.fixture(autouse=True, scope='function')


Test patch:
diff --git a/tests/test_environment.py b/tests/test_environment.py
--- a/tests/test_environment.py
+++ b/tests/test_environment.py
@@ -18,42 +18,17 @@
 # This is possible with some of the testdir magic, but this is the lazy way
 # to do it.
 
-
-class Test_direct_mailbox_access_not_allowed():
-
-    def test_len(self):
-        with pytest.raises(AssertionError):
-            len(mail.outbox)
-
-    def test_indexing(self):
-        with pytest.raises(AssertionError):
-            mail.outbox[0]
-
-    def test_bool(self):
-        with pytest.raises(AssertionError):
-            if mail.outbox:
-                pass
-
-    def test_equality(self):
-        with pytest.raises(AssertionError):
-            mail.outbox == 'whatever'
-
-    def test_not_equality(self):
-        with pytest.raises(AssertionError):
-            mail.outbox != 'whatever'
-
-    def test_unpacking(self):
-        with pytest.raises(AssertionError):
-            (foo,) = mail.outbox
-
-    def test_iteration(self):
-        with pytest.raises(AssertionError):
-            for x in mail.outbox:
-                pass
-
-
-def test_direct_mailbox_proection_should_not_break_sending_mail():
-    mail.send_mail('subject', 'body', 'from@example.com', ['to@example.com'])
+@pytest.mark.parametrize('subject', ['subject1', 'subject2'])
+def test_autoclear_mailbox(subject):
+    assert len(mail.outbox) == 0
+    mail.send_mail(subject, 'body', 'from@example.com', ['to@example.com'])
+    assert len(mail.outbox) == 1
+
+    m = mail.outbox[0]
+    assert m.subject == subject
+    assert m.body == 'body'
+    assert m.from_email == 'from@example.com'
+    assert m.to == ['to@example.com']
 
 
 class TestDirectAccessWorksForDjangoTestCase(TestCase):