Repository: MongoEngine/mongoengine

Issue:
Uniqueness not working across multiple test cases
I have a simple test that tries to verify the uniqueness of a certain field of a document. However, the uniqueness setting seems to not working without dropping the entire collection in addition to dropping the database between tests. I am not sure if I misunderstood something here or it's a bug of some kind. Any advice would be much appreciated. Thanks!

``` python
import unittest
from mongoengine import *

class User(Document):
    email = EmailField(max_length=200, required=True, unique=True)

class TestBase(unittest.TestCase):
    def setUp(self):
        self.db = connect('test_database')

    def tearDown(self):
        # User.drop_collection() # Uncomment this line to get the expected results.
        self.db.drop_database('test_database')
        self.db.close()

    def test_duplicate(self):
        User(email='test@test.com').save()
        User(email='test@test.com').save()

    def test_duplicate_again(self):
        User(email='test@test.com').save()
        User(email='test@test.com').save()

    def test_duplicatw_once_again(self):
        User(email='test@test.com').save()
        User(email='test@test.com').save()

if __name__ == '__main__':
    unittest.main()
```

The expected result here is that all three test cases would fail with the `NotUniqueError` error. However, only the first test case failed and the rest of the tests actually passed.

```
(venv)$ python -m unittest -v tests.users
test_duplicate (tests.users.TestBase) ... ERROR
test_duplicate_again (tests.users.TestBase) ... ok
test_duplicatw_once_again (tests.users.TestBase) ... ok

======================================================================
ERROR: test_duplicate (tests.users.TestBase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/users.py", line 18, in test_duplicate
    User(email='test@test.com').save()
  File "/Users/starsirius/Code/santa/venv/src/mongoengine/mongoengine/document.py", line 312, in save
    raise NotUniqueError(message % unicode(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error index: test_database.user.$email_1  dup key: { : "test@test.com" })

----------------------------------------------------------------------
Ran 3 tests in 1.138s

FAILED (errors=1)
```

If I uncomment the `User.drop_collection()` line (hence drop the collection between tests), the result is correct and all three tests failed.

```
(venv)$ python -m unittest -v tests.users
test_duplicate (tests.users.TestBase) ... ERROR
test_duplicate_again (tests.users.TestBase) ... ERROR
test_duplicatw_once_again (tests.users.TestBase) ... ERROR

======================================================================
ERROR: test_duplicate (tests.users.TestBase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/users.py", line 18, in test_duplicate
    User(email='test@test.com').save()
  File "/Users/starsirius/Code/santa/venv/src/mongoengine/mongoengine/document.py", line 312, in save
    raise NotUniqueError(message % unicode(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error index: test_database.user.$email_1  dup key: { : "test@test.com" })

======================================================================
ERROR: test_duplicate_again (tests.users.TestBase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/users.py", line 22, in test_duplicate_again
    User(email='test@test.com').save()
  File "/Users/starsirius/Code/santa/venv/src/mongoengine/mongoengine/document.py", line 312, in save
    raise NotUniqueError(message % unicode(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error index: test_database.user.$email_1  dup key: { : "test@test.com" })

======================================================================
ERROR: test_duplicatw_once_again (tests.users.TestBase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/users.py", line 26, in test_duplicatw_once_again
    User(email='test@test.com').save()
  File "/Users/starsirius/Code/santa/venv/src/mongoengine/mongoengine/document.py", line 312, in save
    raise NotUniqueError(message % unicode(err))
NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error index: test_database.user.$email_1  dup key: { : "test@test.com" })

----------------------------------------------------------------------
Ran 3 tests in 1.200s

FAILED (errors=3)
```

So, two things that I don't understand here. Firstly, what's the difference between only dropping the database and dropping collections in addition to dropping the database. I thought dropping the database should be enough to reset the state. Secondly, no matter what, the uniqueness behaves weirdly here and perhaps I did something incorrectly?

Any ideas? Thanks a lot!



Discussion:
I have a feeling (without running the same tests locally) that the unique index is not being re-created in the database between tests, causing the unique constraint to not be enforced.

MongoEngine uses Unique Indices in the Collection to enforce unique fields. Perhaps try running the tests again and ensure that the indices do/don't exist?

@mmelliso Thanks a lot for the response. I am pretty new to this, and could you please point me to how to check the existence of the unique indices?

Also, how would you recommend me to reset the database state between tests, if dropping the database isn't enough? Thanks again!

Dropping the database _should_ be enough, but after testing it appears there is a bug where MongoEngine is not ensuring indices after a database has been dropped. 

MongoDB command to view indices:

``` javascript
db.user.getIndexes()
```

After the first test (before drop):

``` json
{
    "0" : {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test_database.user"
    },
    "1" : {
        "v" : 1,
        "unique" : true,
        "key" : {
            "email" : 1
        },
        "name" : "email_1",
        "ns" : "test_database.user",
        "background" : false,
        "sparse" : false,
        "dropDups" : false
    }
}
```

Subsequent tests (after first drop):

``` json
{
    "0" : {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test_database.user"
    }
}
```

If you have the `"unique": true` key/value pair, then the index will not allow duplicate entries, but as you can see above, the index for `email` is never re-created after the database is dropped. _This is clearly a bug._

You can try running `ensure_indexes()` on your Document class to force creation of the indices. Ugly workaround, but it works.

``` python
import unittest
from mongoengine import *

class User(Document):
    email = EmailField(max_length=200, required=True, unique=True)

class TestBase(unittest.TestCase):
    def setUp(self):
        self.db = connect('test_database')

    def tearDown(self):
        #User.drop_collection() # Uncomment this line to get the expected results.
        self.db.drop_database('test_database')
        self.db.close()

    def test_duplicate(self):
        User(email='test@test.com').save()
        User(email='test@test.com').save()

    def test_duplicate_again(self):
        User(email='test@test.com').save()
        User.ensure_indexes()
        User(email='test@test.com').save()

    def test_duplicatw_once_again(self):
        User(email='test@test.com').save()
        User.ensure_indexes()
        User(email='test@test.com').save()

if __name__ == '__main__':
    unittest.main()
```

As for dropping the collection vs dropping the database, obviously if you drop the entire database, the collection is dropped as well, so there would be no reason to call both back to back like that.

In my opinion, if you are trying to clear state between test cases and keep your MongoDB clean I would consider doing the following: 
1. `User.drop_collection()` inside of `tearDown` to clean the collection between test cases.
2. `self.db.drop_database('test_database')` inside of `tearDownClass` to remove the database after all the tests have been run.


Resolving patch:
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -285,6 +285,8 @@ def save(self, force_insert=False, validate=True, clean=True,
 
         try:
             collection = self._get_collection()
+            if self._meta.get('auto_create_index', True):
+                self.ensure_indexes()
             if created:
                 if force_insert:
                     object_id = collection.insert(doc, **write_concern)


Test patch:
diff --git a/tests/document/indexes.py b/tests/document/indexes.py
--- a/tests/document/indexes.py
+++ b/tests/document/indexes.py
@@ -18,7 +18,7 @@
 class IndexesTest(unittest.TestCase):
 
     def setUp(self):
-        connect(db='mongoenginetest')
+        self.connection = connect(db='mongoenginetest')
         self.db = get_db()
 
         class Person(Document):
@@ -795,6 +795,33 @@ class Book(Document):
         key = indexes["title_text"]["key"]
         self.assertTrue(('_fts', 'text') in key)
 
+    def test_indexes_after_database_drop(self):
+        """
+        Test to ensure that indexes are re-created on a collection even
+        after the database has been dropped.
+
+        Issue #812
+        """
+        class BlogPost(Document):
+            title = StringField()
+            slug = StringField(unique=True)
+
+        BlogPost.drop_collection()
+
+        # Create Post #1
+        post1 = BlogPost(title='test1', slug='test')
+        post1.save()
+
+        # Drop the Database
+        self.connection.drop_database(BlogPost._get_db().name)
+
+        # Re-create Post #1
+        post1 = BlogPost(title='test1', slug='test')
+        post1.save()
+
+        # Create Post #2
+        post2 = BlogPost(title='test2', slug='test')
+        self.assertRaises(NotUniqueError, post2.save)
 
 if __name__ == '__main__':
     unittest.main()