Repository: MongoEngine/mongoengine

Issue:
#457 breaks legacy/undeclared fields
#457 is a very big change.

It makes the application not resilient to data change, which is one strong feature of mongo.
It prevent you to load data when all the fields are not declared in your application model.

When you share data between application using mongo, it does not allow to fetch update and save the "known model" without touching unkown fields.

I have tons of documents with legacy fields I don't want to delete or clean now, but I don't want them declared in my application.
It's not possible anymore with #457 and I can't update to 0.9 because my data is unloadable.

I know a workaround is to declare model as dynamic, but it has some side effects I don't want to manage.

I think it should be an optionnal feature. Is it possible to disable it yet ?

Another alternative may be to perform the check only when using the constructor or the `Model.objects.create` method, not when loading existing data.



Discussion:
I don't see that #457 was merged.
How come it creates a problem for you?

Yes it is, but not done through the github pull-request merge feature:
https://github.com/MongoEngine/mongoengine/commits?author=gitaarik

What are the side effects of turning your models into dynamic ones?

I simply don't want extra attributes, not already existing in db, to be saved.
The old mode allowed to preserved legacy ones without adding new one.

I totaly understand #457 , I just want to have the ability to control whether its activated or not, either by model (with a meta attribute) or globally with configuration option.

Does the dynamic document add more fields?
Or is it that you simply don't want them to ever load?

I don't think that reactivating a bug as a configuration option is a good idea.
![Workflow](http://imgs.xkcd.com/comics/workflow.png)

I don't think this is a bug when this a known and explicit behavior, even more when this the default behavior since the first version.

Preventing constructor bad usage, which is good and the purpose of #457, is different than preventing loading data from a schema-less database into a typed and structured model (mostly a vue of the stored data), which is the side effect from #457.
The schema notion doesn't exists in MongoDB and only exists on the application side.

If this feature is not optionnal, it force me to declare any existing attribute in data, even those not related to my application.
If I have another tool setting some data, not declared in my Mongoengine model, I can't load my data anymore because it raises an exception.

On the other side, using a dynamic model saves the extra attributes in DB and prevent to use temporary/client-side only attributes (a little bit sad when using Python)

Like I said: The old mode allowed to preserve legacy/unused/unkown attributes without adding new ones (save perform an update only on changed AND declared fields)

I insist on the fact that I don't want this feature disabled, I want to be able to load data with field not declared in my application model without using DynamicDocument, which was possible in previous versions and was not the purpose of #457 


Resolving patch:
diff --git a/mongoengine/base/document.py b/mongoengine/base/document.py
--- a/mongoengine/base/document.py
+++ b/mongoengine/base/document.py
@@ -70,9 +70,9 @@ def __init__(self, *args, **values):
 
         signals.pre_init.send(self.__class__, document=self, values=values)
 
-        # Check if there are undefined fields supplied, if so raise an
-        # Exception.
-        if not self._dynamic:
+        # Check if there are undefined fields supplied to the constructor,
+        # if so raise an Exception.
+        if not self._dynamic and (self._meta.get('strict', True) or _created):
             for var in values.keys():
                 if var not in self._fields.keys() + ['id', 'pk', '_cls', '_text_score']:
                     msg = (
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -135,6 +135,11 @@ class Document(BaseDocument):
     doesn't contain a list) if allow_inheritance is True. This can be
     disabled by either setting cls to False on the specific index or
     by setting index_cls to False on the meta dictionary for the document.
+
+    By default, any extra attribute existing in stored data but not declared
+    in your model will raise a :class:`~mongoengine.FieldDoesNotExist` error.
+    This can be disabled by setting :attr:`strict` to ``False``
+    in the :attr:`meta` dictionnary.
     """
 
     # The __metaclass__ attribute is removed by 2to3 when running with Python3
diff --git a/mongoengine/errors.py b/mongoengine/errors.py
--- a/mongoengine/errors.py
+++ b/mongoengine/errors.py
@@ -42,7 +42,14 @@ class NotUniqueError(OperationError):
 
 
 class FieldDoesNotExist(Exception):
-    pass
+    """Raised when trying to set a field
+    not declared in a :class:`~mongoengine.Document`
+    or an :class:`~mongoengine.EmbeddedDocument`.
+
+    To avoid this behavior on data loading,
+    you should the :attr:`strict` to ``False``
+    in the :attr:`meta` dictionnary.
+    """
 
 
 class ValidationError(AssertionError):


Test patch:
diff --git a/tests/document/instance.py b/tests/document/instance.py
--- a/tests/document/instance.py
+++ b/tests/document/instance.py
@@ -16,7 +16,8 @@
 
 from mongoengine import *
 from mongoengine.errors import (NotRegistered, InvalidDocumentError,
-                                InvalidQueryError, NotUniqueError)
+                                InvalidQueryError, NotUniqueError,
+                                FieldDoesNotExist)
 from mongoengine.queryset import NULLIFY, Q
 from mongoengine.connection import get_db
 from mongoengine.base import get_document
@@ -2467,6 +2468,114 @@ class Group(Document):
         group = Group.objects.first()
         self.assertEqual("hello - default", group.name)
 
+    def test_load_undefined_fields(self):
+        class User(Document):
+            name = StringField()
+
+        User.drop_collection()
+
+        User._get_collection().save({
+            'name': 'John',
+            'foo': 'Bar',
+            'data': [1, 2, 3]
+        })
+
+        self.assertRaises(FieldDoesNotExist, User.objects.first)
+
+    def test_load_undefined_fields_with_strict_false(self):
+        class User(Document):
+            name = StringField()
+
+            meta = {'strict': False}
+
+        User.drop_collection()
+
+        User._get_collection().save({
+            'name': 'John',
+            'foo': 'Bar',
+            'data': [1, 2, 3]
+        })
+
+        user = User.objects.first()
+        self.assertEqual(user.name, 'John')
+        self.assertFalse(hasattr(user, 'foo'))
+        self.assertEqual(user._data['foo'], 'Bar')
+        self.assertFalse(hasattr(user, 'data'))
+        self.assertEqual(user._data['data'], [1, 2, 3])
+
+    def test_load_undefined_fields_on_embedded_document(self):
+        class Thing(EmbeddedDocument):
+            name = StringField()
+
+        class User(Document):
+            name = StringField()
+            thing = EmbeddedDocumentField(Thing)
+
+        User.drop_collection()
+
+        User._get_collection().save({
+            'name': 'John',
+            'thing': {
+                'name': 'My thing',
+                'foo': 'Bar',
+                'data': [1, 2, 3]
+            }
+        })
+
+        self.assertRaises(FieldDoesNotExist, User.objects.first)
+
+    def test_load_undefined_fields_on_embedded_document_with_strict_false_on_doc(self):
+        class Thing(EmbeddedDocument):
+            name = StringField()
+
+        class User(Document):
+            name = StringField()
+            thing = EmbeddedDocumentField(Thing)
+
+            meta = {'strict': False}
+
+        User.drop_collection()
+
+        User._get_collection().save({
+            'name': 'John',
+            'thing': {
+                'name': 'My thing',
+                'foo': 'Bar',
+                'data': [1, 2, 3]
+            }
+        })
+
+        self.assertRaises(FieldDoesNotExist, User.objects.first)
+
+    def test_load_undefined_fields_on_embedded_document_with_strict_false(self):
+        class Thing(EmbeddedDocument):
+            name = StringField()
+
+            meta = {'strict': False}
+
+        class User(Document):
+            name = StringField()
+            thing = EmbeddedDocumentField(Thing)
+
+        User.drop_collection()
+
+        User._get_collection().save({
+            'name': 'John',
+            'thing': {
+                'name': 'My thing',
+                'foo': 'Bar',
+                'data': [1, 2, 3]
+            }
+        })
+
+        user = User.objects.first()
+        self.assertEqual(user.name, 'John')
+        self.assertEqual(user.thing.name, 'My thing')
+        self.assertFalse(hasattr(user.thing, 'foo'))
+        self.assertEqual(user.thing._data['foo'], 'Bar')
+        self.assertFalse(hasattr(user.thing, 'data'))
+        self.assertEqual(user.thing._data['data'], [1, 2, 3])
+
     def test_spaces_in_keys(self):
 
         class Embedded(DynamicEmbeddedDocument):
diff --git a/tests/fields/fields.py b/tests/fields/fields.py
--- a/tests/fields/fields.py
+++ b/tests/fields/fields.py
@@ -3160,7 +3160,7 @@ class Doc(Document):
 
     def test_undefined_field_exception(self):
         """Tests if a `FieldDoesNotExist` exception is raised when trying to
-        set a value to a field that's not defined.
+        instanciate a document with a field that's not defined.
         """
 
         class Doc(Document):
@@ -3171,6 +3171,21 @@ def test():
 
         self.assertRaises(FieldDoesNotExist, test)
 
+    def test_undefined_field_exception_with_strict(self):
+        """Tests if a `FieldDoesNotExist` exception is raised when trying to
+        instanciate a document with a field that's not defined,
+        even when strict is set to False.
+        """
+
+        class Doc(Document):
+            foo = StringField(db_field='f')
+            meta = {'strict': False}
+
+        def test():
+            Doc(bar='test')
+
+        self.assertRaises(FieldDoesNotExist, test)
+
 
 class EmbeddedDocumentListFieldTestCase(unittest.TestCase):