internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: MongoEngine/mongoengine

Issue:
Rename modifier missing from update
Not sure if this is intentional or not but it would be useful to have the `$rename` operator (or "modifier" for the update method for QuerySet and Document) available.

I'm currently working around it with `exec_js`, like so:

``` python
Document.objects.exec_js("""
function() {
  db[collection].update({}, {$rename: {foo: 'bar'}});
}""")
```



Discussion:
+1

Docs for this operator: http://docs.mongodb.org/manual/reference/operator/update/rename/


Resolving patch:
diff --git a/mongoengine/base/common.py b/mongoengine/base/common.py
--- a/mongoengine/base/common.py
+++ b/mongoengine/base/common.py
@@ -5,7 +5,7 @@
 
 UPDATE_OPERATORS = set(['set', 'unset', 'inc', 'dec', 'pop', 'push',
                         'push_all', 'pull', 'pull_all', 'add_to_set',
-                        'set_on_insert', 'min', 'max'])
+                        'set_on_insert', 'min', 'max', 'rename'])
 
 
 _document_registry = {}


Test patch:
diff --git a/tests/document/instance.py b/tests/document/instance.py
--- a/tests/document/instance.py
+++ b/tests/document/instance.py
@@ -1232,6 +1232,19 @@ def test_update(self):
         self.assertEqual(person.name, None)
         self.assertEqual(person.age, None)
 
+    def test_update_rename_operator(self):
+        """Test the $rename operator."""
+        coll = self.Person._get_collection()
+        doc = self.Person(name='John').save()
+        raw_doc = coll.find_one({'_id': doc.pk})
+        self.assertEqual(set(raw_doc.keys()), set(['_id', '_cls', 'name']))
+
+        doc.update(rename__name='first_name')
+        raw_doc = coll.find_one({'_id': doc.pk})
+        self.assertEqual(set(raw_doc.keys()),
+                         set(['_id', '_cls', 'first_name']))
+        self.assertEqual(raw_doc['first_name'], 'John')
+
     def test_inserts_if_you_set_the_pk(self):
         p1 = self.Person(name='p1', id=bson.ObjectId()).save()
         p2 = self.Person(name='p2')
Repository: MongoEngine/mongoengine

Issue:
__neq__ protocol in datastructure.py
```python
    def __eq__(self, other):
        return self.items() == other.items()

    def __neq__(self, other):
        return self.items() != other.items()
```
defined in [base/datastructures.py](https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/base/datastructures.py#L432), not sure it's a typo for `__ne__` or called in another place?


Discussion:
Agreed, there is no `__neq__` built-in method in python... Great find @lanfon72 ! Would you like to send a PR with a unit test covering this?

Resolving patch:
diff --git a/mongoengine/base/datastructures.py b/mongoengine/base/datastructures.py
--- a/mongoengine/base/datastructures.py
+++ b/mongoengine/base/datastructures.py
@@ -429,7 +429,7 @@ def __len__(self):
     def __eq__(self, other):
         return self.items() == other.items()
 
-    def __neq__(self, other):
+    def __ne__(self, other):
         return self.items() != other.items()
 
     @classmethod


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Invalid QuerySet count caching 
Steps to reproduce:
```
class User(Document):
    name = StringField()

User.drop_collection()
User.objects.create(name='aaa')
User.objects.create(name='bbb')
User.objects.create(name='ccc')
qs = User.objects
print qs.count(with_limit_and_skip=True)  # 3
User.objects.create(name='ddd')
print qs.count(with_limit_and_skip=True)  # still 3 :(
```

Expected behavior: the 2nd count returns 4.
Actual behavior: the 2nd count returns 3, because the query set's previous count result is cached.


Discussion:

Resolving patch:
diff --git a/mongoengine/queryset/base.py b/mongoengine/queryset/base.py
--- a/mongoengine/queryset/base.py
+++ b/mongoengine/queryset/base.py
@@ -396,7 +396,7 @@ def count(self, with_limit_and_skip=False):
             :meth:`skip` that has been applied to this cursor into account when
             getting the count
         """
-        if self._limit == 0 and with_limit_and_skip or self._none:
+        if self._limit == 0 and with_limit_and_skip is False or self._none:
             return 0
         return self._cursor.count(with_limit_and_skip=with_limit_and_skip)
 
@@ -775,10 +775,11 @@ def limit(self, n):
         """Limit the number of returned documents to `n`. This may also be
         achieved using array-slicing syntax (e.g. ``User.objects[:5]``).
 
-        :param n: the maximum number of objects to return
+        :param n: the maximum number of objects to return if n is greater than 0. 
+        When 0 is passed, returns all the documents in the cursor
         """
         queryset = self.clone()
-        queryset._limit = n if n != 0 else 1
+        queryset._limit = n
 
         # If a cursor object has already been created, apply the limit to it.
         if queryset._cursor_obj:


Test patch:
diff --git a/tests/queryset/queryset.py b/tests/queryset/queryset.py
--- a/tests/queryset/queryset.py
+++ b/tests/queryset/queryset.py
@@ -125,6 +125,11 @@ def test_limit(self):
         self.assertEqual(len(people2), 1)
         self.assertEqual(people2[0], user_a)
 
+        # Test limit with 0 as parameter
+        people = self.Person.objects.limit(0)
+        self.assertEqual(people.count(with_limit_and_skip=True), 2)
+        self.assertEqual(len(people), 2)
+
         # Test chaining of only after limit
         person = self.Person.objects().limit(1).only('name').first()
         self.assertEqual(person, user_a)
@@ -5272,6 +5277,16 @@ class BlogPost(Document):
         # in a way we'd expect) should raise a TypeError, too
         self.assertRaises(TypeError, BlogPost.objects(authors__in=author).count)
 
+    def test_create_count(self):
+        self.Person.drop_collection()
+        self.Person.objects.create(name="Foo")
+        self.Person.objects.create(name="Bar")
+        self.Person.objects.create(name="Baz")
+        self.assertEqual(self.Person.objects.count(with_limit_and_skip=True), 3)
+
+        newPerson = self.Person.objects.create(name="Foo_1")
+        self.assertEqual(self.Person.objects.count(with_limit_and_skip=True), 4)
+
 
 if __name__ == '__main__':
     unittest.main()
Repository: MongoEngine/mongoengine

Issue:
Error in validate method, TypeError: not enough arguments for format string
https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/fields.py#L1100


Discussion:
could you provide sample code?
The problem is pretty clear here:

```
        if self.document_type._meta.get('abstract') and \
                not isinstance(value, self.document_type):
            self.error(
                '%s is not an instance of abstract reference type %s' % (
                    self.document_type._class_name)
            )
```

There are two `%s` for a single parameter passed.

Resolving patch:
diff --git a/mongoengine/fields.py b/mongoengine/fields.py
--- a/mongoengine/fields.py
+++ b/mongoengine/fields.py
@@ -1109,7 +1109,7 @@ def validate(self, value):
                 not isinstance(value, self.document_type):
             self.error(
                 '%s is not an instance of abstract reference type %s' % (
-                    self.document_type._class_name)
+                    value, self.document_type._class_name)
             )
 
     def lookup_member(self, member_name):


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Push to a list of cached reference field does not save the cache fields
Push to a list of cached reference field does not save the cache fields, it saves only the reference


Discussion:

Resolving patch:
diff --git a/mongoengine/fields.py b/mongoengine/fields.py
--- a/mongoengine/fields.py
+++ b/mongoengine/fields.py
@@ -1260,7 +1260,11 @@ def prepare_query_value(self, op, value):
             if value.pk is None:
                 self.error('You can only reference documents once they have'
                            ' been saved to the database')
-            return {'_id': value.pk}
+            value_dict = {'_id': value.pk}
+            for field in self.fields:
+                value_dict.update({field: value[field]})
+
+            return value_dict
 
         raise NotImplementedError
 


Test patch:
diff --git a/tests/fields/fields.py b/tests/fields/fields.py
--- a/tests/fields/fields.py
+++ b/tests/fields/fields.py
@@ -4379,6 +4379,51 @@ class SocialData(Document):
         self.assertEqual(SocialData.objects(person__group=g2).count(), 1)
         self.assertEqual(SocialData.objects(person__group=g2).first(), s2)
 
+    def test_cached_reference_field_push_with_fields(self):
+        class Product(Document):
+            name = StringField()
+
+        Product.drop_collection()
+
+        class Basket(Document):
+            products = ListField(CachedReferenceField(Product, fields=['name']))
+
+        Basket.drop_collection()
+        product1 = Product(name='abc').save()
+        product2 = Product(name='def').save()
+        basket = Basket(products=[product1]).save()
+        self.assertEqual(
+            Basket.objects._collection.find_one(),
+            {
+                '_id': basket.pk,
+                'products': [
+                    {
+                        '_id': product1.pk,
+                        'name': product1.name
+                    }
+                ]
+            }
+        )
+        # push to list
+        basket.update(push__products=product2)
+        basket.reload()
+        self.assertEqual(
+            Basket.objects._collection.find_one(),
+            {
+                '_id': basket.pk,
+                'products': [
+                    {
+                        '_id': product1.pk,
+                        'name': product1.name
+                    },
+                    {
+                        '_id': product2.pk,
+                        'name': product2.name
+                    }
+                ]
+            }
+        )
+
     def test_cached_reference_field_update_all(self):
         class Person(Document):
             TYPES = (
Repository: MongoEngine/mongoengine

Issue:
raise StopIteration in generators is deprecated
```
...
/Users/Vic/projects/brain2/brain/utils/user_progress.py:686: DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
  for action in actions:
/Users/Vic/projects/brain2/brain/utils/user_course_state.py:317: DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
  PlayedTask(
/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/mongoengine/dereference.py:33: DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
...
```
https://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and

https://github.com/MongoEngine/mongoengine/blob/5bdd35464b89dd2def1dcb0464e244322f1bc757/mongoengine/queryset/queryset.py#L95


Discussion:
I also have this problem when looping over a queryset.

```
for u in User.objects(name='John'):
    ...

DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
```

Resolving patch:
diff --git a/mongoengine/queryset/queryset.py b/mongoengine/queryset/queryset.py
--- a/mongoengine/queryset/queryset.py
+++ b/mongoengine/queryset/queryset.py
@@ -92,7 +92,7 @@ def _iter_results(self):
             # Raise StopIteration if we already established there were no more
             # docs in the db cursor.
             if not self._has_more:
-                raise StopIteration
+                return
 
             # Otherwise, populate more of the cache and repeat.
             if len(self._result_cache) <= pos:


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Document.ensure_indexes check is_mongos is wrong
in function ensure_indexes inside Document at line 857.
   if not collection.is_mongos and collection.read_preference > 1:
The check on collection.is_mongos generate a new collection a new collection named as
 something.is_mongos

The statement should write as:
   if not collection.database.client.is_mongos and collection.read_preference > 1:


Discussion:
can you add failing test for the current code?

Resolving patch:
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -854,7 +854,7 @@ def ensure_indexes(cls):
         collection = cls._get_collection()
         # 746: when connection is via mongos, the read preference is not necessarily an indication that
         # this code runs on a secondary
-        if not collection.is_mongos and collection.read_preference > 1:
+        if not collection.database.client.is_mongos and collection.read_preference > 1:
             return
 
         # determine if an index which we are creating includes


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Modifying a BaseList while iterating on it makes the iterator buggy.
The surcharged `BaseList.__iter__` assumes that the list won't be modified during iteration, and bugs out if it is (raises KeyError or does not go through all the content).

I believe it could be fixed by looping on `list.__iter__` in that method and passing the yielded value through the code of `__getitem__`. Happy to make a PR if you're interested.


Discussion:

Resolving patch:
diff --git a/mongoengine/base/datastructures.py b/mongoengine/base/datastructures.py
--- a/mongoengine/base/datastructures.py
+++ b/mongoengine/base/datastructures.py
@@ -128,8 +128,8 @@ def __getitem__(self, key, *args, **kwargs):
         return value
 
     def __iter__(self):
-        for i in six.moves.range(self.__len__()):
-            yield self[i]
+        for v in super(BaseList, self).__iter__():
+            yield v
 
     def __setitem__(self, key, value, *args, **kwargs):
         if isinstance(key, slice):
@@ -138,7 +138,7 @@ def __setitem__(self, key, value, *args, **kwargs):
             self._mark_as_changed(key)
         return super(BaseList, self).__setitem__(key, value)
 
-    def __delitem__(self, key, *args, **kwargs):
+    def __delitem__(self, key):
         self._mark_as_changed()
         return super(BaseList, self).__delitem__(key)
 
@@ -187,7 +187,7 @@ def remove(self, *args, **kwargs):
         self._mark_as_changed()
         return super(BaseList, self).remove(*args, **kwargs)
 
-    def reverse(self, *args, **kwargs):
+    def reverse(self):
         self._mark_as_changed()
         return super(BaseList, self).reverse()
 


Test patch:
diff --git a/tests/test_datastructures.py b/tests/test_datastructures.py
--- a/tests/test_datastructures.py
+++ b/tests/test_datastructures.py
@@ -1,6 +1,21 @@
 import unittest
 
-from mongoengine.base.datastructures import StrictDict
+from mongoengine.base.datastructures import StrictDict, BaseList
+
+
+class TestBaseList(unittest.TestCase):
+
+    def test_iter_simple(self):
+        values = [True, False, True, False]
+        base_list = BaseList(values, instance=None, name='my_name')
+        self.assertEqual(values, list(base_list))
+
+    def test_iter_allow_modification_while_iterating_withou_error(self):
+        # regular list allows for this, thus this subclass must comply to that
+        base_list = BaseList([True, False, True, False], instance=None, name='my_name')
+        for idx, val in enumerate(base_list):
+            if val:
+                base_list.pop(idx)
 
 
 class TestStrictDict(unittest.TestCase):
Repository: MongoEngine/mongoengine

Issue:
Querying  EmbeddedDocumentListField with operators 'Client' object has no attribute 'name__icontains'
I have an EmbeddedDocumentListField with Following Fields 
Clients [{"name","email","mobile",..},.....]

now i am able to filter pure on pure fld=value

but unable to use any of the operators along with the fields.
is there a way to do this using mongoengine, can someone help me


If not i would implementing something of my own


Discussion:
can you provide sample code to reproduce this error?
I believe @rp-21 is talking about the .filter of the EmbeddedDocumentList like in:
```
class Client(me.EmbeddedDocument):
    name = me.StringField()

class Doc(me.Document):
    clients = me.EmbeddedDocumentListField(Client)

d = Doc(clients=[Client(name='John')]).save()
d.clients.filter(name='John')			# Out: [<Client: Client object>]
d.clients.filter(name__icontains='J')		# raises AttributeError: 'Client' object has no attribute 'name__icontains'
```
That EmbeddedDocumentList.filter does not support operators like __gte, __lte, __icontains, etc, it only supports simple comparison. I understand that users can get confused with the queryset.filter that has those fancy operators. Implementing those operators here would be overkill, I see 2 options:
- Improve the doc of the EmbeddedDocumentList.filter and mention that it does not supports fancy operator like queryset.filter
- Get rid of EmbeddedDocumentList.filter, I believe we should do this on the long run because the benefits of this method are limited and its confusing with queryset.filter (Just my opinion :))

I'll push a PR to improve the doc shortly

Resolving patch:
diff --git a/mongoengine/base/datastructures.py b/mongoengine/base/datastructures.py
--- a/mongoengine/base/datastructures.py
+++ b/mongoengine/base/datastructures.py
@@ -234,6 +234,9 @@ def filter(self, **kwargs):
         Filters the list by only including embedded documents with the
         given keyword arguments.
 
+        This method only supports simple comparison (e.g: .filter(name='John Doe'))
+        and does not support operators like __gte, __lte, __icontains like queryset.filter does
+
         :param kwargs: The keyword arguments corresponding to the fields to
          filter on. *Multiple arguments are treated as if they are ANDed
          together.*


Test patch:
Repository: MongoEngine/mongoengine

Issue:
DatetimeField - how to accept values like "new Date()" or "new ISODate()"
Hi,
I would like to set a datetime value to, let's say, now, with now being the mongo's server current datetime.
Fact is that if we use python :

`obj.my_datetime_field = datetime.utcnow()`

The datetime will be calculate on the client site, at the time the value is set and pushed to the server.

But if I want the datetime to be server's time (often machines have shifted times) as a reference. The way we can do that in mongo shell is to use Javascript code like this : 

`{ "my_datetime_field" = new Date(), ... }`

In that case, the value will be calculated as the now of the mongo's server.

This case can also be useful for a delayed call to the *save()* method on the object. A field can be set to the date it was saved but not the date the field was first set in python.

The way I can cope with that at the moment, is to use a *DynamicField* and do like this : 

`obj.my_datetime_field = bson.code.Code("new Date()")`

That would be great to be able to use a *Code* object as well as a valid value for a *DatetimeField*.

Does that make sense to you? Would you accept a pull request about that ?

While here, thanks much for the awesome work. Mongoengine really rocks.




Discussion:
Well actually, in this example I wrote earlier :

`obj.my_datetime_field = bson.code.Code("new Date()")`

This does not behave as expected as the value is not calculated but stored as a JS callable in the database, so date is always recalculated when queried : 
```
> db.mycol.find()
{ "_id" : ObjectId("595d5d2e91fdcd02cc7d5fb1"), "created" : ISODate("2017-07-05T21:45:10.488**Z") }
> db.mycol.find() # one minute + secs later
{ "_id" : ObjectId("595d5d2e91fdcd02cc7d5fb1"), "created" : ISODate("2017-07-05T21:46:15.051Z") }
```
See the date that changes... So it is always the now date and always gets recalculated...

But if I insert a document like this : 

`db.mycol.insert({"created" = new Date()})`

It works fine and the date stays the right insert moment. 

Any idea on how we could achieve the same with mongoengine ?
@ronhanson I am using
```py
from datetime import datetime
class User(Document):
     created_at = DateTimeField(default=datetime.utcnow)

Resolving patch:
diff --git a/mongoengine/fields.py b/mongoengine/fields.py
--- a/mongoengine/fields.py
+++ b/mongoengine/fields.py
@@ -462,6 +462,8 @@ class DateTimeField(BaseField):
     installed you can utilise it to convert varying types of date formats into valid
     python datetime objects.
 
+    Note: To default the field to the current datetime, use: DateTimeField(default=datetime.utcnow)
+
     Note: Microseconds are rounded to the nearest millisecond.
       Pre UTC microsecond support is effectively broken.
       Use :class:`~mongoengine.fields.ComplexDateTimeField` if you


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Name of text index
Is it possible to set name of text index?

I have nearly 10 fields which I want to use in text index and I can't, because limited by length of index name (see: https://docs.mongodb.com/v3.2/reference/limits/#Index-Name-Length)

Also, I don't want to use Wildcard index (btw, is it possible in mongoengine?)

Thanks!


Discussion:
I've been digging in also trying to figure out how we can set custom index names...

I found * index_specs* in the document classes, but I can't figure out how to use this to set a custom index name. This may not even be the proper place to look....

But I too am having this issue.

I could disable the index creation in mongoengine for this document, and then create the index manually... but it would be nice if mongoengine handled this natively.
HA!!!!

It's not documented, but... from looking at how the args are passed down to pymongo from the index spec...

You can just do something like this:

```python
meta = {
    'indexes': [
        {
            'fields': (
                'one', 
                'two', 
                'three', 
                'four', 
                'five', 
                'six', 
                'seven', 
                'eight', 
                'nine', 
                'ten', 
                'really_really_long_field_name_to_break_auto_generated_index_names'
                ),
            'name': 'put_custom_index_name_here' # <- put your own index name here
        },
    ]
}
```
@gordol , wow, man, it's awesome, thx!
i missed the nuance that you wanted text indexes... i believe you just append the field names with `$` in that case. cheers!

Resolving patch:
diff --git a/mongoengine/context_managers.py b/mongoengine/context_managers.py
--- a/mongoengine/context_managers.py
+++ b/mongoengine/context_managers.py
@@ -215,7 +215,7 @@ def _get_count(self):
         """Get the number of queries."""
         ignore_query = {'ns': {'$ne': '%s.system.indexes' % self.db.name}}
         count = self.db.system.profile.find(ignore_query).count() - self.counter
-        self.counter += 1
+        self.counter += 1       # Account for the query we just fired
         return count
 
 


Test patch: