internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: MongoEngine/mongoengine

Issue:
raise StopIteration in generators is deprecated
```
...
/Users/Vic/projects/brain2/brain/utils/user_progress.py:686: DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
  for action in actions:
/Users/Vic/projects/brain2/brain/utils/user_course_state.py:317: DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
  PlayedTask(
/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/mongoengine/dereference.py:33: DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
...
```
https://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and

https://github.com/MongoEngine/mongoengine/blob/5bdd35464b89dd2def1dcb0464e244322f1bc757/mongoengine/queryset/queryset.py#L95


Discussion:
I also have this problem when looping over a queryset.

```
for u in User.objects(name='John'):
    ...

DeprecationWarning: generator 'QuerySet._iter_results' raised StopIteration
```

Resolving patch:
diff --git a/mongoengine/queryset/queryset.py b/mongoengine/queryset/queryset.py
--- a/mongoengine/queryset/queryset.py
+++ b/mongoengine/queryset/queryset.py
@@ -89,7 +89,7 @@ def _iter_results(self):
                 yield self._result_cache[pos]
                 pos += 1
 
-            # Raise StopIteration if we already established there were no more
+            # return if we already established there were no more
             # docs in the db cursor.
             if not self._has_more:
                 return


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Limit function behaviour when 0 is passed
It seems that using the limit function with 0 parameters overrides the underlying behaviour of pyMongo limit function. One would expect that passing 0 as a parameter will not limit the output, but only one element is returned


Discussion:

Resolving patch:
diff --git a/mongoengine/queryset/base.py b/mongoengine/queryset/base.py
--- a/mongoengine/queryset/base.py
+++ b/mongoengine/queryset/base.py
@@ -396,7 +396,7 @@ def count(self, with_limit_and_skip=False):
             :meth:`skip` that has been applied to this cursor into account when
             getting the count
         """
-        if self._limit == 0 and with_limit_and_skip or self._none:
+        if self._limit == 0 and with_limit_and_skip is False or self._none:
             return 0
         return self._cursor.count(with_limit_and_skip=with_limit_and_skip)
 
@@ -775,10 +775,11 @@ def limit(self, n):
         """Limit the number of returned documents to `n`. This may also be
         achieved using array-slicing syntax (e.g. ``User.objects[:5]``).
 
-        :param n: the maximum number of objects to return
+        :param n: the maximum number of objects to return if n is greater than 0.
+        When 0 is passed, returns all the documents in the cursor
         """
         queryset = self.clone()
-        queryset._limit = n if n != 0 else 1
+        queryset._limit = n
 
         # If a cursor object has already been created, apply the limit to it.
         if queryset._cursor_obj:


Test patch:
diff --git a/tests/queryset/queryset.py b/tests/queryset/queryset.py
--- a/tests/queryset/queryset.py
+++ b/tests/queryset/queryset.py
@@ -125,6 +125,11 @@ def test_limit(self):
         self.assertEqual(len(people2), 1)
         self.assertEqual(people2[0], user_a)
 
+        # Test limit with 0 as parameter
+        people = self.Person.objects.limit(0)
+        self.assertEqual(people.count(with_limit_and_skip=True), 2)
+        self.assertEqual(len(people), 2)
+
         # Test chaining of only after limit
         person = self.Person.objects().limit(1).only('name').first()
         self.assertEqual(person, user_a)
@@ -5272,6 +5277,16 @@ class BlogPost(Document):
         # in a way we'd expect) should raise a TypeError, too
         self.assertRaises(TypeError, BlogPost.objects(authors__in=author).count)
 
+    def test_create_count(self):
+        self.Person.drop_collection()
+        self.Person.objects.create(name="Foo")
+        self.Person.objects.create(name="Bar")
+        self.Person.objects.create(name="Baz")
+        self.assertEqual(self.Person.objects.count(with_limit_and_skip=True), 3)
+
+        newPerson = self.Person.objects.create(name="Foo_1")
+        self.assertEqual(self.Person.objects.count(with_limit_and_skip=True), 4)
+
 
 if __name__ == '__main__':
     unittest.main()
Repository: MongoEngine/mongoengine

Issue:
Deprication warning for Python 3.8 (collections)
```
    python3.7/site-packages/mongoengine/base/document.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
      from collections import Hashable
```


Discussion:

Resolving patch:
diff --git a/mongoengine/base/document.py b/mongoengine/base/document.py
--- a/mongoengine/base/document.py
+++ b/mongoengine/base/document.py
@@ -1,6 +1,5 @@
 import copy
 import numbers
-from collections import Hashable
 from functools import partial
 
 from bson import ObjectId, json_util
@@ -19,6 +18,7 @@
 from mongoengine.common import _import_class
 from mongoengine.errors import (FieldDoesNotExist, InvalidDocumentError,
                                 LookUpError, OperationError, ValidationError)
+from mongoengine.python_support import Hashable
 
 __all__ = ('BaseDocument', 'NON_FIELD_ERRORS')
 
diff --git a/mongoengine/python_support.py b/mongoengine/python_support.py
--- a/mongoengine/python_support.py
+++ b/mongoengine/python_support.py
@@ -19,3 +19,10 @@
         pass
     else:
         StringIO = cStringIO.StringIO
+
+
+if six.PY3:
+    from collections.abc import Hashable
+else:
+    # raises DeprecationWarnings in Python >=3.7
+    from collections import Hashable


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Deprication Warnings with pymongo 3.7.1
It looks like some method calls have changed in the latest pymongo version.

See:
```
.../mongoengine/document.py:434: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead object_id = collection.save(doc, **write_concern)
```

Maybe it is a good idea to adapt the internal function calls accordingly?


Discussion:
Found the same problem here. Is there a plan to fix this?
[Stop using deprecated collection methods for PyMongo v3.x](https://github.com/MongoEngine/mongoengine/issues/1491) 
I think this the story they are working on.

I added below configuration to `pytest.ini` to block the warning from pytest
```
[pytest]
filterwarnings =
    ignore::DeprecationWarning:mongoengine
```
Can you check on mongoengine==0.16.0?
@erdenezul checked, warning still exists on `mongoengine==0.16.0`...

Code is here:
https://github.com/MongoEngine/mongoengine/blob/v0.16.0/mongoengine/document.py#L436

```
> pytest
.../site-packages/mongoengine/document.py:436: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
    object_id = collection.save(doc, **write_concern)
```
related to #1491 
I will fix and release 0.16.1

Resolving patch:
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -12,7 +12,9 @@
                               TopLevelDocumentMetaclass, get_document)
 from mongoengine.common import _import_class
 from mongoengine.connection import DEFAULT_CONNECTION_NAME, get_db
-from mongoengine.context_managers import switch_collection, switch_db
+from mongoengine.context_managers import (set_write_concern,
+                                          switch_collection,
+                                          switch_db)
 from mongoengine.errors import (InvalidDocumentError, InvalidQueryError,
                                 SaveConditionError)
 from mongoengine.python_support import IS_PYMONGO_3
@@ -429,11 +431,18 @@ def _save_create(self, doc, force_insert, write_concern):
         Helper method, should only be used inside save().
         """
         collection = self._get_collection()
-
-        if force_insert:
-            return collection.insert(doc, **write_concern)
-
-        object_id = collection.save(doc, **write_concern)
+        with set_write_concern(collection, write_concern) as wc_collection:
+            if force_insert:
+                return wc_collection.insert_one(doc).inserted_id
+            # insert_one will provoke UniqueError alongside save does not
+            # therefore, it need to catch and call replace_one.
+            if '_id' in doc:
+                raw_object = wc_collection.find_one_and_replace(
+                    {'_id': doc['_id']}, doc)
+                if raw_object:
+                    return doc['_id']
+
+            object_id = wc_collection.insert_one(doc).inserted_id
 
         # In PyMongo 3.0, the save() call calls internally the _update() call
         # but they forget to return the _id value passed back, therefore getting it back here


Test patch:
Repository: MongoEngine/mongoengine

Issue:
DictField key can contains dollar sign for #1758
I need save key with $ char in the middle of keys. Mongodb can accept $ sign in the middle of key (contains).
Related issue : #1758 


Discussion:

Resolving patch:
diff --git a/mongoengine/fields.py b/mongoengine/fields.py
--- a/mongoengine/fields.py
+++ b/mongoengine/fields.py
@@ -944,7 +944,7 @@ def key_has_dot_or_dollar(d):
     dictionary contains a dot or a dollar sign.
     """
     for k, v in d.items():
-        if ('.' in k or '$' in k) or (isinstance(v, dict) and key_has_dot_or_dollar(v)):
+        if ('.' in k or k.startswith('$')) or (isinstance(v, dict) and key_has_dot_or_dollar(v)):
             return True
 
 
@@ -977,7 +977,7 @@ def validate(self, value):
             self.error(msg)
         if key_has_dot_or_dollar(value):
             self.error('Invalid dictionary key name - keys may not contain "."'
-                       ' or "$" characters')
+                       ' or startswith "$" characters')
         super(DictField, self).validate(value)
 
     def lookup_member(self, member_name):


Test patch:
diff --git a/tests/fields/fields.py b/tests/fields/fields.py
--- a/tests/fields/fields.py
+++ b/tests/fields/fields.py
@@ -1698,6 +1698,10 @@ class BlogPost(Document):
         post.info = {'title': 'test'}
         post.save()
 
+        post = BlogPost()
+        post.info = {'title' : 'dollar_sign', 'details' : {'te$t' : 'test'} }
+        post.save()
+
         post = BlogPost()
         post.info = {'details': {'test': 'test'}}
         post.save()
@@ -1706,12 +1710,15 @@ class BlogPost(Document):
         post.info = {'details': {'test': 3}}
         post.save()
 
-        self.assertEqual(BlogPost.objects.count(), 3)
+        self.assertEqual(BlogPost.objects.count(), 4)
         self.assertEqual(
             BlogPost.objects.filter(info__title__exact='test').count(), 1)
         self.assertEqual(
             BlogPost.objects.filter(info__details__test__exact='test').count(), 1)
 
+        post = BlogPost.objects.filter(info__title__exact='dollar_sign').first()
+        self.assertIn('te$t', post['info']['details'])
+         
         # Confirm handles non strings or non existing keys
         self.assertEqual(
             BlogPost.objects.filter(info__details__test__exact=5).count(), 0)
Repository: MongoEngine/mongoengine

Issue:
Queryset.all() not returning all documents
According to the [API Documentation](http://docs.mongoengine.org/apireference.html#mongoengine.queryset.QuerySet.all), QuerySet.all() **returns all documents**.

So I just assuming `all()` will return a list of `Document` Object

But `all()` returning QuerySet object actually.

Is this a bug or a easily misunderstanding documentation description?

And is there any way to get a list of `Document` conveniencely like `doc_list = queryset.all()`
rather than using a generator like `[doc for doc in queryset]` ?


Discussion:
Maybe should mark this issue as a Question :P
Hi @StoneMoe, I agree that `.all()` doc is a bit confusing. I believe .all() was added to mimic django's [.all()](https://docs.djangoproject.com/en/2.1/ref/models/querysets/#django.db.models.query.QuerySet.all) method but since you can use .objects(), most of the time you won't need .all in mongoengine. I believe it is intended that it returns a queryset but I'll update the doc to make that more clear.

Note that the easiest (and most explicit by the way) to get a list from a queryset is to simply cast it into a list: `doc_list = list(queryset)`, this does exactly the same as `doc_list=[doc for doc in queryset]`

Resolving patch:
diff --git a/mongoengine/queryset/base.py b/mongoengine/queryset/base.py
--- a/mongoengine/queryset/base.py
+++ b/mongoengine/queryset/base.py
@@ -215,7 +215,7 @@ def __bool__(self):
     # Core functions
 
     def all(self):
-        """Returns all documents."""
+        """Returns a copy of the current QuerySet."""
         return self.__call__()
 
     def filter(self, *q_objs, **query):


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Missuse of write_concern in Document.save
It is possible to define write_concern on the connection.
However, while calling save method on a document, the following code (line 229 in document.py) tells you that if it's not define on save call, it is erased, whatever is your settings on the connection: 

```
        if write_concern is None:
            write_concern = {"w": 1}
```

The idea is to delete those two lines to fallback on connection settings.



Discussion:
Hum, my bad after some reflexion, deleting the 2 quoted lines is not enough. The default value for write_concern should be an empty dict, since it is used as kwargs. And it shouldn't be added into the cascade save kwargs in case it is empty.

You should pass `w=0` to the `connect` function as well

This is a bit unfortunate, since it will always override the global write concern (configured when a connection is registered). The Bug label would more appropriate I think.

This is still the case with the latest master -> https://github.com/MongoEngine/mongoengine/blob/24d15d427482f39f7cfa0e9ac0a8b44393aa9ddf/mongoengine/document.py#L358-L359

Resolving patch:
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -375,7 +375,7 @@ def save(self, force_insert=False, validate=True, clean=True,
             self.validate(clean=clean)
 
         if write_concern is None:
-            write_concern = {'w': 1}
+            write_concern = {}
 
         doc = self.to_mongo()
 


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Clarify usage of the shard_key meta attribute
Use `_id`  as ShardKey, and set meta `meta['shard_key'] = ('id',)`.
When try to `reload()` an object, it will raise:
```python
AttributeError: 'Test' object has no attribute '_id'
```

The issue was in this code.  
https://github.com/MongoEngine/mongoengine/blob/e8dbd12f2250fa68cd03b33b8c4d14cf75b92841/mongoengine/document.py#L563
Here will parse `id` to actual_key `_id`, and in 
https://github.com/MongoEngine/mongoengine/blob/e8dbd12f2250fa68cd03b33b8c4d14cf75b92841/mongoengine/document.py#L566
MongoEngine will try to get `_id` from the object, but the object only have `object.id`.



Discussion:
Hi @Ricky-Hao! I agree that the error message could be improved here, or it could *just work*. That said, is there a reason you're setting `meta['shard_key']` to just the `('id',)`?

Specifying a shard key really only makes sense if you shard by something else than the ID, since ID is already included as the "query" part of every update/modify/delete operation:
* If you update a document in a collection sharded by `_id`, we'll already emit the right `update({_id: 'id_of_the_updated_doc'}, {...updated_data...})` by default.
* If you update a document in a collection sharded e.g. by `('id', 'other_field')`, then filtering by `_id` is not enough for the most optimal shard routing. Specifying the `shard_key` in *this* case makes sense and makes us emit `update({_id: 'id', other_field: 'val'}, {...updated_data...})`.
Thanks, problem solved. Maybe we could add this case to the document?

Resolving patch:
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -544,7 +544,7 @@ def cascade_save(self, **kwargs):
 
     @property
     def _qs(self):
-        """Return the queryset to use for updating / reloading / deletions."""
+        """Return the default queryset corresponding to this document."""
         if not hasattr(self, '__objects'):
             self.__objects = QuerySet(self, self._get_collection())
         return self.__objects
@@ -552,9 +552,11 @@ def _qs(self):
     @property
     def _object_key(self):
         """Get the query dict that can be used to fetch this object from
-        the database. Most of the time it's a simple PK lookup, but in
-        case of a sharded collection with a compound shard key, it can
-        contain a more complex query.
+        the database.
+
+        Most of the time the dict is a simple PK lookup, but in case of
+        a sharded collection with a compound shard key, it can contain a more
+        complex query.
         """
         select_dict = {'pk': self.pk}
         shard_key = self.__class__._meta.get('shard_key', tuple())


Test patch:
Repository: MongoEngine/mongoengine

Issue:
Misleading error message on bulkwrite
Error message was like `Document must not have _id value before bulk write.`. But it was misleading. Because of that, I have wasted my one day to debugging data. 


Discussion:

Resolving patch:
diff --git a/mongoengine/errors.py b/mongoengine/errors.py
--- a/mongoengine/errors.py
+++ b/mongoengine/errors.py
@@ -12,6 +12,7 @@
     "InvalidQueryError",
     "OperationError",
     "NotUniqueError",
+    "BulkWriteError",
     "FieldDoesNotExist",
     "ValidationError",
     "SaveConditionError",
@@ -51,6 +52,10 @@ class NotUniqueError(OperationError):
     pass
 
 
+class BulkWriteError(OperationError):
+    pass
+
+
 class SaveConditionError(OperationError):
     pass
 
diff --git a/mongoengine/queryset/base.py b/mongoengine/queryset/base.py
--- a/mongoengine/queryset/base.py
+++ b/mongoengine/queryset/base.py
@@ -20,6 +20,7 @@
 from mongoengine.connection import get_db
 from mongoengine.context_managers import set_write_concern, switch_db
 from mongoengine.errors import (
+    BulkWriteError,
     InvalidQueryError,
     LookUpError,
     NotUniqueError,
@@ -355,8 +356,8 @@ def insert(
         except pymongo.errors.BulkWriteError as err:
             # inserting documents that already have an _id field will
             # give huge performance debt or raise
-            message = u"Document must not have _id value before bulk write (%s)"
-            raise NotUniqueError(message % six.text_type(err))
+            message = u"Bulk write error: (%s)"
+            raise BulkWriteError(message % six.text_type(err.details))
         except pymongo.errors.OperationFailure as err:
             message = "Could not save document (%s)"
             if re.match("^E1100[01] duplicate key", six.text_type(err)):


Test patch:
Repository: MongoEngine/mongoengine

Issue:
MongoDB 4.2: mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
```
mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
    at modify (/opt/python3.7/lib/python3.7/site-packages/mongoengine/queryset/base.py:650)
```

Sorry, don't have exact steps to repro this. Caught this as we were trying out 4.2 and I'm busy reverting back to 4.0 :)

Some basic info:
python: 3.7.4
mongoengine: 0.18.2
pymongo: 3.9.0

Error does not appear when running against server 4.0.12.
Upon upgrading to 4.2, error starts popping up during update queries.


Discussion:
We ran into the same problem.
This seems to be an undocumented breaking change in Mongo 4.2. Should this be reported as a bug on that side?

On another note, pymongo only documents no_cursor_timeout for some operations, so perhaps this was never supported for update?

In any case, the problem is caused by this: https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/queryset/base.py#L1579

Mongoengine _always_ adds no_cursor_timeout, the only thing that changes is whether it's true or false. This is most likely the wrong thing to do.

Resolving patch:
diff --git a/mongoengine/queryset/base.py b/mongoengine/queryset/base.py
--- a/mongoengine/queryset/base.py
+++ b/mongoengine/queryset/base.py
@@ -1576,7 +1576,10 @@ def _cursor_args(self):
         if self._snapshot:
             msg = "The snapshot option is not anymore available with PyMongo 3+"
             warnings.warn(msg, DeprecationWarning)
-        cursor_args = {"no_cursor_timeout": not self._timeout}
+
+        cursor_args = {}
+        if not self._timeout:
+            cursor_args["no_cursor_timeout"] = True
 
         if self._loaded_fields:
             cursor_args[fields_name] = self._loaded_fields.as_dict()


Test patch:
diff --git a/tests/queryset/test_queryset.py b/tests/queryset/test_queryset.py
--- a/tests/queryset/test_queryset.py
+++ b/tests/queryset/test_queryset.py
@@ -5809,9 +5809,19 @@ def test_create_count(self):
         self.Person.objects.create(name="Baz")
         self.assertEqual(self.Person.objects.count(with_limit_and_skip=True), 3)
 
-        newPerson = self.Person.objects.create(name="Foo_1")
+        self.Person.objects.create(name="Foo_1")
         self.assertEqual(self.Person.objects.count(with_limit_and_skip=True), 4)
 
+    def test_no_cursor_timeout(self):
+        qs = self.Person.objects()
+        self.assertEqual(qs._cursor_args, {})  # ensure no regression of  #2148
+
+        qs = self.Person.objects().timeout(True)
+        self.assertEqual(qs._cursor_args, {})
+
+        qs = self.Person.objects().timeout(False)
+        self.assertEqual(qs._cursor_args, {"no_cursor_timeout": True})
+
 
 if __name__ == "__main__":
     unittest.main()
Repository: MongoEngine/mongoengine

Issue:
BaseDocument __init__ description improved for PyCharm code inspection
In PyCharm (2019.2), it gives inspection warning "Expected type dict; got str instead" for codes like:

`User(email='ross@example.com')`

Suggested change:

**base/document.py**
`:param **dict** values: A dictionary of keys and values for the document.`
by removing "dict" type
`:param values: A dictionary of keys and values for the document.`

It helps inspection to display normally.




Discussion:

Resolving patch:
diff --git a/mongoengine/base/document.py b/mongoengine/base/document.py
--- a/mongoengine/base/document.py
+++ b/mongoengine/base/document.py
@@ -62,13 +62,13 @@ def __init__(self, *args, **values):
         """
         Initialise a document or an embedded document.
 
-        :param dict values: A dictionary of keys and values for the document.
+        :param values: A dictionary of keys and values for the document.
             It may contain additional reserved keywords, e.g. "__auto_convert".
-        :param bool __auto_convert: If True, supplied values will be converted
+        :param __auto_convert: If True, supplied values will be converted
             to Python-type values via each field's `to_python` method.
-        :param set __only_fields: A set of fields that have been loaded for
+        :param __only_fields: A set of fields that have been loaded for
             this document. Empty if all fields have been loaded.
-        :param bool _created: Indicates whether this is a brand new document
+        :param _created: Indicates whether this is a brand new document
             or whether it's already been persisted before. Defaults to true.
         """
         self._initialised = False


Test patch: