Repository: MongoEngine/mongoengine

Issue:
Failed migration from 'BaseQuerySet.aggregate_sum()' to 'BaseQuerySet.sum()'
I use models based on QuerySet class in my web application. After update to from 0.10.6 to 0.10.7 I started getting errors: AttributeError: 'MyQuerySet' object has no attribute 'aggregate_sum' because 'aggregate_sum' has been removed in new version from BaseQuerySet class.

Why 'aggregate_sum' method has been removed without prior mark as deprecated?

What should I do? Use 'BaseQuerySet.sum' method based on map-reduce or realize their own method of replacing it?


Discussion:
See changelog:

> Replaced map-reduce based QuerySet.sum/average with aggregation-based implementations [#1336](https://github.com/MongoEngine/mongoengine/pull/1336)

Useless old map-reduce methods have been removed and aggregation-based methods have been renamed as simply sum/average.

You're right it might have been wise to keep the old names as deprecated aliases.

Just rename your methods.
Strangely enough, but in my case, a simple change of name of the method did not work.
My model field:

` total = IntField(db_field='t', default=0) `

my using aggregate_sum:

`total_views = self.aggregate_sum("t")`

i simply rename 'aggregate_sum' -> 'sum' and caught error:

```
  File "/webapp/.venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1283, in sum
    field_instances = self._document._lookup_field(field_parts)
  File "/webapp/.venv/local/lib/python2.7/site-packages/mongoengine/base/document.py", line 969, in _lookup_field
    % field_name)
LookUpError: Cannot resolve field "t"
```

what's wrong?
Surprising. I'm not sure it is related. The error seems to lie in the fact that _lookup_field fails while looking for "t", which should be there AFAIU.

You may want to show more of your code, and maybe try to call _lookup_field directly, see how it goes. And try on a field with no db_field attribute, to see if it makes a difference.

Reopening the issue as I'm not sure you can reopen yourself but you could rename it or even open a new one if you confirm the issue is unrelated to aggregate_sum -> sum.
When I call "sum" with field name of my model class ("total") not db field name ("t") I always get result == 0.

In old realization aggregate method work with db_field names, passed from 'field' argument:

```python
    def aggregate_sum(self, field):
        """Sum over the values of the specified field.

        :param field: the field to sum over; use dot-notation to refer to
            embedded document fields

        This method is more performant than the regular `sum`, because it uses
        the aggregation framework instead of map-reduce.
        """
        result = self._document._get_collection().aggregate([
            {'$match': self._query},
            {'$group': {'_id': 'sum', 'total': {'$sum': '$' + field}}}
        ])
        if IS_PYMONGO_3:
            result = list(result)
        else:
            result = result.get('result')
        if result:
            return result[0]['total']
        return 0
```

In new **sum()** realization **aggregate()** works with model fields names, that resolved in 'cls._fields' without considering that **field_name** and **db_field** may be different:

https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/base/document.py#L951

I think it is bug. It is don't work if your model's fields have db_field different from model field names.

@zeezdev sorry about that. I take the blame for not being more cautious and starting with a deprecation warning.

As for the issue you're experiencing, it seems that the new `BaseQuerySet.sum` doesn't respect the `db_field`:

```
In [11]: class TestDoc(Document):
    total = IntField(db_field='t', default=0)
   ....:

In [12]: TestDoc.objects.create(total=20)
Out[12]: <TestDoc: TestDoc object>

In [13]: TestDoc.objects.create(total=10)
Out[13]: <TestDoc: TestDoc object>

In [14]: TestDoc.objects.create(total=5)
Out[14]: <TestDoc: TestDoc object>

In [15]: TestDoc.objects.sum('total')
Out[15]: 0

In [16]: TestDoc.objects.sum('t')
---------------------------------------------------------------------------
LookUpError                               Traceback (most recent call last)
<ipython-input-16-00bc26fc7158> in <module>()
----> 1 TestDoc.objects.sum('t')

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/queryset/base.pyc in sum(self, field)
   1281         ListField = _import_class('ListField')
   1282         field_parts = field.split('.')
-> 1283         field_instances = self._document._lookup_field(field_parts)
   1284         if isinstance(field_instances[-1], ListField):
   1285             pipeline.insert(1, {'$unwind': '$' + field})

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/document.pyc in _lookup_field(cls, parts)
    967                 else:
    968                     raise LookUpError('Cannot resolve field "%s"'
--> 969                                       % field_name)
    970             else:
    971                 ReferenceField = _import_class('ReferenceField')

LookUpError: Cannot resolve field "t"
```

Working on a fix as we speak!

Resolving patch:
diff --git a/mongoengine/queryset/base.py b/mongoengine/queryset/base.py
--- a/mongoengine/queryset/base.py
+++ b/mongoengine/queryset/base.py
@@ -1271,9 +1271,10 @@ def sum(self, field):
         :param field: the field to sum over; use dot notation to refer to
             embedded document fields
         """
+        db_field = self._fields_to_dbfields([field]).pop()
         pipeline = [
             {'$match': self._query},
-            {'$group': {'_id': 'sum', 'total': {'$sum': '$' + field}}}
+            {'$group': {'_id': 'sum', 'total': {'$sum': '$' + db_field}}}
         ]
 
         # if we're performing a sum over a list field, we sum up all the
@@ -1300,9 +1301,10 @@ def average(self, field):
         :param field: the field to average over; use dot notation to refer to
             embedded document fields
         """
+        db_field = self._fields_to_dbfields([field]).pop()
         pipeline = [
             {'$match': self._query},
-            {'$group': {'_id': 'avg', 'total': {'$avg': '$' + field}}}
+            {'$group': {'_id': 'avg', 'total': {'$avg': '$' + db_field}}}
         ]
 
         # if we're performing an average over a list field, we average out


Test patch:
diff --git a/tests/queryset/queryset.py b/tests/queryset/queryset.py
--- a/tests/queryset/queryset.py
+++ b/tests/queryset/queryset.py
@@ -2838,6 +2838,34 @@ def test_sum(self):
             sum([a for a in ages if a >= 50])
         )
 
+    def test_sum_over_db_field(self):
+        """Ensure that a field mapped to a db field with a different name
+        can be summed over correctly.
+        """
+        class UserVisit(Document):
+            num_visits = IntField(db_field='visits')
+
+        UserVisit.drop_collection()
+
+        UserVisit.objects.create(num_visits=10)
+        UserVisit.objects.create(num_visits=5)
+
+        self.assertEqual(UserVisit.objects.sum('num_visits'), 15)
+
+    def test_average_over_db_field(self):
+        """Ensure that a field mapped to a db field with a different name
+        can have its average computed correctly.
+        """
+        class UserVisit(Document):
+            num_visits = IntField(db_field='visits')
+
+        UserVisit.drop_collection()
+
+        UserVisit.objects.create(num_visits=20)
+        UserVisit.objects.create(num_visits=10)
+
+        self.assertEqual(UserVisit.objects.average('num_visits'), 15)
+
     def test_embedded_average(self):
         class Pay(EmbeddedDocument):
             value = DecimalField()