MongoEngine/mongoengine — pull request #193 by jorgebastida (2012-12-07 17:08:15+00:00)
Title: New QuerySet method as_pymongo

# Introduction
In some situations where documents are really complex and you only need to extract some few fields from thousands of objects (i.e 50k user emails) using `scalar('email')` is not enough fast, or at least as fast as executing the same query using `pymongo`.

My first approach was to do:

User._get_collection().find(User.objects.filter(...)._query, {'email:1})

Basically, get the `pymongo` connection, create a `pymongo` query using `mongoengine` and add the fields I want to extract using `pymongo` api... It works... but it seems some kind of wrong not to use the `QuerySet` itself.

And why not only use `pymongo`? Some queries we are running are really complex and because maintainability and readability we really prefer to write them using `mongoengine` syntax.

This Pull Request adds a new method named `as_pymongo` to every `QuerySet` so you can bypass mongoengine's `_document._from_son` (the "expensive" part) and use pymongo's raw response.

You can also use `as_pymongo` with `coerce_types=True` so (i.e for `DecimalFields`) it'll return `Decimal` objects and not strings.

# Benchmark
Results for retrieving 50k emails from a complex User Document (47 fields, 3 of them EmbeddedDocument with 32 more fields in total)

1. User.objects.all().scalar('email') `29.027s` `0.5805ms/email`
2. User.objects.all().only('email').as_pymongo() `1.202s` `0.0260ms/email`
3. User._get_collection().find({}, {'email:1}) `0.6640s` `0.0132ms/email`

There is obviously some overhead but quite less than using `scalar`.