MongoEngine/mongoengine — pull request #447 by FrankSomething (2013-08-17 18:46:19+00:00)
Title: Enhance "$pull" semantics

Currently, mongoengine appears to limit .update($pull) beyond the inherent limitations of MongoDB. For example, assume a MongoDB document as follows:

```
{
_id : ObjectId(xxx),
"top_field" : {
"nested_array" : [
{"key" : "value1"},
{"key" : "value2"}
]
}
}
```

If a user wanted to $pull from the nested_array all objects where "key" == "value2", in the MongoDB shell, the user can enter:```db.mycollection.update({},{$pull:{"top_field.nested_array":{"key":"value1"}}})```. However, if appears that mongoengine's pull_ semantics prevent this type of query from being issued through mongoengine.

Specifically, in mongoengine/queryset/transform.py, when the update op is a $pull type operation, the update kwarg key is handled in a particular way that limits mongoengine's capabilities. In all other cases, the update kwarg key is split by __'s and then joined with .'s to specify the key of the subdocument to which the update should be applied. E.g., if the kwarg key,value pair is ```set__top_field__nested_array=["foo","bar"]```, this is converted to a MongoDB-style ```{$set:{"top_field.nested_array":["foo","bar"]}}```. However, for pull-type operations, only the first portion of the kwarg key is used to specify the subdocument being modified; all other portions of the key are assumed to be part of the specification identifying which items to pull. E.g., if the kwarg key,value pair is ```pull__top_field__nested_array__key="value1"```, this is converted to the MongoDB-style ```{$pull:{"top_field":{"nested_array":{"key":"value1"}}}}``` (which results in an error, because "top_field" is not an array), rather than, e.g., ```{$pull:{"top_field.nested_array":{"key":"value1"}}}```.

At this point, I observe that this different in parsing the kwarg key seems unnecessary; if pull kwarg keys were parsed the same as other ops, a user could still specify nested queries by putting the nested dict in the value of the kwarg. E.g., assuming pull kwargs keys were parsed as other keys, a user could type ```pull__top_field__nested_array={"key":"value1"}``` to achieve the desired effect. This patter obviates the need for pull special case handling, and provides more flexibility is query specification.

That said, it might be preferable to permit this query flexibility in a backwards-compatible way. To that end, this pull request introduces an additional, backwards-compatible syntax for pull requests. Under this PR, if a user wants a portion of the kwarg key to be parsed as the field specification (rather than the query specification), that portion of the kwarg key should precede an additional "__". E.g., under this pull request, a user can type ```pull__top_field__nested_array____key="value"```. The portion of the key before the extra __ will be concatenated to the field key (e.g., ```"top_field.nested_array"```), while the portion of the key after the extra __ (if any) will be unwrapped into the nested dict structure (e.g., ```{"key" : "value1"}```). If there is no extra __ in the kwarg key, then behavior should remain as before.

Please let me know if you have any comments or questions.