LearnBoost/mongoose — pull request #1292 by aheckmann (2013-01-15 18:22:39+00:00)
Title: [wip] Populate.next roadmap - feedback welcome

DO NOT MERGE
---------------------------------

For release 3.6 our primary objective is a rewrite of the `populate` functionality. Currently it suffers from several limitations:

### 1) Poor support for nested path population

Given a schema like the following:

new Schema({
name: String
, em: [{ arr: [{ person: { type: Schema.ObjectId, ref: 'Person'}}] }]
});

An attempt to populate the following path will not work as expected:

Model.find().populate('em.arr.person').exec(console.log)
// Every em.arr.person is still an ObjectId

### 2) Cannot further populate a document after population

Model.findByid(id, function (err, doc) {
doc.populate('after.the.fact', callback)
})

### 3) Recursive sub-document population not supported

References more than one level deep are not resolved.

a.populate('b') ->
b.populate('c')

### 4) Original _ids not found in populate query do not exist in returned document

Given the following document:

{ _id: 3, friends: [7,8,34] }

and we query/populate the documents friends

Model.find().populate('friends').exex(cb)

All `friends` ids that are not found during population are dropped on the floor. For example, if friends 7 and 34 are found, 8 is dropped from the returned document:

Model.find().populate('friends').exex(function (err, doc) {
console.log(doc.friends) // [{ _id: 7, name: 'luigi' }, { _id: 34, name: 'mario' }]
})

### 5) Inefficient queries

We currently execute a separate query per document per path.

### 6) Clumsy population syntax when specifying multiple params

Model.find().populate(path, [fields], [model], [conditions], [options])

http://mongoosejs.com/docs/api.html#query_Query-populate

### 7) Assigning values to populated paths casts the value back to an _id

https://github.com/LearnBoost/mongoose/issues/570

### 8) Lean queries cannot be populated

https://github.com/LearnBoost/mongoose/issues/1260

### NOTES

So far, items #1, #2, #5, #6, #8 have been fixed in the populate-wip branch.

### #1, #2, #3, #6, #8 solutions

The core populate functionality is being rewritten and exposed through `Model.populate(docs, paths, cb)`. This core functionality will now also be made accessible through `document#populate(path, options, cb)`. `Query#populate` will still exist as in previous versions.

This refactor and exposure opens the door to populating plain js objects as well as mongoose documents, meaning population of documents returned from `lean` queries will now be supported.

The syntax of `Query#populate()` is being cleaned up (yet backward compatible). Its interface will support passing an options object containing all knobs instead of having to pass nulls as placeholders:

// < 3.6
Model.find().populate(path, null, null, { sort: { name: -1 }})

// >= 3.6
var o = {};
o.path = path;
o.options = { sort: { name: -1 }}};
Model.find().populate(o)

This syntax also supports populating multiple paths with the same options:

// >= 3.6
var o = { path: 'path1 path2', options: { sort: { name: -1 }}};
Model.find().populate(o)

As for auto-populating multi-level deep refs, this will be supported through the public exposure of `Model.populate(docs, options, cb)` which can be utilized to continue populating returned documents ad infinitum.

### #4 solution may require breaking changes

> Original _ids not found in populate query do not exist in returned document

#### arrays

For populated arrays we could associate the original ids as an ad-hoc property so we don't lose anything. However, it makes managing array manipulation a headache, ensuring the ad-hoc list of original _ids stays in sync with the exposed array of populated documents in cases where the user then saves the original document is pretty silly.

#### single properties

When single fields are populated and no value was found (due to specified query matchers etc) we currently replace the value with `null` (the value returned by the query). This means we lose the original `_id`. No bookkeeping is really necessary in this case b/c the `null` value will not get saved unless the user modifies it somehow. But, we still lose access to the original _id.

#### solution

When no matching document is found for a given `_id`, we leave that `_id` in the document. This means that no `_ids` would be lost after population and the risk of manipulating a populated array and saving the document back to the db would be mitigated. The result is that populated arrays may contain a mix of `_ids` and documents and it will be up to your code to manage either type. No bookkeeping required. Same goes for populating an individual path, the result could be a path with either the original `_id` or the found document. At this time, this is the direction we are taking but by no means is this final yet. The goal is to remain 3.x backward compatible and hide this new behavior behind and option. Feedback is welcome.